The Ministry of the Environment, whose mandate also includes protection of animals, says "There is no effect on humans" if one stays away far enough.
What about the birds, having grown up in a nest with 1.4 million becquerels/kg of radioactive cesium with 2.6 microsieverts/hour surface radiation?
From Kyodo News (3/23/2012):
1.4 million becquerels of radioactive cesium from a swallow's nest, "No effect" if you stay away
The Ministry of the Environment announced on March 23 that 1.4 million becquerels/kg of radioactive cesium (Cs-134 and Cs-137) was detected from a swallow's nest taken from the wall of a building in Okuma-machi in Fukushima Prefecture, about 3 kilometer from Fukushima I Nuclear Power Plant.
According to the Ministry, the nest was built from the mud from a nearby rice paddy and dead leaves with high radioactive cesium density. The nest was taken to the National Institute of Radiological Sciences (in Chiba City, Chiba), where the radiation levels were measured. It was 2.6 microsieverts/hour on the surface of the nest, but the radiation level dropped off to 0.08 microsievert/hour at 50 centimeters from the nest. The Ministry says that "if one stays away from the nest, it can be safely concluded that any health effect on humans can be ignored".
I can't find any press release on the subject at the Ministry of the Environment website, which is a disorganized mess (probably on purpose).
The Yamashina Institute for Ornithology announced last year that it would collect swallows' nests from various locations in Japan to test for radioactive materials. I wonder if they had done it. I would trust the institute much more than the Ministry of the Environment. Looking at their website, they are still collecting the nests from 2011.