They contain potassium-40, a naturally radioactive isotope
A single banana gives you about 0.01 millirem of radiation. You'd need to eat around 50 million bananas at once for a lethal dose. Nuclear physicists actually use the 'Banana Equivalent Dose' as an informal unit to explain radiation levels to the public.
