The remarkable sensitivity of depleted silicon to ionizing radiation is a nuisance to astronomers. "Cosmic rays" degrade images because of struck pixels, leading to modified observing strategies and the development of algorithms to remove the unwanted artifacts. In the new-generation CCD's with thick sensitive regions, cosmic-ray muons make recognizable straight tracks and there is enhanced sensitivity to ambient gamma radiation via Compton-scattered electrons ("worms"). Beta emitters inside the dewar, for example high-potassium glasses such as BK7, also produce worm-like tracks. The cosmic-ray muon rate is irreducible and increases with altitude. The gamma rays are mostly by-products of the U and Th decay chains; these elements always appear as traces in concrete and other materials. The Compton recoil event rate can be reduced significantly by the choice of materials in the environment and dewar and by careful shielding. Telescope domes appear to be significantly cleaner than basement laboratories and Coude spectrograph rooms. Radiation sources inside the dewar can be eliminated by judicious choice of materials. Cosmogenic activation during high-altitude flights does not appear to be a problem. Our conclusions are supported by tests at the Lawrence Berkeley National Laboratory low-level counting facilities in Berkeley and at Oroville, California (180 m underground).