Integration of contextual sensors into vehicle-borne mobile radiation detection systems delivers a rich description of the environment to inform estimates of the complex and variable gamma-ray signals observed in urban areas. Models based on these data streams could provide realistic inputs to urban radiological search algorithms and potentially improve the system's sensitivity to detect illicit radiological and nuclear materials. In this work, LiDAR and inertial data are combined using simultaneous localization and mapping techniques to create a three-dimensional (3D) representation of the surrounding scenery. Semantic segmentation of concurrently collected video imagery enables the division of the 3D model into distinct material categories. The radioactive flux of surfaces associated with these categories are inferred through maximum likelihood estimation maximization and the activity of the three most common isotopes (K-40, U-238 series, and Th-232 series) in the respective materials is predicted. The results, found to be in agreement with ground truth measurements performed at the facility, suggest that it is possible to quickly infer the composition of naturally occurring materials in structures that comprise a radiological scene. Such a capability could be used to inform radiological search algorithms and enable data-driven modeling of radiological search problems, which could facilitate system testing and operator training activities.