Sound speed is a critical parameter in ocean acoustic studies, as it determines the propagation and interpretation of recorded sounds. The potential for exploiting oceanic vessel noise as a sound source of opportunity to estimate ocean sound speed profile is investigated. A deep learning-based inversion scheme, relying upon the underwater radiated noise of moving vessels measured by a single hydrophone, is proposed. The dataset used for this study consists of Automatic Identification System data and acoustic recordings of maritime vessels transiting through the Santa Barbara Channel between January 2015 and December 2017. The acoustic recordings and vessel descriptors are used as predictors for regressing sound speed for each meter in the top 200 m of the water column, where sound speeds are most variable. Multiple (typically ranging between 4 and 10) transits were recorded each day; therefore, this dataset provides an opportunity to investigate whether multiple acoustic observations can be leveraged together to improve inversion estimates. The proposed single-transit and multi-transit models resulted in depth-averaged root-mean-square errors of 1.79 and 1.55 m/s, respectively, compared to the seasonal average predictions of 2.80 m/s.