Surface enhanced Raman scattering (SERS) is a vibrational spectroscopy method that enables the quantification of the concentration of small molecules. SERS sensing has been demonstrated in a wide variety of applications, from explosive and drug detection, to monitoring of bacteria growth. Underpinning SERS sensing are the sensor surfaces that are composed of vast quantities of metal nanostructures which confine light into small gaps called "hotspots", enhancing Raman scattering. While these surfaces are essential for increasing Raman scattering intensity so that analyte signal may be observed in small concentrations, they introduce signal variations due to spatial distributions of Raman enhancement and hotspot volume. In this work, we introduce a convolutional neural network model that improves concentration regressions in SERS sensors by learning the distributions of sensor surface dependent latent variables. We demonstrate that this model significantly improves predictions compared to a traditional multilayer perceptron approach, and that the model uses analyte spectral information and is capable of reasonable interpolations.