Structural colors in nature have inspired the design of diverse photonic structures, which can interact with light via interference, diffraction or scattering. Among them, responsive soft material-involved photonic structures uniquely feature large volumetric changes upon external stimuli. The volumetric changes result in peak/valley shift of reflection spectra and perceptible color changes, providing responsive soft material-based structural color systems capability of serving as sensors for detecting chemical and biological analytes. Synthetic polymers and some natural materials are the most studied and utilized responsive soft materials for constructing structural color sensors, by tuning the thickness and morphology of formed films, or incorporating them into template structures, or their self-assembling. In this review article, structural colors in nature are firstly introduced, followed by discussing recent developments of promising responsive soft material-based structural color sensors, including the design of structural color sensors based on synthetic polymers and natural materials, as well as their applications for chemical sensing, biosensing, and multi-analyte sensing with sensor arrays. For specific sensing of chemicals and biomolecules, the sensing performance is evaluated in terms of detection range, sensitivity, response time, and selectivity. For multi-analyte sensing, cross-reactive structural sensor arrays based on simply a single soft material will be shown capable of discriminating various series of similar compounds. The future development of structural color sensors is also proposed and discussed.