Background NOAA’s Hazard Mapping System (HMS) smoke product comprises smoke plumes digitised from satellite imagery. Recent studies have used HMS as a proxy for surface smoke presence. Aims We compare HMS with airport observations, air quality station measurements and model estimates of near-surface smoke. Methods We quantify the agreement in numbers of smoke days and trends, regional discrepancies in levels of near-surface smoke fine particulate matter (PM2.5) within HMS polygons, and separation of total PM2.5 on smoke and non-smoke days across the contiguous US and Alaska from 2010 to 2021. Key results We find large overestimates in HMS-derived smoke days and trends if we include light smoke plumes in the HMS smoke day definition. Outside the western US and Alaska, near-surface smoke PM2.5 within areas of HMS smoke plumes is low and almost indistinguishable across density categories, likely indicating frequent smoke aloft. Conclusions Compared with airport, Environmental Protection Agency (EPA) and model-derived estimates, HMS most closely reflects surface smoke in the Pacific and Mountain regions and Alaska when smoke days are defined using only heavy plumes or both medium and heavy plumes. Implications We recommend careful consideration of biases in the HMS smoke product for air quality and public health assessments of fires.