Determining the origin of volatiles on terrestrial planets and quantifying atmospheric loss during planet formation is crucial for understanding the history and evolution of planetary atmospheres. Using geochemical observations of noble gases and major volatiles we determine what the present day inventory of volatiles tells us about the sources, the accretion process and the early differentiation of the Earth. We further quantify the key volatile loss mechanisms and the atmospheric loss history during Earth’s formation. Volatiles were accreted throughout the Earth’s formation, but Earth’s early accretion history was volatile poor. Although nebular Ne and possible H in the deep mantle might be a fingerprint of this early accretion, most of the mantle does not remember this signature implying that volatile loss occurred during accretion. Present day geochemistry of volatiles shows no evidence of hydrodynamic escape as the isotopic compositions of most volatiles are chondritic. This suggests that atmospheric loss generated by impacts played a major role during Earth’s formation. While many of the volatiles have chondritic isotopic ratios, their relative abundances are certainly not chondritic again suggesting volatile loss tied to impacts. Geochemical evidence of atmospheric loss comes from the He3/22Ne, halogen ratios (e.g., F/Cl) and low H/N ratios. In addition, the geochemical ratios indicate that most of the water could have been delivered prior to the Moon forming impact and that the Moon forming impact did not drive off the ocean. Given the importance of impacts in determining the volatile budget of the Earth we examine the contributions to atmospheric loss from both small and large impacts. We find that atmospheric mass loss due to impacts can be characterized into three different regimes: 1) Giant Impacts, that create a strong shock transversing the whole planet and that can lead to atmospheric loss globally. 2) Large enough impactors (mcap≳2ρ0(πhR)3/2, rcap∼25km for the current Earth), that are able to eject all the atmosphere above the tangent plane of the impact site, where h, R and ρ0 are the atmospheric scale height, radius of the target, and its atmospheric density at the ground. 3) Small impactors (mmin> 4 πρ0h3, rmin∼1km for the current Earth), that are only able to eject a fraction of the atmospheric mass above the tangent plane. We demonstrate that per unit impactor mass, small impactors with rmin< r< rcap are the most efficient impactors in eroding the atmosphere. In fact for the current atmospheric mass of the Earth, they are more than five orders of magnitude more efficient (per unit impactor mass) than giant impacts, implying that atmospheric mass loss must have been common. The enormous atmospheric mass loss efficiency of small impactors is due to the fact that most of their impact energy and momentum is directly available for local mass loss, where as in the giant impact regime a lot of energy and momentum is ’wasted’ by having to create a strong shock that can transverse the entirety of the planet such that global atmospheric loss can be achieved. In the absence of any volatile delivery and outgassing, we show that the population of late impactors inferred from the lunar cratering record containing 0.1% M⊕ is able to erode the entire current Earth’s atmosphere implying that an interplay of erosion, outgassing and volatile delivery is likely responsible for determining the atmospheric mass and composition of the early Earth. Combining geochemical observations with impact models suggest an interesting synergy between small and big impacts, where giant impacts create large magma oceans and small and larger impacts drive the atmospheric loss.