Single atoms and molecules represent the ultimate spatial limit of chemistry and device fabrication. Scanning probe microscopy remains a central tool in atomic-scale research due to its unparalleled spatial resolution and versatility. Recent advances in this field have seen the use of single molecules attached to the scanning tip as sensing devices capable of detecting short range electrostatic fields, van der Waals attractive forces and short-range repulsive Coulomb or Pauli forces. However, it is also possible for the quantum states of the scanned molecule sensor to couple with the quantum states of adsorbates, potentially leading to novel physics which can be probed continuously at sub-nanometer scales.

In this dissertation we demonstrate the ability to visualize quantum state coupling between a scanning single molecule tip and other molecules adsorbed on metal surfaces. To highlight the sensitivity of single molecules to their local environment, inelastic electron tunneling spectroscopy is used to study the overtone vibrations of carbon monoxide molecules in different adsorption sites, revealing site-dependent anisotropies and anharmonicities. Next, the mechanical coupling of quantum vibration states of a CO molecule attached to an STM tip with another CO adsorbed on the surface is shown to be sensitively dependent on lateral intermolecular distance and reveals new information about the force fields experienced by the surface adsorbate. In this way the anisotropy of the adsorbate’s lateral confinement potential is visualized for the first time.

Going further, we then extend the sensing capabilities of single molecule scanned probes to the detection of spin densities on the surface. A single nickelocene molecule adsorbed on the tip is used to initiate and continuously tune superexchange interactions with another magnetic adsorbate. We then show that the spin-spin coupling can be visualized in real space through imaging, demonstrating the ability of a single molecule sensor to function as a magnetometer. Finally, we show that inelastic scattering of tunneling electrons can be used to excite single nickelocene molecules to combined spin-vibration excited states, providing a route toward the later study of spin-vibration coupling.

Inelastic Electron Tunneling Spectroscopy with the Scanning Tunneling Microscope (STM-IETS) is a powerful technique used to characterize vibration and spin at the single molecule level. While IETS lacks hard selection rules, historically it has been assumed that vibrational overtones are rarely seen or even absent. Here we provide direct experimental evidence that the hindered rotation overtone excitation of CO molecules on Ag(110) can be detected with STM-IETS by isotope substitution. We also show that the anharmonicity of the overtone excitation can be characterized and compared between unique adsorption sites, and find evidence of anisotropy in the vibrational anharmonicity for CO adsorbed on the [1 1 0] step edge.