The famous Casimir effect was first predicted by Hendrik Casimir in 1948. Two conducting parallel electrically neutral plates placed in vacuum were predicted to have an attractive force in disagreement with classical physics. The Casimir effect is intriguing because the zero-point energy in quantum vacuum is demonstrated by this macroscopic quantum phenomenon. Nowadays, the Casimir effect is still a fascinating research area. We will first discuss the physical foundation of the Casimir effect from the perspective of quantum field theory and the Lifshitz theory for real materials.

In this dissertation, three experimental projects are presented. Current Casimir effect theories with real materials boundaries treat zero-point photon and real photon scattering similarly based on the fluctuation dissipation theorem. Previous experiments below 1 micrometer separation have shown that dissipation has to be ignored for zero-point photon scattering from free electrons in the material. To understand this puzzle, we developed a higher sensitivity frequency shift technique to perform Casimir force measurement in ultra-high vacuum. Using UV radiation and Ar ions bombardment, we eliminated the role of detrimental electrostatic forces to remove ambiguity in the measurements. The cantilever was treated with a special procedure to reduce its spring constant leading to a sixfold improvement in sensitivity. The Casimir force gradient was measured to the largest separation distance of 950 nm. The data agree with theory if the dissipation from free electron scattering is neglected. This was confirmed to a separation of 820 nm at 67 % CL. Previous work had only shown it for a separation of 420 nm. To study the Casimir force to even larger separation, a difference force measurement experiment has been designed to further improve the force sensitivity. The difference in force between Au surface and a vacuum trench is measured. An air bearing spindle is used and measurement systems have been developed. In related work, the entropic Casimir force originating from thermal fluctuation of polymers in solution confined between a pyramidal tip and a flat plate was experimental studied using the AFM. The results have been analyzed and compared to two theoretical models.