This dissertation provides a comprehensive study of non-Hermitian systems and applications. The coupling phenomena occur in three-dimensional plasmonic nanostructure lead to a peculiar spectral response. Modification of near-field coupling in nanorod pairs enables to invert plasmon hybridization scheme, which offers engineering resonance. The first half of this dissertation investigates the optical properties of the plasmon hybridization system based on the multilayered structure. The proposed hybridized sensing platforms can improve sensing capabilities than a standard single plasmonic resonance system.

In the second half of this dissertation, we propose a novel approach to exceptional points (EPs) where eigenvalues and eigenvectors coalesce simultaneously. The plasmonic EPs are based on the hybridization of detuned resonances in multilayered plasmonic structures to reach a critical complex coupling rate between nanoantenna arrays, resulting in the simultaneous coalescence of the resonances and loss rates. The hybridization of optically dissimilar resonators, however, led to two-hybrid modes with crossing and avoided crossing of both the resonances and loss rates, unambiguously demonstrating the existence of a plasmonic EP where resonances and loss rates become simultaneously degenerate. By breaking the symmetry (that is, making the bars optically different) the hybridized modes are no longer purely symmetric or anti-symmetric, making interference via radiation possible. Therefore, the interplay between near-field Coulomb interactions (mostly controlled by dx) and radiative coupling via interferences (enabled by symmetry breaking and mostly controlled by Py) enables the coalescence of the hybrid modes, i.e., lead to the existence of EP. The exceptional points are used as sensors of anti-immunoglobulin G, the most abundant immunoglobulin isotype in human serum. We demonstrated, for the first time, a new type of sensors that are 267 more sensitive than the current states of the art of nanosensors.