This dissertation investigates deep learning-based signal processing techniques for wireless communication system about two research topics: modulation classification in orthogonal frequency division multiplexing (OFDM) signals for intelligent spectrum sensing, and remote adversarial attacks on Wi-Fi-based human activity recognition (HAR) systems for privacy.

Deep learning-based modulation classification of OFDM signals following Wi-Fi 6 and 5G downlink specifications for spectrum sensing is studied. Since intelligent spectrum sensing often targets diverse wireless technologies, protocol-specific preambles or channel allocation might not be available. To reliably estimate modulation under this constraint, this dissertation proposes algorithm to estimate essential OFDM parameters and extract a feature characterizing modulation to make it suitable to be an input to deep learning-based classifier. The OFDM parameters, symbol duration and cyclic prefix length, are estimated utilizing the cyclic autocorrelation properties of OFDM waveforms. Based on these estimates, a feature characterizing modulation of OFDM signals is designed to mitigate synchronization errors arising from unknown symbol boundaries. The extracted features are represented as two-dimensional histograms of amplitude and phase, which serve as inputs to a convolutional neural network (CNN)-based classifier. The classifier's performance, evaluated using synthetic and real-world measured over-the-air (OTA) datasets, achieves a minimum accuracy of 97% accuracy with OTA data when SNR is above the required SNR for reliable data transmission.

Remote adversarial attacks on neural network-based HAR classifier that utilize Wi-Fi channel state information (CSI) has been investigated. The capability of Wi-Fi routers to perform human activity recognition through CSI raises privacy concerns. To address this issue, this dissertation proposes a novel remote adversarial attack deploying geneative adversarial imitation learning (GAIL). The proposed method degrades the accuracy of HAR classifiers deployed at Wi-Fi routers by manipulating the channel estimation signals transmitted from user devices. Unlike gradient-based attacks, which require complete knowledge of the target model's inputs or future CSI, the GAIL-based algorithm generates effective perturbation signals without explicit knowledge of future CSI or details of the target HAR models. Comprehensive experimental evaluations performed across seven distinct environments and six target HAR models--including both deep learning and non-deep learning classifiers--demonstrate the versatility of the proposed adversarial method. The GAIL-based approach reduces HAR classifier accuracy to approximately 50%, requiring only a minimal increase (0.5 dB) in average perturbation amplitude compared to gradient-based methods based on impractical assumptions.