UC Merced

Modern computing systems increasingly rely on heterogeneous architectures that integrate general-purpose processors with specialized accelerators such as GPUs and deep learning engines. To meet the performance and energy demands of modern workloads, these systems depend heavily on optimization techniques including speculative execution, transfer learning, hardware acceleration, and dynamic power management. While these mechanisms substantially improve efficiency, they also introduce new forms of shared state, resource contention, and externally observable behavior, thereby expanding the attack surface. As a result, security vulnerabilities in modern platforms often emerge not from a single component, but from interactions across multiple layers of the computing stack. This dissertation investigates such cross-layer vulnerabilities in heterogeneous computing systems and develops new insights into how performance-oriented design choices can unintentionally expose sensitive information. It presents three case studies spanning machine learning systems, edge GPU platforms, and CPU microarchitecture. First, this dissertation presents Decepticon, a model extraction attack on large-scale transfer-learned models. Decepticon exploits execution fingerprints inherited from shared pre-trained models to identify the underlying model used by a black-box victim and significantly reduce the effort required to reconstruct a high-fidelity clone. The results demonstrate that even large transformer-based models remain vulnerable when transfer learning and GPU execution characteristics are jointly exploited. Second, this dissertation investigates the power and frequency behavior of edge GPU platforms and their security implications. Through detailed characterization of instruction-level and application-level telemetry on commercial NVIDIA Jetson devices, this work demonstrates that GPU and deep learning accelerator behavior can be leveraged to construct covert channels. It further evaluates mitigation strategies based on telemetry access restriction, resolution reduction, and perturbation, highlighting the trade-offs between observability and security. Third, this dissertation introduces SCPC (Securing Cross-Process Collision-Based Transient Attacks), a low-overhead defense against cross-process collision-based transient execution attacks in modern CPUs. SCPC protects speculative execution structures by selectively virtualizing vulnerable predictor state while preserving much of the performance benefit of shared hardware resources. Evaluation shows that the mechanism significantly improves security with minimal performance overhead and can be generalized to other speculative structures. Together, these contributions show that information leakage in modern heterogeneous systems is often a byproduct of the same mechanisms that enable high performance. This dissertation argues that securing future computing platforms requires a cross-layer perspective in which machine learning frameworks, accelerator behavior, and processor microarchitecture are analyzed jointly rather than in isolation. By exposing these vulnerabilities and exploring corresponding defenses, this work advances the design of more secure and robust heterogeneous computing systems.