Security and privacy in distributed systems are long-standing hard problems. On the one hand, solutions for anonymous communications over the Internet are either vulnerable to traffic analysis or offer poor performance. On the other hand, compromises within enterprises remain hard to track down due to complex dependencies between hosts, applications, and their data.

In this thesis, I develop two solutions to improve the anonymity vs. performance trade-off for communications over the Internet. LASTor improves performance of Tor by modifying path selection algorithm and it also mitigate traffic analysis attack by detecting common autonomous system (AS) across the entry and exit segments of a circuit and avoiding using those paths. LASTor reduces median latencies of visiting top 200 websites by 25% while the false negative rate of not detecting a potential snooping AS from 57% to 11%. Next solution, Innominate, is a new framework for anonymous online communication that both offers traffic analysis resistant strong anonymity and scalable performance. Innominate adopts relay-based technique for low latency communication, however instead of a single client servers as a relay, group relay is used to provide strong anonymity.

As of security inside enterprises, I develop DeltaTrack, the first enterprise attack

forensics system that leverages differential dependency tracking to automate the pruning of irrelevant nodes and edges in the backtracking graph. DeltaTrack continuously monitors system call events from all hosts and summarizes their common execution behaviors in a reference model. Then, the reference model is leveraged to prune away frequently observed events across many hosts since they are unlikely to be relevant to the intrusion. DeltaTrack can reduce the number of nodes and edges of the backtracking graph by up to 131x and 512x, respectively, while maintaining its accuracy.