Department of Computer Science & Engineering

Solving "Big Data" problems requires bridging massive quantities of compute, memory, and storage, which requires significant amounts of bisection bandwidth. Topologies like Fat-tree, VL2, and HyperX achieve a scale-out design by leveraging multiple paths from source to destination. However, traditional routing protocols are not able to effectively utilize these links while also harnessing spare network capacity to better statistically multiplex network resources. In this work we present Dahu, a switch mechanism for efficiently load balancing traffic in multipath networks. Dahu avoids congestion hot-spots by dynamically spreading traffic uniformly across links, and forwarding traffic over non-minimal paths where possible. By performing load balancing primarily using local information, Dahu can act more quickly than centralized approaches, and responds to failure gracefully. Our evaluation shows that Dahu delivers up to 50% higher throughput relative to ECMP in an 8,192 server Fat-tree network and up to 500% improvement in throughput in large scale HyperX networks with over 130,000 servers.

Pre-2018 CSE ID: CS2013-0992