UC San Diego

The rapid growth of Internet applications and services such as search, social networking, and cloud computing services in conjunction with the adoption of commoditized hardware has led to the emergence of data centers consisting of tens to hundreds of thousands of computers. A common characteristic of applications running in these data centers is their use of parallelism in their design to enable system performance to grow by scaling out as opposed to scaling up. User interactions with these services often involve forwarding requests to thousands of computers in parallel. A user search request may involve accessing an inverted index stored on thousands of computers. In addition to the network traffic generated from users of front-end facing systems, there is also a significant amounts of intra-data center network traffic generated by backend applications such as MapReduce, Hadoop, and Dryad. These backend applications typically perform analytical analysis on petabyte-sized datasets. Constructing a network fabric that can meet the performance requirements of large scale data centers in an economical manner is an immense challenge. In this thesis, we describe the implementation of a managed network fabric intended for large scale data centers. Our system incorporates the best features of layer 2 and layer 3 networks while seeking to avoid the shortcomings of each protocol. Using commoditized hardware and software primitives, we construct a functioning prototype that treats the network fabric as a single logical unit and enables traffic to be dynamically provisioned in a fault tolerant manner. Our prototype demonstrates the feasibility of building such a system and that high performance can be achieved