The tremendous success of Internet services has led to the rapid growth of Warehouse-Scale

Computers (WSCs). The networking infrastructure has become one of the most vital components

in a datacenter. With the rapid evolving set of workloads and software, evaluating

network designs really requires simulating a computer system with three key features: scale,

performance, and accuracy. To avoid the high capital cost of hardware prototyping, many

designs have only been evaluated with a very small testbed built with off-the-shelf devices,

often running unrealistic microbenchmarks or traces collected from an old cluster. Many

evaluations assume the workload is static and that computations are only loosely coupled

with the very adaptive networking stack. We argue the research community is facing a

hardware-software co-evaluation crisis.

In this dissertation, we propose a novel cost-efficient evaluation methodology, called

Datacenter-in-a-Box at Low cost (DIABLO), which uses Field-Programmable Gate Arrays

(FPGAs) and treats datacenters as whole computers with tightly integrated hardware and

software. Instead of prototyping everything in FPGAs, we build realistic reconfigurable abstracted

performance models at scales of O(10,000) servers. Our server model runs the full

Linux operating system and open-source datacenter software stack, including production

software such as memcached. It achieves two orders of magnitude simulation speedup over

software-based simulators. This speedup enables us to run the full datacenter software stack

for O(100) seconds of simulated time. We have built a DIABLO prototype of a 2,000-node

simulated cluster with runtime-configurable 10 Gbps interconnect using 6 multi-FPGA BEE3

boards.