Lawrence Berkeley National Laboratory

The path towards realizing peta-scale computing is increasingly dependent on building supercomputers with unprecedented numbers of processors. To prevent the interconnect from dominating the overall cost of these ultra-scale systems, there is a critical need for high-performance network solutions whose costs scale linearly with system size. This work makes several unique contributions towards attaining that goal. First, we conduct one of the broadest studies to date of high-end application communication requirements, whose computational methods include: finite-difference, lattice-bolzmann, particle in cell, sparse linear algebra, particle mesh ewald, and FFT-based solvers. To efficiently collect this data, we use the IPM (Integrated Performance Monitoring) profiling layer to gather detailed messaging statistics with minimal impact to code performance. Using the derived communication characterizations, we next present fit-trees interconnects, a novel approach for designing network infrastructure at a fraction of the component cost of traditional fat-tree solutions. Finally, we propose the Hybrid Flexibly Assignable Switch Topology (HFAST) infrastructure, which uses both passive (circuit) and active (packet) commodity switch components to dynamically reconfigure interconnects to suit the topological requirements of scientific applications. Overall our exploration leads to a promising directions for practically addressing the interconnect requirements of future peta-scale systems.