UC San Diego

Reconfigurable networks that use optical circuit switches are slowly paving their way into datacenters due to their ability to offer high bandwidth at low cost and power. However, the majority of these reconfigurable networks are aimed at conventional datacenter clusters where the traffic patterns are unpredictable. While dynamically reconfigurable network topologies have shown promise for general datacenter workloads, in this work, I investigate the extent to which they may benefit machine learning (ML) collectives. Several algorithms have been proposed to implement collectives on static networks (of different topologies), but less well studied has been the interaction between these algorithms and reconfigurable network fabrics. Therefore, I seek to quantify the potential benefit that a reconfigurable network layer can bring to the large-scale collectives that underpin large DNN training. Following this, I study the extent of fragmentation in ML torus clusters. In multi-tenant environments, resource allocations in torus clusters often lead to fragmentation, reducing system utilization and increasing queue times. My work demonstrates how enhancing the flexibility of a torus network through reconfiguration can significantly mitigate this fragmentation.