UC San Diego

Optical networks have become the central elements in many scientific applications. And the emerging of those data intensive applications also brings about new challenges to the approaches on how to efficiently use the abundant bandwidth provided by optical networks. From some of the applications, we observe a parallel data access model, and are thus motivated to study a new approach to use parallel data streams to better feed the parallel data access model. Parallelization of data streams leads to new challenges which are not sufficiently addressed in previous related works. The new approach we propose in this thesis improves existing approaches through tightly coupling data streams using Cross-Stream coding. It can effectively resolve unevenly distributed code packet loss at the receiving side to eliminate the necessity of frequently sending acknowledgments back to the senders requesting missing data. We further prove that the Cross- Stream coding approach is scalable, and give out practical algorithms to find optimal solutions for large number of parallel streams, for any given objective Cross-Stream packet loss pattern. We experimented with our Cross-Stream coding approach using computer clusters. We also observe, from other applications, that optical networks can improve traditional digital archiving methods, and will include our preliminary research of using optical networks to facilitate a workflow based digital archiving system, named CineGrid Exchange. The CineGrid Exchange has been practically online for about a year, storing and managing 20 Tera bytes digital contents. In summary, both the requirements of large scale network applications and the available bandwidth of optical networks increase faster than the development of middleware between them. And there exist significant challenges to alleviate the difference. The research in this thesis is motivated by these challenges, and explores possible directions towards future petascale data transfer and preservation