UC Santa Cruz

Storage system solutions have historically been dominated by proprietary offerings designed around a fixed set of common interfaces such as the POSIX file abstraction. However, as the scalability requirements of applications have grown, these interfaces and their implementations have not kept pace. This has forced developers to rely on middleware and external services to address these limitations. These solutions introduce new complexity into the system in the form of duplicated software and can lead to increased costs and reduced reliability. However, the recent availability of high-performance open-source storage systems is allowing developers to explore alternative storage interfaces that directly meet the needs of applications without the fear of vendor lock-in.

We introduce programmable storage as a means by which existing internal storage system abstractions can be generalized and reused to support applications through the creation of domain-specific interfaces. By reusing internal, code-hardened sub-systems applications can avoid duplicating complex software and increase reliability, as well as realize application-specific optimizations. We demonstrate programmable storage by mapping a wide range of common application and storage services requirements onto existing abstractions found in distributed storage systems.

We show that programmable storage introduces real challenges for issues of portability and maintenance, and that the design space for new storage interfaces is intractable for non-expert developers. To address this limitation we propose that new storage interfaces and services be expressed using a declarative language that abstracts across the differences in internal storage system interfaces to allow application developers to create new storage services without becoming storage system experts. While a declarative approach to building storage interfaces resolves many issues, it doesn't address the implications of developing and evolving storage interfaces in a real-world system. To address this we propose a set of abstractions for developing new interfaces that are aligned with existing software development workflows and source code control systems.