With the dramatic advances in electronic device industry, the availability of high speed non-volatile memory (NVRAM) has introduced a new tier into the storage hierarchy, and holds great promise for reduction in latency, power consumption, and improved performance. Among them, flash-memory has become a popular storage medium to replace hard disk as a permanent storage device and DRAM as a temporary storage device. Yet, despite their fast random I/O performance, the design of a flash-based storage system achieves suboptimal performance or suffers from reduced endurance due to the nature of flash memory, for example, out of place update, asymmetric read-write throughput, and limited write cycles. Lack of application aware design makes flash memory less efficient, and hence cannot meet various performance requirements. To this end, we investigate the roles flash memory plays in different storage applications and their performance and reliability requirements. By examining the behavior of these systems and their consequent data access characteristics, as well as the performance impact, we propose solutions that tradeoff performance, cost, endurance and reliability to achieve high efficiency for flash memory in different storage applications with reduced overhead.

We first explore the use of flash memory as a write-through cache in a tiered storage. We demonstrate the individual and cumulative contributions of cache admission policy, cache eviction policy, flash garbage collection policy, and flash device configuration on a flash caching device. We show that workloads on Solid State Caches (SSCs) have significantly greater write pressures than their storage counterparts. we propose HEC, a High Endurance Cache that aims to improve overall device endurance via reduced media writes and erases while increasing or maintaining cache hit rate performance.

To further characterize the behavior of flash memory used in different storage applications, we focus on flash as a primary backing storage device in the second part of this dissertation. We explore the drawbacks of a typical log-structured file system on top of a log-structured FTL flash device. We characterize the interactions between multiple levels of independent logs, and describe several practical scenarios which arises in real log-on-log systems. We then propose a log-aware coordination to tune the layout of logs, so that when multiple layers of logs exist in the system we can still achieve high performance with minimum interference among each log.

In the third part of this dissertation, we explore the approaches to collapsing logs. While there are several popular ways that utilize the nature of flash memory translation layer to eliminate multiple layers of logs in the entire system, we focus on the benefit we can obtain from a log-less object-based flash aware system. We show from simulation experiments that advanced features could be embedded to improve overall performance for object-based flash system with low overhead through a rich interface.

In an information-driven world, the ability to capture and store data in real time is of the utmost importance. The scope and intent of such data capture, however, varies widely. Individuals record television programs for later viewing, governments maintain vast sensor networks to warn against calamity, scientists conduct experiments requiring immense data collection, and automated monitoring tools supervise a host of processes which human hands rarely touch. All such tasks have the same basic requirements --- guaranteed capture, management, storage, and analysis of streaming real-time data --- but with greatly differing parameters. Our ability to process and interpret data has grown faster than our ability to store and manage it, a characteristic which now hinders our ability to exploit it.

The work presented in this dissertation demonstrates a means of integrating data management with the physical storage layer in order to gain superior performance not otherwise achievable. By fusing a close understanding of the disk hardware with the necessary components of a high-performance storage system, a unique method of data handling is constructed. This approach allows for hard performance guarantees and quality of service regulation at near-maximum hardware capabilities in a transient data environment for indefinite periods of time. The core storage system is made to understand the data as more than just a stream of bytes, uniting indexing and query capabilities into basic operations. All such gains are fully compatible with the accoutrements of a large-scale storage system, such as reliability and control mechanisms, which results in a fully viable new storage architecture.

Big data stores are becoming increasingly important in a variety of domains including scientific computing, internet applications, and business applications. For price and performance reasons, such storage is comprised of magnetic hard drives. To achieve the necessary degree of performance and reliability, the drives are configured into storage subsystems based on RAID (Redundant Array of Independent Disks). Because of their mechanical nature, hard drives are relatively power-hungry and slow compared to most other computer components. Big data centers spend tremendous amounts on power, including cooling, adding significantly to their overall costs. Additionally, drives are orders of magnitude slower than electrical computer components, resulting in significant performance challenges any time disk I/O is required. Recently, SSDs (solid state drives) have emerged based on flash memory technology. Although too expensive to replace magnetic disks altogether, SSDs use less power and are significantly faster for certain operations.

This dissertation examines several new architectures that use a limited amount of faster hardware to decrease the power consumption and increase the performance or data redundancy of RAID storage subsystems. We present RAID4S, which uses SSDs for parity storage in a disk-SSD hybrid RAID system. Because of its

better performance characteristics, SSD parity storage reduces the

disk overhead associated with parity storage and thereby significantly reduces the disk overheads caused by RAID. This decreases the power consumption and can be used to increase the performance of simple RAID schemes or increase redundancy by enabling the use of higher order RAID schemes with less performance penalty. Storing parity on SSDs can reduce the average number of I/Os serviced by the remaining disks by up to 25-50%. By replacing some hard drives with SSDs, we reduce power and improve performance. Our RAID4S-modthresh optimization improves performance in certain workloads.

The other two architectures expose RAID's inability to handle heterogeneity in workloads and hardware. RAIDE is motivated by a workload imbalance we detected in stripe-unaligned workloads. By placing faster hardware to handle the higher workload at the edges of the array, we observed higher throughput. RAIDH places parity on faster devices based on device weights. Higher weighted devices are faster and thus store additional parity. The slowest devices may store no parity at all. This technique can be used on any heterogeneous array to enable faster random write throughput.

We consider the problem of scheduling a collection of processes, or tasks, on a multiprocessor platform. The tasks in question have real-time computing requirements, meaning that they have frequent processing deadlines that must be met. We are interested in optimal scheduling algorithms, which find a correct schedule for a set of tasks whenever it is possible to do so.

Recent work in the field has used the notions of fluid scheduling and deadline partitioning to guarantee optimality and improve performance. In the first part of this dissertation, we develop a unifying theory of existing approaches with the DP-Fair scheduling policy, and show how it easily proves and explains existing approaches. We then present a simple DP-Fair scheduling algorithm, DP-Wrap, which serves as a least common ancestor to many recent algorithms. We also show how to extend DP-Fair to the scheduling of sporadic tasks with arbitrary deadlines.

While easy to understand and implement, DP-Fair algorithms have the drawback of incurring a significant overhead in preemptions and migrations. In the second part of this dissertation, we present RUN, the first optimal algorithm which does not rely on a DP-Fair approach. Instead, RUN reduces the multiprocessor problem to a series of much easier uniprocessor problems. RUN's average preemptions per job never exceeded 3 in any simulation, and has a provable upper bound of 4 for most task sets. It also reduces to Partitioned EDF whenever a proper task-to-processor partitioning is found, and significantly outperforms all existing optimal algorithms.

We increasingly depend on well-behaved networks in the course of every-day activities for business, community, government, science, and recreation. And with more people demanding a greater variety of services comes sharp disagreement about which needs are most important. Unfortunately, today’s technology is inadequate to guarantee the performance of dynamic and diverse workloads in such a way that we are prevented from hurting each other’s goals.

No one likes waiting in traffic, whether on a road or on a computer network. Stuttering progress and slow interactive feedback annoy everyone and cost time and money. A network should aim to (1) minimize latency, (2) maximize bandwidth, (3) share resources according to agreements, (4) enable incremental deployment, and (5) minimize administrative overhead. Many technologies have been developed, but none yet satisfactorily address all five of these goals. The best performing solutions to date achieve some success in goals 1-3, but they often downplay the importance of goals 4-5. Solutions almost always require coordinated configurations across many pieces of a network, but they end up being either impractical or suffer poor performance when even a single piece of a network is not cooperating.

In this dissertation I present an overview of the multilayered problem surrounding network performance, a transport-level solution called TCP Inigo, and foundations for a clean-

slate queueing discipline solution based on a new combination of recent real-time and economic scheduling algorithms. TCP Inigo uses independent delay-based algorithms on the sender and receiver. In emulated experiments with single administrative domains, a situation common in data centers or a single HPC cluster, TCP Inigo’s fairness, bandwidth, and latency indices are up to 1.3× better than the best available solution. When deployed in a more complex environment, such as across administrative domains, TCP Inigo performs up to 42× better.

In the last decade, our ability to store data has grown at a greater rate than our ability to process that data via existing methods. This has given rise to an entire industry focused on developing new approaches to large-scale data-intensive computing that is commonly referred to as "big data".

The scientific community has seen a similar growth in the sizes of the datasets that they must analyze in order to advance their respective fields. Unfortunately, the unique characteristics of scientific data and computations are sufficiently distinct from those that garner the focus of the larger "big data" community that the direct application of existing systems is not feasible. The absence of a suitable platform for scalable data-intensive scientific computing has already become a hindrance to some scientific communities and the negative impact on research will only increase as datasets continue to grow.

In response to this clear and present need, we present our research into designing and building a distributed computing platform appropriate for researchers in a variety of fields that are continually evolving their approaches to processing scientific data. Our work extends Hadoop, an existing open source system based on the research described in the seminal 2004 MapReduce paper, in order to benefit from the existing state of the art. We first identify the barriers to executing common scientific computations over data residing in domain-specific file formats. Once those issues are addressed, we then consider how the unique properties of scientific data and computations can be leveraged to improve the system. The resulting alterations are then evaluated, both in terms of their impact to the MapReduce design as well as their impact on the performance characteristics of the system. The culmination of our work is a viable platform for scientific computing that has been utilized and validated by external researchers.