Emerging large-scale scientific applications involve massive, distributed, shared data collections (petabytes), and require robust, high performance for read-dominated workloads. Achieving robust performance (low variability) in storage systems is difficult. We propose RobuSTore, a novel storage technique, which combines erasure codes and speculative access to reduce performance variability and increase performance. RobuSTore uses erasure codes to add flexible redundancy then spreads the encoded data across a large number of disks. Speculative access to the redundant data from multiple disks enables application requests to be satisfied with only early-arriving blocks, reducing performance dependence on the behavior of individual disks. We present the design and an evaluation of RobuSTore which shows improved robustness, reducing the standard deviation of access latencies by as much as 5-fold vs. traditional RAID. In addition, RobuSTore improves access bandwidth by as much as 15-fold. A typical 1 GB read from 64 disks has average latency of 2 seconds with standard deviation of only 0.5 seconds or 25%. RobuSTore secures these benefits at the cost of a 2-3x storage capacity overhead and ~1.5x network and disk I/O overhead.

Pre-2018 CSE ID: CS2006-0851

A generic objective for a sensor network application is the gathering of data from a field of sensors. Because energy is often scarce in sensor networks, many techniques have been proposed to reduce data size within the network. These techniques either ignore the accuracy of the resulting data, or more often, provide no means for applications to control the resulting accuracy. However in many cases, applications have a quantitative requirement for sensor data accuracy, and the underlying system should meet that efficiently. In this paper, we describe a distributed algorithm that approximates and gathers data in an energy-efficient manner and strictly satisfies an application-provided accuracy requirement. This approximation is based on a hybrid data representation based on linear regression. A distinguishing feature of the proposed algorithm is that it absolutely does not require any models on statistical properties of data and noise, and needs only few general assumptions on sensor node topology. This feature enables the algorithm to serve as a general-purpose mechanism that can be widely used in many scenarios for data gathering-type applications. Simulation experiments with data traces from real environmental data show that it leverages the accuracy requirement to significantly reduce energy consumption.

Pre-2018 CSE ID: CS2006-0854

Overlay networks (proxy networks) have been used as a communication infrastructure to allow applications to communicate with users without revealing their IP addresses. Such proxy networks are used to enhance application security; including protecting applications from direct attacks and infrastructure Denial-of-Service attacks. However, the conditions under which such approaches can hide application location are not well understood. To shed light on this question, we develop a formal framework for proxy network approach to location-hiding which encompasses most of the proposed approaches. This framework is used to analyze the effectiveness of location-hiding: characterizing how attacks, defenses, and correlated host vulnerabilities affect feasibility. We find that existing approaches employing static structure (e.g. SOS and I3) cannot hide application location because attackers gain information monotonically and in a short period of time penetrate the proxy network. However, we find that adding defenses such as network reconfiguration or migration, which invalidate the information attackers have, makes location-hiding feasible for resisting penetration attacks. We also characterize when proxy networks are effective and stable in location-hiding. In these systems, proxy network depth and reconfiguration rates are critical factors for effectiveness. These results provide both deeper understanding of the location-hiding problem and guidelines for proxy network design.

Pre-2018 CSE ID: CS2004-0788

We examine the properties and performance of LT Codes for use in a high performance distributed file system. Different variations to the algorithm were created to achieve key characteristics, such as speed and data reliability. Finally, we evaluate parameter choices in order to balance the system and network loads.

Pre-2018 CSE ID: CS2004-0798

Accurate and continuous monitoring and profiling are important issues of performance tuning and scheduling optimization. In desktop grid systems based on sandboxing techniques these issues are particularly challenging because (1) subjobs inside sandboxes are executed in a virtual environment and (2) sandboxes are usually reset to an initial (empty) state at each subjob termination. To address this problem, we present in this paper DGMonitor, a monitoring tool to build a global, accurate and continuous view of real resource utilization for desktop grids based on sandboxing techniques. Our monitoring tool provides unobtrusive and reliable performance measures, uses a simple performance data model and is easy to use. Our work demonstrates that DGMonitor can easily take over the monitoring of large desktop grids (up to 12000 workers) maintaining low load in terms of resource consumption due to the monitoring process (less than 0.1%) on desktop PCs. Although we use DGMonitor with the Entropia DGrid platform, it can be easily integrated in other desktop grids and its data can be used as an information source for existing information services for performance tuning and scheduling optimization. Keywords: Performance monitoring and profiling, desktop grids, sandboxing techniques, distributed computing.

Pre-2018 CSE ID: CS2003-0770

A critical technology for Grid computing is the ability to describe, select, and bind appropriate resources for synchronous use by applications. Our Virtual Grid Description Language (vgDL) allows applications to describe and manage resources conveniently via application-level abstractions and enables a novel approach to application-driven resource management called "finding and binding". We explore the viability of this strategy to enables applications to obtain complex resource collections in competitive resource environments. Our evaluation shows that combined resource "selection and binding" can scale well to millions of hosts, identifying good matches in a few seconds for the most complex resource requests. The combined selection and binding improves success rates for complex requests and the advantage increases for more complex requests and higher resource competition. Combined Selection and Binding can double the resource utilization (from 30% to 70%) at which synchronous resource allocation and use across as many as sixteen resource managers is possible.

Pre-2018 CSE ID: CS2005-0825

Simple resource specification, resource selection, and effective binding are critical capabilities for Grid middleware. We describe the Virtual Grid, an abstraction for dynamic grid applications to deal with complex resource environments. Elements of the Virtual Grid include a novel resource description language (vgDL) and a resource selection and binding component (vgFAB), which accepts a vgDL specification and returns a Virtual Grid, that is, a set of selected and bound resources. The goals of vgFAB are efficiency, scalability, robustness to high resource contention, and the ability to produce results with quantifiable high quality. We present the design of vgDL, showing how it captures application-level resource abstractions using resource aggregates and connectivity amongst them.

Pre-2018 CSE ID: CS2005-0817

Desktop distributed computing allows companies to exploit the idle cycles on pervasive desktop PC systems to increase the available computing power by orders of magnitude (10x - 1000x). Applications are submitted, distributed, and run on a grid of desktop PCs. Since the applications may be malformed, or malicious, the key challenges for a desktop grid are how to 1) control the distributed computing application's resource usage and behavior as it runs on the desktop PC, 2) provide protection for the distributed application's program and its data, and 3) provide a virtualized execution environment, decoupling correct application execution from desktop configuration heterogeneity. We describe the Entropia Virtual Machine, and the solutions it embodies for each of these challenges. We first outline the distinct requirements for a virtual machine in a desktop grid environment, which differ from web-based and hardware-level virtual machines. We then describe how the Entropia Virtual Machine (EVM) uses binary rewriting to solve these problems, guaranteeing the usability and protection of the desktop machine in the face of malicious or malformed application code. We also show that our approach provides security for the applications and their data. We detail how the virtual machine supports a virtualized execution environment. We present performance results for the EVM which show its performance overhead is modest, less than 6% for the applications studied.

Pre-2018 CSE ID: CS2003-0773

Media companies and other organizations with large amounts of digital content require transfer of extremely large files in a short time from a single source to a collection of geographically dispersed destinations. Due to the high cost of terrestrial networks of sufficient scope and bandwidth, satellite networks are the most common means for performing such transfers. However, satellite file transfer relies on expensive error correction based on a combination of forward error correction and whole-file retransmission. This paper presents a hybrid solution that combines the advantages of satellite and terrestrial networks to provide cost-effective reliable file transfer. Specifically, we propose a new peer-to-peer (P2P) scheme that exploits fast terrestrial networks and the availability of multiple receivers to recover from high loss rates (5% or more) in near real-time (latency<400ms). This solution is efficient, robust under variable packet loss and connectivity, user tunable, scales to hundreds of nodes, and doubles bandwidth compared to existing approaches. The system has been validated via extensive simulations using a terrestrial network based on the AT&T common backbone core network.

Pre-2018 CSE ID: CS2005-0842