In the last few years, the field of data science has been growing rapidly as various businesses have adopted statistical and machine learning techniques to empower their decision makings and applications. Scaling up analysis, possibly including the application of custom machine learning models, to large volumes of data, requires the utilization of distributed frameworks which can introduce serious technical challenges to data analysts and reduce their productivity.

In order to efficiently support the full Big Data analysis lifecycle without requiring extensive distributed systems knowledge, we extend data scientists’ familiar tool, Pandas dataframe, to operate on managed data at scale. We introduce AFrame, a new scalable analysis package that integrates a Pandas-like user experience with data management systems to provide analysts with a familiar working environment while scaling out the evaluation of the analytical operations over a large data cluster to enable analysis on large-scale managed datasets.

There are four aspects involved in this dissertation: The first is constructing a new framework ("AFrame"). We have implemented AFrame on top of Apache AsterixDB by transparently converting dataframe operations to SQL++ queries. The second aspect is making AFrame more flexible for deployment with other composable query languages by retargeting AFrame’s incremental query formation to other query-based database systems. The third aspect is creating a benchmark to evaluate our framework's performance. The fourth and final aspect is to demonstrate the feasibility and efficacy of our framework through a case study analysis.

With our world being more digitized than ever, handling Big Data has become a fundamental challenge in building modern applications and services. Although both academia and industry have developed a plethora of systems in recent years to help developers working with Big Data, many of them still follow the pattern of passively responding to users' queries, rather than processing and delivering data to interested users actively. We need systems for activating Big Data at scale and reducing the users' effort in working with Big Active Data.

In this dissertation, we explore three problems related to activating Big Data at scale. We first investigate the problem of enabling data enrichment during data ingestion. We discuss the needs and challenges in enriching data during data ingestion, and we introduce a new ingestion framework into AsterixDB - dynamic data feeds - that supports complex data enrichment functions and captures relevant data changes in the system during ingestion. We show the design and implementation of the new ingestion framework and evaluate its performance using different enrichment use cases.

Then, we look at the Big Active Data (BAD) challenge. We describe a BAD world that consists of different types of users and requests, and we propose a BAD system for providing BAD services for BAD users. We first review the initial prototype of the BAD system - BAD-RQ - and we discuss its limitations in BAD continuous use cases. We introduce a new BAD service - BAD-CQ - for providing continuous query semantics in the BAD system. Further, we use an alternative system constructed by gluing together multiple existing Big Data systems to show the challenges in providing BAD services without the BAD system. We measure the performance of BAD-CQ with various workloads and compare that with the alternative system's performance.

Last but not least, we study how to allow users to declaratively create scalable data sharing services between multiple BAD system instances without having to create and manage dedicated programs/services. We describe the notion of BAD islands that consist of multiple BAD instances and introduce new features to the BAD system for "bridging" multiple BAD instances together. We use a sample use case to illustrate how to create bridges between different BAD systems. To this end, we present a demonstration system that also involves the use of dynamic data feeds and BAD-CQ to show how BAD islands work.

Recently, there is a growing need for distributed graph processing systems that are capable of gracefully scaling to very large datasets. In the mean time, in real-world applications, it is highly desirable to reduce the tedious, inefficient ETL (extract, transform, load) gap between tabular data processing systems and graph processing systems. Unfortunately, those challenges have not been easily met due to the intense memory pressure imposed by process-centric, message passing designs that many graph processing systems follow, as well as the separation of tabular data processing runtimes and graph processing runtimes.

In this thesis, we explore the application of programming techniques and algorithms from the database systems world to the problem of scalable graph analysis. We first propose a bloat-aware design paradigm towards the development of efficient and scalable Big Data applications in object-oriented, GC enabled languages and demonstrate that programming under this paradigm does not incur significant programming burden but obtains remarkable performance gains (e.g., 2.5X).

Based on the design paradigm, we then build Pregelix, an open source distributed graph processing system which is based on an iterative dataflow design that is better tuned to handle both in-memory and out-of-core workloads. As such, Pregelix offers improved performance characteristics and scaling properties over current open source systems (e.g., we have seen up to 15X speedup compared to Apache Giraph and up to 35X speedup compared to distributed GraphLab).

Finally, we integrate Pregelix with the open source Big Data management system AsterixDB to offer users a mix of a vertex-oriented programming model and a declarative query language for richer forms of Big Graph analytics with reduced ETL pains.

In this thesis, we develop the support for continuous data ingestion in AsterixDB, an open-source Big Data Management System (BDMS) that provides a platform for storage and analysis of large volumes of semi-structured data. Data feeds are a mechanism for having continuous data arrive into a BDMS from external sources and incrementally populate a persisted dataset and associated indexes. The need to persist and index "fast-flowing'' high-velocity data (and support ad hoc analytical queries) is ubiquitous. However, the state of the art today involves 'gluing' together different systems. AsterixDB is different in being a unified system with "native support'' for data ingestion.

We discuss the challenges and present the design and implementation of the concepts involved in modeling and managing data feeds in AsterixDB. AsterixDB allows the runtime behavior, allocation of resources, and the offered degree of robustness to be customized (by associating an ingestion policy) to suit the application(s) that wish to consume the ingested data. Results from experiments that evaluate the scalability and fault-tolerance of the AsterixDB data feeds facility are reported. We include an evaluation of the built-in ingestion policies and study their effect as well on throughput and latency. An evaluation and comparison with a 'glued' together system formed from popular engines - Storm (for streaming) and MongoDB (for persistence) - is also included.

Big Data is turning to be a key basis for the competition and growth among various businesses. The emerging need to store and process huge volumes of data has resulted in the appearance of different Big Data serving systems with fundamental differences. Big Data benchmarking is a means to assist users to pick the correct system to fulfill their applications' needs. It can also help the developers of these systems to make the correct decisions in building and extending them. While there have been several major efforts in benchmarking Big Data systems, there are still a number of challenges and unmet needs in this area. This dissertation is aimed at contributing to the performance evaluation of Big Data systems from two major aspects. First, it uses both new and existing benchmarks to do deep performance analysis of two major classes of modern Big Data systems, namely NoSQL request-serving systems and Big Data analytics platforms. As its second contribution, this dissertation looks at Big Data benchmarking from a new angle by comparing Big Data systems with respect to their features and functionality. This effort is specifically important in the context of increasing interest in using a unified system which is rich in functionality to serve different types of workloads, especially as the Big Data applications evolve and become more complex.

In the last two decades, relational databases for analytics have been specialized to address the needs of analytical workloads. For instance, analytical workloads often focus on a few attributes (or columns) instead of the whole tuple. Thus, many analytical databases have opted to store the data in a columnar layout to reduce the I/O cost. Additionally, relational databases for analytics have shifted from interpreter structures for query processing to compiling queries into executable programs that can achieve the performance of hand-written specialized programs.

Document store database systems have gained traction for storing and querying large volumes of semi-structured data without requiring the users to pre-define a schema. This flexibility allows users to change the structure of incoming records without worrying about taking the system offline or hindering the performance of currently running queries. Despite their popularity, document stores have lacked the advances made for analytics-specialized relational databases for handling analytical workloads. In fact, the redundant information in the records (``embedded schemas'') stored in document stores can introduce an unnecessary storage overhead that can render document stores even less performant than non-analytics-specialized relational databases. Despite the performance gap between relational and document databases, many users have no choice but to use the slower yet flexible document stores. In this dissertation, we aim to optimize document stores and show that users can enjoy a similar performance of the analytics-specialized relational databases without sacrificing the flexibility of the document model. Specifically, this dissertation makes the following main contributions:

We first address the lack of schemas and the storage overhead in document stores by proposing the tuple compactor, a framework for inferring and extracting the schemas of self-describing semi-structured records. Our tuple compactor exploits the events of Log-structured Merge tree (LSM) storage, namely the flush operation, to infer and extract the schema during data ingestion. We experimentally demonstrate the efficiency of our tuple compactor on real, large datasets.

Second, we use the inferred schema and introduce two layouts for storing nested semi-structured data in columnar formats. We also propose several extensions to the Dremel format, a popular columnar format for nested data, to comply with document stores' flexible data model. Our experiments show significant performance gains, improving the query execution time by orders of magnitude while minimally impacting ingestion~performance.

Third, we present our code generation framework, which can handle the document data model's polymorphic nature and improve query execution performance in Apache AsterixDB. We analytically and empirically show that the CPU overhead of an interpreter-style execution engine can be a bottleneck even in disk-based databases like AsterixDB. However, our code generation framework can remedy the CPU overhead and significantly improve the performance of analytical queries. We also propose several improvements to AsterixDB's aggregation framework to optimize group-by queries, a crucial class of queries in analytical workloads.

In addition to storing and managing the data and providing capabilities to query them, aDatabase Management System (DBMS) tries to achieve performance goals. High resource utilization, high throughput, and low query execution time are a few of the performance goals that are considered for various DBMSs. The system’s success in achieving its performance goals highly depends on the performance of queries and their operators. Many factors can impact a query’s performance, including how much of its resource requirements are satisfied, when it is scheduled for execution, and which other queries will execute concurrently with it. This thesis is an experimental study focusing on resource management and scheduling techniques to assist a database management system in reaching its performance goals. We begin this thesis by exploring the design space for a robust dynamic Hybrid Hash Join operator, one of the main and most common types of memory-intensive database operators. Our variant of this operator is specifically designed to perform well even when the required statistics and information for a Hybrid Hash Join operator are unavailable or inaccurate. Next, we explore various memory management and execution strategies for efficiently executing queries containing multiple join operators. We specifically study variations of Left Deep Trees, Right Deep Trees, and Bushy Trees containing one to eight join operators. We evaluate their performance under different memory availabilities, join and scan selectivities, degrees of parallelism, storage types, and query complexities. Lastly, we study and evaluate the performance of various schedulers designed to schedule queries with highly different memory requirements and execution times in a concurrent environment. Our performance goal is to design a fair scheduler that keeps different classes of queries in admission and resource control queues in proportion to their execution times.

As big data evolves, more and more databases are incorporating a parallel architecture. Sorting a dataset on a key value is a required operation of any database. The hope is that a linear increase in computing power given to a database results in a linear increase in performance. One of these databases is AsterixDB, which has Repartitioning Parallel Sort, or RPS, as its sort operator.

The goal of this thesis is to optimize RPS to fully utilize the parallel nature of AsterixDB in all cases. Currently, the sort operator performs poorly when faced with an input dataset whose sort attribute is skewed on one or more identical values. In this thesis, we first discuss the current state of sorting in AsterixDB and the problems associated with it. Second, we go over a proposed optimization to the sort operator. Third, we compare the old approach with the new one with performance testing. Finally, we discuss some future work that can be done to further improve sorting in AsterixDB.

The increasing prevalence of large graph data has produced a variety of research and applications tailored toward graph data management. Users aiming to perform graph analytics will typically start by importing existing data into a separate graph-purposed storage engine. The cost of maintaining a separate system (e.g., the data copy, the associated queries, etc...) just for graph analytics may be prohibitive for users with Big Data. Furthermore, using separate systems for mixed-model analytics (e.g., JSON and graph) requires specialized solutions. In this thesis, we introduce Graphix and show how it enables property graph views of existing document data in AsterixDB, a Big Data management system boasting a partitioned-parallel query execution engine.

This thesis starts with a description of how Graphix property graphs naturally extend the AsterixDB document model to define vertices and edges. We detail how users can specify Graphix graphs in a manner that handles a wide variety of document-to-graph mappings while maintaining the schema-flexibility offered by AsterixDB. Next, we explain how users can query their Graphix graphs. The Graphix query language (gSQL++) minimally extends AsterixDB's query language (SQL++) to express synergistic graph and traditional (multi-model) analytics. After describing the user model aspects of Graphix, we detail how AsterixDB was extended to accommodate a recursive graph query construct: path finding. We focus on how the AsterixDB runtime layer was extended to realize semi-synchronous, partitioned-parallel recursion. We later discuss how to extend the query optimization layer as well as the query parsing and AST rewriting layer to reuse as much of AsterixDB as possible. This thesis concludes with an evaluation of Graphix against a native graph database, Neo4j. We show that Graphix is able to scale horizontally to perform on-par with (and in some cases, even outperform) Neo4j for many kinds of operational and analytical queries -- ultimately illustrating that users might not need a separate graph database just to issue graph queries.

In recent years, the world has seen an explosion in the amount of data being generated. Google proposed the MapReduce framework to allow programmers easily process massive amounts of data in parallel using a cluster of shared-nothing commodity machines. What started out as a tool for human efficiency subsequently began to be used as an intermediate representation for queries compiled from higher-level declarative languages. In this thesis, we present an alternate software stack for building scalable Big Data systems. We specifically focus on two parts of the stack. Hyracks is a new partitioned-parallel runtime layer that provides an efficient, generalized model for executing data-processing jobs on a cluster of commodity machines. Algebricks is a compiler framework that helps to build high-level declarative language compilers for parallel processing on top of Hyracks.