User-generated content on the Internet has been explosively growing in the current Web 2.0 era. This has been facilitated through widespread user access to the web through mobile devices, the rapid growth of social media applications, and review-based provider websites. The majority of this data is in the form of free text, as in social posts. Storing and querying this massive unstructured textual data is a challenging task that has been studied extensively recently.

Current search solutions, such as Google, Bing and Amazon’s internal search, are effective in allowing users to find relevant documents in large collections. Those solutions rely on several content and reputation-based factors including document relevance to the user query. However, capturing and exploiting user intent particularly, in a domain-specific setting, remains an open problem with a variety of research challenges. In this thesis, we study several such settings where existing search techniques are inadequate.

In particular, we studied the following subproblems where we are showcasing the benefit of leveraging domain-specific knowledge and user-generated content: 1) We argue for more effective item ranking for crowd-sourced review platforms and provide efficient algorithms to support it. 2) We provide a practical high-quality solution to build domain-specific ontologies from unstructured text documents. We describe our approach and provide fast and simple algorithms to use the generated ontology in extracting domain-specific features from the textual data. In particular, we describe our approach using a real-estate agency case study where domain agents are interested in evaluating the textual property descriptions. 3) We study how to search for similar documents, given a set of input documents, when the data source can only be accessed through a query interface (such as Google search). We propose a ranking model to extract effective query keywords from the input documents to retrieve similar documents through keyword-based search APIs. 4) We use data mining techniques to classify user-generated content on online forums in terms of its characteristics, such as bullying behavior. In particular, we crawl Yik Yak, an anonymous social media, to detect potentially harmful behaviors.

Social media users generate large volumes of data every day. Analysis of this data is an important tool in several areas. For example, the study of users' opinions, behaviors, and topics of discussion can be of use in the field of health care. The first part of my research involves using existing tools to perform analysis of Web content. Specifically, it first compares between health care provider attributes and quality measures of the insurance plans they accept. This is followed by analyses of how real estate prices and related metrics are affected by proximity to a university or hospital. Further, this research studies user behaviors and discussion topics to find differences in how various demographic groups generate content on health-related Web forums and on health-related discussions in general social media. The remainder of this dissertation shifts its focus from analysis of Web content to proposing new tools to perform similar analyses. It first proposes and evaluates natural language processing-based methods to automatically classify patient opinions in doctor reviews. This work also introduces a variant of the review classification problem where class labels can represent two opposing opinions that are not necessarily positive or negative. This is followed by an exploration of methods to effectively filter social media posts according to a user's interests. The key challenge behind this work is to determine how to use this information to maximize a trained text classification model's performance in classifying new posts. Finally, this dissertation proposes a multimodal Twitter embedding model that can leverage information from several parts of a tweet, such as text, image, and location. Such a model can have several applications for both researchers and Twitter users without the need to train a separate model for each application.

In this cloud era, data is being generated rapidly from billions of network users, mobile devices, social networks, sensors, and many other devices and applications. Compared to traditional relational databases which were optimized for read-heavy workloads, many modern NoSQL database systems choose log-structured merge (LSM) architectures to support high write throughput, including AsterixDB, Bigtable, Cassandra, Dynamo, HBase, LevelDB, and RocksDB. My research interests focus on the architectural design and optimization of the storage engines of such LSM systems. Specifically, my thesis targets three aspects: merge policies, spatial data, and partitioning.

First, a merge policy, also known as compaction strategy, is a critical component of an LSM system. It defines how data is organized on disk and highly affects the system's read and write performance as well as space utilization. Five state-of-the-art merge policies from existing LSM systems, including Bigtable, Constant, Exploring, Tiered, and Leveled, with two recently proposed policies, Binomial and MinLatency, are selected for comparison and evaluation of write, read and transient space amplification. We build and experimentally compare all these policies on the same platform. The experimental results show these new policies outperform the other strategies, as they offer a better trade-off between write and read amplification.

Second, most of the existing LSM systems are optimized only for single dimensional data, that is, they lack support for spatial indexes for spatial queries. To support spatial indexes, an LSM system must either index spatial data by mapping the spatial keys into single dimensional keys or provide native support for a secondary LSM R-tree index. Using an OpenStreetMap dataset and a synthetic dataset, we experimentally compare LSM R-tree indexes with four different merge policies: Concurrent, Binomial, Tiered, and Leveled (with three partitioning algorithms). We discuss our observations and recommendations with respect to the merge policy, comparator, and partitioning in Leveled policy.

Third, the incremental merge style of the Leveled policy makes it possible to break a big merge into multiple small sub-merges via partitioning. For certain workloads, such as sequential insertions, Leveled policy supports trivial-moves, where a whole partition is moved to the next level without any processing. Such features are missing from stack-based merge policies, such as Tiered, which often have many time-consuming large merges, and have no effective support for trivial moves to minimize disk I/O. We propose a novel global-range partitioning algorithm for stack-based merge policies to 1) improve the parallelism of merges to improve the overall write throughput; 2) increase opportunities for trivial-moves; and 3) enable a hybrid of stack-based and leveled merge policies.

The increased popularity of social media and the copious amount of user-generated data in the last few years have impacted various aspects of individuals’ lives. The use of social media for health care related purposes, which is the focus of this thesis, has increased exponentially.

This provides the researchers with a massive volume of data that can augment traditional health-related data sources (like electronic medical records) if properly mined and analyzed. Despite the advances in text analytics, it is challenging to analyze this data, due to its specialized vocabulary, the data collection, and the missing values.

In this thesis, we focus on two research directions: (a) Analyzing the demographics of users who participate in health-related social media, along with their posted content across a wide range of sources, and highlight specific health issues reported by users. (b) Effectively querying health-related social media or other health-related documents (can be generalized to the problem of querying annotated document).

Specifically, in our first contribution, we study the demographics of users who participate in health-related social media, to identify possible links to health care disparities. Using these demographics, our second contribution analyzes the content of posts grouped by demographic segments by implementing information extraction methods to extract medical concepts, identify top distinctive terms, and measure sentiment and emotion. We also extend our content analysis in the third contribution by studying the intent of posts generated by users for different data sources. Lastly, we focus on a specific domain, electronic cigarettes, and analyze the health-related effects reported by online users.

In the second direction of this thesis, we developed a query framework to help users efficiently explore health-related data, present in either online social media or other medical documents, by exploiting the relationships between the network users or the concepts inside the documents. Our solution is generalized to other domains with similar properties, such as general purpose social networks. We refer to this problem as keyword querying on graph-annotated documents, where we query documents annotated by interconnected entities, which are related to each other through association graphs. Our novel framework balances the importance of text relevance and semantic relevance through the graph.

The age of Big data has transformed into the era of Internet of Things (IoT) where massive scale data is generated, stored, and used by a diverse set of physical objects: devices, vehicles, buildings, software, sensors, GPS and networks. It has become an open challenge for researchers in academia and industry to find the best ways to ingest, replicate, manage, read and deliver this massively growing data efficiently to millions of users in real time. Big data storage systems -- especially NoSQL databases like LevelDB, Cassandra, BigTable and AsterixDB -- have become extremely popular in the last decade for managing large amounts of data that don't require the stringent concurrency or transaction management guarantees.

In such settings, NoSQL systems achieve high rates of data writes. My research interests focus on Input/Output (I/O) optimizations of such state-of-the-art big data storage systems. Specially my thesis aims mainly at three aspects of optimization: Indexing, Partitioning and Replication.

a) Indexing of non-key attributes: Current state-of-the-art big data storage systems have limited support for

secondary attribute lookup queries or continuous lookup queries. To tackle these limitations, first we introduce and implement five secondary indexes on a NoSQL database. Specifically, we use the popular LevelDB database, which employs Log-Structured Merge-Tree (LSM) for organizing its data.

Our comprehensive experimental study and theoretical evaluation provide empirical guidelines for optimal choice of secondary index, depending on the workload of different applications.

b) Indexing for publish-subscribe systems: We propose and compare several publish/subscribe storage architectures, based on the popular NoSQL LSM storage paradigm, to support high-throughput and highly dynamic continuous lookup queries. Our framework naturally supports subscriptions on both historic and future streaming data, and generates instant notifications.

c) Data partitioning: We create optimization techniques for spatial indexes via intelligent partitioning. Currently NoSQL based databases do not offer any spatial partitioning to achieve faster spatial query response. We propose a level-based organization of disk components and two novel component merge techniques that leverage their spatial properties.

d) Data replication: Another important feature of big storage systems is its availability and reliability, which is achieved through replication. Paxos is a widely used replication policy to ensure the replicas are in sync. We develop an I/O optimized Paxos-based fault-tolerant block storage replication engine.

With the ever-increasing volume of user-generated text (e.g., product reviews, doctor notes, chat logs), there is a need to distill valuable semantic information from such un-structured sources. We initially focus on product reviews, which conceptually consist of concepts (or aspects) such as “screen brightness”, and user opinions on these concepts such as "very positive". First, we present a novel review summarization framework that advances the state-of-the-art by leveraging a domain hierarchy of concepts to handle the semantic overlap among the aspects, and by accounting for different opinion levels. Second, we argue and empirically show that the current style of soliciting customer opinion by asking them to write free-form text reviews is suboptimal, as few aspects receive most of the ratings. Therefore, we propose various techniques to dynamically select which aspects to ask users to rate given the current review history of a product.

The last body of work leverages user chat logs to continuously optimize the workflow of a goal-oriented chatbot, such as a pizza ordering bot. On one hand, diagram-based chatbots are simple and interpretable but only support limited predefined conversation scenarios. On the other hand, the state-of-the-art Reinforcement Learning (RL) models can handle more scenarios but are not interpretable. We propose a hybrid method, which enforces workflow constraints in a chatbot, and uses RL to select the best chatbot response given the specified constraints.

In the last decade there has been a shift towards online

user-generated content. Services such Twitter, Facebook, Instagram or

Tumblr allow users to share their content and ideas in a simple

manner. The lower barrier of access (more non-technical users),

ubiquity and social capabilities generate the creation of content that

is user-centered: opinions, reviews, discussions and

interests. Effective and Efficient exploration and analysis of

user-generate content is an open and evolving question, which gained

increased interest in recent years.

The aim of my dissertation was to explore method and algorithms to

analyze and leverage the user-generated content. In particular our

objectives are as follows: 1) Study how we can leverage social network

data for prediction, recommendation or classification tasks; 2) Define

novel features that can be used to improve the efficiency/efficacy of

the models generated from user-content; and 3) Define tools and

algorithms that facilitate the use of user generated content.

This work has studied: how to collaboratively annotate user-generated

content in specific domains, how to define features that improve the

prediction accuracy of time series based on the social network

activity, how to define algorithms and measures to detect

spatio-temporal changes of topics, and how to improve accessibility to

the online social data, which is publicly available using a

restrictive API provided by the content provider.

Background

Objective

Methods

Results

Conclusions

Background

Objective

Methods

Results

Conclusions