Programmers often write code with similarity to existing code written somewhere. Code search tools can help developers find similar solutions and identify possible improvements. For code search tools, good search results rely on valid data collection. Social coding websites, such as Question & Answer forum Stack Overflow (SO) and project repository GitHub, are popular destinations when programmers look for how to achieve certain programming tasks. Over the years, SO and GitHub have accumulated an enormous knowledge base of, and around, code. Since these software artifacts are publicly available, it is possible to leverage them in code search tools. This dissertation explores the opportunities of leveraging software artifacts from the social coding websites in searching for not just similar, but related, code.

Programmers query SO and GitHub extensively to search for suitable code for reuse, however, not much is known about the usability or quality of the available code from each website. This dissertation first investigates under what circumstances the software artifacts found in social coding websites can be leveraged for purposes other than their immediate use by developers. It points out a number of problems that need to be addressed before those artifacts can be leveraged for code search and development tools. Specifically, triviality, fragility, and duplication, dominate these artifacts. However, when these problems are addressed, there is still a considerable amount of good quality artifacts that can be leveraged.

SO and GitHub are not only two separate data resources, moreover, they together, belong to a larger system of software development process: the same users that rely on facilities of GitHub often seeks support on SO for their problems, and return to GitHub to apply the knowledge acquired. This dissertation further studies the crossover of software artifacts between SO and GitHub, and categorizes the adaptations from a SO code snippet to its GitHub counterparts.

Existing search tools only recommend other code locations that are syntactically or semantically similar to the given code but do not reason about other kinds of relevant code that a developer should also pay attention to, e.g., auxiliary code to accomplish a complete task. With the good quality software artifacts and crossover between the two systems available, this dissertation presents two approaches that leverage these artifacts in searching for related code. Aroma indexes GitHub projects, takes a partial code snippet as input, searches the corpus for methods containing the partial code snippet, and clusters and intersects the results of the search to recommend. Aroma is evaluated on randomly selected queries created from the GitHub corpus, as well as queries derived from SO code snippets. It recommends related code for error checking and handling, objects configuring, etc. Furthermore, a user study is conducted where industrial developers are asked to complete programming tasks using Aroma and provide feedback. The results indicate that Aroma is capable of retrieving and recommending relevant code snippets efficiently. CodeAid reuses the crossover between SO and GitHub and recommends related code outside of a method body. For each SO snippet as a query, CodeAid retrieves the co-occurring code fragments for its GitHub counterparts and clusters them to recommend common ones. 74% of the common co-occurring code fragments represent related functionality that should be included in code search results. Three major types of relevancy--complementary, supplementary, and alternative methods, are identified.

Distributed systems are the backbone of a wide range of highly important applications, but developing and debugging such systems become tricky with the synchronization mechanism needing to meet critical consistency and latency requirements. A range of models, frameworks and languages have been developed to tackle such complexity in allowing a shared computing state. In this thesis, based on the Global Object Tracker (GoT) distributed programming model proposed in previous research, I present a re-implementation of the core components of Spacetime, a GoT based Python framework. The C++ implementation tackles multiple limitations the pure Python implementation was faced with, and further improves the performance with redesigned core structures. The internal designs and comparison with previous designs will be discussed throughout this thesis. The C++ core also allows interoperability with other high level languages, such as javascript. Through benchmarks comparing both implementations and the Parse platform with respect to update latency, we are able to show significant performance improvement over the pure Python implementation when the job scales up, and have a competitive performance against Parse in many configurations.

Simulations are powerful computational tools to comprehend complex systems in a large number of scientific domains. Even with the advance of computational power, the capability to model the real-world in full detail is still far in the future. But at every step forward in computational power, more can be done with discrete simulations to predict and explain physical systems in domains where continuous mathematical representations are insufficient or unknown. The field of distributed simulations was born to leverage simulation capabilities past what a singular computer can perform. Furthermore, distributed simulation frameworks enables a collaborative development environment, where multiple developers can work in independent parts of the same problem – similarly to large software problems being solved by the collaborative effort of software developers working on semi-independent modules. Unfortunately, like in most distributed system applications, distributed simulations are very complex to design and develop. While there are supporting frameworks, like the High Level Architecture (HLA), there is room for improvement in reducing the technical knowledge required to develop for such frameworks.

The goal of this dissertation is to expand the expressiveness of distributed simulation frameworks while reducing the technical complexity to develop integrated distributed simulations independently. This dissertation presents a novel architecture for distributed interactive simulations (DIS) called Collaborative Aspect-oriented DIS (CADIS). The architecture is divided in two frameworks called PCC and spacetime. PCC (Predicate Collection Classes) is an object-oriented programming model for representing collections of objects. These collections of objects are called relational data types – types that are reclassified based on runtime values. Relational types allows the expressiveness of queries that are common to relational databases to be defined as abstract data types. Spacetime adds automatic synchronization and recalculation of PCC data types in discrete time distributed simulations. Through a simple push, update, and pull process, simulations operate with newly reclassified objects, and any modifications are automatically pushed to the server.

CADIS is evaluated in four different ways. First, a benchmark measures the performance impact of each individual relational operation, demonstrating the many performance improvements in the implementation required to reduce the data exchange between simulation and servers to an acceptable level. Second, a feasibility evaluation using a realistic urban simulation scenario is presented, demonstrating effectiveness in a real-world problem. Third, a case study using CADIS in a graduate course is presented, demonstrating how CADIS can be taught to developers with little simulation background in a small period of time. Finally, a parallel between HLA and CADIS is presented, by analyzing a converted urban simulation event routing system to HLA.

The architecture of CADIS provides two major contributions to the field of distributed simulations: a) it allows the development effort to be partitioned and split among independent groups of developers; b) thanks to relational data types in PCC, simulations can join at runtime with new data models without affecting existing simulations. Furthermore, existing simulations can handle and use new data models that are related to known ones. The larger impact of CADIS is creating a scalable service-oriented development environment, where simulations are providers of data types with specific features, allowing developers to add complexity to data models based off previous implementations.

Game designers and researchers agree that the main motivation for starting playing a game is challenge. It is only a small step to say that when the game becomes too difficult, players can become frustrated and quit. While extensive work shows that challenge is central in player enjoyment, its influence on player retention has received little attention. Establishing this influence is difficult for several reasons: (i) Definitions of challenge tend to be game- or genre-specific. (ii) Measuring retention accurately requires real-life data, which is often hard to access. (iii) Challenge can be intertwined with other factors, such as social interactions in multi-player games.

Addressing these three limitations requires a mixed-method, holistic, and cross-game approach paying close attention to the game mechanics. Therefore, we use a diverse mix of data sources, ranging from interviews to online questionnaires and telemetry logs, to explore player behavior in five games: Ragnarok Online, World of Warcraft, Forza Motorsport 4, Jelly Splash, and an undisclosed commercial mobile game.

This work provides empirical evidence that challenge does influence retention, and that many contextual factors, namely player segments, social interactions, time, and money, nuance and mediate this influence.

These findings may help game developers make better games, expand our knowledge of challenge, retention, and their relationship, and improve the bottom-line of game companies by making games more engaging and players stay longer.

Clone detection locates exact or similar pieces of code, known as clones, within or between software systems. With the amount of source code increasing steadily, large-scale clone detection has become a necessity. Large code bases and repositories of projects have led to several new use cases of clone detection including mining library candidates, detecting similar mobile applications, detection of license violations, reverse engineering product lines, finding the provenance of a component, and code search.

While several techniques have been proposed for clone detection over many years, accuracy and scalability of clone detection tools and techniques still remains an active area of research. Specifically, there is a marked lack in clone detectors that scale to large systems or repositories, particularly for detecting near-miss clones where significant editing activities may have taken place in the cloned code.

The problem stated above motivates the need for clone detection techniques and tools that satisfy the following requirements: (1) accurate detection of near-miss clones, where minor to significant editing changes occur in the copy/pasted fragments; (2) scalability to hundreds of millions of lines of code and several thousand projects; and (3) minimal dependency on programming languages.

To that effect, this dissertation presents SourcererCC, an accurate, near-miss clone detection tool that scales to hundreds of millions of lines of code (MLOC) on a single standard machine. The core idea of SourcererCC is to build an optimized index of code blocks and compare them using a simple bag-of-tokens strategy, which is very effective in detecting near-miss clones. Coupled with several filtering heuristics that reduce the size of the index, this approach is also very efficient, as it reduces the number of code block comparisons to detect the clones.

This dissertation evaluates scalability, execution time, and accuracy of SourcererCC against four state-of-the-art open-source tools: CCFinderX, Deckard, iClones, and NiCad. To measure scalability, the performance of the tools is evaluated on inter-project software repository IJaDataset-2.0, consisting of 25,000 projects, containing 3 million files and 250 MLOC. To measure precision and recall, two recent benchmarks are used: (1) a benchmark of real clones, BigCloneBench, that spans the four primary clone types and the full spectrum of syntactical similarity in three different languages (Java, C, and C#); and (2) a Mutation/Injection-based framework of thousands of fine-grained artificial clones. The results of these experiments suggest that SourcererCC improves the state-of-the-art in code clone detection by being the most scalable technique known so far, with accuracy at par with the current state-of-the-art tools.

Additionally, this dissertation presents two tools built on top of SourcererCC: (i) SourcererCC-D: a distributed version of SourcererCC that exploits the inherent parallelism present in SourcererCC's approach to scale horizontally on a cluster of commodity machines for large scale code clone detection. Our experiments demonstrate SourcererCC-D's ability to achieve ideal speed-up and near linear scale-up on large datasets; and (ii) SourcererCC-I: an interactive and real-time version of SourcererCC that is integrated with the Eclipse development environment. SourcererCC-I is built to support developers in clone-aware development and maintenance activities. Finally, this dissertation concludes by presenting two empirical studies conducted using SourcererCC to demonstrate its effectiveness in practice.

Code Clone detection is important in software engineering as it aims at solving various problems like code maintenance, identification, code reuse, scalability, and plagiarism. Software development revolves around implementing logic using tools and technologies where every developer has a different coding style and logical approach to reach required goals. However, the end result of many implementations can be the same. This is where the need for code maintainability, reusability, and optimization arises. Code clone detection can help to leverage the immensely large source codes available on the web to attenuate code writing time by reusing sources available online. Clone detection in source code is based on the similarity of the program content or similarity in the program functionality. There are many techniques that have been tried and tested in the literature. However, these naïve approaches do not perform adequately for higher-level clones. In this thesis, I am exploring deep learning based technique Code2Vec. In order to identify, compare, and reuse an existing piece of code, deep learning techniques can help to predict if a similar implementation source code exists in a codebase or a dataset of codes. In this thesis, the approach of representing code in the form of vectors and applying Natural Language Processing for code clone detection has been discussed. The scope of the thesis is to devise an approach for the detection of similar functional methods in Java GitHub code repositories, expanding on the Code2Vec model [1]. I demonstrate the capabilities of applying the Code2Vec model to Java source code in order to determine the path vectors for method similarity detection. Furthermore, I discuss the design, architecture, and usage of the model. An in-depth analysis of preprocessing mechanisms, data collection, data preprocessing is also highlighted. In this thesis, I apply a normalization technique to update the variable names in the Code2Vec approach and compare the baseline Code2Vec model with the normalized model. The comparison shows improved precision results in the normalized code clone detection approach. Finally, the benefits of my approach and a detailed analysis of results on a dataset with Java methods are presented. The results are evaluated on the basis of Recall and Precision. I evaluate the recall with the help of BigCloneBench and precision using InspectorClone and MeasurePrecision open-source tools.

Similarity detection in software systems, also known as clone detection, has been a focus of software engineering in the past years. Cloning can be defined based on simple similarity concepts, such as similarity in pure syntactic features, or based on more sophisticated notions of similarity, where the ultimate functionality (or behavior), with less or no syntactic signals, is the focus. The latter type of clones are known as functional clones.A special case of functional clones, which is becoming more prevalent with the advances in artificial intelligence, is concerned with the behavioral similarity among deep neural network (DNN) models. DNN models are functions as they define a portion of a software system's functionality. The wide adoption of these components in software brings new challenges to similarity detection techniques. DNN models are black boxes containing matrices of numbers learned from a training dataset. The training code contains little knowledge about the ultimate model's behavior, and similar training scripts and network architectures may end up producing completely different models. Model comparison, therefore, cannot rely on the structural properties of the models or their training scripts. Instead, it must compare the models' outputs on canonical inputs which generally are the training or testing datasets. However, such datasets may not always be available due to reasons such as the independence of models' deployment from their datasets, and privacy or security concerns. These issues motivate the need for an approach that can automatically detect functional similarity among DNN models in the absence of canonical inputs. This dissertation starts by looking into the problem of functional clone detection by presenting Oreo, a code clone detection tool focused on clones with diminished syntactic but high semantic similarity. It then presents a systematic study on precision of code clone detection tools, highlighting the importance of looking into clone types (and more importantly, types with decreased syntactic similarity) when measuring the precision. The dissertation continues by formulating the problem of DNN functional similarity detection in the absence of canonical inputs. To solve this problem, it then introduces RICA, a technique that works by generating random inputs to be used instead of canonical inputs for the purpose of similarity detection. Three similarity metrics that can be used with RICA are presented and their strengths and weaknesses are highlighted. The evaluation of RICA is done by performing extensive experiments using a dataset of more than 56K classifiers collected from GitHub. RICA's evaluation shows that it has high precision and recall, and highlights the effectiveness of the similarity metrics used by it. Running RICA on the entire dataset of 56K classifiers results in performing more than 7 million comparisons and finding a cloning percentage of 26% among the analyzed models. This is followed by showing how RICA's applicability can be extended beyond classifiers: applying RICA on a regression task demonstrates its effectiveness in finding clones of regression models. Furthermore, a sensitivity analysis reveals how certain model and training properties affect the performance of RICA. Finally, a taxonomy of DNN clone types is proposed which helps in specifying the ultimate capabilities and limitations of RICA, as well as being helpful in future studies of DNN similarity detection.

A large number of open source projects are hosted on the Internet by popular repository sites like GitHub, SourceForge, BitBucket, etc. These repositories are becoming more popular and growing in size. There are many research projects that mine these software repositories for valuable information.

Compiling the projects found in these repositories can help lter out the good, and usable

projects. It gives us a guarantee that the source code is syntactically correct, and that

all the dependencies of the project are either self contained or accessible on the Internet.

Projects can be maintained and organized in dierent ways depending on the developer culture and practice. Unfortunately, very often repositories fail to capture the environmental assumptions made by the developers such as build tools, versions, presence of external dependencies, etc. This heterogeneous nature of the projects makes the successful compilation of large numbers of projects a challenging task as one solution cannot be applied to all. It is impractical to manually correct the compilation of every project. We designed several heuristics to maximize the number of projects compiling successfully in a repository.

Sourcerer is an infrastructure for large-scale collection and analysis of open-source code. It crawls open-source Java projects from various sources on the Internet and builds an aggregated repository, and database. We used the information found in the database to automatically compile more than 55,000 Java projects in the Sourcerer repository. We propose several general, language independent heuristics to tackle the most common errors.

Using these heuristics, we were capable of building 33.18% of the projects in the repository, successfully building more than 18,000 Java projects.

Synchronization in distributed applications with shared, highly mutable replicated state often requires complex engineering to maintain low latency for the propagation of updates. At their core, distributed applications with replicated state need to implement some form of version control to be able to deal with concurrent updates and still maintain some notion of consistency. This need leads to complex and ad-hoc code that is hard to maintain and easy to break. To simplify the creation of such applications, I propose the Global Object Tracker (GoT) model, an object-oriented programming model based on causal consistency, whose design and interfaces mirror those found in decentralized version control systems: a version graph, working data, diffs, commit, checkout, fetch, push, and merge. I have implemented GoT in a framework called Spacetime, written in Python.

GoT is designed to ensure causal consistency with low latency updates in all sorts of distributed application topologies. These include peer-to-peer and applications subject to, or requiring, temporary isolation of nodes. The key to being able to do this is a novel approach to storing and merging concurrent updates within the version graph. GoT stores the version history as a graph of updates instead of calculating the whole state at each version. This allows GoT to send delta updates between nodes without having to calculate the deltas. The ability to send delta updates along with the lack of heavy serialization and deserialization allows for the updates to be quickly available. The final state at each node is only constructed when needed.

In addition to the lower latency of updates, GoT addresses three concerns. First, object-based version control systems are typically not suitable for peer to peer networks. The merge strategies used are incompatible with the communication patterns seen in peer to peer. GoT addresses this concern with a novel approach to merging concurrent updates, called the small-step merge. Second, GoT makes the state changes in the system more visible, allowing for powerful tools such as interactive debuggers to be built and integrated, helping developers find and fix errors. Finally, GoT provides these benefits to not just push-style communication but also pull-style, allowing for isolation and partition recovery.

In its traditional form, GoT is impractical for real systems, because of the unbounded growth of the version graph. I present my solution to this problem that adds constraints to GoT applications but makes the model feasible in practice. I show that Spacetime is not just feasible, but viable for real applications, and present performance results showing low update latency.

Debugging distributed systems is difficult. Most of the techniques that have been developed for debugging such systems use either model checking, or postmortem analysis of logs and traces. Interactive debugging is a widely-used technique for debugging single threaded applications. But only a few interactive debuggers for distributed systems are available as the techniques used for building traditional debuggers for sequential single-threaded programs cannot be easily transferred to a distributed setting. In this thesis, I look at the topic of designing interactive debuggers for distributed systems and introduce GoTcha which is an interactive debugging tool for distributed systems based on the GoT distributed programming model. GoTcha supports the core features of traditional single-threaded debuggers like step-through debugging and breakpoints so that users can wield these powerful tools in a distributed context. I demonstrate the capabilities of GoTcha through a series of usage examples, and discuss the design and architecture of GoTcha.