A membership inference attack is a method of extracting the training data from machine learning models. Previous analysis has characterized the worst case vulnerability to membership inference by instantiating the attack algorithm as the Bayes Optimal Classifier. We extend these findings by developing practical estimators for the worst case vulnerability on a sub-class of membership inference problems that are easy to compute without resorting to computationally expensive privacy auditing techniques. Extensive simulation studies are conducted on real world data sets to show that privacy auditing techniques, such as shadow modeling, can be replaced with the proposed worst case estimators. Furthermore, we examine the notion of disparity in membership inference: that some subgroups of the population are easier to identify in the training data set than others. We use a framework to quantify the degree of disparity and demonstrate that several real world models exhibit disparity in membership inference. We advocate that average metrics of attack accuracy, commonly usedin the privacy auditing literature, do not reliably convey the difference in privacy risks across different levels of the population.

In today’s world with ever increasing need for data collection, there is a rise in demand for privacy-preserving synthetic data generation and privacy auditing techniques to safeguard sensitive user information and data from privacy attacks. This paper explores the adversarial privacy auditing of synthetically generated data produced by Large Language Models (LLMs) using the TAPAS “Toolbox for Adversarial Privacy Auditing of Synthetic Data” framework. This paper uses a healthcare dataset with sensitive user information of Breast Cancer to evaluate the privacy of the data using adversarial techniques. The paper compares and contrasts the data quality, data distributions and privacy-preserving metrics of the real dataset with synthetically generated datasets from several sources including LLMs such as the GReaT framework and OpenAI's GPT4, Generative Adversarial Networks (GANs), and an AI-generated dataset produced using a proprietary technique from an industry startup, mostly.ai.

Cost estimation in the early mission life cycle is typically a difficult task, with the standard being utilizing historical analogues and heavy reliance on systems engineers with exceptional domain expertise to guide analysis. However, the lack of data points (previous flown missions) introduces large amounts of uncertainty and can call into question the validity of statistical results. Nonetheless, this task is of utmost importance in the context of exploring mission feasibility and particularly when trying to solicit work for a NASA Announcement of Opportunity. This paper explores the utilization of generative data science in the low-data paradigm, in order to train more effective and robust cost models. The development of generative methods and models to protect privacy and assist various machine learning tasks has been given much theoretical focus, but these frameworks have not been adopted widely in many realms that present great opportunities. Furthermore, we compare different generative methods by assessing the quality of marginal distributions, maintenance of conditional relationships that are grounded in physical spacecraft parameters, and by exploring their utility in simple machine learning tasks that represent the types of analyses done in mission formulation.

Synthetic data is a promising technology with numerous benefits for data sharing and machine learning workflows, such as augmenting available data, bolstering fairness, and implementing privacy. As synthetic data becomes more and more prevalent, understanding its methodology and strengths and weaknesses in regards to the above promises becomes crucial. This thesis in particular explores the privacy implications of generative models for synthetic data by investigating whether these models deliver on their proposed privacy guarantees. We analyze the increasingly popular GAN-based methods for generating synthetic data, including CTGAN, DP-CTGAN, and PATE-GAN. To investigate the privacy delivered by these methods, we focus on the approach of adversarial privacy auditing, which utilizes a toolbox of adversarial attacks to detect privacy leakage in (differentially private) algorithms. We aim to extend previous work in privacy auditing, which typically focuses on general machine learning algorithms, to the scarcely examined case of synthetic data generators by analyzing a range of models and datasets. Our goal is to demonstrate the need for further exploration and application of privacy auditing in the scenario of synthetic data, provide insights by comparing behavior across different datasets, and offer simulation results for future investigations into various privacy preservation patterns. For example, experimental results on various datasets and target data records reveal differences in privacy outcomes, highlighting the important role of the data, independent of the synthetic data generator, in privacy preservation. The findings highlight the importance of data-centric privacy evaluations and the need for further work to achieve a truly tight privacy analysis. This research hopes to serve as a foundation for further studying adversarial privacy auditing and possibly for the development of robust defenses against privacy leakage in the future.

Membership inference attacks can threaten the privacy of records in machine learning modelsby enabling adversaries to determine whether or not a record was used to train said model. In this paper we will be exploring the use of synthetic training data to defend against this form of attack. Synthetic data here keeps the attributes of the original training data set while maintaining machine learning utility. We use CTGAN and DP-CTGAN in order to generate high quality tabular synthetic training data. We evaluate the effectiveness of this approach empirically by comparing the vulnerability and utility of models trained with synthetic and real data. We also analyze the privacy-utility trade-off that comes with using synthetic data. Synthetic data seems to be a promising defense mechanism against membership inference attacks by providing increased privacy at reasonable utility losses.

Two-sided online matching is a crucial aspect of optimizing social welfare sequentially within economic frameworks, achieved through pairing participants via third-party platforms. These platforms are utilized across various marketplaces such as college admissions, ride-sharing, doctor placement, dating, and job applications. Typically, these markets allocate indivisible ``good'' to multiple agents based on mutual compatibility, with preferences often being unknown due to the large participant volume, making it explicitly challenging. Moreover, matching markets inherently involve scarcity due to limited resources on both sides. This dissertation presents significant advances in statistical sequential modeling for two-sided online matching markets, considering dynamic markets, quota constraints, and participants' incentive compatibility. Situated at the intersection of sequential decision-making algorithm design and economics, this work introduces new algorithms, theories, and insights with applications spanning economics, statistics, and machine learning.

Part I establishes foundational concepts of statistical sequential decision making and relevant economic terminology. Chapter 1 explores bandit algorithms, probability theory, and concentration inequalities, while Chapter 2 elucidates essential concepts of two-sided matching markets from an economic perspective, laying the groundwork for subsequent applications.

Part II presents a theoretical framework for multi-agent competitive two-sided matching markets, crucial for online recommendation systems in job markets. The first project, detailed in Chapter 3, introduces an online statistical ridge estimation method for the dynamic matching problem (DMP) with its application in the LinkedIn text data. The second project, discussed in Chapter 4, presents an online statistical sequential decision-making method for the competing matching under complementary preferences recommendation problem (CMCPR), along with a novel algorithm addressing both complementary preferences and quota constraints simultaneously.