UC San Diego

In many high-stakes machine learning problems, outcomes for a given input are only observed if a certain decision is made. For example, in loan prediction, an applicant’s loan repayment is only observed if the loan is provided. In these cases, the resulting missing data can lead to uncertainty in models trained on that data, and even with decisions that are globally optimal across both groups, the model can have differences in uncertainty between groups. In this paper, we use active learning with mutual information, or infomax learning, to establish that it could be more informative for a model to select from groups with more missing values. This establishes an epistemic argument, rather than a moral one, for intervening by sampling more from one group than another, indicating new opportunities and questions for fair, accurate prediction over time in these settings.