This dissertation is composed of projects on three aspects of gathering and learning from data in the social sciences: drawing representative samples, taking valid measurement, and making warranted inductive inferences. Chapter one studies the challenge of drawing representative human samples. It is well documented that most samples used by studies in psychology-related fields are composed of Euro-American undergraduate students. Most writers agree that this is a serious problem for the generalizability of study results, but little improvement has occurred. By tracing the history of sampling, I identify the scientific and statistical rationale of sampling as a method of induction. I explain how the design-based approach, where to sample representatively is to sample randomly, became dominant. I show that this approach faces too many practical challenges within the social sciences to be useful as a guiding framework. In its place, I argue that the model-based approach, initially disfavoured for its theoretical shortcomings, is a better framework for the social sciences, because it allows the systematic integration of multiple imperfect samples. Instead of relying on one general framework to provide `in-principle' justifications to all cases, the model-based approach allows context and background knowledge to inform practice. Chapter two discusses measurement validity in the social sciences. Through an examination of the historical evolution of measurement and validity theories and the relationship between them, I argue that we should reject the view that measurement should be about an attribute that exists in the world in some robust sense, and that a pronouncement of measurement validity is a vindication of such an existence. First, I argue that this view, while attractive, has numerous theoretical difficulties and practical limitations. Next, I show that a rejection of this view, exemplified by the modern argument-based approach validity, presents a better perspetive in analyzing complex measurement problems in non-laboratory contexts. I conclude by pointing out that, based on the argument advanced in this paper, we should be more skeptical of ontological claims made on the basis of valid testing alone. Chapter three studies the problem of induction in the context of statistical learning theory. I examine a claim in the literature that the Vapnik-Chervonenkis (VC) theorem, which specifies conditions under which a problem is machine-learnable, offers a response to the problem of induction. I prove that the problem of when this learnability condition applies in general is uncomputable. Hence this solution strategy fails. If statistical learning theory is trustworthy at all, the justification of this trust must be in parallel with other inductive methodologies and, consequently, subject to the same challenges. I conclude this dissertation by arguing that a naturalistic, practice-first approach to philosophy of social science must pay attention not only to how a scientific method works in theory, but also to how it has been changed to accommodate resource-limited contexts.

At the heart of evolutionary theory is the concept of 'fitness', which is, standardly, an organism's reproductive success. Many evolutionary theorists argue, however, that to explain the evolution of social traits, such as altruism, we must use a different notion of fitness. This 'inclusive fitness', which includes the reproductive success of relatives, is seen as indispensable for studying social evolution. Recently, however, both biologists and philosophers have critically scrutinized its significance. My thesis explores the explanatory value of inclusive fitness, while attempting to resolve significant conceptual confusions. I argue that although inclusive fitness is not necessary for evolutionary explanations, it can nonetheless provide an extremely useful way of conceptualizing the evolutionary process.

This dissertation aims to naturalize decision theory by creating a model where an agent views herself as both a decision-maker and part of the natural world. The key contribution is a family of formal conditions that identify when an agent views herself as having control. In a slogan, an agent takes herself to control a partition if probability track her desirability. I call this approach a "desirability tracking" account of agency. I argue that this condition provides a place for individual purpose and effort, even for an agent who views herself as part of nature. I show how a desirability tracking approach allows us to chart a nuanced course between both sides of the "deliberation crowds out prediction debate" debate.

Over the past 40 years, decision theorists have produced an impressive body of literature employing dynamic choice arguments to defend the standard principles of Bayesian decision theory, with Peter Hammond's work serving as the locus classicus. However, examination of the import of these arguments has largely been restricted to the context of Savage-style decision theories that posit a sharp distinction between acts, states, and outcomes, while the more general framework developed in Richard Jeffrey's Logic of Decision has remained mostly neglected. This is remarkable given both the widely recognized appeal of dynamic choice arguments and the broad popularity that Jeffrey's framework enjoys among philosophical decision theorists. My dissertation aims to remedy this situation by extracting and defending what I take to be the core insights of dynamic choice arguments and exploring their significance in the context of Jeffrey-style decision theories.

What is the right statistical theory for scientific practice? What are the distinctive dynamics of social learning and conformity? And what can disagreement between theoretical models teach us? This dissertation consists of essays that engage just these questions.

Chapter 1 examines explanations for a broadly maligned research practice, HARKing, which has been called one of the ‘four horsemen’ of the replication crisis in the social and biomedical sciences (alongside publication bias, low statistical power, and p-hacking). In it, I demonstrate how classical accounts of why HARKing undermines the reliability of scientific findings must be wrong, and proffer what I take to be the correct, Bayesian account for when and why HARKing is in fact bad. Further, I consider the implications of all this for a prominent proposal for methodological reform in the context of the replication crisis.

Chapter 2 consists of more exploratory work, produced in collaboration with my colleague Cole Williams. We propose a simple model of social learning on networks under the influence of conformity bias. In our model, heterogeneous agents express public opinions where those expressions are driven by the competing priorities of accuracy and of conformity to one’s peers. Agents learn, by Bayesian conditionalization, from private evidence from nature, and from the public declarations of other agents. Our key findings are that networks that produce configurations of social relation- ships that sustain a diversity of opinions empower honest communication and more accurate beliefs and that the networks that do this best turn out to be those which are both less centralized and less connected.

Finally, chapter 3 consists of elucidating the relationship between two key models in evolutionary game theory–the replicator dynamics and Moran process. These models are connected by a mean-field relation- ship—the former describes the expected behavior of the latter. However, there are conditions under which their predictions diverge. I demonstrate that the divergence between their predictions is a function of standard techniques used in their analysis, and of differences in the idealizations involved in each. My analysis reveals problems for stochastic stability analysis in a broad class of games. I demonstrate a novel domain of agreement between the dynamics and consider a simple moral for scientific modeling.