Prologue

Symposium on Model Uncertainty and Model Selection

In sociological inquiry, topics of model uncertainty and model selection raise some of the most fundamental issues that researchers may confront: what is the “right model” given the theory and the data at hand? Although the typical researcher makes decisions of model selection using routine, or standard, procedures, model uncertainty and model selection are topics that deserve more nuanced attention. Thinking seriously about the traditions within sociology of hypothesis testing and model selection can help improve our understanding of human social behavior. These issues have been around for a long time; there is a previous literature on this topic in econometrics and in statistics (e.g., see Leamer 1983, 1985; Sala-i-Martin 1997). In this symposium the articles and the commentaries present a variety of points of view on issues related to model uncertainty and model selection. In sociological applications, model uncertainty involves the absence of knowledge regarding proper variable selection, as well as uncertainty about the values of model parameters. Writing specifically about macrosociological research, Western (1996) linked model uncertainty to the existence of vague theory and the lack of prior knowledge about the phenomenon of interest, concerning both theoretical variables of interest and key control variables. He noted that the theories that drive research provide too little guidance beyond suggesting large sets of candidate variables for quantitative analysis. His argument also applies to other types of sociological research, and as Western observed, it is certainly not limited to statistical analysis; but what is clear is that all methods share a common concern that inferences that result from uncertainty can produce misleading conclusions. The problem of model selection is typically thought of as the flip side of model uncertainty; this refers to selecting a model, given the constraints of available data, from a set of candidate models that vary in their complexity, their fit to the data (and/or their predictive power), and their compatibility with existing theory.

More recently, Young (2009) noted that where, given an outcome of interest and a set of potential predictors, there is no one “true” specification of a model that estimates the effects of specific variables, the problem of “parameter uncertainty” frequently occurs. There is a range of circumstances that may confront the researcher in this case. Considerations are different in well-developed research areas, where there are prior expectations for the nature of the models, especially in areas in which the goal is to add important information regarding explanatory mechanisms to the understanding of particular outcomes. In contrast, in less well developed research areas, in which there are few if any prior expectations, a bigger problem exists that may lead to the capitalization on chance. Eliminating false positives becomes of primary concern in this case, and these problems are addressed by John Muñoz and Cristobal Young in “We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness.” This article echoes similar previous considerations in the econometrics literature, and their title is reminiscent of the title of Xavier Sala-i-Martin’s 1997 American Economic Review article, “I Just Ran Two Million Regressions.” Muñoz and Young have raised the computational ante, and by computational methods they draw attention to the possibilities of learning from data in a context in which there is weak prior information. Specifically, they consider model uncertainty in the familiar applied situation in which interest focuses on one predictor in a regression analysis, but the researcher is unsure which control variables to include in the model. Given a degree of multicollinearity, this is a situation in which false positives may arise, and their article demonstrates that these problems can be evaluated through the use of computational model robustness analysis.

The commentaries by Robert M. O’Brien and Bruce Western, both quantitative sociologists who have wrestled with these issues, raise a number of questions and provide an interesting traverse through this terrain. O’Brien approaches the article by raising a series of questions that should be on the minds of readers confronting these issues, including the distinction between model influence (or the stability of parameter estimates) and model fit as a criterion for model selection. Western, who has visited and revisited the problems of vagueness of theoretical propositions in macrosociology, suggests that much sociological analysis involves weak prior expectations and involves an iterative and inductive process of learning from the data at hand. Western’s approach is from a Bayesian perspective, suggesting that this framework provides a few extensions to their analysis. In their rejoinder, Muñoz and Young raise the question of whether models can be weighted by their probability of being true. The collective conclusion is that this work provides a strong contribution to the discussion of model uncertainty and model selection.

Concerns with model selection are not unique to the mathematical and statistical sciences. Indeed, the field of philosophy of science has historically pioneered in considerations of model selection, from all the way back to Aristotle and Ptolemy to Ockham’s principle of parsimony (see Sober 2015). The concept of underdetermination is used as a starting point by Michael Schultz in “The Problem of Underdetermination in Model Selection.” Schultz provides an interesting counterpoint to the computational approaches considered by Muñoz and Young. He argues that the procedures in conventional sociological model selection emphasize their ability to best represent the data at hand, and in so doing they ignore their dependence on theoretical and methodological assumptions. He notes that standard model selection criteria used in sociology, such as the Akaike information criterion (AIC) (Akaike 1974), the Bayesian information criterion (Raftery 1995), or the minimum description length principle (Rissanen 1978), are concerned solely with the empirical fit of the model to the data at hand, and they do not take theoretical criteria into account. He argues that rather than ignoring theoretical underdetermination, modeling can be made more rigorous and effective by acknowledging and addressing the problem more directly. Schultz develops an alternative to standard model selection approaches and proposes a new modified version of the AIC involving nested models, which he calls the inferential information criterion (IIC). This index measures the trade-off between a given model’s AIC and its conditional probability. Commentaries on Schultz’s argument by David L. Weakliem and Olav B. Vassend raise a number of technical and substantive issues with Schultz’s approach. Vassend, a philosopher, analyzes the IIC index from a Bayesian point of view and draws several conclusions about the utility of this index. Similarly, Weakliem agrees that there are fundamental problems with model selection and the testing of sociological theories, and he suggests that more can be learned from Schultz’s alternative approach. In his rejoinder, Schultz concedes there is more work to be done in developing a workable approach to model selection that takes theoretical underdetermination into account.

Taken together, the articles and commentaries within this symposium further reinforce the idea that the consideration of issues of model uncertainty and model selection move us well beyond traditional approaches to hypothesis testing. These discussions open up the larger questions of how we can learn from data without blindly following standard procedures for examining data and without ignoring sociological theory—even if the theory may be vague and offer weak prior expectations. These discussions suggest that we can improve our approaches to both theory and data by moving beyond mindless approaches that result in the acceptance of false notions about the relative importance of the events and exposures of social life. Issues of model uncertainty and model selection should motivate us to take a broader approach to data analysis and the examination of sociological theory.

Integrating Event History and Sequence Analysis

As more longitudinal data become available, sociological research is increasingly attending to questions about processes, transitions, and change (see Alwin 2012). Two of the most common methods sociologists use for analyzing such topics are event history analysis and sequence analysis. Event history analysis, particularly the subclass of multistate models, focuses on the hazards of transitioning between states and, implicitly, the duration of time spent in different states. In event history models, transitions and durations are the focal outcomes, which allow researchers to study how time-varying covariates affect different types of transitions. Sequence analysis, by contrast, focuses on the relative ordering of successive states, the dominant trajectories that people or other entities take through the set of possible states. In sequence analysis, the trajectories are the focal outcome, and researchers are typically unable to examine time-varying covariates.

In “Estimating the Relationship between Time-varying Covariates and Trajectories: The Sequence Analysis Multistate Model Procedure,” Matthias Studer, Emanuela Struffolino, and Anette E. Fasang describe a new method that allows researchers to blend some of the desirable features of multistate event history models—the ability to examine time-varying covariates—with the desirable features of sequence analysis—a focus on trajectories. They refer to their model as the sequence analysis multistate model (SAMM) and describe how to implement it in two successive steps. First, SAMM uses sequence analysis to determine the dominant trajectories in the data. Second, it uses multistate models to estimate how time-varying covariates are associated with the risks of transitioning from initial states to each sequence. To illustrate the utility of the SAMM approach, the authors then apply it to the interesting case of women’s employment trajectories in East Germany and West Germany during and after reunification. Their primary interest is in how family formation processes affect employment and how and whether these might be moderated by reunification. Their case study illustrates the role that family formation plays in women’s employment sequencing. It also shows that these processes were stronger for East German than West German women with both convergence and divergence, depending on the sequence in question, after reunification. The SAMM approach will be of great interest to sociologists interested in developing new theoretical models of processes, transitions, and change. As noted by the authors, how longitudinal processes are associated with time-varying covariates is a question that spans numerous subfields in sociology, including aging and the life-course; organizations, occupations, and work; family; political sociology; collective behavior and social movements; comparative-historical sociology; and science, knowledge, and technology; among many others.

Network Interference Models and Causal Inference

In nearly every setting, people are linked to one another in social networks that crystalize around interactions and interpersonal influence. Most sociologists recognize this fact, but few sociological methods account for it, which can bias estimation (Friedkin 1990). One area of particular concern is how networks of interaction and influence can bias causal inferences made from the results of experiments, randomized trials, or “A/B” testing. The central premise of many experimental designs is that treated and untreated units can be reliably distinguished, that there is no interference wherein the untreated control units receive the treatment without the researcher being aware. If interactions and social influence lead to interference, which is likely to occur in settings in which individuals are embedded in dense social networks such as schools, then causal inferences will be biased. Of course, sociologists often study and conduct experiments within such settings. The two articles in this section address the problems that network interference generates for causal inference. In “Limitations of Design-based Causal Inference and A/B Testing under Arbitrary and Network Interference,” Guillaume W. Basse and Edoardo M. Airoldi begin with a thorough documentation of the threats. In “Causal Inference with Networked Treatment Diffusion,” Weihua An proposes potential solutions that can be applied in several cases. As such, both articles push the envelope theoretically as well as methodologically, because they directly connect the relational focus that has grown into a core insight of sociological theory with some of the most important methods that sociologists use to make causal inferences.

Basse and Airoldi use analytical methods to show how arbitrary and network interference constrain the ability of researchers to draw causal conclusions from experiments. Their key contribution is to formalize intuition about the challenges that network interference poses for causal inference, even in experimental settings, much as prior work has done for observational network studies (Shalizi and Thomas 2011). Importantly, their findings show that the variance of unbiased estimators of treatment effects depends on network structure and does not necessarily go to zero as the sample size goes to infinity. This is a problem: even with massive samples, such as those used for A/B testing on online social networking platforms, inference in the face of network interference is imprecise. In the case of interference on Erdős-Renyi random graphs, Basse and Airoldi document a phase transition linked to the connectivity parameter p where the asymptotic variance properties break down; this finding directly connects the overall graph structure—specifically features related to its component size distribution—to the variance of causal inference estimators. In settings in which a large portion of respondents can reach one another through chains of interaction or influence, network interference poses a significant threat to causal inference. In practice, these results mean that researchers must make explicit assumptions about the structure of the interference in order to rely on asymptotic properties for causal inference.

The article by Weihua An offers one example of how researchers might proceed in cases of network interference. An proposes explicitly formalizing and measuring the interference structure with careful data collection. Using analytical methods, he shows how, armed with such data, researchers can identify and estimate causal effects that interference would otherwise obscure. For instance, his methods allow estimation of the direct treatment effect, the treatment interference effect, and the treatment effect on interference. He then demonstrates the applicability of this approach on a case study of a smoking prevention intervention in six middle schools in China, where students are connected in dense networks and there is substantial evidence of treatment interference. He also offers guidance for other researchers interested in collecting the types of data that would be necessary to decompose the treatment effects as he does in this case. By measuring and testing the actual interference structure explicitly with social network data collection, the methods that An proposes allow researchers to go beyond assuming treatment interference is negligible or assuming it follows an overly simplistic, and likely inaccurate, structure. This article thus encompasses both innovations in data collection as well as innovations in statistical inference. Practitioners may find this discussion especially valuable because it walks interested readers through the exact methodological steps to take in different circumstances, the assumptions that are associated with such choices, and the potential limitations they introduce.

Analysis of Residential Segregation

Residential segregation is a topic of perennial interest to sociologists, particularly those engaged in demographic research. The literature has devoted special attention to the spatial dimensions of segregation in recent years, particularly by asking how the sizes of geographic units influence traditional metrics and by developing new measures that quantify segregation at different geographic scales (as in Lee et al. 2008). However, residential segregation is determined by more than proximity, and current methods do not account for how features of the built and natural environment—from roads to physical barriers such as rivers—influence the effective distances between groups. For instance, two homes may be only a few dozen yards apart, but if they are separated by a highway, a few dozen yards may necessitate miles of travel. In “The Spatial Proximity and Connectivity Method for Measuring and Analyzing Residential Segregation,” Elizabeth Roberto proposes a new method that accounts for both proximity and connectivity. The core innovation is to measure distance between places using shortest paths along road networks, which accounts for intersite connectivity and the barriers and shortcuts that the built and physical environment place on commuting and interaction. Roberto demonstrates the spatial proximity and connectivity model using two examples. First, she looks at a stylized city with a north-south pattern of residential segregation but also with substantial physical barriers between places. Second, she looks at racial segregation in Pittsburgh, Pennsylvania, a city traversed by numerous rivers and other physical features that cut off otherwise nearby neighborhoods from one another. The results provide a more nuanced view that aligns experiences of residential segregation in terms of getting from place to place with its measurement. It also addresses well-known problems in the residential segregation literature, including the checkerboard problem and the modifiable areal unit problem, by accounting for proximity and allowing comparisons at multiple geographic scales. The spatial proximity and connectivity methodology has great potential to refine measurement of residential segregation and may in turn change understandings of how segregation influences inequality and “how the built environment influences the patterns, processes, and consequences of segregation” (this volume, p. 217). A related innovation of this work is its use of computational methods and the increasing availability of detailed measures through new, “big data” tools and approaches.

Survey Measurement

There are three general sets of factors that interact to shape the quality of survey data: those associated with questions, respondents, and interviewers (see Schaeffer 1991). Kristen Olson, Jolene D. Smyth, and Beth Cochran focus on the interaction of these factors in “Item Location, the Interviewer–Respondent Interaction, and Responses to Battery Questions in Telephone Surveys,” a study that has implications for study design. The authors deal with a fundamental problem in survey questionnaire design: how to structure the questions within a questionnaire. They recognize that when survey researchers put questions together to form questionnaires, they have a number of choices about how to group questions. Their focus is on the practice of grouping questions into topical batteries, which involve lists of questions that are part of a topical series, with a common question stem and the same response options. Battery questions are very common in major national surveys in use by sociologists, such as the General Social Survey and the National Election Study. In many applications, the battery stem provides instructions that indicate there will be a series of questions, all using the same set of response categories (see Dillman, Smyth, and Christian 2014), but not all batteries are preceded by instructions. These forms of questions create a number of problems for the respondent, as well as the interviewer, and the authors provide a thorough discussion of our state of knowledge regarding these issues. In modern computer-assisted surveys, the order of presentation of the questions in a battery may be randomized in order to disassociate the location of a question in the battery and the content of the question; but traditionally the order is fixed. There is some evidence that questions in batteries tend to have greater errors of measurement compared with other types of survey questions, suggesting that by “streamlining” the questions in the questionnaire, the investigator prompts those processes that lead to greater errors of measurement (see Andrews 1984; Alwin 2007). As Olson and her colleagues point out, there is virtually no research on the dynamics of respondent-interviewer interactions in responses to battery questions, and this is the first study of how battery questions are currently being administered in telephone surveys. The authors focus on whether respondent and interview behavior differ with respect to item location within batteries and whether the nature of this interaction results in differences in responses to these questions. Using data from a nationally representative telephone survey, they address three specific questions with regard to the design and administration of batteries in telephone surveys: (1) Does the behavior of survey actors differ with respect to the location of battery items? (2) Do respondent answers differ across the location of items in batteries? and (3) Do interview and respondent behaviors predict responses to battery items? Item location was randomized, so that the location of the battery item is independent of content. Using a random subset of audio recordings of the interviews, implementing behavior coding of respondent and interviewer interaction, and using cross-classified random effects models in the analysis of the data, the authors find strong evidence for a primacy effect—that is, there is more respondent-interviewer interaction around items administered earlier in the batteries. Other findings suggest that both interviewer and respondent behaviors are associated with item location. Their results support the conclusion that the design of batteries should include randomization of the order of presentation of battery items and the importance of using standardization of interviewer behavior in the administration of battery questions. This innovative mixed-methods study provides some initial insight into the potential causes of reduced reliability of questions in batteries, as they are currently employed in telephone surveys. The article also showcases the myriad contributions to sociological methodology that can result from blending qualitative and quantitative approaches.

Latent Variable Approaches

Latent variable models have been popular in sociology over the past 50 or more years. Work on latent structure analysis by Lazarsfeld and Henry (1968) paved the way for the development of latent class (LC) models (e.g., see Goodman 1974), but until relatively recent years, these models saw infrequent use. Factor analysis (e.g., Lawley and Maxwell 1971) and classical true-score models (Lord and Novick 1968) were strongly endorsed as statistical models worthy of serious attention. And when sociologists discovered that they could combine basic regression-type models with traditional or classical measurement models (factor models and true-score models), a new genre of sociological methodology (called structural equation modeling) was born. Such latent variable approaches have been an important development within sociological methodology. We include two articles involving latent variables—one from the latent growth model tradition, the other from the LC tradition—that advance work in these areas.

Shawn Bauldry and Kenneth A. Bollen, in their article “Nonlinear Autoregressive Latent Trajectory Models,” provide an extension of the work of Bollen and Curran (2004, 2006) on autoregressive latent trajectory (ALT) models. The ALT models combine the essential features of two traditions: (1) the autoregressive latent variable models, otherwise known as simplex models (e.g., Heise 1969; Jöreskog 1970), and (2) latent growth curve models (e.g., McArdle and Bell 2000). The ALT model parameterizes both the mean structure over time (constituting the latent growth model), and the intercorrelation among latent variables (constituting the autoregressive model). Bauldry and Bollen’s article relaxes the assumption of linear growth in the analysis of change, referring to nonlinear ALT (NLALT) models. Although allowing for some forms of nonlinearity in latent trajectories is often straightforward, several important features of these models have not been systematically studied. Bauldry and Bollen focus on two forms of nonlinear functional forms in the model: a quadratic NLALT model and a latent basis (or free loadings) NLALT model. They present a discussion of parameter estimation and testing in the context of nested models as well as a simulation study that illustrates some of the potential problems that arise from fitting such models to longitudinal data. The authors conclude with an empirical example demonstrating these models and their approach to estimating and testing them.

Classification is a basic goal of any science, and in sociology LC models have become an increasingly popular tool for clustering respondents into LCs. The goal of LC analysis is to discover a set of relatively homogenous subgroups on the basis of their responses to a set of categorical measures. SM has over the years published numerous articles that explicate the analysis of LCs. In recent years, LC tree (LCT) modeling has been proposed as an alternative to standard LC analysis. In “Deciding on the Starting Number of Classes of a Latent Class Tree,” Mattis van den Bergh, Geert H. van Kollenburg, and Jeroen K. Vermunt present the argument for the superiority of the LCT approach over standard LC analysis. The basic idea is that a hierarchical tree structure can be developed using an exploratory stepwise procedure of forming subclasses from the initial “root” classes. These subclasses can be divided further to build additional classes within these subclasses. One notable feature of the tree approach—that is, the stepwise splitting of LCs—is that it can be applied to any LC model with any number of starting classes. The article provides an introductory discussion of the basic LC model and how it can be used to build an LCT model, describing measures of relative fit that can help determine decisions to split classes at the root level. The authors include empirical examples that illustrate how the measures of fit, along with substantive theory, can be used to determine the appropriate number of classes at the first split of an LCT. Their work nicely complements the growing literature on developments in LCT analysis.

Aggregation and Statistical Estimation

Almost any sociologist who has conducted quantitative data analysis will have encountered the issues addressed by Paul A. Jargowsky and Christopher A. Wheeler’s article, “Estimating Income Statistics from Grouped Data: Mean-constrained Integration over Brackets.” The authors look at cases in which income data are grouped into brackets or bands of potentially unequal length, rather than offering precise amounts. For instance, instead of a data set saying that one person’s income is $9,205 and another’s is $36,812, data collectors or providers might report only that the first person’s income is less than $10,000 and the second person’s is in the $10,000 to $50,000 band. These decisions are often taken to protect respondent confidentiality, or to simplify measurement. Sometimes only the highest or lowest or both income ranges are grouped (i.e., top coding and bottom coding, as in the U.S. Public Use Microdata Samples [PUMS]; Ruggles et al. 2017), whereas others, such as the General Social Survey’s variable “income” (Smith et al. n.d.), use brackets throughout the range. In each case, the challenge arises because “the exact arrangement of the individual observations within the brackets is unknown and could have large effects on summary statistics and inequality measures” (this volume, p. 338). Given the primacy placed on studies of inequality, poverty, and elites in contemporary sociology, that top and bottom coding obscure distinctions among individuals in these groups is an especially daunting challenge. Jargowski and Wheeler’s solution, mean-constrained integration over brackets (MCIB), uses a three-step process to improve on commonly used approaches in the literature, such as using bracket midpoints. The authors demonstrate this procedure empirically using PUMS data about real and bracketed household income in 297 metropolitan areas in 2011. Their results clearly highlight MCIB’s substantial improvements over traditional and contemporary approaches to estimating a wide range of income statistics from grouped data. For instance, MCIB estimates of the Gini coefficient with raw (“groundtruth”) and grouped data are correlated at 0.998, while the correlations between estimates made with raw and grouped data are considerably lower for a wide variety of other methods currently in use (e.g., see von Hippel, Scarpino, and Holas 2016). The benefits of MCIB over existing approaches can also be seen when looking at many other common income and inequality statistics and measures, such as mean income, the standard deviation of income, or the Theil index. In short, across a wide range of situations and income statistics of interest, MCIB produces less biased and more accurate results with fewer assumptions, and it is less sensitive to the placement and width of the brackets, than the currently dominant approaches for working with grouped income data (see also von Hippel, Hunter, and Drown 2017). Such improvements are likely to be of interest to sociologists working in a wide range of subfields.