Advances in Methods and Practices in Psychological Science

Supervised machine learning (ML) is becoming an influential analytical method in psychology and other social sciences. However, theoretical ML concepts and predictive-modeling techniques are not yet widely taught in psychology programs. This tutorial is intended to provide an intuitive but thorough primer and introduction to supervised ML for psychologists in four consecutive modules. After introducing the basic terminology and mindset of supervised ML, in Module 1, we cover how to use resampling methods to evaluate the performance of ML models (bias-variance trade-off, performance measures, k-fold cross-validation). In Module 2, we introduce the nonlinear random forest, a type of ML model that is particularly user-friendly and well suited to predicting psychological outcomes. Module 3 is about performing empirical benchmark experiments (comparing the performance of several ML models on multiple data sets). Finally, in Module 4, we discuss the interpretation of ML models, including permutation variable importance measures, effect plots (partial-dependence plots, individual conditional-expectation profiles), and the concept of model fairness. Throughout the tutorial, intuitive descriptions of theoretical concepts are provided, with as few mathematical formulas as possible, and followed by code examples using the mlr3 and companion packages in R. Key practical-analysis steps are demonstrated on the publicly available PhoneStudy data set ( N = 624), which includes more than 1,800 variables from smartphone sensing to predict Big Five personality trait scores. The article contains a checklist to be used as a reminder of important elements when performing, reporting, or reviewing ML analyses in psychology. Additional examples and more advanced concepts are demonstrated in online materials ( https://osf.io/9273g/ ).

Florian Pargent

Ramona Schoedel

Clemens Stachl

Best Practices in Supervised Machine Learning: A Tutorial for Psychologists

监督机器学习中的最佳实践：心理学家的教程

Intervention research is often time- and resource-intensive, with numerous participants involved over extended periods of time. To maximize the value of intervention studies, multiple outcome measures are often included, either to ensure a diverse set of outcomes is being assessed or to refine assessments of specific outcomes. Here, we advocate for combining assessments, rather than relying on individual measures assessed separately, to better evaluate the effectiveness of interventions. Specifically, we argue that by pooling information from individual measures into a single outcome, composite scores can provide finer estimates of the underlying theoretical construct of interest while retaining important properties more sophisticated methods often forgo, such as transparency and interpretability. We describe different methods to compute, evaluate, and use composites depending on the goals, design, and data. To promote usability, we also provide a preregistration template that includes examples in the context of psychological interventions with supporting R code. Finally, we make a number of recommendations to help ensure that intervention studies are designed in a way that maximizes discoveries. A Shiny app and detailed R code accompany this article and are available at https://osf.io/u96em/ .

David Moreau

Kristina Wiebels

Assessing Change in Intervention Research: The Benefits of Composite Outcomes

评估干预研究中的变化：复合结果的益处

Correlation does not imply causation; but often, observational data are the only option, even though the research question at hand involves causality. This article discusses causal inference based on observational data, introducing readers to graphical causal models that can provide a powerful tool for thinking more clearly about the interrelations between variables. Topics covered include the rationale behind the statistical control of third variables, common procedures for statistical control, and what can go wrong during their implementation. Certain types of third variables—colliders and mediators—should not be controlled for because that can actually move the estimate of an association away from the value of the causal effect of interest. More subtle variations of such harmful control include using unrepresentative samples, which can undermine the validity of causal conclusions, and statistically controlling for mediators. Drawing valid causal inferences on the basis of observational data is not a mechanistic procedure but rather always depends on assumptions that require domain knowledge and that can be more or less plausible. However, this caveat holds not only for research based on observational data, but for all empirical research endeavors.

Julia M. Rohrer

Thinking Clearly About Correlations and Causation: Graphical Causal Models for Observational Data

清晰思考相关性与因果关系：观察数据的图形因果模型

Power analysis for first-order interactions poses two challenges: (a) Conducting an appropriate power analysis is difficult because the typical expected effect size of an interaction depends on its shape, and (b) achieving sufficient power is difficult because interactions are often modest in size. This article consists of three parts. In the first part, we address the first challenge. We first use a fictional study to explain the difference between power analyses for interactions and main effects. Then, we introduce an intuitive taxonomy of 12 types of interactions based on the shape of the interaction (reversed, fully attenuated, partially attenuated) and the size of the simple slopes (median, smaller, larger), and we offer mathematically derived sample-size recommendations to detect each interaction with a power of .80/.90/.95 (for two-tailed tests in between-participants designs). In the second part, we address the second challenge. We first describe a preregistered metastudy (159 studies from recent articles in influential psychology journals) showing that the median power to detect interactions of a typical size is .18. Then, we use simulations (≈900,000,000 data sets) to generate power curves for the 12 types of interactions and test three approaches to increase power without increasing sample size: (a) preregistering one-tailed tests (+21% gain), (b) using a mixed design (+75% gain), and (c) preregistering contrast analysis for a fully attenuated interaction (+62% gain). In the third part, we introduce INT×Power ( www.intxpower.com ), a web application that enables users to draw their interaction and determine the sample size needed to reach the power of their choice with the option of using/combining these approaches.

Nicolas Sommet

David L. Weissman

Nicolas Cheutin

Andrew J. Elliot

How Many Participants Do I Need to Test an Interaction? Conducting an Appropriate Power Analysis and Achieving Sufficient Power to Detect an Interaction

我需要多少参与者来测试交互作用？进行适当的效能分析并实现足够的效能以检测交互作用

Model generalizability describes how well the findings from a sample are applicable to other samples in the population. In this Tutorial, we explain model generalizability through the statistical concept of model overfitting and its outcome (i.e., validity shrinkage in new samples), and we use a Shiny app to simulate and visualize how model generalizability is influenced by three factors: model complexity, sample size, and effect size. We then discuss cross-validation as an approach for evaluating model generalizability and provide guidelines for implementing this approach. To help researchers understand how to apply cross-validation to their own research, we walk through an example, accompanied by step-by-step illustrations in R. This Tutorial is expected to help readers develop the basic knowledge and skills to use cross-validation to evaluate model generalizability in their research and practice.

Q. Chelsea Song

Chen Tang

Serena Wee

Making Sense of Model Generalizability: A Tutorial on Cross-Validation in R and Shiny

理解模型泛化能力：R与Shiny中的交叉验证教程

One of the most common applications of Bayes’s theorem for inferential purposes consists of computing the ratio between the probability of the alternative and the null hypotheses via a Bayes factor, which allows quantifying the most likely explanation of the data between the two. However, the actual scientific questions that researchers are interested in are rarely well represented by the classically defined alternative and null hypotheses. Bayesian informative-hypothesis testing offers a valid and easy way to overcome such limitations via a model-selection procedure that allows comparing highly specific hypotheses formulated in terms of equality (A = B) or inequality (A &gt; B) constraints among parameters. Although packages for testing informative hypotheses in the most used statistical software have been developed in recent years, they are still rarely used, possibly because their implementation and interpretation may not be straightforward. Starting from a brief theoretical overview of the Bayesian theorem and its applications in statistical inference (i.e., Bayes factor and Bayesian informative hypotheses), in this article, we provide two step-by-step tutorials illustrating how to test, interpret, and report in a scientific article Bayesian informative hypotheses using JASP and R/RStudio software for running a 2 × 2 analysis of variance and a multiple linear regression. The complete JASP files, R code, and data sets used in the article are freely available on the OSF page at https://osf.io/dez9b/ .

Sara Garofalo

Gianluca Finotti

Matteo Orsoni

Sara Giovagnoli

Mariagrazia Benassi

Testing Bayesian Informative Hypotheses in Five Steps With JASP and R

使用JASP和R分五步测试贝叶斯信息假设

When planning experimental research, determining an appropriate sample size and using suitable statistical models are crucial for robust and informative results. The recent replication crisis underlines the need for more rigorous statistical methodology and adequately powered designs. Generalized linear mixed models (GLMMs) offer a flexible statistical framework to analyze experimental data with complex (e.g., dependent and hierarchical) data structures. However, available methods and software for a priori sample-size planning for GLMMs are often limited to specific designs. Tailored data-simulation approaches offer a more flexible alternative. Based on a practical case study in which we focus on a binomial GLMM with two random intercepts and discrete predictor variables, the current tutorial equips researchers with a step-by-step guide and corresponding code for conducting tailored a priori sample-size planning with GLMMs. We not only focus on power analysis but also explain how to use the precision of parameter estimates to determine appropriate sample sizes. We conclude with an outlook on the increasing importance of simulation-based sample-size planning.

Timo K. Koch

Anne-Kathrin Kleine

Eva Lermer

Susanne Gaube

A Tutorial on Tailored Simulation-Based Sample-Size Planning for Experimental Designs With Generalized Linear Mixed Models

基于模拟的自定义样本量规划教程：适用于广义线性混合模型的实验设计

The prototype approach provides a theoretically supported basis for novel research, detailing “typical” cognitive representations of targets in question (e.g., groups, experiences). Fairly recently, in social and cognitive psychology, this approach has emerged to understand how people categorize and conceptualize everyday phenomena. Although this approach has previously been used to study everyday concepts, it has predominantly been overlooked in cross-cultural research. Prototype analyses are flexible enough to allow for the identification of both universal and culture-specific elements, offering a more comprehensive and nuanced understanding of the concept in question. We highlight theoretical, empirical, and practical reasons why prototype analyses offer an important tool in cross-cultural and interdisciplinary research while also addressing the potential for reducing construct bias in research that spans multiple cultural contexts. The advantages and risks of conducting prototype analyses are discussed in detail along with novel ways of integrating computational approaches with traditional prototype-analyses methods to assist in their implementation.

Yuning Sun

Elaine L. Kinsella

Eric R. Igou

A Guide to Prototype Analyses in Cross-Cultural Research: Purpose, Advantages, and Risks

跨文化研究中的原型分析指南：目的、优势与风险

The stimuli presented in cognitive experiments have a crucial role in the ability to isolate the underlying mechanism from other interweaved mechanisms. New ideas aimed at unveiling cognitive mechanisms are often realized through introducing new stimuli. This, in turn, raises challenges in reconciling results to literature. We demonstrate this challenge in the field of numerical cognition. Stimuli used in this field are designed to present quantity in a non symbolic manner. Physical properties, such as surface area and density, inherently correlate with quantity, masking the mechanism underlying numerical perception. Different generation methods (GMs) are used to control these physical properties. However, the way a GM controls physical properties affects numerical judgments in different ways, compromising comparability and the pursuit of cumulative science. Here, using a novel data-driven approach, we provide a methodological review of non symbolic stimuli GMs developed since 2000. Our results reveal that the field thrives and that a wide variety of GMs are tackling new methodological and theoretical ideas. However, the field lacks a common language and means to integrate new ideas into the literature. These shortcomings impair the interpretability, comparison, replication, and reanalysis of previous studies that have considered new ideas. We present guidelines for GMs relevant also to other fields and tasks involving perceptual decisions, including (a) defining controls explicitly and consistently, (b) justifying controls and discussing their implications, (c) considering stimuli statistical features, and (d) providing complete stimuli set, matching responses, and generation code. We hope these guidelines will promote the integration of findings and increase findings’ explanatory power.

Yoel Shilat

Avishai Henik

Hanit Galili

Shir Wasserman

Alon Salzmann

Moti Salti

A Methodological Framework for Stimuli Control: Insights From Numerical Cognition

刺激控制的方法论框架：来自数值认知的启示

Testing is used to inform a range of critical decisions that help structure much of contemporary society. An unavoidable aspect of testing is that test scores are not infallible. As a result, individual test scores should be accompanied by an interval that indicates the uncertainty surrounding the score. There are a number of different test-score intervals that can be created from different error terms. Unfortunately, there are pervasive misinterpretations of these errors and their intervals. Many of these interpretations can be found in authoritative sources on psychological measurement, which has resulted in stubborn and persistent confusion about what these intervals mean. In the current article, we clarify two important error terms and their intervals: (a) the Standard Error of Estimation and (b) the Standard Error of Measurement. We explicate the meaning and interpretation of these errors by examining their statistical foundations. Specifically, we detail how these terms are formulated from different statistical models and the implications of these models for their different interpretations. We use classical test theory, bivariate linear regression, R activities, and algebra to illustrate the key concepts and differences.

David J. Stanley

Jeffrey R. Spence

The Comedy of Measurement Errors: Standard Error of Measurement and Standard Error of Estimation

测量误差的喜剧：测量标准误与估计标准误

Abbreviation	Adv. Methods. Pract. Psychol. Sci.
Journal Impact	15.65
Quartiles(Global)	PSYCHOLOGY, MULTIDISCIPLINARY(Q1)

ISSN	2515-2459, 2515-2467
h-index	42

Publication Information	Publisher: SAGE Publications Inc.，Publishing cycle: ，Journal Type: journal，Open Access Journals: No
Basic data	Year of publication: 2018，Proportion of original research papers: 100.00%，Self Citation Rate:3.20%， Gold OA Rate: 94.19%

Advances in Methods and Practices in Psychological Science

Journal Citation Format

A journal article with 1 author

A journal article with 2 authors

A journal article with 3 authors

A journal article with 5 or more authors

Books Citation Format

Thesis Citation Format

Web sites Citation Format

Patent Citation Format

Staying up late manually editing references? ivySCI automatically matches journals and helps you generate references with a single click.

Share Submission Experience