Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

The University of British Columbia

$18,000.00

Development and evaluation of R-squared effect size measures and methods for mixed effects models

FY 2023-24

Summary
Hierarchical data structures are ubiquitous in many areas of scientific research, and are particularly common in psychology. Typical examples include students nested within schools, or repeated observations nested within individuals. Such data are often analyzed with mixed effects models, which extend standard regression models by allowing coefficients to vary by cluster (e.g., school). Despite their increasing popularity in psychology and other disciplines, one major limitation in the use of mixed effects models is that researchers rarely report measures of effect size (a quantitative indication of the magnitude of effects/ relationships among variables) and instead typically summarize results primarily with p-values. This stands in sharp contrast to widespread recommendations to convey practical significance rather than relying exclusively on statistical significance. In particular, one effect size measure routinely reported in standard regression-R-squared (R2), reflecting the proportion of variance explained-is rarely reported for mixed effects models. Computing R2 in mixed effects models is markedly more complex than in standard regression, and there has been confusion about how best to do so. In my previous work, I explained how the methodological literature on mixed effect model R2 has historically suffered from many shortcomings, leading to researchers either not reporting such measures at all or, perhaps more troublingly, using and interpreting them incorrectly. To address these shortcomings, I provided a unifying framework to defining, computing, and interpreting R2 for mixed effects models (Rights & Sterba, 2019, in press). Several key shortcomings persist, however, in that (a) this framework does not apply to every possible model specification, (b) little work has been done to assess the statistical properties of the available mixed effect R2 measures, and (c) there is a lack of available software to compute measures. In this proposal, I aim to address these shortcomings. Objective 1 is to extend the Rights & Sterba R2 framework to a wider variety of modeling contexts, such as generalized linear mixed effects models, cross-classified models, longitudinal models with complex residual variance structures, and latent variable models. Objective 2 is to formally evaluate both preexisting and newly developed R2 methods in terms of bias, efficiency, and coverage via simulation. Different methods of bias adjustment and confidence interval construction will be considered and developed. Objective 3 is to ease the computational burden on researchers by providing open-source software to compute these measures and implement these methods. Overall, the long-term aim of this research program is to enable researchers to better compute effect size for mixed effects models. This responds to widespread recommendations to consider practical significance of statistical results, and will be useful in many research disciplines in psychology and beyond.
Program
Discovery Grants Program - Individual
Location
British Columbia, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RGPIN-2020-06132
Name nom
Rights, Jason
Department département
Psychology
Organization id
2
Institution établissement
The University of British Columbia
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2020
Award amount
18000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1508
Committee name en
Mathematics and Statistics
Committee name fr
Mathématiques et statistique
Area of application code
1210
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Psychology
Area of application fr
Psychologie
Research subject code
6300
Research subject group en
Psychology
Research subject group fr
Psychology
Research subject en
Psychologie
Research subject fr
Psychologie
Application title
Development and evaluation of R-squared effect size measures and methods for mixed effects models
Keyword
effect size; hierarchical linear models; mixed effects models; multilevel models; R-squared
Application summary
Hierarchical data structures are ubiquitous in many areas of scientific research, and are particularly common in psychology. Typical examples include students nested within schools, or repeated observations nested within individuals. Such data are often analyzed with mixed effects models, which extend standard regression models by allowing coefficients to vary by cluster (e.g., school). Despite their increasing popularity in psychology and other disciplines, one major limitation in the use of mixed effects models is that researchers rarely report measures of effect size (a quantitative indication of the magnitude of effects/ relationships among variables) and instead typically summarize results primarily with p-values. This stands in sharp contrast to widespread recommendations to convey practical significance rather than relying exclusively on statistical significance. In particular, one effect size measure routinely reported in standard regression-R-squared (R2), reflecting the proportion of variance explained-is rarely reported for mixed effects models. Computing R2 in mixed effects models is markedly more complex than in standard regression, and there has been confusion about how best to do so. In my previous work, I explained how the methodological literature on mixed effect model R2 has historically suffered from many shortcomings, leading to researchers either not reporting such measures at all or, perhaps more troublingly, using and interpreting them incorrectly. To address these shortcomings, I provided a unifying framework to defining, computing, and interpreting R2 for mixed effects models (Rights & Sterba, 2019, in press). Several key shortcomings persist, however, in that (a) this framework does not apply to every possible model specification, (b) little work has been done to assess the statistical properties of the available mixed effect R2 measures, and (c) there is a lack of available software to compute measures. In this proposal, I aim to address these shortcomings. Objective 1 is to extend the Rights & Sterba R2 framework to a wider variety of modeling contexts, such as generalized linear mixed effects models, cross-classified models, longitudinal models with complex residual variance structures, and latent variable models. Objective 2 is to formally evaluate both preexisting and newly developed R2 methods in terms of bias, efficiency, and coverage via simulation. Different methods of bias adjustment and confidence interval construction will be considered and developed. Objective 3 is to ease the computational burden on researchers by providing open-source software to compute these measures and implement these methods. Overall, the long-term aim of this research program is to enable researchers to better compute effect size for mixed effects models. This responds to widespread recommendations to consider practical significance of statistical results, and will be useful in many research disciplines in psychology and beyond.
The University of British Columbia - $18,000 Grant (2… | Canada Spends