Power analysis

Alexander Strobel
Faculty of Psychology, TU Dresden

Introduction

At AG.DPP, it is mandatory that you perform power calculations (or at least elaborate on power considerations) in any study that leaves the lab. Ideally, we use such analyses to determine sample size a priori. Sometimes, however, we have to rely on convenience samples and can calculate power only post hoc. Both approaches are appropriate, but must be named as such. In the former case, we would state in our manuscript:

Sample size was determined a priori based on a meta-analytically derived effect size for the effect of interest of r = .28. With a power of 1-β = .80 and an α = .05 (two-tailed), the required sample size was N = 97 as determined using the R package pwr.

In the later case, we would write something like:

For our online survey, we set out to collect as many participants as possible within a time window of one month. We eventually had complete datasets of N = 348 participants. With this sample size, we were able to detect correlations of r ≥ .15 at a power of 1-β = .80 and an α = .05 (two-tailed) as determined using the R package pwr.

Effect size estimation

If the literature tells you what effect size to expect, this effect size is most likely inflated due to publication bias. Evidence from replication research (e.g., Open Science Collaboration, 2015) suggests that even replicable effects are about half of the size as those originally reported. It may thus be wise to divide the effect size you found in the literature by two. This may be too conservative if you ground your power calculation on meta-analytically derived effect sizes and use that estimate for your power analysis. Yet, also meta-analyses suffer from publication bias and may overestimate effect sizes. Thus, it is perhaps a good idea to use the lower bound of the confidence interval of the effect size in question as estimate for your power analysis. Calculate the required sample size with a desired power of at least 80% and a significance level that accounts for possible multiple testing.

If you have no idea what effect size to expect, Cohen‘s classification most likely will not reflect the typical effect sizes in your area of research. If there are no established guidelines (such as those of Gignac & Szodorai, 2016, for individual differences research), assume a correlation of r = .20 (or any derivative such as an explained variance of .04, see Fraley & Vazire, 2014). A small to medium effect is more likely than a large one.

Software to use for power analysis

The software used for power analysis makes no difference. Yet, G*Power is more powerful than other software such as jamovi or the R package pwr under R. If there is no power analysis software for your specific effect size, run simulations.

Power analysis for special analysis types

As said, if there are no established routines for power analysis of your desired analysis routine, simulations may be the way to go. Yet, before we go at great lengths to program such simulations, we should always refer to the literature whether meanwhile there are published and perhaps even approved (by means of having been cites a lot) routines for the analysis type we want to apply. Such analysis routines include:

1. mediation analysis
2. structural equation modeling
3. ...

Approaches for these types of analyses are outlined below.

Power for mediation analysis

...

Power for structural equation modeling

There seem to be three commonly accepted approaches to run a power analysis for structural equation modeling (SEM) or any derivative of such as a more complex path analysis, the power of which cannot be determined in terms of regression analysis:

1. Satorra and Saris (1985)
2. MacCallum, Browne and Sugawara (1996)
3. Muthén and Muthén (2002)

Satorra and Saris (1985)

In first approach, you compare two models: one with a parameter of interest (e.g., a regression weight) set free (the preferred alternative model) and one with the respective parameter fixed to zero (the null model).

MacCallum, Browne and Sugawara (1996)

The second approach, suggested by MacCallum et al. (1996), determines power based on the root mean square error of approximation (RMSEA). As far as I understand, this approach sets out to estimate the power to detect misspecification of your structural equation model. Basically, you state an RMSEA for the null hypothesis (H0) and another for the alternative hypothesis (H1). The MacCallum et al. paper is not very precise on what grounds you should choose the values for RMSEA_H0 and RMSEA_H1. In one part if their paper, they compare an RMSEA of .05 vs. .08, in another part an RMSEA of .00 vs. .05. The nice thing is that there is an R package called semPower that implements the MacCallum et al. approach. It was written by Morten Moshagen (professor for Psychological Research Methods at Ulm University) and Edgar Erdfelder (professor for Cognitive Psychology and Individual Differences who also contributed to G*Power). So we can be quite confident that this package will do its job properly. As compared to other packages you could employ to run a power analysis based on the MacCallum et al. (1996) approach such as WebPower, with semPower you do not have to specify RMSEA_H0, but only RMSEA_H1. RMSEA_H0 seems to be zero per default. So the only question you have to deal with is what value to choose for RMSEA_H1. I suggest to employ RMSEA_H1=.06 for a priori power analysis, as this is the suggested cut-off value in Hu and Bettler (1999), and we usually refer to this paper when defining cut-off values to evaluate the fit of our structural equation models. For post hoc power analysis, you would of course use the RMSEA you achieved with your model.

As an example for an a priori sample size determination, you would state:

required_n = semPower.aPriori(effect         = .06, 
                              effect.measure = 'RMSEA',
                              alpha          = .05, 
                              power          = .80, 
                              df             = df)$requiredN

You see that you need to know the degrees of freedom of you model (i.e., argument df). You can calculate the degrees of freedom from the number of (co)variances in your data (i.e., the lower triangle and the diagonal of a covariances matrix) minus the number of estimated parameters in your model or simply specify your model using lavaan syntax and use the function semPower.getDf(model).

As an example for a post hoc power calculation, you would use the N and the df of your fitted model. For a sample size of N = 321 and df = 24, post hoc power would be:

posthoc_power = semPower.postHoc(effect         = .06, 
                                 effect.measure = 'RMSEA', 
                                 alpha          = .05, 
                                 N              = 321, 
                                 df             = 24)$power

The power here is 1-β = .90 to detect misspecification of your model at RMSEA = .06 (our cut-off). If we entered our actually achieved RMSEA, say RMSEA = .04, we would have had a power of only 1-β = .46 to detect such a low level of misspecification with this sample size and model df. As a rule of thumb, the higher the df, the lower is the sample size needed to have adequate power to detect misspecification (see Tables 4 and 5 in MacCallum et al., 1996).

Muthén and Muthén (2002)

The third approach involves Monte Carlo simulations, which sounds too fancy for our mainly rather plain purposes, so we discard this method for now (but see Muthén & Muthén, 2002)

Concluding remark

For details on power analysis, see the following presentation:

• Power Analyses

---

References

MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130–149. https://doi.org/10.1037/1082-989X.1.2.130

Moshagen, M., & Erdfelder, E. (2016). A new strategy for testing structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 23(1), 54–60. https://doi.org/10.1080/10705511.2014.950896

Muthén, L. K. & Muthén, B. O. (2002) How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling: A Multidisciplinary Journal, 9(4), 599-620, https://doi.org/10.1207/S15328007SEM0904_8

Satorra, A., & Saris, W. E. (1985). Power of the likelihood ratio test in covariance structure analysis. Psychometrika, 50(1), 83–90. https://doi.org/10.1007/BF02294150