Post Hoc Tests

Familywise Error Also known as alpha inflation or cumulative Type I error. Familywise error (FWE) represents the probability that any one of a set of comparisons or significance tests is a Type I error. As more tests are conducted, the likelihood that one or more are significant just due to chance (Type I error) increases. One can estimate familywise error with the following formula:

[pic]

where αFWE is the familywise error rate, αEC is the alpha rate for an individual test (almost always considered to be .05), and c is the number of comparisons. c as used in the formula is an exponent, so the parenthetical value is raised to the cth power.

Bonferroni The Bonferroni simply calculates a new pairwise alpha to keep the familywise alpha value at .05 (or another specified value). The formula for doing this is as follows:

[pic]

where αB is the new alpha based on the Bonferroni test that should be used to evaluate each comparison or significance test, αFWE is the familywise error rate as computed in the first formula, and c is the number of comparisons (statistical tests).

The Bonferroni is probably the most commonly used post hoc test, because it is highly flexible, very simple to compute, and can be used with any type of statistical test (e.g., correlations)—not just post hoc tests with ANOVA. The traditional Bonferroni, however, tends to lack power. The loss of power occurs for several reasons: (1) the familywise error calculation depends on the assumption that, for all tests, the null hypothesis is true. This is unlikely to be the case, especially after a significant omnibus test; (2) all tests are assumed to be orthogonal (i.e., independent or nonoverlapping) when calculating the familywise error test, and this is usually not the case when all pairwise comparisons are made; (3) the test does not take into account whether the findings are consistent with theory and past research. If consistent with previous findings and theory, an individual result should be less likely to be a Type I error; and (4) Type II error rates are too high for individual tests. In other words, the Bonferroni overcorrects for Type I error.

Modified Bonferroni Approaches

Several alternatives to the traditional Bonferroni have been developed, including those developed by Holm, Holland and Copenhaver, Hommel, Rom, and others (see Olejnik, Li, Supattathum, & Huberty, 1997 for a review). These tests have greater power than the Bonferroni while retaining its flexible approach that allows for use with any set of statistical tests (e.g., t-tests, correlations, chi-squares).

Sidak-Bonferroni. Sidak (1967) suggested a relatively simple modification of the Bonferroni formula that would have less of an impact on statistical power but retain much of the flexibility of the Bonferroni method (Keppel & Wickens, 2004 discuss this testing approach). Instead of dividing by the number of comparisons, there is a slightly more complicated formula:
[pic]
where αS-B is the Sidak-Bonferroni alpha level used to determine significance (something less than .05), αFWE is the computed familywise error according to the formula at the top of the first page, and c is the number of comparisons or statistical tests conducted in the “family.” The p-values obtained from the computer printout must be smaller than αS-B to be considered significant. One can also extend this test to other statistical tests, such as correlations. In the case of correlations, one could replace dfA with the number of variables that are used in the group of correlations tests. c would represent the number of correlations in the correlation matrix. This approach is convenient and easy to do but has not received any systematic study, and it is likely that a single, simple correction will not result in the most efficient balance of Type I and Type II errors.

Hochberg’s Sequential Method. This test uses a specific sequential method