A comparison of independent samples t-test, approximate randomization test and bootstrap randomization test to departures from population normality : a Monte Carlo study
Date
1997
Authors
Miles, Stephen Allen
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
A Monte Carlo study was undertaken to compare the two independent sample t-test (parametric) with the approximate randomization test and the bootstrap randomization test using data sampled from the Micceri distributions, a set of non-normal distributions. The three tests were compared with respect to Type I errors for varying sample sizes.
Following the method of Sawilowsky and Blair ( 1992), Monte Carlo methods were used to sample the eight Micceri distributions with sample sizes of (5, 15), (7, 7), (10, 10), (10, 30), (20, 20), (20, 60), (60, 60). Observations were sampled independently with replacement from each of the eight distributions. Replicating Sawilowsky and Blair (1992), the t-test showed marked non-robustness when matched with many of Micceri's distributions and the departures in Type I error from the five percent pre-set value of the study were greater than those found in previous studies. These results suggest that the probability estimates associated with the t-test are unreliable. The present study extends these' results to the alternative nonparametric methods of the bootstrap and approximate randomization tests as follows: when the distribution is extremely asymmetric, the bootstrap method outperforms both the t-test and approximate randomization test. Also, for non-symmetric distributions, pooled sample sizes of at least 30 to 40 would appear necessary to overcome the tendency of the t-test to underestimate the preset value when compared to the bootstrap test. For symmetric distributions (skew of an absolute value less than one), the t-test meets the criterion (i.e. it is within 10% of the preset value) for pooled sample sizes of 20 or more. The t-test lacks robustness for small (less than 20) unequal sample sizes from non-symmetric distributions and also is not robust for the very small and equal samples (7, 7) even for symmetric distributions. The approximate randomization test is consistently conservative for small sample sizes or unequal sample sizes. Whereas for the large sample sizes (60,60), this test produces a Type I error 4.6% of the time versus the 5% present value. The approximate randomization test never over estimates the preset value under the criterion.
Prior to further research, I recommend that anyone using small sample sizes (less than 20 subjects in either group) consider using both parametric and nonparametric tests. A frequency plot of the sample data should be examined for breaches of the symmetry assumption of the normal distribution. If such a breach does occur then nonparametric tests should be applied.