Changes

::::Your claim that Monte Carlo methods are “inevitably more accurate” than other tests is obviously wrong because the accuracy of MC methods always depends on the number of realizations used. You should have written <math>\alpha=1\%</math>, not <math>\alpha<1\%</math>. If 1,000 random realizations are generated, the number of realizations above the true <math>\alpha=1\%</math> level is binomial with mean 10 and variance about 10. Thus, the standard deviation of the MC estimate is >0.003. In this example, a Monte Carlo p-value could be off by 30% and still be within a standard deviation. Is that really “pretty accurate”?

::::Your claim that Monte Carlo methods are “inevitably more accurate” than other tests is obviously wrong because the accuracy of MC methods always depends on the number of realizations used. You should have written <math>\alpha=1\%</math>, not <math>\alpha<1\%</math>. If 1,000 random realizations are generated, the number of realizations above the true <math>\alpha=1\%</math> level is binomial with mean 10 and variance about 10. Thus, the standard deviation of the MC estimate is >0.003. In this example, a Monte Carlo p-value could be off by 30% and still be within a standard deviation. Is that really “pretty accurate”?

::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 10%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 3%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::::You're talking about miniscule differences in the accuracy of a test. 0.013 isn't very different from 0.007. In either case, it's very unlikely the experimenter would have obtained that result if the null hypothesis were true. If you're bothered about differences in P-values to the third decimals (which would make you unusual!), just run more MC realisations, that's all. Not really a problem. [[User:FredFerguson|FredFerguson]] 11:53, 12 March 2009 (EDT)

:::::You're talking about miniscule differences in the accuracy of a test. 0.013 isn't very different from 0.007. In either case, it's very unlikely the experimenter would have obtained that result if the null hypothesis were true. If you're bothered about differences in P-values to the third decimals (which would make you unusual!), just run more MC realisations, that's all. Not really a problem. [[User:FredFerguson|FredFerguson]] 11:53, 12 March 2009 (EDT)