55
edits
Changes
Jump to navigation
Jump to search
Line 46:
Line 46: + +
Talk:Significance of E. Coli Evolution Experiments (view source)
Revision as of 15:53, March 12, 2009
, 15:53, March 12, 2009Accuracy
::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 10%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)
::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 10%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)
:::::You're talking about miniscule differences in the accuracy of a test. 0.013 isn't very different from 0.007. In either case, it's very unlikely the experimenter would have obtained that result if the null hypothesis were true. If you're bothered about differences in P-values to the third decimals (which would make you unusual!), just run more MC realisations, that's all. Not really a problem. [[User:FredFerguson|FredFerguson]] 11:53, 12 March 2009 (EDT)
There’s still confusion about the difference between test statistics and Monte Carlo methods. Before you find a Monte Carlo estimate of a p-value, you need to select a test statistic to reduce the data set to a scalar. I am interested in hearing which test statistic you believe should be used in place of the chi-square test and why. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)
There’s still confusion about the difference between test statistics and Monte Carlo methods. Before you find a Monte Carlo estimate of a p-value, you need to select a test statistic to reduce the data set to a scalar. I am interested in hearing which test statistic you believe should be used in place of the chi-square test and why. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)