37
edits
Changes
Jump to navigation
Jump to search
Line 21:
Line 21: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Talk:Significance of E. Coli Evolution Experiments (view source)
Revision as of 21:38, March 5, 2009
, 21:38, March 5, 2009Responses to questions and comments
I hope this helps.--[[User:Able806|Able806]] 10:27, 5 March 2009 (EST)
I hope this helps.--[[User:Able806|Able806]] 10:27, 5 March 2009 (EST)
:I’m still waiting to hear which literature says that “Monte Carlo resampling” is “more accurate than the chi-squared test”. The page mentioned above [http://faculty.vassar.edu/lowry/chi_beta.html] is a discussion of why statisticians “fail to reject the null” rather than “accepting the null” when the p-value is above 0.05 or so. The page says nothing about superiority of Monte Carlo methods. Why were alternate hypothesis distributions mentioned? Only the null hypothesis distribution is used to calculate a p-value. Yates’s correction is for 2x2 contingency tables [http://en.wikipedia.org/wiki/Yates%27_correction_for_continuity]. It doesn’t apply in this case. Finally, what the heck do “covariance structure analysis” and “allele frequencies at a highly polymorphic locus” have to do with this problem? [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)
----
----
Quick question for SJohnson: How many degrees of freedom did you choose when calculating the p-value? I'd like to know upon what condition you base that number. Thanks.--[[User:Argon|Argon]] 11:05, 5 March 2009 (EST)
Quick question for SJohnson: How many degrees of freedom did you choose when calculating the p-value? I'd like to know upon what condition you base that number. Thanks.--[[User:Argon|Argon]] 11:05, 5 March 2009 (EST)
:The degree of freedom for a contingency table is rows minus one times columns minus one. That is, <math> (r-1)(c-1) </math>. Here’s a pretty good tutorial I came across: [http://faculty.uncfsu.edu/dwallace/lesson%2020.pdf]. For the experiments from [http://www.pnas.org/content/105/23/7899.full.pdf], the DOFs are 11, 11, and 13. For experiment one, the chi-square test statistic is
::<math>
X^2
=\sum\limits_i\sum\limits_j
\frac{\left(n_{i,j}-E\left[n_{i,j}\right]\right)^2}
{E\left[n_{i,j}\right]}
</math>
::<math>
=\frac{\left(0-1/3\right)^2}{1/3}
+\frac{\left(6-17/3\right)^2}{17/3}
+\frac{\left(0-1/3\right)^2}{1/3}
+\ldots+
</math>
::<math>
+\frac{\left(2-1/3\right)^2}{1/3}
+\frac{\left(4-17/3\right)^2}{17/3}
</math>
::<math>
\approx
14.82
</math>
:where <math>n_{i,j}</math> is the observed value and <math>E\left[n_{i,j}\right]</math> is the expected null hypothesis value. So if you have MS Excel, another way to arrive at the p-value of 0.19 is to type “=CHIDIST(14.82,11)” into a cell. Cheers! [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)
----
Let’s go back to the beginning. There appears to be confusion about the difference between test statistics and methods for computing p-values. As is noted at the beginning of the page [http://www.conservapedia.com/Significance_of_E._Coli_Evolution_Experiments], the fundamental problem with the paper is that it used a flawed test statistic, not that it used Monte Carlo methods to find the p-value for that flawed statistic.
Every hypothesis test uses a test statistic to reduce the data to a single number. The p-value for the test statistic can be calculated analytically (as I’ve done for the chi-square test statistic) or by Monte Carlo methods. In the paper, Monte Carlo methods were used to compute the p-value of the “mutation generation” test statistic. The key problem with the analysis from the paper is that it doesn’t work to use a weighted average to test for variations in mutation rate. This is like trying to use the sample variance to test for an increase in the mean in Gaussian-distributed data. A statistic should be selected based on the null and alternate hypothesis distributions of the data. The chi-square test (unlike the weighted average from the paper) is a reasonable choice for data that mutates at a constant rate under the null hypothesis, but mutates at varying rates under the alternate hypothesis.
Able806, you made a good point about the contingency table cell frequencies being relatively low, but were wrong when you said ”the data set is actually too small to use the chi square method correctly”. In the low cell frequency case the chi-square test is still effective, but the null hypothesis distribution of the chi-square statistic starts to look less like the chi-square distribution. Thus, p-values calculated using the chi-square distribution may be a bit off. However, Monte Carlo p-values are always imperfect as well because it's impossible to generate an infinite number of random realizations. There are imperfections in p-values generated by analytic and Monte Carlo methods. However, low cell frequencies does not explain the >20x and >2.5x differences between chi-square p-values and p-values from the paper for experiments one and three. The reason for those huge differences was the use of the flawed test statistic (“mutation generation”) in the paper. [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)