37
edits
Changes
Jump to navigation
Jump to search
Line 1:
Line 1: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Significance of E. Coli Evolution Experiments (view source)
Revision as of 19:48, March 15, 2009
, 19:48, March 15, 2009Added section on test statistics and described why the test from the paper doesn't work
Blount, Borland, and [[Lenski]]<ref> http://www.pnas.org/content/105/23/7899.full.pdf</ref> claimed that a key evolutionary innovation was observed during a [[laboratory]] [[experiment]]. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.
Blount, Borland, and [[Lenski]]<ref> http://www.pnas.org/content/105/23/7899.full.pdf</ref> claimed that a key evolutionary innovation was observed during a [[laboratory]] [[experiment]]. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.
==Test Statistics==
The test statistic used in Blout, Borland, and Lenski was mean mutation generation. For example, the experiment one mean mutation generation is (see Tables 1 and 2 of the paper)
:<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)
=31750.
</math>
The null hypothesis from the paper was constant mutation rate over all generations. However, measurements of mean mutation
generation can fail to observe deviations from that null
hypothesis. Consider an experiment where the mutation
probabilities for generations 1, 2, and 3 are <math>p_1</math>, <math>p_2</math>, and
<math>p_3</math>, respectively. If the mutation probabilities per generation
are <math>p_1=p_2=p_3=p</math>, then the mean mutation generation is 2.
However, if the mutation probabilities are <math>p_1=p_3=p/2</math> and
<math>p_2=2p</math>, then the mean mutation generation is still 2. In the
latter example the experiment has deviated from the null
hypothesis, but the mean mutation
generation is insensitive to the
change. Because this test can fail to observe deviations from the null hypothesis, levels of statistical significance computed using this test (p-values) are meaningless.
If the trials from an experiment have two outcomes (e.g. success or failure), then the chi-square test for independence can be written
:<math>
X^2
=\sum\limits_{i=1}^M
\frac{\left(x_i-\hat{p}N_i\right)^2}
{\hat{p}(1-\hat{p})N_i}
</math>
where <math>x_i</math> is the number of successes (e.g. mutations) in the ''i''-th experiment, <math>N_i</math> is the number of trials in the ''i''-th experiment, and
:<math>
\hat{p}=\frac{\sum\limits_{i=1}^Mx_i}{\sum\limits_{i=1}^MN_i}
</math>
is the fraction of trials that are successful in the all experiments. The chi-square test is, on average, at a minimum when all success probabilities are equal (<math>p_i=p</math> for <math>i=1,2,\ldots,M</math>) and will, on average, increase whenever the data follows any other hypothesis (<math>\sum\limits_{i=1}^M
\left(p_i-\bar{p}\right)^2>0</math>) where <math>\bar{p}</math> is the mean of the success probabilities. Thus the chi-square test is an effective hypothesis test for the data and hypotheses from Blount et al.
==Experiment One Data==
==Experiment One Data==