Significance of E. Coli Evolution Experiments

Blount, Borland, and Lenski claimed that a key evolutionary innovation was observed during a laboratory experiment. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.

Experiment One Data

The data from experiment one of the paper is shown below (see Table 1 of the paper). The expected outcomes under the null hypothesis (rate of mutation is constant over time) are also shown.

Generation	Trials	Mutants	Statics	Expected Mutants	Expected Statics
0	6	0	6	0.333	5.667
10000	6	0	6	0.333	5.667
20000	6	0	6	0.333	5.667
25000	6	0	6	0.333	5.667
27500	6	0	6	0.333	5.667
29000	6	0	6	0.333	5.667
30000	6	0	6	0.333	5.667
30500	6	1	5	0.333	5.667
31000	6	0	6	0.333	5.667
31500	6	1	5	0.333	5.667
32000	6	0	6	0.333	5.667
32500	6	2	4	0.333	5.667
Total	72	4	68	4	68

When the test used by Blount is used to compute the significance of this data, the p-value is 0.0085 (see Table 2 of the paper). This p-value is considered statistically significant, implying that the rate of mutation increases over time. However, when the data is analyzed using a standard method (the chi-square test) the p-value is 0.19. This p-value suggests that the mutation rate is constant over time. Note, however, that the chi-squared test is conservative when any expected value is less than 1 and unreliable when, as in the present case, many expected values are less than 0.5 . Consequently, while the p-value of 0.19 appears to suggest there is no reason to reject the null hypothesis, it is in fact much too high and the results of the chi-squared test are unreliable.

In statistical terms, conservative means that the p-value is higher than it ought to be, causing the experimenter to accept the null hypothesis when it should in fact be rejected.

The chi-square test is a common statistical method. It can be implemented in Microsoft Excel. If the numbers from the last four columns of the experiment one data table (excluding the “totals” row) are entered into Excel in rows 1-12 and columns A-D, then the p-value can be computed by entering “=CHITEST(A1:B12,C1:D12)” into any empty cell of the spreadsheet.

Experiment Three Data

The experiment three data from Blount et al. is shown in the table below. The expected numbers of mutants under the null hypothesis (constant mutation rate) is also shown. As for Experiment 1, the p-value of 0.22 might appear to suggest that the null hypothesis should be accepted. But for exactly the same reason as in Experiment 1, this p-value is highly conservative (in statistical terms) and cannot be considered a reliable test of the null hypothesis.

Generation	Trials	Mutants	Statics	Expected Mutants	Expected Statics
0	200	0	200	0.571	199.429
10000	200	0	200	0.571	199.429
20000	200	0	200	0.571	199.429
25000	200	0	200	0.571	199.429
27500	200	2	198	0.571	199.429
29000	200	0	200	0.571	199.429
30000	200	2	198	0.571	199.429
30500	200	0	200	0.571	199.429
31000	200	0	200	0.571	199.429
31500	200	0	200	0.571	199.429
32000	200	1	199	0.571	199.429
32500	200	1	199	0.571	199.429
Total	2800	8	2792	8	2792

Comparison of p-Values

The following table compares the p-values reported in Table 2 of Blount et al. to the chi-square p-values for the same experiments. For experiments one and three, the chi-square p-values are much larger than the "mean generation" test p-values from the paper. This is because the small expected number of mutants causes the chi-squared test to be conservative, meaning that the null hypothesis may be accepted when in fact it ought to be rejected. The chi-squared test is unreliable in this situation.

	Experiment 1	Experiment 2	Experiment 3
p-Value from Paper	0.0085	0.0007	0.082
Chi-square p-value	0.19	0.0004	0.22

See Also

http://www.sciencenews.org/index/feature/activity/view/id/40006/title/Molecular_Evolution

http://sciencenews.org/view/generic/id/40649/title/FOR_KIDS_Hitting_the_redo_button_on_evolution