Difference between revisions of "Flaws in Richard Lenski Study"

Revision as of 18:13, August 21, 2011

Regarding his experiment on historical contingency in evolution, Richard Lenski rejected a request to fully release his bacteria mutation data to the public. An analysis of his work^[1], however, reveals serious flaws even without a full disclosure of the data. Note that the peer review on Lenski's paper took somewhere between 0 (non-existent) and at most 14 days (including administrative time), and Lenski himself does not have any obvious expertise in statistics. In fact, Richard Lenski admits in his paper that he based his statistical conclusions on use of a website called "statistics101".

1. Lenski's "historical contingency" hypothesis, as specifically depicted in Figure 3, is contradicted by the data presented in the Third Experiment in Table 1 of his paper. Figure 3 proposes a step-up in mutation rate to Cit⁺ due to a historical contingency (potentiating mutation) occurring at about the 31,000th generation, yet the Third (and largest) Experiment in Table 1 shows Cit⁺ arising just as often before the 31,000th generation as after. The abstract, in further contradiction with Figure 3, suggests that the historical contingency (potentiating mutation) occurred prior to the 20,000th generation.

2. Lenski's two alternative hypotheses suggest a fixed mutation rate, but the failure of the mutations in his experiments to increase based on scale (number of samples) tends to disprove both of Lenski's alternative hypotheses. Yet Lenski's paper fails to address adequately this obvious flaw in the paper.

3. Richard Lenski incorrectly included generations of the E. coli already known to contain Cit⁺ variants in his experiments.^[2] Once these generations are removed from the analysis, the data disprove Lenski's hypothesis.

4. The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events. The Third Experiment results are consistent with the null hypothesis, contrary to the paper's claim.

5. Lenski's largest experiment (Third Experiment) failed to support his hypothesis with statistical significance. Even though this largest experiment was nearly ten times the size of his other experiments, Richard Lenski did not weight this largest experiment correctly in combining his results.

6. It was error to include generations of the E. coli already known to contain trace Cit+ variants. The highly improbable occurrence of four Cit+ variants from the 32,000th generation in the Second Experiment suggests an origin from undetected, pre-existing Cit+ variants.

7. The Third Experiment was erroneously combined with the other two experiments based on outcome rather than sample size, thereby yielding a false claim of overall statistical significance. Lenski's paper applied the Whitlock Z-transformation incorrectly, perhaps intentionally so, in making a claim that Lenski's results were "extremely significant": "We also used the Z-transformation method to combine the probabilities from our three experiments, and the result is extremely significant (P < 0.0001) whether or not the experiments are weighted by the number of independent Cit+ mutants observed in each one."^[3] Lenski's "whether or not" refers to two incorrect applications of the Whitlock technique, obscuring how the straightforward, correct weighting based on sample size was not used. A reader could conclude that the Lenski paper deliberately conceals the misapplication.

8. Lenski's paper is not clear in explaining how the results of his largest experiment (Third Experiment) failed to confirm his hypothesis with statistical significance, even with the incorrect inclusion of the Cit⁺ variant generations. Instead, his paper refers to his largest experiment as "marginally ... significant," which serves to obscure its statistical insignificance. Other works published in PNAS are clear in defining statistical significance in the traditional way, which Lenski's Third Experiment (even with incorrect inclusion of the above-referenced generations) failed to satisfy.^[4]

9. The long lag time (over 12,000 generations) between the historical contingency (potentiating mutation) in the largest experiment disproves Lenski's implicit assumption that the potentiating mutation likely occurred in proximity with the occurrence of the Cit⁺ variant, and that the first occurrence of the Cit⁺ variant in the Third Experiment at the 20,000th generation somehow implies that a potentiating mutation occurred in its proximity.

10. Lenski's paper claims that "During [30,000 generations], each population experienced billions of mutations,^[5] far more than the number of possible point mutations in the [approximately] 4.6-million-bp genome. This ratio implies, to a first approximation, that each population tried every typical one-step mutation many times." Lenski's conclusion is nonsensical because it assumes that the mutations are completely random and that each mutation has a roughly equal probability.

11. In Table 2 of [1], the expected mean should be 26,382 generations, not 28,382.

12. The p-value computed for experiment two was incorrectly listed as 0.0007 instead of 0.0006 in [2]. These p-values are meaningless because the paper used a flawed test statistic (see: Significance of E. Coli Evolution Experiments#Test Statistics). However, the error illustrates the need to use enough random realizations when using Monte Carlo methods to estimate p-values.

References

1. ↑ Blount et al., "Historical contingency and the evolution of a key innovation in an experimental population of Escherichia coli, 105 PNAS 7899-7906 (June 10, 2008).
2. ↑ Richard Lenski incorrectly included generations 31,500, 32,000 and 32,500.
3. ↑ Lenski paper at 7902 (citation to Whitlock paper omitted, emphasis added).
4. ↑ See, e.g., Cholera toxin induces malignant glioma cell differentiation
5. ↑ Lenski cites one of his own prior articles for this.

See also

• Letter to PNAS
• Significance of E. Coli Evolution Experiments