Conservapedia - User contributions [en]

User:SJohnson

2009-11-30T03:54:44Z

SJohnson:

Data on the math aptitude of graduate school applicants, by major [http://www.ets.org/Media/Tests/GRE/pdf/gre_0809_interpretingscores.pdf]:

{|class="wikitable" style="text-align:center"
|-
!Intended Graduate Major
!GRE Quantitative (Math) Score
|-
|Engineering
|717
|-
|Physical Sciences
|692
|-
|Business
|594
|-
|Life Sciences
|575
|}

Which group should be developing new statistical hypothesis tests?

Significance of E. Coli Evolution Experiments

2009-07-02T13:34:40Z

SJohnson: Put p-value comparison table closer to beginning

Blount, Borland, and [[Lenski]]<ref> http://www.pnas.org/content/105/23/7899.full.pdf</ref> claimed that a key evolutionary innovation was observed during a [[laboratory]] [[experiment]]. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.

==Test Statistics==

The test statistic used in Blout, Borland, and Lenski was mean mutation generation. For example, the experiment one mean mutation generation is (see Tables 1 and 2 of the paper)
:<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)
=31750.
</math>

The null hypothesis from the paper was constant mutation rate over all generations. However, measurements of mean mutation
generation can fail to observe deviations from that null
hypothesis. Consider an experiment where the mutation
probabilities for generations 1, 2, and 3 are <math>p_1</math>, <math>p_2</math>, and
<math>p_3</math>, respectively. If the mutation probabilities per generation
are <math>p_1=p_2=p_3=p</math>, then the mean mutation generation is 2.
However, if the mutation probabilities are <math>p_1=p_3=p/2</math> and
<math>p_2=2p</math>, then the mean mutation generation is still 2. In the
latter example the experiment has deviated from the null
hypothesis, but the mean mutation
generation is insensitive to the
change. Because this test can fail to observe deviations from the null hypothesis, levels of statistical significance computed using this test (p-values) are meaningless.

If the trials from an experiment have two outcomes (e.g. success or failure), then the chi-square test for independence can be written
:<math>
X^2
=\sum\limits_{i=1}^M
\frac{\left(x_i-\hat{p}N_i\right)^2}
{\hat{p}(1-\hat{p})N_i}
</math>
where <math>x_i</math> is the number of successes (e.g. mutations) in the ''i''-th experiment, <math>N_i</math> is the number of trials in the ''i''-th experiment, and
:<math>
\hat{p}=\frac{\sum\limits_{i=1}^Mx_i}{\sum\limits_{i=1}^MN_i}
</math>
is the fraction of trials that are successful in all experiments. The chi-square test is, on average, at a minimum when all success probabilities are equal (<math>p_i=p</math> for <math>i=1,2,\ldots,M</math>) and will, on average, increase whenever the data follows any other hypothesis (<math>\sum\limits_{i=1}^M
\left(p_i-\bar{p}\right)^2>0</math> where <math>\bar{p}</math> is the mean of the success probabilities). Thus the chi-square test is an effective hypothesis test for the data and hypotheses from Blount et al.

==Comparison of p-Values==

The following table compares the p-values reported in Table 2 of Blount et al. to the chi-square p-values for the same experiments. For experiments one and three, the chi-square p-values are much larger than the "mean generation" test p-values from the paper.

{|class="wikitable" style="text-align:center"
|-
|
!Experiment 1
!Experiment 2
!Experiment 3
|-
!p-Value from Paper
|0.0085
|0.0007
|0.082
|-
!Chi-square p-value
|0.19
|0.0004
|0.22
|}

The chi-square test p-values are computed by comparing the test statistic to the chi-square distribution. It is generally assumed that the cell frequencies should be greater than five so that the statistic's distribution follows chi-square distribution. However, there is no consensus about what minimum cell frequency is necessary or how many expected values need to cross that threshold.

===Experiment One Data===

The data from experiment one of the paper is shown below (see Table 1 of the paper). The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown.

{|class="wikitable" style="text-align:center"
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|6
|0
|6
|0.333
|5.667
|-
|10000
|6
|0
|6
|0.333
|5.667
|-
|20000
|6
|0
|6
|0.333
|5.667
|-
|25000
|6
|0
|6
|0.333
|5.667
|-
|27500
|6
|0
|6
|0.333
|5.667
|-
|29000
|6
|0
|6
|0.333
|5.667
|-
|30000
|6
|0
|6
|0.333
|5.667
|-
|30500
|6
|1
|5
|0.333
|5.667
|-
|31000
|6
|0
|6
|0.333
|5.667
|-
|31500
|6
|1
|5
|0.333
|5.667
|-
|32000
|6
|0
|6
|0.333
|5.667
|-
|32500
|6
|2
|4
|0.333
|5.667
|-
|-
!Total
!72
!4
!68
!4
!68
|}

When the flawed test is used to compute the significance of this data, the p-value is 0.0085 (see Table 2 of the paper). This p-value is considered statistically significant. However, when the data is analyzed using a standard method (the chi-square test) the p-value is 0.19. This p-value is much larger than the one from the paper and indicates that there is no reason to reject the null hypothesis. The chi-square test p-value for experiment two is small (0.0004). However, experiment three is not statistically significant because its p-value is 0.22.

The chi-square test is a common statistical method.<ref>''Mathematical Statistics with Applications'' by Wackerly, Mendenhall, and Scheaffer, Section 14.4.</ref> It can be implemented in Microsoft Excel. If the numbers from the last four columns of the experiment one data table (excluding the “totals” row) are entered into Excel in rows 1-12 and columns A-D, then the p-value can be computed by entering “=CHITEST(A1:B12,C1:D12)” into any empty cell of the spreadsheet.

===Experiment Three Data===

The experiment three data from Blount et al. is shown in the table below. The expected numbers of mutants under the null hypothesis (constant mutation rate) is also shown.

{|class="wikitable" style="text-align:center"
|-
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|200
|0
|200
|0.571
|199.429
|-
|10000
|200
|0
|200
|0.571
|199.429
|-
|20000
|200
|0
|200
|0.571
|199.429
|-
|25000
|200
|0
|200
|0.571
|199.429
|-
|27500
|200
|2
|198
|0.571
|199.429
|-
|29000
|200
|0
|200
|0.571
|199.429
|-
|30000
|200
|2
|198
|0.571
|199.429
|-
|30500
|200
|0
|200
|0.571
|199.429
|-
|31000
|200
|0
|200
|0.571
|199.429
|-
|31500
|200
|0
|200
|0.571
|199.429
|-
|32000
|200
|1
|199
|0.571
|199.429
|-
|32500
|200
|1
|199
|0.571
|199.429
|-
|-
!Total
!2800
!8
!2792
!8
!2792
|}

==References==
<references/>
[[Category:Statistics]]

==See Also==
http://www.sciencenews.org/index/feature/activity/view/id/40006/title/Molecular_Evolution

http://sciencenews.org/view/generic/id/40649/title/FOR_KIDS_Hitting_the_redo_button_on_evolution

[[Flaws_in_Richard_Lenski_Study|Flaws in Richard Lenski Study]]

Flaws in Richard Lenski Study

2009-04-11T00:40:42Z

SJohnson:

[[Richard Lenski]] rejected a request to release his bacteria mutation data to the public,<ref>See [[Conservapedia:Lenski dialog]].</ref> but the following serious flaws are emerging about his work<ref>Blount et al., "Historical contingency and the [[evolution]] of a key innovation in an experimental population of ''Escherichia coli'', 105 PNAS 7899-7906 (June 10, 2008).</ref> even without a full disclosure of the data. Note that the peer review on Lenski's paper took somewhere between 0 (non-existent) and at most 14 days (including administrative time), and Lenski himself does not have any obvious expertise in statistics. In fact, Richard Lenski admits in his paper that he based his statistical conclusions on use of a website called "statistics101".

1. Lenski's "historical contingency" hypothesis, as specifically depicted in Figure 3, is contradicted by the data presented in the Third Experiment in Table 1 of his paper. Figure 3 proposes a step-up in mutation rate to Cit+ due to a historical contingency (potentiating mutation) occurring at about the 31,000th generation, yet the Third (and largest) Experiment in Table 1 shows Cit+ arising just as often before the 31,000th generation as after. The abstract, in further contradiction with Figure 3, suggests that the historical contingency (potentiating mutation) occurred prior to the 20,000th generation.

2. Lenski's two alternative hypotheses suggest a fixed mutation rate, but the failure of the mutations in his experiments to increase based on scale (number of samples) tends to disprove both of Lenski's alternative hypotheses. Yet Lenski's paper fails to address adequately this obvious flaw in the paper.

3. Richard Lenski incorrectly included generations of the ''E. coli'' already known to contain Cit+ variants in his experiments.<ref>Richard Lenski incorrectly included generations 31,500, 32,000 and 32,500.</ref> Once these generations are removed from the analysis, the data disprove Lenski's hypothesis.

4. The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events. The Third Experiment results are consistent with the null hypothesis, contrary to the paper's claim.

5. Lenski's largest experiment (Third Experiment) failed to support his hypothesis with statistical significance. Even though this largest experiment was nearly ten times the size of his other experiments, Richard Lenski did not weight this largest experiment correctly in combining his results.

6. It was error to include generations of the E. coli already known to contain trace Cit+ variants. The highly improbable occurrence of four Cit+ variants from the 32,000th generation in the Second Experiment suggests an origin from undetected, pre-existing Cit+ variants.

7. The Third Experiment was erroneously combined with the other two experiments based on outcome rather than sample size, thereby yielding a false claim of overall statistical significance. Lenski's paper applied the Whitlock Z-transformation incorrectly, perhaps intentionally so, in making a claim that Lenski's results were "extremely significant": "We also used the Z-transformation method to combine the probabilities from our three experiments, and '''the result is extremely significant (P < 0.0001) whether or not''' the experiments are weighted by the number of independent Cit+ mutants observed in each one."<ref>Lenski paper at 7902 (citation to Whitlock paper omitted, emphasis added).</ref> Lenski's "whether or not" refers to two incorrect applications of the Whitlock technique, obscuring how the straightforward, correct weighting based on sample size was ''not'' used. A reader could conclude that the Lenski paper deliberately conceals the misapplication.

8. Lenski's paper is not clear in explaining how the results of his largest experiment (Third Experiment) failed to confirm his hypothesis with statistical significance, even with the incorrect inclusion of the Cit+ variant generations. Instead, his paper refers to his largest experiment as "marginally ... significant," which serves to obscure its statistical insignificance. Other works published in PNAS are clear in defining statistical significance in the traditional way, which Lenski's Third Experiment (even with incorrect inclusion of the above-referenced generations) failed to satisfy.<ref>See, e.g., [http://www.pnas.org/cgi/content/full/0701990104 Cholera toxin induces malignant glioma cell differentiation]</ref>

9. The long lag time (over 12,000 generations) between the historical contingency (potentiating mutation) in the largest experiment disproves Lenski's implicit assumption that the potentiating mutation likely occurred in proximity with the occurrence of the Cit+ variant, and that the first occurrence of the Cit+ variant in the Third Experiment at the 20,000th generation somehow implies that a potentiating mutation occurred in its proximity.

10. Lenski's paper claims that "During [30,000 generations], each population experienced billions of mutations,<ref>Lenski cites one of his own prior articles for this.</ref> far more than the number of possible point mutations in the [approximately] 4.6-million-bp genome. This ratio implies, to a first approximation, that each population tried every typical one-step mutation many times." Lenski's conclusion is nonsensical because it assumes that the mutations are completely random '''and''' that each mutation has a roughly equal probability.

11. In Table 2 of [http://www.pnas.org/content/105/23/7899.full.pdf], the expected mean should be 26,382 generations, not 28,382.

12. The p-value computed for experiment two was incorrectly listed as 0.0007 instead of 0.0006 in [http://www.pnas.org/content/105/23/7899.full.pdf]. These p-values are meaningless because the paper used a flawed test statistic (see: [[Significance of E. Coli Evolution Experiments#Test Statistics]]). However, the error illustrates the need to use enough random realizations when using Monte Carlo methods to estimate p-values.

== References ==

<references/>

== See also ==

*[[Letter to PNAS]]
*[[Significance of E. Coli Evolution Experiments]]
[[Category:Science]]

Significance of E. Coli Evolution Experiments

2009-04-04T03:43:26Z

SJohnson: /* Test Statistics */ typo

Blount, Borland, and [[Lenski]]<ref> http://www.pnas.org/content/105/23/7899.full.pdf</ref> claimed that a key evolutionary innovation was observed during a [[laboratory]] [[experiment]]. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.

==Test Statistics==

The test statistic used in Blout, Borland, and Lenski was mean mutation generation. For example, the experiment one mean mutation generation is (see Tables 1 and 2 of the paper)
:<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)
=31750.
</math>

The null hypothesis from the paper was constant mutation rate over all generations. However, measurements of mean mutation
generation can fail to observe deviations from that null
hypothesis. Consider an experiment where the mutation
probabilities for generations 1, 2, and 3 are <math>p_1</math>, <math>p_2</math>, and
<math>p_3</math>, respectively. If the mutation probabilities per generation
are <math>p_1=p_2=p_3=p</math>, then the mean mutation generation is 2.
However, if the mutation probabilities are <math>p_1=p_3=p/2</math> and
<math>p_2=2p</math>, then the mean mutation generation is still 2. In the
latter example the experiment has deviated from the null
hypothesis, but the mean mutation
generation is insensitive to the
change. Because this test can fail to observe deviations from the null hypothesis, levels of statistical significance computed using this test (p-values) are meaningless.

If the trials from an experiment have two outcomes (e.g. success or failure), then the chi-square test for independence can be written
:<math>
X^2
=\sum\limits_{i=1}^M
\frac{\left(x_i-\hat{p}N_i\right)^2}
{\hat{p}(1-\hat{p})N_i}
</math>
where <math>x_i</math> is the number of successes (e.g. mutations) in the ''i''-th experiment, <math>N_i</math> is the number of trials in the ''i''-th experiment, and
:<math>
\hat{p}=\frac{\sum\limits_{i=1}^Mx_i}{\sum\limits_{i=1}^MN_i}
</math>
is the fraction of trials that are successful in all experiments. The chi-square test is, on average, at a minimum when all success probabilities are equal (<math>p_i=p</math> for <math>i=1,2,\ldots,M</math>) and will, on average, increase whenever the data follows any other hypothesis (<math>\sum\limits_{i=1}^M
\left(p_i-\bar{p}\right)^2>0</math> where <math>\bar{p}</math> is the mean of the success probabilities). Thus the chi-square test is an effective hypothesis test for the data and hypotheses from Blount et al.

==Experiment One Data==

The data from experiment one of the paper is shown below (see Table 1 of the paper). The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown.

{|class="wikitable" style="text-align:center"
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|6
|0
|6
|0.333
|5.667
|-
|10000
|6
|0
|6
|0.333
|5.667
|-
|20000
|6
|0
|6
|0.333
|5.667
|-
|25000
|6
|0
|6
|0.333
|5.667
|-
|27500
|6
|0
|6
|0.333
|5.667
|-
|29000
|6
|0
|6
|0.333
|5.667
|-
|30000
|6
|0
|6
|0.333
|5.667
|-
|30500
|6
|1
|5
|0.333
|5.667
|-
|31000
|6
|0
|6
|0.333
|5.667
|-
|31500
|6
|1
|5
|0.333
|5.667
|-
|32000
|6
|0
|6
|0.333
|5.667
|-
|32500
|6
|2
|4
|0.333
|5.667
|-
|-
!Total
!72
!4
!68
!4
!68
|}

When the flawed test is used to compute the significance of this data, the p-value is 0.0085 (see Table 2 of the paper). This p-value is considered statistically significant. However, when the data is analyzed using a standard method (the chi-square test) the p-value is 0.19. This p-value is much larger than the one from the paper and indicates that there is no reason to reject the null hypothesis. The chi-square test p-value for experiment two is small (0.0004). However, experiment three is not statistically significant because its p-value is 0.22.

The chi-square test is a common statistical method.<ref>''Mathematical Statistics with Applications'' by Wackerly, Mendenhall, and Scheaffer, Section 14.4.</ref> It can be implemented in Microsoft Excel. If the numbers from the last four columns of the experiment one data table (excluding the “totals” row) are entered into Excel in rows 1-12 and columns A-D, then the p-value can be computed by entering “=CHITEST(A1:B12,C1:D12)” into any empty cell of the spreadsheet.

==Experiment Three Data==

The experiment three data from Blount et al. is shown in the table below. The expected numbers of mutants under the null hypothesis (constant mutation rate) is also shown.

{|class="wikitable" style="text-align:center"
|-
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|200
|0
|200
|0.571
|199.429
|-
|10000
|200
|0
|200
|0.571
|199.429
|-
|20000
|200
|0
|200
|0.571
|199.429
|-
|25000
|200
|0
|200
|0.571
|199.429
|-
|27500
|200
|2
|198
|0.571
|199.429
|-
|29000
|200
|0
|200
|0.571
|199.429
|-
|30000
|200
|2
|198
|0.571
|199.429
|-
|30500
|200
|0
|200
|0.571
|199.429
|-
|31000
|200
|0
|200
|0.571
|199.429
|-
|31500
|200
|0
|200
|0.571
|199.429
|-
|32000
|200
|1
|199
|0.571
|199.429
|-
|32500
|200
|1
|199
|0.571
|199.429
|-
|-
!Total
!2800
!8
!2792
!8
!2792
|}

==Comparison of p-Values==

The following table compares the p-values reported in Table 2 of Blount et al. to the chi-square p-values for the same experiments. For experiments one and three, the chi-square p-values are much larger than the "mean generation" test p-values from the paper.

{|class="wikitable" style="text-align:center"
|-
|
!Experiment 1
!Experiment 2
!Experiment 3
|-
!p-Value from Paper
|0.0085
|0.0007
|0.082
|-
!Chi-square p-value
|0.19
|0.0004
|0.22
|}

The chi-square test p-values are computed by comparing the test statistic to the chi-square distribution. It is generally assumed that the cell frequencies should be greater than five so that the statistic's distribution follows chi-square distribution. However, there is no consensus about what minimum cell frequency is necessary or how many expected values need to cross that threshold.

==References==
<references/>
[[Category:Statistics]]

==See Also==
http://www.sciencenews.org/index/feature/activity/view/id/40006/title/Molecular_Evolution

http://sciencenews.org/view/generic/id/40649/title/FOR_KIDS_Hitting_the_redo_button_on_evolution

[[Flaws_in_Richard_Lenski_Study|Flaws in Richard Lenski Study]]

Significance of E. Coli Evolution Experiments

2009-03-29T21:08:14Z

SJohnson: /* See Also */

Blount, Borland, and [[Lenski]]<ref> http://www.pnas.org/content/105/23/7899.full.pdf</ref> claimed that a key evolutionary innovation was observed during a [[laboratory]] [[experiment]]. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.

==Test Statistics==

The test statistic used in Blout, Borland, and Lenski was mean mutation generation. For example, the experiment one mean mutation generation is (see Tables 1 and 2 of the paper)
:<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)
=31750.
</math>

The null hypothesis from the paper was constant mutation rate over all generations. However, measurements of mean mutation
generation can fail to observe deviations from that null
hypothesis. Consider an experiment where the mutation
probabilities for generations 1, 2, and 3 are <math>p_1</math>, <math>p_2</math>, and
<math>p_3</math>, respectively. If the mutation probabilities per generation
are <math>p_1=p_2=p_3=p</math>, then the mean mutation generation is 2.
However, if the mutation probabilities are <math>p_1=p_3=p/2</math> and
<math>p_2=2p</math>, then the mean mutation generation is still 2. In the
latter example the experiment has deviated from the null
hypothesis, but the mean mutation
generation is insensitive to the
change. Because this test can fail to observe deviations from the null hypothesis, levels of statistical significance computed using this test (p-values) are meaningless.

If the trials from an experiment have two outcomes (e.g. success or failure), then the chi-square test for independence can be written
:<math>
X^2
=\sum\limits_{i=1}^M
\frac{\left(x_i-\hat{p}N_i\right)^2}
{\hat{p}(1-\hat{p})N_i}
</math>
where <math>x_i</math> is the number of successes (e.g. mutations) in the ''i''-th experiment, <math>N_i</math> is the number of trials in the ''i''-th experiment, and
:<math>
\hat{p}=\frac{\sum\limits_{i=1}^Mx_i}{\sum\limits_{i=1}^MN_i}
</math>
is the fraction of trials that are successful in the all experiments. The chi-square test is, on average, at a minimum when all success probabilities are equal (<math>p_i=p</math> for <math>i=1,2,\ldots,M</math>) and will, on average, increase whenever the data follows any other hypothesis (<math>\sum\limits_{i=1}^M
\left(p_i-\bar{p}\right)^2>0</math> where <math>\bar{p}</math> is the mean of the success probabilities). Thus the chi-square test is an effective hypothesis test for the data and hypotheses from Blount et al.

==Experiment One Data==

The data from experiment one of the paper is shown below (see Table 1 of the paper). The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown.

{|class="wikitable" style="text-align:center"
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|6
|0
|6
|0.333
|5.667
|-
|10000
|6
|0
|6
|0.333
|5.667
|-
|20000
|6
|0
|6
|0.333
|5.667
|-
|25000
|6
|0
|6
|0.333
|5.667
|-
|27500
|6
|0
|6
|0.333
|5.667
|-
|29000
|6
|0
|6
|0.333
|5.667
|-
|30000
|6
|0
|6
|0.333
|5.667
|-
|30500
|6
|1
|5
|0.333
|5.667
|-
|31000
|6
|0
|6
|0.333
|5.667
|-
|31500
|6
|1
|5
|0.333
|5.667
|-
|32000
|6
|0
|6
|0.333
|5.667
|-
|32500
|6
|2
|4
|0.333
|5.667
|-
|-
!Total
!72
!4
!68
!4
!68
|}

When the flawed test is used to compute the significance of this data, the p-value is 0.0085 (see Table 2 of the paper). This p-value is considered statistically significant. However, when the data is analyzed using a standard method (the chi-square test) the p-value is 0.19. This p-value is much larger than the one from the paper and indicates that there is no reason to reject the null hypothesis. The chi-square test p-value for experiment two is small (0.0004). However, experiment three is not statistically significant because its p-value is 0.22.

The chi-square test is a common statistical method.<ref>''Mathematical Statistics with Applications'' by Wackerly, Mendenhall, and Scheaffer, Section 14.4.</ref> It can be implemented in Microsoft Excel. If the numbers from the last four columns of the experiment one data table (excluding the “totals” row) are entered into Excel in rows 1-12 and columns A-D, then the p-value can be computed by entering “=CHITEST(A1:B12,C1:D12)” into any empty cell of the spreadsheet.

==Experiment Three Data==

The experiment three data from Blount et al. is shown in the table below. The expected numbers of mutants under the null hypothesis (constant mutation rate) is also shown.

{|class="wikitable" style="text-align:center"
|-
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|200
|0
|200
|0.571
|199.429
|-
|10000
|200
|0
|200
|0.571
|199.429
|-
|20000
|200
|0
|200
|0.571
|199.429
|-
|25000
|200
|0
|200
|0.571
|199.429
|-
|27500
|200
|2
|198
|0.571
|199.429
|-
|29000
|200
|0
|200
|0.571
|199.429
|-
|30000
|200
|2
|198
|0.571
|199.429
|-
|30500
|200
|0
|200
|0.571
|199.429
|-
|31000
|200
|0
|200
|0.571
|199.429
|-
|31500
|200
|0
|200
|0.571
|199.429
|-
|32000
|200
|1
|199
|0.571
|199.429
|-
|32500
|200
|1
|199
|0.571
|199.429
|-
|-
!Total
!2800
!8
!2792
!8
!2792
|}

==Comparison of p-Values==

The following table compares the p-values reported in Table 2 of Blount et al. to the chi-square p-values for the same experiments. For experiments one and three, the chi-square p-values are much larger than the "mean generation" test p-values from the paper.

{|class="wikitable" style="text-align:center"
|-
|
!Experiment 1
!Experiment 2
!Experiment 3
|-
!p-Value from Paper
|0.0085
|0.0007
|0.082
|-
!Chi-square p-value
|0.19
|0.0004
|0.22
|}

The chi-square test p-values are computed by comparing the test statistic to the chi-square distribution. It is generally assumed that the cell frequencies should be greater than five so that the statistic's distribution follows chi-square distribution. However, there is no consensus about what minimum cell frequency is necessary or how many expected values need to cross that threshold.

==References==
<references/>
[[Category:Statistics]]

==See Also==
http://www.sciencenews.org/index/feature/activity/view/id/40006/title/Molecular_Evolution

http://sciencenews.org/view/generic/id/40649/title/FOR_KIDS_Hitting_the_redo_button_on_evolution

[[Flaws_in_Richard_Lenski_Study|Flaws in Richard Lenski Study]]

Significance of E. Coli Evolution Experiments

2009-03-29T13:36:52Z

SJohnson: /* See Also */

Blount, Borland, and [[Lenski]]<ref> http://www.pnas.org/content/105/23/7899.full.pdf</ref> claimed that a key evolutionary innovation was observed during a [[laboratory]] [[experiment]]. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.

==Test Statistics==

The test statistic used in Blout, Borland, and Lenski was mean mutation generation. For example, the experiment one mean mutation generation is (see Tables 1 and 2 of the paper)
:<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)
=31750.
</math>

The null hypothesis from the paper was constant mutation rate over all generations. However, measurements of mean mutation
generation can fail to observe deviations from that null
hypothesis. Consider an experiment where the mutation
probabilities for generations 1, 2, and 3 are <math>p_1</math>, <math>p_2</math>, and
<math>p_3</math>, respectively. If the mutation probabilities per generation
are <math>p_1=p_2=p_3=p</math>, then the mean mutation generation is 2.
However, if the mutation probabilities are <math>p_1=p_3=p/2</math> and
<math>p_2=2p</math>, then the mean mutation generation is still 2. In the
latter example the experiment has deviated from the null
hypothesis, but the mean mutation
generation is insensitive to the
change. Because this test can fail to observe deviations from the null hypothesis, levels of statistical significance computed using this test (p-values) are meaningless.

If the trials from an experiment have two outcomes (e.g. success or failure), then the chi-square test for independence can be written
:<math>
X^2
=\sum\limits_{i=1}^M
\frac{\left(x_i-\hat{p}N_i\right)^2}
{\hat{p}(1-\hat{p})N_i}
</math>
where <math>x_i</math> is the number of successes (e.g. mutations) in the ''i''-th experiment, <math>N_i</math> is the number of trials in the ''i''-th experiment, and
:<math>
\hat{p}=\frac{\sum\limits_{i=1}^Mx_i}{\sum\limits_{i=1}^MN_i}
</math>
is the fraction of trials that are successful in the all experiments. The chi-square test is, on average, at a minimum when all success probabilities are equal (<math>p_i=p</math> for <math>i=1,2,\ldots,M</math>) and will, on average, increase whenever the data follows any other hypothesis (<math>\sum\limits_{i=1}^M
\left(p_i-\bar{p}\right)^2>0</math> where <math>\bar{p}</math> is the mean of the success probabilities). Thus the chi-square test is an effective hypothesis test for the data and hypotheses from Blount et al.

==Experiment One Data==

The data from experiment one of the paper is shown below (see Table 1 of the paper). The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown.

{|class="wikitable" style="text-align:center"
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|6
|0
|6
|0.333
|5.667
|-
|10000
|6
|0
|6
|0.333
|5.667
|-
|20000
|6
|0
|6
|0.333
|5.667
|-
|25000
|6
|0
|6
|0.333
|5.667
|-
|27500
|6
|0
|6
|0.333
|5.667
|-
|29000
|6
|0
|6
|0.333
|5.667
|-
|30000
|6
|0
|6
|0.333
|5.667
|-
|30500
|6
|1
|5
|0.333
|5.667
|-
|31000
|6
|0
|6
|0.333
|5.667
|-
|31500
|6
|1
|5
|0.333
|5.667
|-
|32000
|6
|0
|6
|0.333
|5.667
|-
|32500
|6
|2
|4
|0.333
|5.667
|-
|-
!Total
!72
!4
!68
!4
!68
|}

When the flawed test is used to compute the significance of this data, the p-value is 0.0085 (see Table 2 of the paper). This p-value is considered statistically significant. However, when the data is analyzed using a standard method (the chi-square test) the p-value is 0.19. This p-value is much larger than the one from the paper and indicates that there is no reason to reject the null hypothesis. The chi-square test p-value for experiment two is small (0.0004). However, experiment three is not statistically significant because its p-value is 0.22.

The chi-square test is a common statistical method.<ref>''Mathematical Statistics with Applications'' by Wackerly, Mendenhall, and Scheaffer, Section 14.4.</ref> It can be implemented in Microsoft Excel. If the numbers from the last four columns of the experiment one data table (excluding the “totals” row) are entered into Excel in rows 1-12 and columns A-D, then the p-value can be computed by entering “=CHITEST(A1:B12,C1:D12)” into any empty cell of the spreadsheet.

==Experiment Three Data==

The experiment three data from Blount et al. is shown in the table below. The expected numbers of mutants under the null hypothesis (constant mutation rate) is also shown.

{|class="wikitable" style="text-align:center"
|-
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|200
|0
|200
|0.571
|199.429
|-
|10000
|200
|0
|200
|0.571
|199.429
|-
|20000
|200
|0
|200
|0.571
|199.429
|-
|25000
|200
|0
|200
|0.571
|199.429
|-
|27500
|200
|2
|198
|0.571
|199.429
|-
|29000
|200
|0
|200
|0.571
|199.429
|-
|30000
|200
|2
|198
|0.571
|199.429
|-
|30500
|200
|0
|200
|0.571
|199.429
|-
|31000
|200
|0
|200
|0.571
|199.429
|-
|31500
|200
|0
|200
|0.571
|199.429
|-
|32000
|200
|1
|199
|0.571
|199.429
|-
|32500
|200
|1
|199
|0.571
|199.429
|-
|-
!Total
!2800
!8
!2792
!8
!2792
|}

==Comparison of p-Values==

The following table compares the p-values reported in Table 2 of Blount et al. to the chi-square p-values for the same experiments. For experiments one and three, the chi-square p-values are much larger than the "mean generation" test p-values from the paper.

{|class="wikitable" style="text-align:center"
|-
|
!Experiment 1
!Experiment 2
!Experiment 3
|-
!p-Value from Paper
|0.0085
|0.0007
|0.082
|-
!Chi-square p-value
|0.19
|0.0004
|0.22
|}

The chi-square test p-values are computed by comparing the test statistic to the chi-square distribution. It is generally assumed that the cell frequencies should be greater than five so that the statistic's distribution follows chi-square distribution. However, there is no consensus about what minimum cell frequency is necessary or how many expected values need to cross that threshold.

==References==
<references/>
[[Category:Statistics]]

==See Also==
http://www.sciencenews.org/index/feature/activity/view/id/40006/title/Molecular_Evolution

http://sciencenews.org/view/generic/id/40649/title/FOR_KIDS_Hitting_the_redo_button_on_evolution

[[Flaws_in_Richard_Lenski_Study|Flaws_in_Richard_Lenski_Study]]

Talk:Significance of E. Coli Evolution Experiments

2009-03-28T12:58:02Z

SJohnson: /* Caveat */

SJohnson, your assessment, while good in the utilization of the chi-squared test is unfortunately incorrect. The Monte Carlo resampling gives a more accurate p-value than the chi-squared. You may research the literature (i.e. publications in statistical mathematics, many pubs actualy compare Monte Carlo vs Chi Squared) to discover that this method is commonly used in advance statistical work and how it is more accurate than the chi-squared test.--[[User:Able806|Able806]] 17:00, 4 March 2009 (EST)

:It doesn’t make sense to compare the chi-square test, which is a specific statistical hypothesis test, to Monte Carlo methods, which can be used for anything from fluid motion modeling to p-value computations. You can use Monte Carlo methods to compute the p-values of the chi-square test!

:Monte Carlo methods involve the generation of random realizations. Your broad claim the Monte Carlo methods are “more accurate” than the chi-square test is obviously incorrect because the accuracy of Monte Carlo methods always depends on the number of random realizations generated. When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous.

:Which publications compare Monte Carlo to chi-square and show that the former is more accurate? Could you provide specific examples? Thanks. [[User:SJohnson|SJohnson]] 18:50, 4 March 2009 (EST)

:In furtherance of SJohnson's remarks with respect to rarely occurring events, the use of the basic Monte Carlo method is plainly incorrect for modeling a rarely occurring event, as the Lenski paper did. This has long been pointed out in [[Flaws in Richard Lenski Study]]. I know [[evolutionists]] will never admit a flaw in anything promoting their pet theory, but this (and other) flaws in that paper is undeniable.

:Watch how evolutionists defended obvious errors in the Lenski paper, and then realize why the [[Piltdown Man]] fraud was taught for 40 years without evolutionists admitting it was a hoax.--[[User:Aschlafly|Andy Schlafly]] 09:55, 5 March 2009 (EST)

::Andy, how exactly is the Monte Carlo method incorrect to use in this case? I have seen it used in publications with much smaller datasets.--[[User:Able806|Able806]] 10:29, 5 March 2009 (EST)

:::Able806, I'm interested in looking at the publications you mentioned that use Monte Carlo methods to analyze small data sets. Could you provide some examples? Thanks. [[User:SJohnson|SJohnson]] 16:41, 5 March 2009 (EST)

::::SJohnson, here are two papers, [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6WH8-45RFJ1J-19&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=1ad95954654bb97b17e474ce6b469f6e 1] and [http://cat.inist.fr/?aModele=afficheN&cpsidt=787963 2]. Most are in chemistry and genetics where you find the observed to be much smaller and have to use the MCM. You can search on the subject as well and find that how Lenski performed the test is the standard for microbiological genetic analysis.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

:::::Those papers have nothing to do with hypothesis testing. One is an archeology paper. To be blunt, it seems like you’re just doing internet searches on “Monte Carlo” to find these links. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

::::::SJohnson, actually they do, did you read the papers? If so you would see how they used the MCM for their data analysis of small data sets, which indeed was hypothesis testing and answers you inquiry about publications that use MCM for small data set analysis. If you wish I can try to track down some actual mathematical publications, however, I am not as familiar with mathematical journals as I am with science/medical journals (not knowing which mathematical journals are acceptable). I am assuming that you have a background in math and possibly access to mathematical journals, therefore if you know the reputable ones I can do the leg work.
::::::I believe the thing that needs to be looked at is there truly a problem with the choice of test and if so what is an alternative. Bayesian might be an option but seems to be difficult to employ for this situation.--[[User:Able806|Able806]] 12:36, 12 March 2009 (EDT)

:::Able806, you still seem to miss the point about how inappropriate the Monte Carlo method (as used in the Lenski paper) is for evaluating rarely occurring events. You need to open your mind to be productive. If you simply cling to a view that Lenski (who I don't think has any meaningful education in statistics) must somehow be right, then you're not going to make any progress in understanding the flaws.--[[User:Aschlafly|Andy Schlafly]] 17:07, 5 March 2009 (EST)

::::Andy, you still have not answered what you find inappropriate about his use of the Monte Carlo method? I am a reasonable person and with evidence I do have an open mind. I provided examples last week, with a working model, showing that Monte Carlo is better than the chi-square in this case. I have also shown where the Chi-Square was inappropriate due to the occurrence size as well. So if you have any evidence that Monte Carlo should not be used in the way that Lenski used please let it be shown.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

Sjohnson, I believe you just proved my point. In the literature of mean and covariance structure analysis, non-central chi-square distribution is commonly used to describe the behavior of the likelihood ratio statistic under alternative hypothesis; it is widely believed that the non-central chi-square distribution is justified by statistical theory. Actually, when the null hypothesis is not trivially violated, the non-central chi-square distribution cannot describe the LR statistic well even when data are normally distributed and the sample size is large. Monte Carlo results compare the strength of the normal distribution against that of the non-central chi-square distribution. In an association analysis comparing cases and controls with respect to allele frequencies at a highly polymorphic locus, a potential problem is that the conventional chi-squared test may not be valid for a large, sparse contingency table. Reliance on statistics with known asymptotic distribution is unnecessary, as Monte Carlo simulations can be performed to estimate the significance level of the test statistic.

Here is a [http://faculty.vassar.edu/lowry/chi_beta.html link] to a great page the provides an interactive example as to why the Chi Squared test would provide poor results compared to the Monte Carlo in relation to the Lenski data workup.

Something you may have overlooked was that the data set is actually too small to use the chi square method correctly. It is often accepted that is any of the analyzed data falls under 10 for a particular cell of the data set then the Yates correction needs to be applied; unfortunately the Yates correction can over correct thus skewing the p-value. Lenksi seemed to understand this by supporting his Monte Carlo p-value results with the Fisher z-transformation p-value.

I hope this helps.--[[User:Able806|Able806]] 10:27, 5 March 2009 (EST)

:I’m still waiting to hear which literature says that “Monte Carlo resampling” is “more accurate than the chi-squared test”. The page mentioned above [http://faculty.vassar.edu/lowry/chi_beta.html] is a discussion of why statisticians “fail to reject the null” rather than “accepting the null” when the p-value is above 0.05 or so. The page says nothing about superiority of Monte Carlo methods. Why were alternate hypothesis distributions mentioned? Only the null hypothesis distribution is used to calculate a p-value. Yates’s correction is for 2x2 contingency tables [http://en.wikipedia.org/wiki/Yates%27_correction_for_continuity]. It doesn’t apply in this case. Finally, what the heck do “covariance structure analysis” and “allele frequencies at a highly polymorphic locus” have to do with this problem? [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::SJohnson, I am looking for this paper for you, I cited it for one of my past publications dealing with allele frequencies (I believe it came from the Duke Biostatistics group). To answer your question about allele frequencies, that is the issue at hand, more about the genetics than the math, but it is the item being studied. So you stated that Yates can not be used and statistics says the number of occurrences is too small to evaluate using the Chi-Squared test so what would you recommend instead of the Monte-Carlo Method?

:Regarding the "Fisher z-transformation p-value" from the paper, garbage in garbage out. If the p-values were bad to begin with, then why would a combination of them be meaningful? [[User:SJohnson|SJohnson]] 10:49, 9 March 2009 (EDT)

::You are assuming that p-values are wrong based on a test that is inappropriate in this case due to data limitations. Did you perform a z-transformation on the chi-squared for the three data groups?--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

::You asked about the “Fisher z-transformation p-value”. The z-transformation test and Fisher’s method are actually two different things (see Whitlock's 2005 paper - Ref. 49 in Blount et al.). But no, I haven’t tried either. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::There's a large literature on various kinds of Monte Carlo test, a very short summary of which is that they're inevitably more accurate than parametric tests (e.g. F, t, chi-squared, etc) because they don't make assumptions about the distribution of the data under the null hypothesis. See for example ''Introduction to the Bootstrap'' by B. Efron and R. Tibshirani and ''The Jack-knife, the Bootstrap and Other Resampling Plans'', also by Efron. They're certainly applicable to small datasets and their accuracy is really only limited by the number of samples you care to take. E.g. 1000 M-C samples would give you a pretty accurate idea about significance at the alpha<1% level (That book should answer SJohnson's questions of 18:50 on 4/3/09 and 16:38 on 5/3/09 about accuracy and Aschalfly's comment of 17:07 on 5/3/09 about appropriateness of Monte Carlo tests.) [[User:FredFerguson|FredFerguson]] 16:53, 11 March 2009 (EDT)

::::Your claim that Monte Carlo methods are “inevitably more accurate” than other tests is obviously wrong because the accuracy of MC methods always depends on the number of realizations used. You should have written <math>\alpha=1\%</math>, not <math>\alpha<1\%</math>. If 1,000 random realizations are generated, the number of realizations above the true <math>\alpha=1\%</math> level is binomial with mean 10 and variance about 10. Thus, the standard deviation of the MC estimate is >0.003. In this example, a Monte Carlo p-value could be off by 30% and still be within a standard deviation. Is that really “pretty accurate”?

::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 3%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::::You're talking about miniscule differences in the accuracy of a test. 0.013 isn't very different from 0.007. In either case, it's very unlikely the experimenter would have obtained that result if the null hypothesis were true. If you're bothered about differences in P-values to the third decimals (which would make you unusual!), just run more MC realisations, that's all. Not really a problem. [[User:FredFerguson|FredFerguson]] 11:53, 12 March 2009 (EDT)

There’s still confusion about the difference between test statistics and Monte Carlo methods. Before you find a Monte Carlo estimate of a p-value, you need to select a test statistic to reduce the data set to a scalar. I am interested in hearing which test statistic you believe should be used in place of the chi-square test and why. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

----

Quick question for SJohnson: How many degrees of freedom did you choose when calculating the p-value? I'd like to know upon what condition you base that number. Thanks.--[[User:Argon|Argon]] 11:05, 5 March 2009 (EST)

:The degree of freedom for a contingency table is rows minus one times columns minus one. That is, <math> (r-1)(c-1) </math>. Here’s a pretty good tutorial I came across: [http://faculty.uncfsu.edu/dwallace/lesson%2020.pdf]. For the experiments from [http://www.pnas.org/content/105/23/7899.full.pdf], the DOFs are 11, 11, and 13. For experiment one, the chi-square test statistic is
::<math>
X^2
=\sum\limits_i\sum\limits_j
\frac{\left(n_{i,j}-E\left[n_{i,j}\right]\right)^2}
{E\left[n_{i,j}\right]}
</math>
::<math>
=\frac{\left(0-1/3\right)^2}{1/3}
+\frac{\left(6-17/3\right)^2}{17/3}
+\frac{\left(0-1/3\right)^2}{1/3}
+\ldots+
</math>
::<math>
+\frac{\left(2-1/3\right)^2}{1/3}
+\frac{\left(4-17/3\right)^2}{17/3}
</math>
::<math>
\approx
14.82
</math>
:where <math>n_{i,j}</math> is the observed value and <math>E\left[n_{i,j}\right]</math> is the expected null hypothesis value. So if you have MS Excel, another way to arrive at the p-value of 0.19 is to type “=CHIDIST(14.82,11)” into a cell. Cheers! [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::OK, thanks for the info. From what I'd calculated and looked up in tables, the numbers seemed close to a df=11 for a chi-square of ~14. (Aside: With terms having 17/3 in the denominator in the figures above, were you using the test of independence? I was using Pearson's test for [http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_for_fit_of_a_distribution fit of a distribution] which returns a chi-squared value of 14 and roughly matched the p-values you reported, assuming the df was 11).

::Also, the first sentence of the article reads: "Blount, Borland, and Lenski[1] claimed that a key evolutionary innovation was observed during a laboratory experiment. That claim is false." A small correction: There were several claims in the paper. The 'key evolutionary innovation' was acquiring the ability to utilize citrate as a food source. That claim was demonstrated multiple times. The claim, which pertains to this statistics discussion was that the Cit+ phenotype arose in a multi-step process, first requiring a rare, pre-adaptive mutation before additional mutation(s) lead to the subsequent development of citrate utilization.--[[User:Argon|Argon]] 20:46, 5 March 2009 (EST)

:::My biology-degreed wife assures me that mutation does not necessarily mean that evolution occurred. What the paper claimed is that evolution (a “key innovation”) occurred in the lab. The key innovation supposedly increased the mutation rate. In the experiments, the observed mutation rate increased after generation 31,000, but not enough to make a statistically significant claim that the rate is not constant. The analysis in the paper was similar to flipping a coin ten times, counting six heads and claiming that the coin must be biased against tails. In reality, there’s nothing surprising about a fair coin producing slightly more of one outcome than the other. Just like there's nothing surprising about there being slightly more mutations in later generations than early generations given the null hypothesis (constant mutation rate). [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

>>Inserting a later comment first<<
SJohnson, the paper's title is: "Historical contingency '''and the evolution of a key innovation''' in an experimental population of ''Escherichia coli''" As I mentioned earlier, the key innovation is the evolution of the Cit+ phenotype and not the timing or rate of its acquisition. And yes, it *is* evolution (call it microevolution, if you wish). Blount et al went on further to speculate how this evolutionary innovation arose and they proposed the historical contingency hypothesis in which 'pre-adaptive' mutations were required before the Cit+ phenotype developed. It is only this latter hypothesis that you are attempting to address with your chi-square analysis, not the fact that Cit+ mutants arose (which is the evolutionary innovation).--[[User:Argon|Argon]] 21:57, 18 March 2009 (EDT)

::::SJohnson, not to say anything about your wife, but has she had a 400 level molecular genetics course (most general biology degrees do not cover the detail unless they are specialized)? If so, she would have mentioned that if the mutation passes to the offspring and is selectively beneficial to the population then it is a step of evolution as along as the conditions continue through the sharing of the mutation with the population and the environment is such that reduces the growth rate of the non-transformed population. While not all mutations are signs that evolution occurred the mutations that pass to offspring and provide a benefit compared to other offspring are very strong indicators. In the case of this paper the population that evolved the cit+ was able to metabolize a chemical in their environment which allowed for an adaptation advantage compared to the non-transformed colonies.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

----

Let’s go back to the beginning. There appears to be confusion about the difference between test statistics and methods for computing p-values. As is noted at the beginning of the page [http://www.conservapedia.com/Significance_of_E._Coli_Evolution_Experiments], the fundamental problem with the paper is that it used a flawed test statistic, not that it used Monte Carlo methods to find the p-value for that flawed statistic.

Every hypothesis test uses a test statistic to reduce the data to a single number. The p-value for the test statistic can be calculated analytically (as I’ve done for the chi-square test statistic) or by Monte Carlo methods. In the paper, Monte Carlo methods were used to compute the p-value of the “mutation generation” test statistic. The key problem with the analysis from the paper is that it doesn’t work to use a weighted average to test for variations in mutation rate. This is like trying to use the sample variance to test for an increase in the mean in Gaussian-distributed data. A statistic should be selected based on the null and alternate hypothesis distributions of the data. The chi-square test (unlike the weighted average from the paper) is a reasonable choice for data that mutates at a constant rate under the null hypothesis, but mutates at varying rates under the alternate hypothesis.

Able806, you made a good point about the contingency table cell frequencies being relatively low, but were wrong when you said ”the data set is actually too small to use the chi square method correctly”. In the low cell frequency case the chi-square test is still effective, but the null hypothesis distribution of the chi-square statistic starts to look less like the chi-square distribution. Thus, p-values calculated using the chi-square distribution may be a bit off. However, Monte Carlo p-values are always imperfect as well because it's impossible to generate an infinite number of random realizations. There are imperfections in p-values generated by analytic and Monte Carlo methods. However, low cell frequencies does not explain the >20x and >2.5x differences between chi-square p-values and p-values from the paper for experiments one and three. The reason for those huge differences was the use of the flawed test statistic (“mutation generation”) in the paper. [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

:SJohnson, the chi-squared test is a valuable statistical tool, but the limitations of the test must be acknowledged. The chi-squared test can only produce valid results if the assumptions that underly the test are not violated. As an analogy, Newtonian models of motion fail to produce accurate results as velocities approach the speed of light; under those circumstances one must switch to a theory that accounts for relativistic effects.

:It seems that you have simply dismissed the [http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm widely-acknowledged] [http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm fact] that the [http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html chi-squared test] is [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236 inappropriate] for use in [http://www.graphpad.com/www/Book/Choose.htm situations] where n in any cell is [http://mysite.du.edu/~jcalvert/econ/chisquar.htm less] less than a [http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1 threshold] [http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html number]. Different authors set different thresholds, but all are well above the numbers seen in your chi-squared analysis - even the most liberal guidelines advise against the chi-squared test when any expected cell frequency is less than one or more than 20% of the table cells are less than 5; others require that expected values in all cells must be more than 5. With smaller amounts of data, the test is insensitive and errs on the side of rejecting the hypothesis. If you attempt your chi-squared statistical analysis with a program that is more sophisticated than MS Excel (as I did), you get an error message indicating that the results are invalid due to low expected cell counts.

:That issue aside, there are other reasons that the chi-squared test is inappropriate here. As the links above point out, the categories tested must be truly independent; one example is that you can't use the chi-squared test to compare age and ability to kick a field goal by testing the same experimental group twice, one year apart; you have to test one group of age A and a different group of age B. In the case of the Blount paper, the categories are not independent. Even if there were adequate numbers to address the low-expected-frequency problem, this would make the chi-squared an invalid test in this case.

:There are other significant problems with the use of the chi-squared test in this circumstance, but they can wait until you address these first major problems.--[[User:ElyM|ElyM]] 12:18, 11 March 2009 (EDT)

::Wackerly et al. says in general it’s assumed that the cell frequencies are above five so that the chi-square statistic (under the null) is approximately chi-square distributed (see p. 703). That book does not say chi-square test results are invalid if frequencies are five or less. Your example of a chi-square test warning message (it said "warning" not "error" as you stated) in Minitab [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236] said “approximation probably invalid” referring to the chi-square distribution approximation to the chi-square test statistic’s distribution. Your example did not say “chi-square test invalid”. I agree that when cell frequencies are low, the chi-square test statistic’s distribution starts to deviate from the chi-square distribution. I maintain that this deviation is not enough to explain the >2.5x and >20x differences in the chi-square test p-values and the p-values from the paper.

::As the numerous links in your post proved, the chi-square test is widely-used by statisticians. Can you give examples of statisticians using mean mutation generation as a test statistic? Also, did your software agree with the chi-square test p-values I presented? Thanks. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::Thank you for giving page references for Wackerly; however it seems we have different editions, since page 703 in my copy (5th ed, 1996) does not deal with chi-squared issues at all. My copy does state the following, on page 622: "Although the mathematical proof is beyond the scope of this text, it can be shown that, when n is large [chi-squared] will possess approximately a chi-square probability distribution in repeated sampling." Then, on page 624: "Experience has shown that cell counts [n sub i] should not be too small in order that the chi-square distribution provide an accurate approximation to the distribution of [chi squared]. As a rule of thumb we require that all expected cell counts equal or exceed 5, although Cochran (1952) has noted that this value can be as low as 1 for some situations." Wackerly then goes on, in the problems sections, to describe the use of the chi-squared test as a "violation of good statistical practice" when "some expected counts [are] <5."

:::It seems that you are already aware that the [chi-square] statistic under the null is no longer chi-square distributed for small n; this is precisely why the test should not be used under those conditions. I can claim to be able to accelerate a 1-kg mass to 10 times the speed of light by applying 1 N of force for 95 years by using F=ma and t= (vf-vi)/a. Plugging the numbers into those equations will produce the same result every time, but the answer is illegitimate because those equations are only valid under certain assumptions, which are violated as velocities approach the speed of light. Similarly, having a statistical program calculate a chi-squared value given the Blount data will produce a number result, but since the assumptions of the test are violated the result is not legitimate. Yes, if I put the Blount data in SAS 9.2, I get the same numerical answer as you do, but I also get the following message: "WARNING: >89% of the cells have expected counts less than 5. Chi-square may not be a valid test." You may argue that that's a warning, not an error; that's a semantic distinction. The reason that the program says that it MAY not be valid is that the chi-squared test skews in the direction of being too conservative at low n values; the test has an acceptable rate of false positives but an unacceptably high rate of false negatives. Comparing the results of the Monte Carlo and chi-squared results in this case is like comparing the results of Newtonian and relativistic equations of motion: they can produce very different results from the same input data.

::::For a finite amount of data, the chi-square statistic is never chi-square distributed under the null. The p-values are always approximate regardless of cell frequencies. The approximation becomes more accurate as the amount of data increases, but I don’t believe that this inaccuracy will change p-values that are about 0.2 (for experiments 1 and 3) into statistically significant p-values. How much do you expect the p-values to change if an exact computation is used in place of the chi-square distribution approximation? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::Your last paragraph has a major non sequitur in it: yes, many statisticians use the chi-square test. As long as the assumptions of the test are not violated, it is a valuable tool. That has nothing to do with the validity of using mean mutation generation as a test statistic. 'Mean number of werewolf attacks in Mumbai in the week centered on the new moon, by month, from 1654 to 1798' is a valid test statistic. I am quite sure that it has never been used in a peer-reviewed paper before. That does not mean that I can't perform valid statistical tests on that statistic. If, however, the incorrect test is applied, the results of the analysis will be flawed. Papers apply a (relatively small) standard repertoire of valid tests to a (potentially infinite) number of test statistics. The particular test statistic used in a paper may never have been used before and may never be used again; that does not address the validity of the analysis. In Blount's case, the test is the Monte Carlo analysis, which is also "widely-used by statisticians".

::::There are an infinite number of ways to reduce a data set to a single number. However, it’s foolish to think every method would be effective. I gave an example of a flawed test statistic in an earlier post [http://www.conservapedia.com/index.php?title=Talk%3ASignificance_of_E._Coli_Evolution_Experiments&diff=635070&oldid=634987]. Another example of a flawed test statistic is the one used in the paper because it does not always detect deviations from the null hypothesis (see: [[Significance of E. Coli Evolution Experiments#Test Statistics]]).

::::Test statistics are typically derived. The likelihood ratio test is a common method used to derive them. The chi-square test for independence is an approximation to the LRT. Where is the derivation saying that mean mutation generation is an appropriate test statistic for this problem? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::We still haven't touched on the issue of the categories not being independent, which by itself is sufficient to invalidate the chi-squared technique. I'm new to this site, so I'm unsure as to the etiquette of making changes to the articles of another person - but the article here should at the very least mention that the chi-square test is being used here in a manner that violates its underlying assumptions in at least two fundamental ways, and the results are therefore suspect.--[[User:ElyM|ElyM]] 17:34, 12 March 2009 (EDT)

:::::When generating random realizations of experiment outcomes, the authors assumed that the total number of mutants was fixed. Thus the paper assumed the numbers of mutants per generation are statistically dependent. Does this seem like a realistic model, or do you think that if the experiments were recreated that the total number of mutants could vary? For example, if experiment one were recreated, would the total number of mutants always be exactly four? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::: It looks to me as though SJohnson has misinterpreted the application of the chi-squared test in quite a fundamental way. His/her analysis of Blount's data are therefore close to meaningless, regardless of whether the test used by Blount is appropriate or not. In my opinion, the entire page should therefore be deleted. [[User:FredFerguson|FredFerguson]] 08:18, 13 March 2009 (EDT)

::::: "Fred", perhaps you mistakenly think this is Wikipedia, where [[censorship]] and deletion of pages for ideological reasons are common. Not here.--[[User:Aschlafly|Andy Schlafly]] 10:23, 14 March 2009 (EDT)

:::::: Umm... I'm suggesting deletion for mathematical reasons, not ideological reasons. Using an argument filled with mathematical errors to try to support your case only detracts from your credibility. [[User:FredFerguson|FredFerguson]] 10:38, 14 March 2009 (EDT)

:::::: Actually, I think correction is better than deletion. So that's what I've done. [[User:FredFerguson|FredFerguson]] 11:01, 14 March 2009 (EDT)

::::::: I find no credibility in your denial of having ideological reasons.--[[User:Aschlafly|Andy Schlafly]] 11:04, 14 March 2009 (EDT)

== Misinterpretation of test ==

SJohnson, Your analysis misinterprets the test. You say the null hypothesis is that this mutation cannot happen. They saw a mutation (4 mutations, in fact, in the data set you show) so the null hypothesis (as you state is) is disproved. That's perfectly straightforward.

I don't know what the "mean mutation generation" test is but you're doing when you apply a chi-squared test to this dataset is to test if the mutations are evenly distributed throughout the generations. Your test says they are, so there's no strong evidence to suppose that mutations are likely to occur in one generation rather than another in the series of tests. Blount's test says thay aren't, so it's more likely that the mutation will occur later in the series of tests. I can't tell which test is right without knowing more about the test that Blount used.

But that point (the foregoing paragraph) has no bearing at all on the null hypothesis, as you describe it. The mutation appeared, so that means the hypothesis that the mutation can't happen is disproved. Very simple. [[User:FredFerguson|FredFerguson]] 21:10, 8 March 2009 (EDT)

:I never said that “the null hypothesis is that this mutation cannot happen”. The chi-square test statistic I'm using wouldn’t be defined if the null hypothesis mutation rate was zero because the <math>E\left[n_{i,j}\right]</math> term in the denominator of the statistic (see above equation) would be zero.

:The test statistic from the paper is the average of the generation numbers of observed mutations. For experiment one this number is
::<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)= 31750.
</math>
:The same number is shown in Table 2 of the paper. [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

:: SJohnson, the way you're calculating the chi-squared statistic implies that you're testing the null hypothesis of a constant mutation rate over time against an alternative hypothesis of a mutation rate which varies over time. [[User:FredFerguson|FredFerguson]] 11:02, 9 March 2009 (EDT)

As it currently stands, the article makes the following statement: "The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown." This misstates the null hypothesis of the paper, which is elaborated in the Introduction section of the paper, and repeated in the section '''Statistical Analysis of the Replay Experiments''':
<blockquote>
For each experiment, we compared the observed mean generation of those clones that yielded Cit+ variants to the mean expected under the null hypothesis that clones from all generations have equal likelihood. The null thus corresponds to the rare-mutation hypothesis laid out in the Introduction."
Block quote
</blockquote><ref>www.pnas.org/cgi/reprint/105/23/7899.pdf</ref>

The article also continues to describe 'mean mutation generation' as a ''test'' rather than a ''statistic'' to which the ''Monte Carlo test'' was applied.--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

I'm a bit confused about why the Chi-squared test, which we're told compares the results to a null hypothesis of a constant mutation rate, seems insensitive to which generations the Cit+ mutations are found. Instead, the chi-square test seems only to be evaluating whether the frequencies of Cit+ mutations in any particular generation are 'expected'. Thus the test is asking whether finding a distribution (e.g. in the first experiment) across nine periods that have no mutations, two periods that have one mutation and one period with two mutations is a statistically significant deviation from what you'd expect of the mutations were randomly distributed. The number returned from the function is the same regardless of the order of Cit+ results. The number of mutations per bin is not the only question being asked. Instead it's the '''order and temporal distribution''' of Cit+ mutants that the analyses probably need to confront. It's not whether one can get nine no-mutants, two single mutants and one double-mutant result, it's a matter of '''when''' they occur and whether that distribution affects the significance of the results. Blount's hypothesis is that mutations should appear later in the experiment. When formulating a suitable null hypothesis, wouldn't one want to take the timing of Cit+ mutants into consideration too?--[[User:Argon|Argon]] 22:25, 18 March 2009 (EDT)

==Unreferenced Claims==

I deleted the claim that mean mutation generation is an appropriate test statistic because no reference was produced that back that claim. No reference was provided to back the claim that the chi-square test p-values are always conservative, either. [[User:SJohnson|SJohnson]] 12:57, 14 March 2009 (EDT)

: The reference is Everitt. I'll check I put it in the right place. [[User:FredFerguson|FredFerguson]] 13:30, 14 March 2009 (EDT)

There was a typo in my edit summaries on the talk page and the main page. I meant to say "Removed unsupported claims" rather than "Removed supported claims". [[User:SJohnson|SJohnson]] 13:13, 14 March 2009 (EDT)

I do not believe that anyone has claimed that 'chi-square test p-values are ''always'' conservative'. The claim that has been made is that ''under certain circumstances'', namely low n and low individual cell values, the chi-square test is an invalid test; that under those circumstances the power of the test is low and it becomes impossible to reject the null hypothesis even when it is false. You may have missed the pertinent sections in my links above, so I will directly quote the relevant sections. All the quoted sections refer to chi-square testing in particular. Any bolding below is mine.

:This edit claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639379]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

<blockquote>
"Assumptions: 

Even though a nonparametric statistic does not require a normally distributed population, there still are some restrictions regarding its use. 

1. Representative sample (Random) 
2. The data must be in frequency form (nominal data) or greater. 
3. The individual observations must be independent of each other. 
4. '''Sample size must be adequate. In a 2 x 2 table, Chi Square should not be used if n is less than 20. In a larger table, no expected value should be less than 1, and not more than 20% of the variables can have expected values of less than 5'''. 
5. Distribution basis must be decided on before the data is collected. 
6. The sum of the observed frequencies must equal the sum of the expected frequencies."
</blockquote>
<ref>http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm
</ref>

<blockquote>
"Assumptions:
* Random sample data are assumed. As with all significance tests, if you have population data, then any table differences are real and therefore significant. If you have non-random sample data, significance cannot be established, though significance tests are nonetheless sometimes utilized as crude "rules of thumb" anyway.
* A sufficiently large sample size is assumed, as in all significance tests. '''Applying chi-square to small samples exposes the researcher to an unacceptable rate of Type II errors. There is no accepted cutoff. Some set the minimum sample size at 50, while others would allow as few as 20'''. Note chi-square must be calculated on actual count data, not substituting percentages, which would have the effect of pretending the sample size is 100.
* '''Adequate cell sizes are also assumed. Some require 5 or more, some require more than 5, and others require 10 or more. A common rule is 5 or more in all cells of a 2-by-2 table, and 5 or more in 80% of cells in larger tables, but no cells with zero count'''. When this assumption is not met, Yates' correction is applied.
* Independence. Observations must be independent. The same observation can only appear in one cell. '''This means chi-square cannot be used to test correlated data (ex., before-after, matched pairs, panel data)'''.
* Similar distribution. Observations must have the same underlying distribution.
* Known distribution. The hypothesized distribution is specified in advance, so that the number of observations that are expected to appear each cell in the table can be calculated without reference to the observed values. Normally this expected value is the crossproduct of the row and column marginals divided by the sample size.
* Non-directional hypotheses are assumed. Chi-square tests the hypothesis that two variables are related only by chance. If a significant relationship is found, this is not equivalent to establishing the researcher's hypothesis that A causes B, or that B causes A.
* Finite values. Observations must be grouped in categories.
* Normal distribution of deviations (observed minus expected values) is assumed. Note chi-square is a nonparametric test in the sense that is does not assume the parameter of normal distribution for the data -- only for the deviations.
* Data level. No assumption is made about level of data. Nominal, ordinal, or interval data may be used with chi-square tests."
</blockquote>
<ref>http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm</ref>

<blockquote>
"Assumptions: 
-None of the expected values may be less than 1 
-No more than 20% of the expected values may be less than 5"</blockquote> <ref>http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html</ref>

<blockquote>
"When performing a chi-square test, your data must satisfy important assumptions. Although these assumptions may be stated differently in different textbooks, they generally assert that: 
1)The sample must be randomly drawn from the population 
'''2)The sample size, n, must be large enough so that the expected cell count in each cell is greater than or equal to 5.''' 
Both assumptions must be met in the process of collecting your data, and violations of the second assumption will appear in the Minitab output when you run the analysis. 
... 
'''You may wonder why the second assumption is necessary for performing the chi-square test. The second assumption arises because the distribution of counts under the null hypothesis is multinomial, and the normal distribution can be used to approximate the multinomial distribution if the sample size is sufficiently large and the probability parameters aren't too small. It can be shown via the Central Limit Theorem that the multinomial distribution converges to the normal distribution as the sample size approaches infinity; however, there is no easy way to show mathematically how and when the convergence fails.'''"
</blockquote>
<ref>http://www.minitab.com/support/docs/Answers/Chi-Square%20Test%20Assumptions.pdf</ref>

<blockquote>
"'''The chi-square test is simpler to calculate but yields only an approximate P value. ... You should definitely avoid the chi-square test when the numbers in the contingency table are very small (any number less than about six)'''."
</blockquote> <ref>http://www.graphpad.com/www/Book/Choose.htm</ref>

<blockquote>
"The most important things to remember to get a valid χ2 test are that the expected values are not too small in any bin (certainly 5 or more), and that the degrees of freedom are properly evaluated. '''Unless you have a very large amount of data, the test is not very sensitive and errs on the side of safety. If you get a significant result, however, it is not likely to be wrong.'''"
</blockquote> <ref>http://mysite.du.edu/~jcalvert/econ/chisquar.htm</ref>

<blockquote>
"The critical assumptions of the chi-square test for k independent samples are similar to those for the chi-square test for two independent samples. ... 
'''4. No more than 20% of the cells may have expected frequencies of less than 5, and no cell should have an expected frequency of less than 1. 
The rule given in Assumption 4 is particularly important for a contingency table that is larger than 2X2'''"
</blockquote> <ref>http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1
</ref>

<blockquote>
"Special problems with small expected cell frequencies for the chi-square test: 
The chi-square test involves using the chi-square distribution to approximate the underlying exact distribution. The approximation becomes better as the expected cell frequencies grow larger, and '''may be inappropriate for tables with very small expected cell frequencies.''' 
'''For tables with expected cell frequencies less than 5, the chi-square approximation may not be reliable. A standard (and conservative) rule of thumb (due to Cochran) is to avoid using the chi-square test for tables with expected cell frequencies less than 1, or when more than 20% of the table cells have expected cell frequencies less than 5.''' 
Another rule of thumb (due to Roscoe and Byars) is that the average expected cell frequency should be at least 1 when the expected cell frequencies are close to equal, and 2 when they are not. (If the chosen significance level is 0.01 instead of 0.05, then double these numbers.) 
Koehler and Larntz suggest that if the total number of observations is at least 10, the number categories is at least 3, and the square of the total number of observations is at least 10 times the number of categories, then the chi-square approximation should be reasonable. 
Care should be taken when cell categories are combined (collapsed together) to fix problems of small expected cell frequencies. Collapsing can destroy evidence of non-independence, so a failure to reject the null hypothesis for the collapsed table does not rule out the possibility of non-independence in the original table. 
'''As with most statistical tests, the power of the chi-square test increases with a larger number of observations. If there are too few observations, it may be impossible to reject the null hypothesis even if it is false.'''"
</blockquote> <ref>http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html</ref>--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

: Thanks for this really excellent contribution, ElyM. The only thing I'd like to add is in relation to your initial statement, "I do not believe that anyone has claimed that 'chi-square test p-values are always conservative'". The question of whether a test is conservative in a particular situation is probabilistic. One can determine whether a test is likely to generate a p-value which is too high in a particular situation (e.g. for a chi-squared test, when there are lots of small expected values) but one needs an exact test (such as an appropriate Monte Carlo randomisation test) to determine whether the p-value in any ''particular'' test is in fact excessively high.

::This edit also claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639373]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

: I hope careful reading of your very clear description will put SJohnson's mind at rest on this subject. [[User:FredFerguson|FredFerguson]] 18:11, 14 March 2009 (EDT)

ElyM, you've provided nothing to address the basic flaw that "The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events." See [[Flaws in Lenski Study]]. Also, do not impose your view on the content page until after SJohnson has had an opportunity to respond to your posting. As to "Fred", his put-downs are getting tiresome and I'm going to review his edit pattern now to see if he's been contributing anything of value to this site.--[[User:Aschlafly|Andy Schlafly]] 14:06, 15 March 2009 (EDT)

::Mr. Schlafly, per your request I have not added anything to the content page as SJohnson has not yet responded to my posts. Since all of my comments have been in regards to SJohnson's use of the chi-square test in this particular article, I'm not sure why you expect me to address Blout's use of Monte Carlo - that issue seems to be addressed on the [[Flaws in Lenski Study]] page. SJohnson has added a reformulation of the chi-square test for two possible outcomes, and stated that the chi-square test is at a minimum when all success probabilities are equal. He then extrapolates from this to claim that the chi-square test is an effective test for the data from Blount.

::The reformulation of the equations for two possible outcomes does not address the underlying problem that the chi-square test has universally accepted parameters outside of which it is considered an invalid test; I have provided references for these parameters and shown that the data from Blount lies outside them. None of the expected cells in SJohnson's analysis have values above one, and the total n is four. SJohnson's own reference states that the application of the chi-square test in this circumstance is a "violation of good statistical practice". Analogously, combining F=ma and t=(vf-vi)/a into t=(vf-vi)m/F and showing that t is a minimum when m approaches zero does not address the fact that those Newtonian equations do not apply as velocities approach the speed of light. The legitimacy of Blount's arguments cannot be determined by the application of illegitimate counterarguments. If SJohnson or others can point to references from the statistical literature that show that Blount has made methodological errors - as I have been able to do with SJohnson's chi-square analysis - I would welcome their input, and no doubt Conservapedia's other readers would as well, and this page would be greatly improved.

::I have not seen a rebuttal from SJohnson in the four days since my last post, although he has added new material to the content page since then. In light of this, I would appreciate some guidelines as to when it is appropriate for me to add my information and references to the content page. I can add citations from the primary mathematical literature if necessary, but in general I find that these are less helpful as they are not easily accessible by readers without access to academic libraries.--[[User:ElyM|ElyM]] 18:07, 18 March 2009 (EDT)

::: You say, "I'm not sure why you expect me to address Blout's use of Monte Carlo." The reason is obvious: the title of the content page is the "Significance of E. Coli Evolution Experiments." You haven't addressed the inappropriateness of using Monte Carlo simulations for assessing the significance rarely occurring events, which was central to Lenski's statistical claims. I suggest you address this flaw if you want to be taken seriously.--[[User:Aschlafly|Andy Schlafly]] 23:12, 18 March 2009 (EDT)

::::ASchlafly, again per your request, and based on your statement regarding the "inappropriateness of using Monte Carlo simulations for assessing the significance of rarely occurring events", I have spent the last several days reviewing the literature available to me on Monte Carlo and other resampling techniques, looking for ways in which Blount may have made a methodological error of the sort that SJohnson has made. I have been unable to find any examples of authors suggesting that Monte Carlo be avoided for low ''n'', or for events with low probability regardless of'' n'', much less providing specific cutoff numbers as are seen in the references that I provided for the chi-square test. Similarly, the technique that Blount used does not require/assume that categories are unrelated, as the chi-square test does. Of course, the absence of evidence is not evidence of absence, and I may have misinterpreted the basis of your objection. At this point I'll need you to explain your objection in more detail if you wish me to find the appropriate literature addressing your concerns. Do you believe that the number of resamplings was too low in Blount's paper? That the analysis should have been performed with a software package other than Statistics101? Some other procedural issue? Some issue of interpretation?

::::The statistical problem that Blount must address is straightforward: given a distribution of mutant cultures that ''appears'' to be skewed toward the higher generations, what is the probability that this same amount of skew (or a greater degree) could arise by chance, given the null hypothesis that every generation is equally likely to produce a mutant? Interestingly, in the case of the first replay experiment, the total number of ways to randomly select (equal probability, no replacement) four cultures from seventy-two is 72x71x70x69, or 24,690,960. This number is small enough that a program can brute-force-calculate the 'mean generation number' of ''all possible'' combinations of four cultures in a reasonable amount of time. An experimentally-derived 'mean generation number' can be checked against this exhaustive list, and the number of means equal to or larger than the experimental mean can be found exactly. Converting this number to a percentage of 24,690,960 provides an exact p-value for any given experimental 'mean generation number'. This exhaustive approach is different than the Monte Carlo technique, in that ''all possible'' outcomes are examined, rather than a ''random subset'' of all possible outcomes. For the first replay experiment, it provides a way to independently check Blount's Monte Carlo results. This approach is not possible for the second and third replay experiments, in which the total number of possible combinations becomes impractically large: 340!/335! = 4.41 x10^12 and 2800!/2792! = 3.74 x 10^27, respectively.

::::I asked a colleague to run just such a brute-force program for me on the first replay data. I also ran several Monte Carlo simulations ('''not''' using Statsistics101) with Blount's data, using twenty-five million, one hundred million, and 493,819,200 resamplings - note that this last is twenty times the number of all possible combinations of 4 samples drawn without replacement from 72. The p-values from the 25M, 100M, and 493M Monte Carlo resamplings (0.00844, 0.00846, and 0.00846, respectively) compare favorably with Blount's 1M value of 0.0085 and the non-Monte-Carlo brute-force exact calculation, which provides a p-value of 0.008457. Thus it appears that Blount's statistical results are confirmed by a ''non-Monte Carlo'' technique, at least for the first replay experiment.

:::::What do you get for the experiment two p-value using the method from the paper and at least ten million realizations? [[User:SJohnson|SJohnson]] 08:48, 25 March 2009 (EDT)

::::::For the second replay experiment, Blount reports that one million resamplings gives a p-value of 0.0007. When I run the Monte Carlo simulations, ten million resamplings give a p-value of 0.00060; one hundred million resamplings give a p-value of 0.00062, and one ''billion'' resamplings give a p of 0.00061.

:::::::I got 0.0006 using ten million realizations and the flawed test statistic. The paper had 0.0007. The authors obviously didn't use enough Monte Carlo realizations. I'm going to add this to the list of flaws in the paper. [http://www.conservapedia.com/Flaws_in_Richard_Lenski_Study] [[User:SJohnson|SJohnson]] 08:51, 26 March 2009 (EDT)

::::::As to the brute-force method for the second replay: the 4.41x10^12 combinations of five cultures picked from 340 actually represents 'only' 36.8 billion unique combinations, since for the purposes of calculating a mean generation value, the ordering of the cultures does not matter: 0, 0, 0, 0, 10 gives the same mean as 10, 0, 0, 0, 0 and 0, 10, 0, 0, 0. With brute force, it turns out that out of the 36,760,655,568 unique combinations possible in the second replay, 22,536,306 have means that are greater than or equal to 32,100.

::::::22,536,306 / 36,760,655,568 = 0.000613 = the ''exact'' p-value derived from exhaustive evaluation rather than Monte Carlo.

::::::The third replay has 9.27 x 10^22 unique combinations; at a billion comparisons a minute it would take over 170,000,000 years to check them all.--[[User:ElyM|ElyM]] 12:36, 25 March 2009 (EDT)

::::::----------
::::::Here are pointers to the freely available Statistics 101 package <ref>http://www.statistics101.net/statistics101web_000003.htm</ref> and the actual programs run through the package by Blount ''et al.'' <ref>http://myxo.css.msu.edu/ecoli/citrate2008/MCprograms.html</ref>. The stats package is written in Java and should run under many operating systems. A 10 million trial run of the second experiment took a bit of time and yielded a p-value of 0.00061. Ten separate, one-million trial runs produced an average p-value of 0.00061 (std.dev=0.00002, n=10). Even with trial sizes of 5K, the numbers averaged about 0.0006 (std.dev=0.0004 n=10).--[[User:Argon|Argon]] 20:52, 25 March 2009 (EDT)

::::My intention is not to get caught up in a digression about Monte Carlo, though - I'd rather keep the focus on the fact that the main article should acknowledge that SJohnson is using chi-square in a way that violates accepted guidelines; this remains true whether Blount's analysis is valid or not.--[[User:ElyM|ElyM]] 12:13, 23 March 2009 (EDT)

==Caveat==
If SJohnson can provide citations to authors who support the use of chi-square where all expected cell counts are less than one, or where categories are not independent, I look forward to evaluating them.--[[User:ElyM|ElyM]] 12:35, 27 March 2009 (EDT)
:Wackerly et al. does not say to avoid the test because of low cell frequencies. You're still making a false claim that p-values are always high if cell frequencies are low. The last paragraph you added is just your opinions about the test being inappropriate. Modeling each trial as a statistically independent Bernoulli trial is reasonable. Thus, the chi-square test is appropriate. The assumption from the paper that the numbers of mutants per experiment would never change is an example of a bad way to model an experiment. Note that all p-values in Blount et al. were calculated under that unreasonable assumption. Do you think that if these experiments were recreated, that the total number of mutants would always be exactly the same? [[User:SJohnson|SJohnson]] 08:58, 28 March 2009 (EDT)

== References ==
{{reflist}}

Significance of E. Coli Evolution Experiments

2009-03-28T12:49:06Z

SJohnson: Undo revision 645164 by ElyM (Talk)

Blount, Borland, and [[Lenski]]<ref> http://www.pnas.org/content/105/23/7899.full.pdf</ref> claimed that a key evolutionary innovation was observed during a [[laboratory]] [[experiment]]. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.

==Test Statistics==

The test statistic used in Blout, Borland, and Lenski was mean mutation generation. For example, the experiment one mean mutation generation is (see Tables 1 and 2 of the paper)
:<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)
=31750.
</math>

The null hypothesis from the paper was constant mutation rate over all generations. However, measurements of mean mutation
generation can fail to observe deviations from that null
hypothesis. Consider an experiment where the mutation
probabilities for generations 1, 2, and 3 are <math>p_1</math>, <math>p_2</math>, and
<math>p_3</math>, respectively. If the mutation probabilities per generation
are <math>p_1=p_2=p_3=p</math>, then the mean mutation generation is 2.
However, if the mutation probabilities are <math>p_1=p_3=p/2</math> and
<math>p_2=2p</math>, then the mean mutation generation is still 2. In the
latter example the experiment has deviated from the null
hypothesis, but the mean mutation
generation is insensitive to the
change. Because this test can fail to observe deviations from the null hypothesis, levels of statistical significance computed using this test (p-values) are meaningless.

If the trials from an experiment have two outcomes (e.g. success or failure), then the chi-square test for independence can be written
:<math>
X^2
=\sum\limits_{i=1}^M
\frac{\left(x_i-\hat{p}N_i\right)^2}
{\hat{p}(1-\hat{p})N_i}
</math>
where <math>x_i</math> is the number of successes (e.g. mutations) in the ''i''-th experiment, <math>N_i</math> is the number of trials in the ''i''-th experiment, and
:<math>
\hat{p}=\frac{\sum\limits_{i=1}^Mx_i}{\sum\limits_{i=1}^MN_i}
</math>
is the fraction of trials that are successful in the all experiments. The chi-square test is, on average, at a minimum when all success probabilities are equal (<math>p_i=p</math> for <math>i=1,2,\ldots,M</math>) and will, on average, increase whenever the data follows any other hypothesis (<math>\sum\limits_{i=1}^M
\left(p_i-\bar{p}\right)^2>0</math> where <math>\bar{p}</math> is the mean of the success probabilities). Thus the chi-square test is an effective hypothesis test for the data and hypotheses from Blount et al.

==Experiment One Data==

The data from experiment one of the paper is shown below (see Table 1 of the paper). The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown.

{|class="wikitable" style="text-align:center"
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|6
|0
|6
|0.333
|5.667
|-
|10000
|6
|0
|6
|0.333
|5.667
|-
|20000
|6
|0
|6
|0.333
|5.667
|-
|25000
|6
|0
|6
|0.333
|5.667
|-
|27500
|6
|0
|6
|0.333
|5.667
|-
|29000
|6
|0
|6
|0.333
|5.667
|-
|30000
|6
|0
|6
|0.333
|5.667
|-
|30500
|6
|1
|5
|0.333
|5.667
|-
|31000
|6
|0
|6
|0.333
|5.667
|-
|31500
|6
|1
|5
|0.333
|5.667
|-
|32000
|6
|0
|6
|0.333
|5.667
|-
|32500
|6
|2
|4
|0.333
|5.667
|-
|-
!Total
!72
!4
!68
!4
!68
|}

When the flawed test is used to compute the significance of this data, the p-value is 0.0085 (see Table 2 of the paper). This p-value is considered statistically significant. However, when the data is analyzed using a standard method (the chi-square test) the p-value is 0.19. This p-value is much larger than the one from the paper and indicates that there is no reason to reject the null hypothesis. The chi-square test p-value for experiment two is small (0.0004). However, experiment three is not statistically significant because its p-value is 0.22.

The chi-square test is a common statistical method.<ref>''Mathematical Statistics with Applications'' by Wackerly, Mendenhall, and Scheaffer, Section 14.4.</ref> It can be implemented in Microsoft Excel. If the numbers from the last four columns of the experiment one data table (excluding the “totals” row) are entered into Excel in rows 1-12 and columns A-D, then the p-value can be computed by entering “=CHITEST(A1:B12,C1:D12)” into any empty cell of the spreadsheet.

==Experiment Three Data==

The experiment three data from Blount et al. is shown in the table below. The expected numbers of mutants under the null hypothesis (constant mutation rate) is also shown.

{|class="wikitable" style="text-align:center"
|-
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|200
|0
|200
|0.571
|199.429
|-
|10000
|200
|0
|200
|0.571
|199.429
|-
|20000
|200
|0
|200
|0.571
|199.429
|-
|25000
|200
|0
|200
|0.571
|199.429
|-
|27500
|200
|2
|198
|0.571
|199.429
|-
|29000
|200
|0
|200
|0.571
|199.429
|-
|30000
|200
|2
|198
|0.571
|199.429
|-
|30500
|200
|0
|200
|0.571
|199.429
|-
|31000
|200
|0
|200
|0.571
|199.429
|-
|31500
|200
|0
|200
|0.571
|199.429
|-
|32000
|200
|1
|199
|0.571
|199.429
|-
|32500
|200
|1
|199
|0.571
|199.429
|-
|-
!Total
!2800
!8
!2792
!8
!2792
|}

==Comparison of p-Values==

The following table compares the p-values reported in Table 2 of Blount et al. to the chi-square p-values for the same experiments. For experiments one and three, the chi-square p-values are much larger than the "mean generation" test p-values from the paper.

{|class="wikitable" style="text-align:center"
|-
|
!Experiment 1
!Experiment 2
!Experiment 3
|-
!p-Value from Paper
|0.0085
|0.0007
|0.082
|-
!Chi-square p-value
|0.19
|0.0004
|0.22
|}

The chi-square test p-values are computed by comparing the test statistic to the chi-square distribution. It is generally assumed that the cell frequencies should be greater than five so that the statistic's distribution follows chi-square distribution. However, there is no consensus about what minimum cell frequency is necessary or how many expected values need to cross that threshold.

==References==
<references/>
[[Category:Statistics]]

==See Also==
http://www.sciencenews.org/index/feature/activity/view/id/40006/title/Molecular_Evolution

http://sciencenews.org/view/generic/id/40649/title/FOR_KIDS_Hitting_the_redo_button_on_evolution

http://www.conservapedia.com/index.php?title=Flaws_in_Richard_Lenski_Study&action=submit

Flaws in Richard Lenski Study

2009-03-26T13:00:13Z

SJohnson: Incorrect p-values in paper

[[Richard Lenski]] rejected a request to release his bacteria mutation data to the public,<ref>See [[Conservapedia:Lenski dialog]].</ref> but the following serious flaws are emerging about his work<ref>Blount et al., "Historical contingency and the [[evolution]] of a key innovation in an experimental population of ''Escherichia coli'', 105 PNAS 7899-7906 (June 10, 2008).</ref> even without a full disclosure of the data. Note that the peer review on Lenski's paper took somewhere between 0 (non-existent) and at most 14 days (including administrative time), and Lenski himself does not have any obvious expertise in statistics. In fact, Richard Lenski admits in his paper that he based his statistical conclusions on use of a website called "statistics101".

1. Lenski's "historical contingency" hypothesis, as specifically depicted in Figure 3, is contradicted by the data presented in the Third Experiment in Table 1 of his paper. Figure 3 proposes a step-up in mutation rate to Cit+ due to a historical contingency (potentiating mutation) occurring at about the 31,000th generation, yet the Third (and largest) Experiment in Table 1 shows Cit+ arising just as often before the 31,000th generation as after. The abstract, in further contradiction with Figure 3, suggests that the historical contingency (potentiating mutation) occurred prior to the 20,000th generation.

2. Lenski's two alternative hypotheses suggest a fixed mutation rate, but the failure of the mutations in his experiments to increase based on scale (number of samples) tends to disprove both of Lenski's alternative hypotheses. Yet Lenski's paper fails to address adequately this obvious flaw in the paper.

3. Richard Lenski incorrectly included generations of the ''E. coli'' already known to contain Cit+ variants in his experiments.<ref>Richard Lenski incorrectly included generations 31,500, 32,000 and 32,500.</ref> Once these generations are removed from the analysis, the data disprove Lenski's hypothesis.

4. The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events. The Third Experiment results are consistent with the null hypothesis, contrary to the paper's claim.

5. Lenski's largest experiment (Third Experiment) failed to support his hypothesis with statistical significance. Even though this largest experiment was nearly ten times the size of his other experiments, Richard Lenski did not weight this largest experiment correctly in combining his results.

6. It was error to include generations of the E. coli already known to contain trace Cit+ variants. The highly improbable occurrence of four Cit+ variants from the 32,000th generation in the Second Experiment suggests an origin from undetected, pre-existing Cit+ variants.

7. The Third Experiment was erroneously combined with the other two experiments based on outcome rather than sample size, thereby yielding a false claim of overall statistical significance.

8. Lenski's paper is not clear in explaining how the results of his largest experiment (Third Experiment) failed to confirm his hypothesis with statistical significance, even with the incorrect inclusion of the Cit+ variant generations. Instead, his paper refers to his largest experiment as "marginally ... significant," which serves to obscure its statistical insignificance. Other works published in PNAS are clear in defining statistical significance in the traditional way, which Lenski's Third Experiment (even with incorrect inclusion of the above-referenced generations) failed to satisfy.<ref>See, e.g., [http://www.pnas.org/cgi/content/full/0701990104 Cholera toxin induces malignant glioma cell differentiation]</ref>

9. The long lag time (over 12,000 generations) between the historical contingency (potentiating mutation) in the largest experiment disproves Lenski's implicit assumption that the potentiating mutation likely occurred in proximity with the occurrence of the Cit+ variant, and that the first occurrence of the Cit+ variant in the Third Experiment at the 20,000th generation somehow implies that a potentiating mutation occurred in its proximity.

10. Lenski's paper claims that "During [30,000 generations], each population experienced billions of mutations,<ref>Lenski cites one of his own prior articles for this.</ref> far more than the number of possible point mutations in the [approximately] 4.6-million-bp genome. This ratio implies, to a first approximation, that each population tried every typical one-step mutation many times." Lenski's conclusion is nonsensical because it assumes that the mutations are completely random '''and''' that each mutation has a roughly equal probability.

11. In Table 2 of [http://www.pnas.org/content/105/23/7899.full.pdf], the expected mean should be 26,382 generations, not 28,382.

12. The p-value computed for experiment two was incorrectly listed as 0.0007 instead of 0.0006 in [http://www.pnas.org/content/105/23/7899.full.pdf]. These p-values are meaningless because the paper used a flawed test statistic (see: [[Significance of E. Coli Evolution Experiments#Test Statistics]]). However, the error illustrates the need to use enough random realizations when using Monte Carlo methods to measure p-values.

== References ==

<references/>

== See also ==

*[[Letter to PNAS]]
*[[Significance of E. Coli Evolution Experiments]]
[[Category:Science]]

Significance of E. Coli Evolution Experiments

2009-03-26T12:54:04Z

SJohnson:

Blount, Borland, and [[Lenski]]<ref> http://www.pnas.org/content/105/23/7899.full.pdf</ref> claimed that a key evolutionary innovation was observed during a [[laboratory]] [[experiment]]. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.

==Test Statistics==

The test statistic used in Blout, Borland, and Lenski was mean mutation generation. For example, the experiment one mean mutation generation is (see Tables 1 and 2 of the paper)
:<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)
=31750.
</math>

The null hypothesis from the paper was constant mutation rate over all generations. However, measurements of mean mutation
generation can fail to observe deviations from that null
hypothesis. Consider an experiment where the mutation
probabilities for generations 1, 2, and 3 are <math>p_1</math>, <math>p_2</math>, and
<math>p_3</math>, respectively. If the mutation probabilities per generation
are <math>p_1=p_2=p_3=p</math>, then the mean mutation generation is 2.
However, if the mutation probabilities are <math>p_1=p_3=p/2</math> and
<math>p_2=2p</math>, then the mean mutation generation is still 2. In the
latter example the experiment has deviated from the null
hypothesis, but the mean mutation
generation is insensitive to the
change. Because this test can fail to observe deviations from the null hypothesis, levels of statistical significance computed using this test (p-values) are meaningless.

If the trials from an experiment have two outcomes (e.g. success or failure), then the chi-square test for independence can be written
:<math>
X^2
=\sum\limits_{i=1}^M
\frac{\left(x_i-\hat{p}N_i\right)^2}
{\hat{p}(1-\hat{p})N_i}
</math>
where <math>x_i</math> is the number of successes (e.g. mutations) in the ''i''-th experiment, <math>N_i</math> is the number of trials in the ''i''-th experiment, and
:<math>
\hat{p}=\frac{\sum\limits_{i=1}^Mx_i}{\sum\limits_{i=1}^MN_i}
</math>
is the fraction of trials that are successful in the all experiments. The chi-square test is, on average, at a minimum when all success probabilities are equal (<math>p_i=p</math> for <math>i=1,2,\ldots,M</math>) and will, on average, increase whenever the data follows any other hypothesis (<math>\sum\limits_{i=1}^M
\left(p_i-\bar{p}\right)^2>0</math> where <math>\bar{p}</math> is the mean of the success probabilities). Thus the chi-square test is an effective hypothesis test for the data and hypotheses from Blount et al.

==Experiment One Data==

The data from experiment one of the paper is shown below (see Table 1 of the paper). The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown.

{|class="wikitable" style="text-align:center"
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|6
|0
|6
|0.333
|5.667
|-
|10000
|6
|0
|6
|0.333
|5.667
|-
|20000
|6
|0
|6
|0.333
|5.667
|-
|25000
|6
|0
|6
|0.333
|5.667
|-
|27500
|6
|0
|6
|0.333
|5.667
|-
|29000
|6
|0
|6
|0.333
|5.667
|-
|30000
|6
|0
|6
|0.333
|5.667
|-
|30500
|6
|1
|5
|0.333
|5.667
|-
|31000
|6
|0
|6
|0.333
|5.667
|-
|31500
|6
|1
|5
|0.333
|5.667
|-
|32000
|6
|0
|6
|0.333
|5.667
|-
|32500
|6
|2
|4
|0.333
|5.667
|-
|-
!Total
!72
!4
!68
!4
!68
|}

When the flawed test is used to compute the significance of this data, the p-value is 0.0085 (see Table 2 of the paper). This p-value is considered statistically significant. However, when the data is analyzed using a standard method (the chi-square test) the p-value is 0.19. This p-value is much larger than the one from the paper and indicates that there is no reason to reject the null hypothesis. The chi-square test p-value for experiment two is small (0.0004). However, experiment three is not statistically significant because its p-value is 0.22.

The chi-square test is a common statistical method.<ref>''Mathematical Statistics with Applications'' by Wackerly, Mendenhall, and Scheaffer, Section 14.4.</ref> It can be implemented in Microsoft Excel. If the numbers from the last four columns of the experiment one data table (excluding the “totals” row) are entered into Excel in rows 1-12 and columns A-D, then the p-value can be computed by entering “=CHITEST(A1:B12,C1:D12)” into any empty cell of the spreadsheet.

==Experiment Three Data==

The experiment three data from Blount et al. is shown in the table below. The expected numbers of mutants under the null hypothesis (constant mutation rate) is also shown.

{|class="wikitable" style="text-align:center"
|-
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|200
|0
|200
|0.571
|199.429
|-
|10000
|200
|0
|200
|0.571
|199.429
|-
|20000
|200
|0
|200
|0.571
|199.429
|-
|25000
|200
|0
|200
|0.571
|199.429
|-
|27500
|200
|2
|198
|0.571
|199.429
|-
|29000
|200
|0
|200
|0.571
|199.429
|-
|30000
|200
|2
|198
|0.571
|199.429
|-
|30500
|200
|0
|200
|0.571
|199.429
|-
|31000
|200
|0
|200
|0.571
|199.429
|-
|31500
|200
|0
|200
|0.571
|199.429
|-
|32000
|200
|1
|199
|0.571
|199.429
|-
|32500
|200
|1
|199
|0.571
|199.429
|-
|-
!Total
!2800
!8
!2792
!8
!2792
|}

==Comparison of p-Values==

The following table compares the p-values reported in Table 2 of Blount et al. to the chi-square p-values for the same experiments. For experiments one and three, the chi-square p-values are much larger than the "mean generation" test p-values from the paper.

{|class="wikitable" style="text-align:center"
|-
|
!Experiment 1
!Experiment 2
!Experiment 3
|-
!p-Value from Paper
|0.0085
|0.0007
|0.082
|-
!Chi-square p-value
|0.19
|0.0004
|0.22
|}

The chi-square test p-values are computed by comparing the test statistic to the chi-square distribution. It is generally assumed that the cell frequencies should be greater than five so that the statistic's distribution follows chi-square distribution. However, there is no consensus about what minimum cell frequency is necessary or how many expected values need to cross that threshold.

==References==
<references/>
[[Category:Statistics]]

==See Also==
http://www.sciencenews.org/index/feature/activity/view/id/40006/title/Molecular_Evolution

http://sciencenews.org/view/generic/id/40649/title/FOR_KIDS_Hitting_the_redo_button_on_evolution

http://www.conservapedia.com/index.php?title=Flaws_in_Richard_Lenski_Study&action=submit

Talk:Significance of E. Coli Evolution Experiments

2009-03-26T12:51:18Z

SJohnson: P-value in paper wrong for flawed test statistic in experiment two

Significance of E. Coli Evolution Experiments

2009-03-26T12:41:09Z

SJohnson: Removing several incorrect claims

Blount, Borland, and [[Lenski]]<ref> http://www.pnas.org/content/105/23/7899.full.pdf</ref> claimed that a key evolutionary innovation was observed during a [[laboratory]] [[experiment]]. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.

==Test Statistics==

The test statistic used in Blout, Borland, and Lenski was mean mutation generation. For example, the experiment one mean mutation generation is (see Tables 1 and 2 of the paper)
:<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)
=31750.
</math>

The null hypothesis from the paper was constant mutation rate over all generations. However, measurements of mean mutation
generation can fail to observe deviations from that null
hypothesis. Consider an experiment where the mutation
probabilities for generations 1, 2, and 3 are <math>p_1</math>, <math>p_2</math>, and
<math>p_3</math>, respectively. If the mutation probabilities per generation
are <math>p_1=p_2=p_3=p</math>, then the mean mutation generation is 2.
However, if the mutation probabilities are <math>p_1=p_3=p/2</math> and
<math>p_2=2p</math>, then the mean mutation generation is still 2. In the
latter example the experiment has deviated from the null
hypothesis, but the mean mutation
generation is insensitive to the
change. Because this test can fail to observe deviations from the null hypothesis, levels of statistical significance computed using this test (p-values) are meaningless.

If the trials from an experiment have two outcomes (e.g. success or failure), then the chi-square test for independence can be written
:<math>
X^2
=\sum\limits_{i=1}^M
\frac{\left(x_i-\hat{p}N_i\right)^2}
{\hat{p}(1-\hat{p})N_i}
</math>
where <math>x_i</math> is the number of successes (e.g. mutations) in the ''i''-th experiment, <math>N_i</math> is the number of trials in the ''i''-th experiment, and
:<math>
\hat{p}=\frac{\sum\limits_{i=1}^Mx_i}{\sum\limits_{i=1}^MN_i}
</math>
is the fraction of trials that are successful in the all experiments. The chi-square test is, on average, at a minimum when all success probabilities are equal (<math>p_i=p</math> for <math>i=1,2,\ldots,M</math>) and will, on average, increase whenever the data follows any other hypothesis (<math>\sum\limits_{i=1}^M
\left(p_i-\bar{p}\right)^2>0</math> where <math>\bar{p}</math> is the mean of the success probabilities). Thus the chi-square test is an effective hypothesis test for the data and hypotheses from Blount et al.

==Experiment One Data==

The data from experiment one of the paper is shown below (see Table 1 of the paper). The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown.

{|class="wikitable" style="text-align:center"
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|6
|0
|6
|0.333
|5.667
|-
|10000
|6
|0
|6
|0.333
|5.667
|-
|20000
|6
|0
|6
|0.333
|5.667
|-
|25000
|6
|0
|6
|0.333
|5.667
|-
|27500
|6
|0
|6
|0.333
|5.667
|-
|29000
|6
|0
|6
|0.333
|5.667
|-
|30000
|6
|0
|6
|0.333
|5.667
|-
|30500
|6
|1
|5
|0.333
|5.667
|-
|31000
|6
|0
|6
|0.333
|5.667
|-
|31500
|6
|1
|5
|0.333
|5.667
|-
|32000
|6
|0
|6
|0.333
|5.667
|-
|32500
|6
|2
|4
|0.333
|5.667
|-
|-
!Total
!72
!4
!68
!4
!68
|}

When the flawed test is used to compute the significance of this data, the p-value is 0.0085 (see Table 2 of the paper). This p-value is considered statistically significant. However, when the data is analyzed using a standard method (the chi-square test) the p-value is 0.19. This p-value is much larger than the one from the paper and indicates that there is no reason to reject the null hypothesis. The chi-square test p-value for experiment two is small (0.0004). However, experiment three is not statistically significant because its p-value is 0.22.

The chi-square test is a common statistical method.<ref>''Mathematical Statistics with Applications'' by Wackerly, Mendenhall, and Scheaffer, Section 14.4.</ref> It can be implemented in Microsoft Excel. If the numbers from the last four columns of the experiment one data table (excluding the “totals” row) are entered into Excel in rows 1-12 and columns A-D, then the p-value can be computed by entering “=CHITEST(A1:B12,C1:D12)” into any empty cell of the spreadsheet.

==Experiment Three Data==

The experiment three data from Blount et al. is shown in the table below. The expected numbers of mutants under the null hypothesis (constant mutation rate) is also shown.

{|class="wikitable" style="text-align:center"
|-
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|200
|0
|200
|0.571
|199.429
|-
|10000
|200
|0
|200
|0.571
|199.429
|-
|20000
|200
|0
|200
|0.571
|199.429
|-
|25000
|200
|0
|200
|0.571
|199.429
|-
|27500
|200
|2
|198
|0.571
|199.429
|-
|29000
|200
|0
|200
|0.571
|199.429
|-
|30000
|200
|2
|198
|0.571
|199.429
|-
|30500
|200
|0
|200
|0.571
|199.429
|-
|31000
|200
|0
|200
|0.571
|199.429
|-
|31500
|200
|0
|200
|0.571
|199.429
|-
|32000
|200
|1
|199
|0.571
|199.429
|-
|32500
|200
|1
|199
|0.571
|199.429
|-
|-
!Total
!2800
!8
!2792
!8
!2792
|}

==Comparison of p-Values==

The following table compares the p-values reported in Table 2 of Blount et al. to the chi-square p-values for the same experiments. For experiments one and three, the chi-square p-values are much larger than the "mean generation" test p-values from the paper.

{|class="wikitable" style="text-align:center"
|-
|
!Experiment 1
!Experiment 2
!Experiment 3
|-
!p-Value from Paper
|0.0085
|0.0007
|0.082
|-
!Chi-square p-value
|0.19
|0.0004
|0.22
|}

The chi-square test p-values are computed by comparing the test statistic to the chi-square distribution. It is generally assumed that the cell frequencies should be greater than five so that the statistic's distribution follows chi-square distribution. However, there is no consensus about what minimum cell frequency is necessary or how many expected values need to cross that threshold.

==References==
<references/>
[[Category:Conservapedia Dealings with PNAS and Lenski]]
[[Category:Statistics]]

==See Also==
http://www.sciencenews.org/index/feature/activity/view/id/40006/title/Molecular_Evolution

http://sciencenews.org/view/generic/id/40649/title/FOR_KIDS_Hitting_the_redo_button_on_evolution

Talk:Significance of E. Coli Evolution Experiments

2009-03-25T12:48:33Z

SJohnson: Experiment two

SJohnson, your assessment, while good in the utilization of the chi-squared test is unfortunately incorrect. The Monte Carlo resampling gives a more accurate p-value than the chi-squared. You may research the literature (i.e. publications in statistical mathematics, many pubs actualy compare Monte Carlo vs Chi Squared) to discover that this method is commonly used in advance statistical work and how it is more accurate than the chi-squared test.--[[User:Able806|Able806]] 17:00, 4 March 2009 (EST)

:It doesn’t make sense to compare the chi-square test, which is a specific statistical hypothesis test, to Monte Carlo methods, which can be used for anything from fluid motion modeling to p-value computations. You can use Monte Carlo methods to compute the p-values of the chi-square test!

:Monte Carlo methods involve the generation of random realizations. Your broad claim the Monte Carlo methods are “more accurate” than the chi-square test is obviously incorrect because the accuracy of Monte Carlo methods always depends on the number of random realizations generated. When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous.

:Which publications compare Monte Carlo to chi-square and show that the former is more accurate? Could you provide specific examples? Thanks. [[User:SJohnson|SJohnson]] 18:50, 4 March 2009 (EST)

:In furtherance of SJohnson's remarks with respect to rarely occurring events, the use of the basic Monte Carlo method is plainly incorrect for modeling a rarely occurring event, as the Lenski paper did. This has long been pointed out in [[Flaws in Richard Lenski Study]]. I know [[evolutionists]] will never admit a flaw in anything promoting their pet theory, but this (and other) flaws in that paper is undeniable.

:Watch how evolutionists defended obvious errors in the Lenski paper, and then realize why the [[Piltdown Man]] fraud was taught for 40 years without evolutionists admitting it was a hoax.--[[User:Aschlafly|Andy Schlafly]] 09:55, 5 March 2009 (EST)

::Andy, how exactly is the Monte Carlo method incorrect to use in this case? I have seen it used in publications with much smaller datasets.--[[User:Able806|Able806]] 10:29, 5 March 2009 (EST)

:::Able806, I'm interested in looking at the publications you mentioned that use Monte Carlo methods to analyze small data sets. Could you provide some examples? Thanks. [[User:SJohnson|SJohnson]] 16:41, 5 March 2009 (EST)

::::SJohnson, here are two papers, [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6WH8-45RFJ1J-19&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=1ad95954654bb97b17e474ce6b469f6e 1] and [http://cat.inist.fr/?aModele=afficheN&cpsidt=787963 2]. Most are in chemistry and genetics where you find the observed to be much smaller and have to use the MCM. You can search on the subject as well and find that how Lenski performed the test is the standard for microbiological genetic analysis.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

:::::Those papers have nothing to do with hypothesis testing. One is an archeology paper. To be blunt, it seems like you’re just doing internet searches on “Monte Carlo” to find these links. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

::::::SJohnson, actually they do, did you read the papers? If so you would see how they used the MCM for their data analysis of small data sets, which indeed was hypothesis testing and answers you inquiry about publications that use MCM for small data set analysis. If you wish I can try to track down some actual mathematical publications, however, I am not as familiar with mathematical journals as I am with science/medical journals (not knowing which mathematical journals are acceptable). I am assuming that you have a background in math and possibly access to mathematical journals, therefore if you know the reputable ones I can do the leg work.
::::::I believe the thing that needs to be looked at is there truly a problem with the choice of test and if so what is an alternative. Bayesian might be an option but seems to be difficult to employ for this situation.--[[User:Able806|Able806]] 12:36, 12 March 2009 (EDT)

:::Able806, you still seem to miss the point about how inappropriate the Monte Carlo method (as used in the Lenski paper) is for evaluating rarely occurring events. You need to open your mind to be productive. If you simply cling to a view that Lenski (who I don't think has any meaningful education in statistics) must somehow be right, then you're not going to make any progress in understanding the flaws.--[[User:Aschlafly|Andy Schlafly]] 17:07, 5 March 2009 (EST)

::::Andy, you still have not answered what you find inappropriate about his use of the Monte Carlo method? I am a reasonable person and with evidence I do have an open mind. I provided examples last week, with a working model, showing that Monte Carlo is better than the chi-square in this case. I have also shown where the Chi-Square was inappropriate due to the occurrence size as well. So if you have any evidence that Monte Carlo should not be used in the way that Lenski used please let it be shown.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

Sjohnson, I believe you just proved my point. In the literature of mean and covariance structure analysis, non-central chi-square distribution is commonly used to describe the behavior of the likelihood ratio statistic under alternative hypothesis; it is widely believed that the non-central chi-square distribution is justified by statistical theory. Actually, when the null hypothesis is not trivially violated, the non-central chi-square distribution cannot describe the LR statistic well even when data are normally distributed and the sample size is large. Monte Carlo results compare the strength of the normal distribution against that of the non-central chi-square distribution. In an association analysis comparing cases and controls with respect to allele frequencies at a highly polymorphic locus, a potential problem is that the conventional chi-squared test may not be valid for a large, sparse contingency table. Reliance on statistics with known asymptotic distribution is unnecessary, as Monte Carlo simulations can be performed to estimate the significance level of the test statistic.

Here is a [http://faculty.vassar.edu/lowry/chi_beta.html link] to a great page the provides an interactive example as to why the Chi Squared test would provide poor results compared to the Monte Carlo in relation to the Lenski data workup.

Something you may have overlooked was that the data set is actually too small to use the chi square method correctly. It is often accepted that is any of the analyzed data falls under 10 for a particular cell of the data set then the Yates correction needs to be applied; unfortunately the Yates correction can over correct thus skewing the p-value. Lenksi seemed to understand this by supporting his Monte Carlo p-value results with the Fisher z-transformation p-value.

I hope this helps.--[[User:Able806|Able806]] 10:27, 5 March 2009 (EST)

:I’m still waiting to hear which literature says that “Monte Carlo resampling” is “more accurate than the chi-squared test”. The page mentioned above [http://faculty.vassar.edu/lowry/chi_beta.html] is a discussion of why statisticians “fail to reject the null” rather than “accepting the null” when the p-value is above 0.05 or so. The page says nothing about superiority of Monte Carlo methods. Why were alternate hypothesis distributions mentioned? Only the null hypothesis distribution is used to calculate a p-value. Yates’s correction is for 2x2 contingency tables [http://en.wikipedia.org/wiki/Yates%27_correction_for_continuity]. It doesn’t apply in this case. Finally, what the heck do “covariance structure analysis” and “allele frequencies at a highly polymorphic locus” have to do with this problem? [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::SJohnson, I am looking for this paper for you, I cited it for one of my past publications dealing with allele frequencies (I believe it came from the Duke Biostatistics group). To answer your question about allele frequencies, that is the issue at hand, more about the genetics than the math, but it is the item being studied. So you stated that Yates can not be used and statistics says the number of occurrences is too small to evaluate using the Chi-Squared test so what would you recommend instead of the Monte-Carlo Method?

:Regarding the "Fisher z-transformation p-value" from the paper, garbage in garbage out. If the p-values were bad to begin with, then why would a combination of them be meaningful? [[User:SJohnson|SJohnson]] 10:49, 9 March 2009 (EDT)

::You are assuming that p-values are wrong based on a test that is inappropriate in this case due to data limitations. Did you perform a z-transformation on the chi-squared for the three data groups?--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

::You asked about the “Fisher z-transformation p-value”. The z-transformation test and Fisher’s method are actually two different things (see Whitlock's 2005 paper - Ref. 49 in Blount et al.). But no, I haven’t tried either. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::There's a large literature on various kinds of Monte Carlo test, a very short summary of which is that they're inevitably more accurate than parametric tests (e.g. F, t, chi-squared, etc) because they don't make assumptions about the distribution of the data under the null hypothesis. See for example ''Introduction to the Bootstrap'' by B. Efron and R. Tibshirani and ''The Jack-knife, the Bootstrap and Other Resampling Plans'', also by Efron. They're certainly applicable to small datasets and their accuracy is really only limited by the number of samples you care to take. E.g. 1000 M-C samples would give you a pretty accurate idea about significance at the alpha<1% level (That book should answer SJohnson's questions of 18:50 on 4/3/09 and 16:38 on 5/3/09 about accuracy and Aschalfly's comment of 17:07 on 5/3/09 about appropriateness of Monte Carlo tests.) [[User:FredFerguson|FredFerguson]] 16:53, 11 March 2009 (EDT)

::::Your claim that Monte Carlo methods are “inevitably more accurate” than other tests is obviously wrong because the accuracy of MC methods always depends on the number of realizations used. You should have written <math>\alpha=1\%</math>, not <math>\alpha<1\%</math>. If 1,000 random realizations are generated, the number of realizations above the true <math>\alpha=1\%</math> level is binomial with mean 10 and variance about 10. Thus, the standard deviation of the MC estimate is >0.003. In this example, a Monte Carlo p-value could be off by 30% and still be within a standard deviation. Is that really “pretty accurate”?

::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 3%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::::You're talking about miniscule differences in the accuracy of a test. 0.013 isn't very different from 0.007. In either case, it's very unlikely the experimenter would have obtained that result if the null hypothesis were true. If you're bothered about differences in P-values to the third decimals (which would make you unusual!), just run more MC realisations, that's all. Not really a problem. [[User:FredFerguson|FredFerguson]] 11:53, 12 March 2009 (EDT)

There’s still confusion about the difference between test statistics and Monte Carlo methods. Before you find a Monte Carlo estimate of a p-value, you need to select a test statistic to reduce the data set to a scalar. I am interested in hearing which test statistic you believe should be used in place of the chi-square test and why. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

----

Quick question for SJohnson: How many degrees of freedom did you choose when calculating the p-value? I'd like to know upon what condition you base that number. Thanks.--[[User:Argon|Argon]] 11:05, 5 March 2009 (EST)

:The degree of freedom for a contingency table is rows minus one times columns minus one. That is, <math> (r-1)(c-1) </math>. Here’s a pretty good tutorial I came across: [http://faculty.uncfsu.edu/dwallace/lesson%2020.pdf]. For the experiments from [http://www.pnas.org/content/105/23/7899.full.pdf], the DOFs are 11, 11, and 13. For experiment one, the chi-square test statistic is
::<math>
X^2
=\sum\limits_i\sum\limits_j
\frac{\left(n_{i,j}-E\left[n_{i,j}\right]\right)^2}
{E\left[n_{i,j}\right]}
</math>
::<math>
=\frac{\left(0-1/3\right)^2}{1/3}
+\frac{\left(6-17/3\right)^2}{17/3}
+\frac{\left(0-1/3\right)^2}{1/3}
+\ldots+
</math>
::<math>
+\frac{\left(2-1/3\right)^2}{1/3}
+\frac{\left(4-17/3\right)^2}{17/3}
</math>
::<math>
\approx
14.82
</math>
:where <math>n_{i,j}</math> is the observed value and <math>E\left[n_{i,j}\right]</math> is the expected null hypothesis value. So if you have MS Excel, another way to arrive at the p-value of 0.19 is to type “=CHIDIST(14.82,11)” into a cell. Cheers! [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::OK, thanks for the info. From what I'd calculated and looked up in tables, the numbers seemed close to a df=11 for a chi-square of ~14. (Aside: With terms having 17/3 in the denominator in the figures above, were you using the test of independence? I was using Pearson's test for [http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_for_fit_of_a_distribution fit of a distribution] which returns a chi-squared value of 14 and roughly matched the p-values you reported, assuming the df was 11).

::Also, the first sentence of the article reads: "Blount, Borland, and Lenski[1] claimed that a key evolutionary innovation was observed during a laboratory experiment. That claim is false." A small correction: There were several claims in the paper. The 'key evolutionary innovation' was acquiring the ability to utilize citrate as a food source. That claim was demonstrated multiple times. The claim, which pertains to this statistics discussion was that the Cit+ phenotype arose in a multi-step process, first requiring a rare, pre-adaptive mutation before additional mutation(s) lead to the subsequent development of citrate utilization.--[[User:Argon|Argon]] 20:46, 5 March 2009 (EST)

:::My biology-degreed wife assures me that mutation does not necessarily mean that evolution occurred. What the paper claimed is that evolution (a “key innovation”) occurred in the lab. The key innovation supposedly increased the mutation rate. In the experiments, the observed mutation rate increased after generation 31,000, but not enough to make a statistically significant claim that the rate is not constant. The analysis in the paper was similar to flipping a coin ten times, counting six heads and claiming that the coin must be biased against tails. In reality, there’s nothing surprising about a fair coin producing slightly more of one outcome than the other. Just like there's nothing surprising about there being slightly more mutations in later generations than early generations given the null hypothesis (constant mutation rate). [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

>>Inserting a later comment first<<
SJohnson, the paper's title is: "Historical contingency '''and the evolution of a key innovation''' in an experimental population of ''Escherichia coli''" As I mentioned earlier, the key innovation is the evolution of the Cit+ phenotype and not the timing or rate of its acquisition. And yes, it *is* evolution (call it microevolution, if you wish). Blount et al went on further to speculate how this evolutionary innovation arose and they proposed the historical contingency hypothesis in which 'pre-adaptive' mutations were required before the Cit+ phenotype developed. It is only this latter hypothesis that you are attempting to address with your chi-square analysis, not the fact that Cit+ mutants arose (which is the evolutionary innovation).--[[User:Argon|Argon]] 21:57, 18 March 2009 (EDT)

::::SJohnson, not to say anything about your wife, but has she had a 400 level molecular genetics course (most general biology degrees do not cover the detail unless they are specialized)? If so, she would have mentioned that if the mutation passes to the offspring and is selectively beneficial to the population then it is a step of evolution as along as the conditions continue through the sharing of the mutation with the population and the environment is such that reduces the growth rate of the non-transformed population. While not all mutations are signs that evolution occurred the mutations that pass to offspring and provide a benefit compared to other offspring are very strong indicators. In the case of this paper the population that evolved the cit+ was able to metabolize a chemical in their environment which allowed for an adaptation advantage compared to the non-transformed colonies.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

----

Let’s go back to the beginning. There appears to be confusion about the difference between test statistics and methods for computing p-values. As is noted at the beginning of the page [http://www.conservapedia.com/Significance_of_E._Coli_Evolution_Experiments], the fundamental problem with the paper is that it used a flawed test statistic, not that it used Monte Carlo methods to find the p-value for that flawed statistic.

Every hypothesis test uses a test statistic to reduce the data to a single number. The p-value for the test statistic can be calculated analytically (as I’ve done for the chi-square test statistic) or by Monte Carlo methods. In the paper, Monte Carlo methods were used to compute the p-value of the “mutation generation” test statistic. The key problem with the analysis from the paper is that it doesn’t work to use a weighted average to test for variations in mutation rate. This is like trying to use the sample variance to test for an increase in the mean in Gaussian-distributed data. A statistic should be selected based on the null and alternate hypothesis distributions of the data. The chi-square test (unlike the weighted average from the paper) is a reasonable choice for data that mutates at a constant rate under the null hypothesis, but mutates at varying rates under the alternate hypothesis.

Able806, you made a good point about the contingency table cell frequencies being relatively low, but were wrong when you said ”the data set is actually too small to use the chi square method correctly”. In the low cell frequency case the chi-square test is still effective, but the null hypothesis distribution of the chi-square statistic starts to look less like the chi-square distribution. Thus, p-values calculated using the chi-square distribution may be a bit off. However, Monte Carlo p-values are always imperfect as well because it's impossible to generate an infinite number of random realizations. There are imperfections in p-values generated by analytic and Monte Carlo methods. However, low cell frequencies does not explain the >20x and >2.5x differences between chi-square p-values and p-values from the paper for experiments one and three. The reason for those huge differences was the use of the flawed test statistic (“mutation generation”) in the paper. [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

:SJohnson, the chi-squared test is a valuable statistical tool, but the limitations of the test must be acknowledged. The chi-squared test can only produce valid results if the assumptions that underly the test are not violated. As an analogy, Newtonian models of motion fail to produce accurate results as velocities approach the speed of light; under those circumstances one must switch to a theory that accounts for relativistic effects.

:It seems that you have simply dismissed the [http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm widely-acknowledged] [http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm fact] that the [http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html chi-squared test] is [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236 inappropriate] for use in [http://www.graphpad.com/www/Book/Choose.htm situations] where n in any cell is [http://mysite.du.edu/~jcalvert/econ/chisquar.htm less] less than a [http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1 threshold] [http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html number]. Different authors set different thresholds, but all are well above the numbers seen in your chi-squared analysis - even the most liberal guidelines advise against the chi-squared test when any expected cell frequency is less than one or more than 20% of the table cells are less than 5; others require that expected values in all cells must be more than 5. With smaller amounts of data, the test is insensitive and errs on the side of rejecting the hypothesis. If you attempt your chi-squared statistical analysis with a program that is more sophisticated than MS Excel (as I did), you get an error message indicating that the results are invalid due to low expected cell counts.

:That issue aside, there are other reasons that the chi-squared test is inappropriate here. As the links above point out, the categories tested must be truly independent; one example is that you can't use the chi-squared test to compare age and ability to kick a field goal by testing the same experimental group twice, one year apart; you have to test one group of age A and a different group of age B. In the case of the Blount paper, the categories are not independent. Even if there were adequate numbers to address the low-expected-frequency problem, this would make the chi-squared an invalid test in this case.

:There are other significant problems with the use of the chi-squared test in this circumstance, but they can wait until you address these first major problems.--[[User:ElyM|ElyM]] 12:18, 11 March 2009 (EDT)

::Wackerly et al. says in general it’s assumed that the cell frequencies are above five so that the chi-square statistic (under the null) is approximately chi-square distributed (see p. 703). That book does not say chi-square test results are invalid if frequencies are five or less. Your example of a chi-square test warning message (it said "warning" not "error" as you stated) in Minitab [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236] said “approximation probably invalid” referring to the chi-square distribution approximation to the chi-square test statistic’s distribution. Your example did not say “chi-square test invalid”. I agree that when cell frequencies are low, the chi-square test statistic’s distribution starts to deviate from the chi-square distribution. I maintain that this deviation is not enough to explain the >2.5x and >20x differences in the chi-square test p-values and the p-values from the paper.

::As the numerous links in your post proved, the chi-square test is widely-used by statisticians. Can you give examples of statisticians using mean mutation generation as a test statistic? Also, did your software agree with the chi-square test p-values I presented? Thanks. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::Thank you for giving page references for Wackerly; however it seems we have different editions, since page 703 in my copy (5th ed, 1996) does not deal with chi-squared issues at all. My copy does state the following, on page 622: "Although the mathematical proof is beyond the scope of this text, it can be shown that, when n is large [chi-squared] will possess approximately a chi-square probability distribution in repeated sampling." Then, on page 624: "Experience has shown that cell counts [n sub i] should not be too small in order that the chi-square distribution provide an accurate approximation to the distribution of [chi squared]. As a rule of thumb we require that all expected cell counts equal or exceed 5, although Cochran (1952) has noted that this value can be as low as 1 for some situations." Wackerly then goes on, in the problems sections, to describe the use of the chi-squared test as a "violation of good statistical practice" when "some expected counts [are] <5."

:::It seems that you are already aware that the [chi-square] statistic under the null is no longer chi-square distributed for small n; this is precisely why the test should not be used under those conditions. I can claim to be able to accelerate a 1-kg mass to 10 times the speed of light by applying 1 N of force for 95 years by using F=ma and t= (vf-vi)/a. Plugging the numbers into those equations will produce the same result every time, but the answer is illegitimate because those equations are only valid under certain assumptions, which are violated as velocities approach the speed of light. Similarly, having a statistical program calculate a chi-squared value given the Blount data will produce a number result, but since the assumptions of the test are violated the result is not legitimate. Yes, if I put the Blount data in SAS 9.2, I get the same numerical answer as you do, but I also get the following message: "WARNING: >89% of the cells have expected counts less than 5. Chi-square may not be a valid test." You may argue that that's a warning, not an error; that's a semantic distinction. The reason that the program says that it MAY not be valid is that the chi-squared test skews in the direction of being too conservative at low n values; the test has an acceptable rate of false positives but an unacceptably high rate of false negatives. Comparing the results of the Monte Carlo and chi-squared results in this case is like comparing the results of Newtonian and relativistic equations of motion: they can produce very different results from the same input data.

::::For a finite amount of data, the chi-square statistic is never chi-square distributed under the null. The p-values are always approximate regardless of cell frequencies. The approximation becomes more accurate as the amount of data increases, but I don’t believe that this inaccuracy will change p-values that are about 0.2 (for experiments 1 and 3) into statistically significant p-values. How much do you expect the p-values to change if an exact computation is used in place of the chi-square distribution approximation? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::Your last paragraph has a major non sequitur in it: yes, many statisticians use the chi-square test. As long as the assumptions of the test are not violated, it is a valuable tool. That has nothing to do with the validity of using mean mutation generation as a test statistic. 'Mean number of werewolf attacks in Mumbai in the week centered on the new moon, by month, from 1654 to 1798' is a valid test statistic. I am quite sure that it has never been used in a peer-reviewed paper before. That does not mean that I can't perform valid statistical tests on that statistic. If, however, the incorrect test is applied, the results of the analysis will be flawed. Papers apply a (relatively small) standard repertoire of valid tests to a (potentially infinite) number of test statistics. The particular test statistic used in a paper may never have been used before and may never be used again; that does not address the validity of the analysis. In Blount's case, the test is the Monte Carlo analysis, which is also "widely-used by statisticians".

::::There are an infinite number of ways to reduce a data set to a single number. However, it’s foolish to think every method would be effective. I gave an example of a flawed test statistic in an earlier post [http://www.conservapedia.com/index.php?title=Talk%3ASignificance_of_E._Coli_Evolution_Experiments&diff=635070&oldid=634987]. Another example of a flawed test statistic is the one used in the paper because it does not always detect deviations from the null hypothesis (see: [[Significance of E. Coli Evolution Experiments#Test Statistics]]).

::::Test statistics are typically derived. The likelihood ratio test is a common method used to derive them. The chi-square test for independence is an approximation to the LRT. Where is the derivation saying that mean mutation generation is an appropriate test statistic for this problem? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::We still haven't touched on the issue of the categories not being independent, which by itself is sufficient to invalidate the chi-squared technique. I'm new to this site, so I'm unsure as to the etiquette of making changes to the articles of another person - but the article here should at the very least mention that the chi-square test is being used here in a manner that violates its underlying assumptions in at least two fundamental ways, and the results are therefore suspect.--[[User:ElyM|ElyM]] 17:34, 12 March 2009 (EDT)

:::::When generating random realizations of experiment outcomes, the authors assumed that the total number of mutants was fixed. Thus the paper assumed the numbers of mutants per generation are statistically dependent. Does this seem like a realistic model, or do you think that if the experiments were recreated that the total number of mutants could vary? For example, if experiment one were recreated, would the total number of mutants always be exactly four? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::: It looks to me as though SJohnson has misinterpreted the application of the chi-squared test in quite a fundamental way. His/her analysis of Blount's data are therefore close to meaningless, regardless of whether the test used by Blount is appropriate or not. In my opinion, the entire page should therefore be deleted. [[User:FredFerguson|FredFerguson]] 08:18, 13 March 2009 (EDT)

::::: "Fred", perhaps you mistakenly think this is Wikipedia, where [[censorship]] and deletion of pages for ideological reasons are common. Not here.--[[User:Aschlafly|Andy Schlafly]] 10:23, 14 March 2009 (EDT)

:::::: Umm... I'm suggesting deletion for mathematical reasons, not ideological reasons. Using an argument filled with mathematical errors to try to support your case only detracts from your credibility. [[User:FredFerguson|FredFerguson]] 10:38, 14 March 2009 (EDT)

:::::: Actually, I think correction is better than deletion. So that's what I've done. [[User:FredFerguson|FredFerguson]] 11:01, 14 March 2009 (EDT)

::::::: I find no credibility in your denial of having ideological reasons.--[[User:Aschlafly|Andy Schlafly]] 11:04, 14 March 2009 (EDT)

== Misinterpretation of test ==

SJohnson, Your analysis misinterprets the test. You say the null hypothesis is that this mutation cannot happen. They saw a mutation (4 mutations, in fact, in the data set you show) so the null hypothesis (as you state is) is disproved. That's perfectly straightforward.

I don't know what the "mean mutation generation" test is but you're doing when you apply a chi-squared test to this dataset is to test if the mutations are evenly distributed throughout the generations. Your test says they are, so there's no strong evidence to suppose that mutations are likely to occur in one generation rather than another in the series of tests. Blount's test says thay aren't, so it's more likely that the mutation will occur later in the series of tests. I can't tell which test is right without knowing more about the test that Blount used.

But that point (the foregoing paragraph) has no bearing at all on the null hypothesis, as you describe it. The mutation appeared, so that means the hypothesis that the mutation can't happen is disproved. Very simple. [[User:FredFerguson|FredFerguson]] 21:10, 8 March 2009 (EDT)

:I never said that “the null hypothesis is that this mutation cannot happen”. The chi-square test statistic I'm using wouldn’t be defined if the null hypothesis mutation rate was zero because the <math>E\left[n_{i,j}\right]</math> term in the denominator of the statistic (see above equation) would be zero.

:The test statistic from the paper is the average of the generation numbers of observed mutations. For experiment one this number is
::<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)= 31750.
</math>
:The same number is shown in Table 2 of the paper. [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

:: SJohnson, the way you're calculating the chi-squared statistic implies that you're testing the null hypothesis of a constant mutation rate over time against an alternative hypothesis of a mutation rate which varies over time. [[User:FredFerguson|FredFerguson]] 11:02, 9 March 2009 (EDT)

As it currently stands, the article makes the following statement: "The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown." This misstates the null hypothesis of the paper, which is elaborated in the Introduction section of the paper, and repeated in the section '''Statistical Analysis of the Replay Experiments''':
<blockquote>
For each experiment, we compared the observed mean generation of those clones that yielded Cit+ variants to the mean expected under the null hypothesis that clones from all generations have equal likelihood. The null thus corresponds to the rare-mutation hypothesis laid out in the Introduction."
Block quote
</blockquote><ref>www.pnas.org/cgi/reprint/105/23/7899.pdf</ref>

The article also continues to describe 'mean mutation generation' as a ''test'' rather than a ''statistic'' to which the ''Monte Carlo test'' was applied.--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

I'm a bit confused about why the Chi-squared test, which we're told compares the results to a null hypothesis of a constant mutation rate, seems insensitive to which generations the Cit+ mutations are found. Instead, the chi-square test seems only to be evaluating whether the frequencies of Cit+ mutations in any particular generation are 'expected'. Thus the test is asking whether finding a distribution (e.g. in the first experiment) across nine periods that have no mutations, two periods that have one mutation and one period with two mutations is a statistically significant deviation from what you'd expect of the mutations were randomly distributed. The number returned from the function is the same regardless of the order of Cit+ results. The number of mutations per bin is not the only question being asked. Instead it's the '''order and temporal distribution''' of Cit+ mutants that the analyses probably need to confront. It's not whether one can get nine no-mutants, two single mutants and one double-mutant result, it's a matter of '''when''' they occur and whether that distribution affects the significance of the results. Blount's hypothesis is that mutations should appear later in the experiment. When formulating a suitable null hypothesis, wouldn't one want to take the timing of Cit+ mutants into consideration too?--[[User:Argon|Argon]] 22:25, 18 March 2009 (EDT)

==Unreferenced Claims==

I deleted the claim that mean mutation generation is an appropriate test statistic because no reference was produced that back that claim. No reference was provided to back the claim that the chi-square test p-values are always conservative, either. [[User:SJohnson|SJohnson]] 12:57, 14 March 2009 (EDT)

: The reference is Everitt. I'll check I put it in the right place. [[User:FredFerguson|FredFerguson]] 13:30, 14 March 2009 (EDT)

There was a typo in my edit summaries on the talk page and the main page. I meant to say "Removed unsupported claims" rather than "Removed supported claims". [[User:SJohnson|SJohnson]] 13:13, 14 March 2009 (EDT)

I do not believe that anyone has claimed that 'chi-square test p-values are ''always'' conservative'. The claim that has been made is that ''under certain circumstances'', namely low n and low individual cell values, the chi-square test is an invalid test; that under those circumstances the power of the test is low and it becomes impossible to reject the null hypothesis even when it is false. You may have missed the pertinent sections in my links above, so I will directly quote the relevant sections. All the quoted sections refer to chi-square testing in particular. Any bolding below is mine.

:This edit claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639379]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

<blockquote>
"Assumptions: 

Even though a nonparametric statistic does not require a normally distributed population, there still are some restrictions regarding its use. 

1. Representative sample (Random) 
2. The data must be in frequency form (nominal data) or greater. 
3. The individual observations must be independent of each other. 
4. '''Sample size must be adequate. In a 2 x 2 table, Chi Square should not be used if n is less than 20. In a larger table, no expected value should be less than 1, and not more than 20% of the variables can have expected values of less than 5'''. 
5. Distribution basis must be decided on before the data is collected. 
6. The sum of the observed frequencies must equal the sum of the expected frequencies."
</blockquote>
<ref>http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm
</ref>

<blockquote>
"Assumptions:
* Random sample data are assumed. As with all significance tests, if you have population data, then any table differences are real and therefore significant. If you have non-random sample data, significance cannot be established, though significance tests are nonetheless sometimes utilized as crude "rules of thumb" anyway.
* A sufficiently large sample size is assumed, as in all significance tests. '''Applying chi-square to small samples exposes the researcher to an unacceptable rate of Type II errors. There is no accepted cutoff. Some set the minimum sample size at 50, while others would allow as few as 20'''. Note chi-square must be calculated on actual count data, not substituting percentages, which would have the effect of pretending the sample size is 100.
* '''Adequate cell sizes are also assumed. Some require 5 or more, some require more than 5, and others require 10 or more. A common rule is 5 or more in all cells of a 2-by-2 table, and 5 or more in 80% of cells in larger tables, but no cells with zero count'''. When this assumption is not met, Yates' correction is applied.
* Independence. Observations must be independent. The same observation can only appear in one cell. '''This means chi-square cannot be used to test correlated data (ex., before-after, matched pairs, panel data)'''.
* Similar distribution. Observations must have the same underlying distribution.
* Known distribution. The hypothesized distribution is specified in advance, so that the number of observations that are expected to appear each cell in the table can be calculated without reference to the observed values. Normally this expected value is the crossproduct of the row and column marginals divided by the sample size.
* Non-directional hypotheses are assumed. Chi-square tests the hypothesis that two variables are related only by chance. If a significant relationship is found, this is not equivalent to establishing the researcher's hypothesis that A causes B, or that B causes A.
* Finite values. Observations must be grouped in categories.
* Normal distribution of deviations (observed minus expected values) is assumed. Note chi-square is a nonparametric test in the sense that is does not assume the parameter of normal distribution for the data -- only for the deviations.
* Data level. No assumption is made about level of data. Nominal, ordinal, or interval data may be used with chi-square tests."
</blockquote>
<ref>http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm</ref>

<blockquote>
"Assumptions: 
-None of the expected values may be less than 1 
-No more than 20% of the expected values may be less than 5"</blockquote> <ref>http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html</ref>

<blockquote>
"When performing a chi-square test, your data must satisfy important assumptions. Although these assumptions may be stated differently in different textbooks, they generally assert that: 
1)The sample must be randomly drawn from the population 
'''2)The sample size, n, must be large enough so that the expected cell count in each cell is greater than or equal to 5.''' 
Both assumptions must be met in the process of collecting your data, and violations of the second assumption will appear in the Minitab output when you run the analysis. 
... 
'''You may wonder why the second assumption is necessary for performing the chi-square test. The second assumption arises because the distribution of counts under the null hypothesis is multinomial, and the normal distribution can be used to approximate the multinomial distribution if the sample size is sufficiently large and the probability parameters aren't too small. It can be shown via the Central Limit Theorem that the multinomial distribution converges to the normal distribution as the sample size approaches infinity; however, there is no easy way to show mathematically how and when the convergence fails.'''"
</blockquote>
<ref>http://www.minitab.com/support/docs/Answers/Chi-Square%20Test%20Assumptions.pdf</ref>

<blockquote>
"'''The chi-square test is simpler to calculate but yields only an approximate P value. ... You should definitely avoid the chi-square test when the numbers in the contingency table are very small (any number less than about six)'''."
</blockquote> <ref>http://www.graphpad.com/www/Book/Choose.htm</ref>

<blockquote>
"The most important things to remember to get a valid χ2 test are that the expected values are not too small in any bin (certainly 5 or more), and that the degrees of freedom are properly evaluated. '''Unless you have a very large amount of data, the test is not very sensitive and errs on the side of safety. If you get a significant result, however, it is not likely to be wrong.'''"
</blockquote> <ref>http://mysite.du.edu/~jcalvert/econ/chisquar.htm</ref>

<blockquote>
"The critical assumptions of the chi-square test for k independent samples are similar to those for the chi-square test for two independent samples. ... 
'''4. No more than 20% of the cells may have expected frequencies of less than 5, and no cell should have an expected frequency of less than 1. 
The rule given in Assumption 4 is particularly important for a contingency table that is larger than 2X2'''"
</blockquote> <ref>http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1
</ref>

<blockquote>
"Special problems with small expected cell frequencies for the chi-square test: 
The chi-square test involves using the chi-square distribution to approximate the underlying exact distribution. The approximation becomes better as the expected cell frequencies grow larger, and '''may be inappropriate for tables with very small expected cell frequencies.''' 
'''For tables with expected cell frequencies less than 5, the chi-square approximation may not be reliable. A standard (and conservative) rule of thumb (due to Cochran) is to avoid using the chi-square test for tables with expected cell frequencies less than 1, or when more than 20% of the table cells have expected cell frequencies less than 5.''' 
Another rule of thumb (due to Roscoe and Byars) is that the average expected cell frequency should be at least 1 when the expected cell frequencies are close to equal, and 2 when they are not. (If the chosen significance level is 0.01 instead of 0.05, then double these numbers.) 
Koehler and Larntz suggest that if the total number of observations is at least 10, the number categories is at least 3, and the square of the total number of observations is at least 10 times the number of categories, then the chi-square approximation should be reasonable. 
Care should be taken when cell categories are combined (collapsed together) to fix problems of small expected cell frequencies. Collapsing can destroy evidence of non-independence, so a failure to reject the null hypothesis for the collapsed table does not rule out the possibility of non-independence in the original table. 
'''As with most statistical tests, the power of the chi-square test increases with a larger number of observations. If there are too few observations, it may be impossible to reject the null hypothesis even if it is false.'''"
</blockquote> <ref>http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html</ref>--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

: Thanks for this really excellent contribution, ElyM. The only thing I'd like to add is in relation to your initial statement, "I do not believe that anyone has claimed that 'chi-square test p-values are always conservative'". The question of whether a test is conservative in a particular situation is probabilistic. One can determine whether a test is likely to generate a p-value which is too high in a particular situation (e.g. for a chi-squared test, when there are lots of small expected values) but one needs an exact test (such as an appropriate Monte Carlo randomisation test) to determine whether the p-value in any ''particular'' test is in fact excessively high.

::This edit also claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639373]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

: I hope careful reading of your very clear description will put SJohnson's mind at rest on this subject. [[User:FredFerguson|FredFerguson]] 18:11, 14 March 2009 (EDT)

ElyM, you've provided nothing to address the basic flaw that "The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events." See [[Flaws in Lenski Study]]. Also, do not impose your view on the content page until after SJohnson has had an opportunity to respond to your posting. As to "Fred", his put-downs are getting tiresome and I'm going to review his edit pattern now to see if he's been contributing anything of value to this site.--[[User:Aschlafly|Andy Schlafly]] 14:06, 15 March 2009 (EDT)

::Mr. Schlafly, per your request I have not added anything to the content page as SJohnson has not yet responded to my posts. Since all of my comments have been in regards to SJohnson's use of the chi-square test in this particular article, I'm not sure why you expect me to address Blout's use of Monte Carlo - that issue seems to be addressed on the [[Flaws in Lenski Study]] page. SJohnson has added a reformulation of the chi-square test for two possible outcomes, and stated that the chi-square test is at a minimum when all success probabilities are equal. He then extrapolates from this to claim that the chi-square test is an effective test for the data from Blount.

::The reformulation of the equations for two possible outcomes does not address the underlying problem that the chi-square test has universally accepted parameters outside of which it is considered an invalid test; I have provided references for these parameters and shown that the data from Blount lies outside them. None of the expected cells in SJohnson's analysis have values above one, and the total n is four. SJohnson's own reference states that the application of the chi-square test in this circumstance is a "violation of good statistical practice". Analogously, combining F=ma and t=(vf-vi)/a into t=(vf-vi)m/F and showing that t is a minimum when m approaches zero does not address the fact that those Newtonian equations do not apply as velocities approach the speed of light. The legitimacy of Blount's arguments cannot be determined by the application of illegitimate counterarguments. If SJohnson or others can point to references from the statistical literature that show that Blount has made methodological errors - as I have been able to do with SJohnson's chi-square analysis - I would welcome their input, and no doubt Conservapedia's other readers would as well, and this page would be greatly improved.

::I have not seen a rebuttal from SJohnson in the four days since my last post, although he has added new material to the content page since then. In light of this, I would appreciate some guidelines as to when it is appropriate for me to add my information and references to the content page. I can add citations from the primary mathematical literature if necessary, but in general I find that these are less helpful as they are not easily accessible by readers without access to academic libraries.--[[User:ElyM|ElyM]] 18:07, 18 March 2009 (EDT)

::: You say, "I'm not sure why you expect me to address Blout's use of Monte Carlo." The reason is obvious: the title of the content page is the "Significance of E. Coli Evolution Experiments." You haven't addressed the inappropriateness of using Monte Carlo simulations for assessing the significance rarely occurring events, which was central to Lenski's statistical claims. I suggest you address this flaw if you want to be taken seriously.--[[User:Aschlafly|Andy Schlafly]] 23:12, 18 March 2009 (EDT)

::::ASchlafly, again per your request, and based on your statement regarding the "inappropriateness of using Monte Carlo simulations for assessing the significance of rarely occurring events", I have spent the last several days reviewing the literature available to me on Monte Carlo and other resampling techniques, looking for ways in which Blount may have made a methodological error of the sort that SJohnson has made. I have been unable to find any examples of authors suggesting that Monte Carlo be avoided for low ''n'', or for events with low probability regardless of'' n'', much less providing specific cutoff numbers as are seen in the references that I provided for the chi-square test. Similarly, the technique that Blount used does not require/assume that categories are unrelated, as the chi-square test does. Of course, the absence of evidence is not evidence of absence, and I may have misinterpreted the basis of your objection. At this point I'll need you to explain your objection in more detail if you wish me to find the appropriate literature addressing your concerns. Do you believe that the number of resamplings was too low in Blount's paper? That the analysis should have been performed with a software package other than Statistics101? Some other procedural issue? Some issue of interpretation?

::::The statistical problem that Blount must address is straightforward: given a distribution of mutant cultures that ''appears'' to be skewed toward the higher generations, what is the probability that this same amount of skew (or a greater degree) could arise by chance, given the null hypothesis that every generation is equally likely to produce a mutant? Interestingly, in the case of the first replay experiment, the total number of ways to randomly select (equal probability, no replacement) four cultures from seventy-two is 72x71x70x69, or 24,690,960. This number is small enough that a program can brute-force-calculate the 'mean generation number' of ''all possible'' combinations of four cultures in a reasonable amount of time. An experimentally-derived 'mean generation number' can be checked against this exhaustive list, and the number of means equal to or larger than the experimental mean can be found exactly. Converting this number to a percentage of 24,690,960 provides an exact p-value for any given experimental 'mean generation number'. This exhaustive approach is different than the Monte Carlo technique, in that ''all possible'' outcomes are examined, rather than a ''random subset'' of all possible outcomes. For the first replay experiment, it provides a way to independently check Blount's Monte Carlo results. This approach is not possible for the second and third replay experiments, in which the total number of possible combinations becomes impractically large: 340!/335! = 4.41 x10^12 and 2800!/2792! = 3.74 x 10^27, respectively.

::::I asked a colleague to run just such a brute-force program for me on the first replay data. I also ran several Monte Carlo simulations ('''not''' using Statsistics101) with Blount's data, using twenty-five million, one hundred million, and 493,819,200 resamplings - note that this last is twenty times the number of all possible combinations of 4 samples drawn without replacement from 72. The p-values from the 25M, 100M, and 493M Monte Carlo resamplings (0.00844, 0.00846, and 0.00846, respectively) compare favorably with Blount's 1M value of 0.0085 and the non-Monte-Carlo brute-force exact calculation, which provides a p-value of 0.008457. Thus it appears that Blount's statistical results are confirmed by a ''non-Monte Carlo'' technique, at least for the first replay experiment.

:::::What do you get for the experiment two p-value using the method from the paper and at least ten million realizations? [[User:SJohnson|SJohnson]] 08:48, 25 March 2009 (EDT)

::::My intention is not to get caught up in a digression about Monte Carlo, though - I'd rather keep the focus on the fact that the main article should acknowledge that SJohnson is using chi-square in a way that violates accepted guidelines; this remains true whether Blount's analysis is valid or not.--[[User:ElyM|ElyM]] 12:13, 23 March 2009 (EDT)

== References ==
{{reflist}}

Talk:Significance of E. Coli Evolution Experiments

2009-03-19T00:49:35Z

SJohnson: More responses

SJohnson, your assessment, while good in the utilization of the chi-squared test is unfortunately incorrect. The Monte Carlo resampling gives a more accurate p-value than the chi-squared. You may research the literature (i.e. publications in statistical mathematics, many pubs actualy compare Monte Carlo vs Chi Squared) to discover that this method is commonly used in advance statistical work and how it is more accurate than the chi-squared test.--[[User:Able806|Able806]] 17:00, 4 March 2009 (EST)

:It doesn’t make sense to compare the chi-square test, which is a specific statistical hypothesis test, to Monte Carlo methods, which can be used for anything from fluid motion modeling to p-value computations. You can use Monte Carlo methods to compute the p-values of the chi-square test!

:Monte Carlo methods involve the generation of random realizations. Your broad claim the Monte Carlo methods are “more accurate” than the chi-square test is obviously incorrect because the accuracy of Monte Carlo methods always depends on the number of random realizations generated. When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous.

:Which publications compare Monte Carlo to chi-square and show that the former is more accurate? Could you provide specific examples? Thanks. [[User:SJohnson|SJohnson]] 18:50, 4 March 2009 (EST)

:In furtherance of SJohnson's remarks with respect to rarely occurring events, the use of the basic Monte Carlo method is plainly incorrect for modeling a rarely occurring event, as the Lenski paper did. This has long been pointed out in [[Flaws in Richard Lenski Study]]. I know [[evolutionists]] will never admit a flaw in anything promoting their pet theory, but this (and other) flaws in that paper is undeniable.

:Watch how evolutionists defended obvious errors in the Lenski paper, and then realize why the [[Piltdown Man]] fraud was taught for 40 years without evolutionists admitting it was a hoax.--[[User:Aschlafly|Andy Schlafly]] 09:55, 5 March 2009 (EST)

::Andy, how exactly is the Monte Carlo method incorrect to use in this case? I have seen it used in publications with much smaller datasets.--[[User:Able806|Able806]] 10:29, 5 March 2009 (EST)

:::Able806, I'm interested in looking at the publications you mentioned that use Monte Carlo methods to analyze small data sets. Could you provide some examples? Thanks. [[User:SJohnson|SJohnson]] 16:41, 5 March 2009 (EST)

::::SJohnson, here are two papers, [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6WH8-45RFJ1J-19&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=1ad95954654bb97b17e474ce6b469f6e 1] and [http://cat.inist.fr/?aModele=afficheN&cpsidt=787963 2]. Most are in chemistry and genetics where you find the observed to be much smaller and have to use the MCM. You can search on the subject as well and find that how Lenski performed the test is the standard for microbiological genetic analysis.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

:::::Those papers have nothing to do with hypothesis testing. One is an archeology paper. To be blunt, it seems like you’re just doing internet searches on “Monte Carlo” to find these links. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

::::::SJohnson, actually they do, did you read the papers? If so you would see how they used the MCM for their data analysis of small data sets, which indeed was hypothesis testing and answers you inquiry about publications that use MCM for small data set analysis. If you wish I can try to track down some actual mathematical publications, however, I am not as familiar with mathematical journals as I am with science/medical journals (not knowing which mathematical journals are acceptable). I am assuming that you have a background in math and possibly access to mathematical journals, therefore if you know the reputable ones I can do the leg work.
::::::I believe the thing that needs to be looked at is there truly a problem with the choice of test and if so what is an alternative. Bayesian might be an option but seems to be difficult to employ for this situation.--[[User:Able806|Able806]] 12:36, 12 March 2009 (EDT)

:::Able806, you still seem to miss the point about how inappropriate the Monte Carlo method (as used in the Lenski paper) is for evaluating rarely occurring events. You need to open your mind to be productive. If you simply cling to a view that Lenski (who I don't think has any meaningful education in statistics) must somehow be right, then you're not going to make any progress in understanding the flaws.--[[User:Aschlafly|Andy Schlafly]] 17:07, 5 March 2009 (EST)

::::Andy, you still have not answered what you find inappropriate about his use of the Monte Carlo method? I am a reasonable person and with evidence I do have an open mind. I provided examples last week, with a working model, showing that Monte Carlo is better than the chi-square in this case. I have also shown where the Chi-Square was inappropriate due to the occurrence size as well. So if you have any evidence that Monte Carlo should not be used in the way that Lenski used please let it be shown.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

Sjohnson, I believe you just proved my point. In the literature of mean and covariance structure analysis, non-central chi-square distribution is commonly used to describe the behavior of the likelihood ratio statistic under alternative hypothesis; it is widely believed that the non-central chi-square distribution is justified by statistical theory. Actually, when the null hypothesis is not trivially violated, the non-central chi-square distribution cannot describe the LR statistic well even when data are normally distributed and the sample size is large. Monte Carlo results compare the strength of the normal distribution against that of the non-central chi-square distribution. In an association analysis comparing cases and controls with respect to allele frequencies at a highly polymorphic locus, a potential problem is that the conventional chi-squared test may not be valid for a large, sparse contingency table. Reliance on statistics with known asymptotic distribution is unnecessary, as Monte Carlo simulations can be performed to estimate the significance level of the test statistic.

Here is a [http://faculty.vassar.edu/lowry/chi_beta.html link] to a great page the provides an interactive example as to why the Chi Squared test would provide poor results compared to the Monte Carlo in relation to the Lenski data workup.

Something you may have overlooked was that the data set is actually too small to use the chi square method correctly. It is often accepted that is any of the analyzed data falls under 10 for a particular cell of the data set then the Yates correction needs to be applied; unfortunately the Yates correction can over correct thus skewing the p-value. Lenksi seemed to understand this by supporting his Monte Carlo p-value results with the Fisher z-transformation p-value.

I hope this helps.--[[User:Able806|Able806]] 10:27, 5 March 2009 (EST)

:I’m still waiting to hear which literature says that “Monte Carlo resampling” is “more accurate than the chi-squared test”. The page mentioned above [http://faculty.vassar.edu/lowry/chi_beta.html] is a discussion of why statisticians “fail to reject the null” rather than “accepting the null” when the p-value is above 0.05 or so. The page says nothing about superiority of Monte Carlo methods. Why were alternate hypothesis distributions mentioned? Only the null hypothesis distribution is used to calculate a p-value. Yates’s correction is for 2x2 contingency tables [http://en.wikipedia.org/wiki/Yates%27_correction_for_continuity]. It doesn’t apply in this case. Finally, what the heck do “covariance structure analysis” and “allele frequencies at a highly polymorphic locus” have to do with this problem? [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::SJohnson, I am looking for this paper for you, I cited it for one of my past publications dealing with allele frequencies (I believe it came from the Duke Biostatistics group). To answer your question about allele frequencies, that is the issue at hand, more about the genetics than the math, but it is the item being studied. So you stated that Yates can not be used and statistics says the number of occurrences is too small to evaluate using the Chi-Squared test so what would you recommend instead of the Monte-Carlo Method?

:Regarding the "Fisher z-transformation p-value" from the paper, garbage in garbage out. If the p-values were bad to begin with, then why would a combination of them be meaningful? [[User:SJohnson|SJohnson]] 10:49, 9 March 2009 (EDT)

::You are assuming that p-values are wrong based on a test that is inappropriate in this case due to data limitations. Did you perform a z-transformation on the chi-squared for the three data groups?--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

::You asked about the “Fisher z-transformation p-value”. The z-transformation test and Fisher’s method are actually two different things (see Whitlock's 2005 paper - Ref. 49 in Blount et al.). But no, I haven’t tried either. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::There's a large literature on various kinds of Monte Carlo test, a very short summary of which is that they're inevitably more accurate than parametric tests (e.g. F, t, chi-squared, etc) because they don't make assumptions about the distribution of the data under the null hypothesis. See for example ''Introduction to the Bootstrap'' by B. Efron and R. Tibshirani and ''The Jack-knife, the Bootstrap and Other Resampling Plans'', also by Efron. They're certainly applicable to small datasets and their accuracy is really only limited by the number of samples you care to take. E.g. 1000 M-C samples would give you a pretty accurate idea about significance at the alpha<1% level (That book should answer SJohnson's questions of 18:50 on 4/3/09 and 16:38 on 5/3/09 about accuracy and Aschalfly's comment of 17:07 on 5/3/09 about appropriateness of Monte Carlo tests.) [[User:FredFerguson|FredFerguson]] 16:53, 11 March 2009 (EDT)

::::Your claim that Monte Carlo methods are “inevitably more accurate” than other tests is obviously wrong because the accuracy of MC methods always depends on the number of realizations used. You should have written <math>\alpha=1\%</math>, not <math>\alpha<1\%</math>. If 1,000 random realizations are generated, the number of realizations above the true <math>\alpha=1\%</math> level is binomial with mean 10 and variance about 10. Thus, the standard deviation of the MC estimate is >0.003. In this example, a Monte Carlo p-value could be off by 30% and still be within a standard deviation. Is that really “pretty accurate”?

::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 3%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::::You're talking about miniscule differences in the accuracy of a test. 0.013 isn't very different from 0.007. In either case, it's very unlikely the experimenter would have obtained that result if the null hypothesis were true. If you're bothered about differences in P-values to the third decimals (which would make you unusual!), just run more MC realisations, that's all. Not really a problem. [[User:FredFerguson|FredFerguson]] 11:53, 12 March 2009 (EDT)

There’s still confusion about the difference between test statistics and Monte Carlo methods. Before you find a Monte Carlo estimate of a p-value, you need to select a test statistic to reduce the data set to a scalar. I am interested in hearing which test statistic you believe should be used in place of the chi-square test and why. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

----

Quick question for SJohnson: How many degrees of freedom did you choose when calculating the p-value? I'd like to know upon what condition you base that number. Thanks.--[[User:Argon|Argon]] 11:05, 5 March 2009 (EST)

:The degree of freedom for a contingency table is rows minus one times columns minus one. That is, <math> (r-1)(c-1) </math>. Here’s a pretty good tutorial I came across: [http://faculty.uncfsu.edu/dwallace/lesson%2020.pdf]. For the experiments from [http://www.pnas.org/content/105/23/7899.full.pdf], the DOFs are 11, 11, and 13. For experiment one, the chi-square test statistic is
::<math>
X^2
=\sum\limits_i\sum\limits_j
\frac{\left(n_{i,j}-E\left[n_{i,j}\right]\right)^2}
{E\left[n_{i,j}\right]}
</math>
::<math>
=\frac{\left(0-1/3\right)^2}{1/3}
+\frac{\left(6-17/3\right)^2}{17/3}
+\frac{\left(0-1/3\right)^2}{1/3}
+\ldots+
</math>
::<math>
+\frac{\left(2-1/3\right)^2}{1/3}
+\frac{\left(4-17/3\right)^2}{17/3}
</math>
::<math>
\approx
14.82
</math>
:where <math>n_{i,j}</math> is the observed value and <math>E\left[n_{i,j}\right]</math> is the expected null hypothesis value. So if you have MS Excel, another way to arrive at the p-value of 0.19 is to type “=CHIDIST(14.82,11)” into a cell. Cheers! [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::OK, thanks for the info. From what I'd calculated and looked up in tables, the numbers seemed close to a df=11 for a chi-square of ~14. (Aside: With terms having 17/3 in the denominator in the figures above, were you using the test of independence? I was using Pearson's test for [http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_for_fit_of_a_distribution fit of a distribution] which returns a chi-squared value of 14 and roughly matched the p-values you reported, assuming the df was 11).

::Also, the first sentence of the article reads: "Blount, Borland, and Lenski[1] claimed that a key evolutionary innovation was observed during a laboratory experiment. That claim is false." A small correction: There were several claims in the paper. The 'key evolutionary innovation' was acquiring the ability to utilize citrate as a food source. That claim was demonstrated multiple times. The claim, which pertains to this statistics discussion was that the Cit+ phenotype arose in a multi-step process, first requiring a rare, pre-adaptive mutation before additional mutation(s) lead to the subsequent development of citrate utilization.--[[User:Argon|Argon]] 20:46, 5 March 2009 (EST)

:::My biology-degreed wife assures me that mutation does not necessarily mean that evolution occurred. What the paper claimed is that evolution (a “key innovation”) occurred in the lab. The key innovation supposedly increased the mutation rate. In the experiments, the observed mutation rate increased after generation 31,000, but not enough to make a statistically significant claim that the rate is not constant. The analysis in the paper was similar to flipping a coin ten times, counting six heads and claiming that the coin must be biased against tails. In reality, there’s nothing surprising about a fair coin producing slightly more of one outcome than the other. Just like there's nothing surprising about there being slightly more mutations in later generations than early generations given the null hypothesis (constant mutation rate). [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

::::SJohnson, not to say anything about your wife, but has she had a 400 level molecular genetics course (most general biology degrees do not cover the detail unless they are specialized)? If so, she would have mentioned that if the mutation passes to the offspring and is selectively beneficial to the population then it is a step of evolution as along as the conditions continue through the sharing of the mutation with the population and the environment is such that reduces the growth rate of the non-transformed population. While not all mutations are signs that evolution occurred the mutations that pass to offspring and provide a benefit compared to other offspring are very strong indicators. In the case of this paper the population that evolved the cit+ was able to metabolize a chemical in their environment which allowed for an adaptation advantage compared to the non-transformed colonies.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

----

Let’s go back to the beginning. There appears to be confusion about the difference between test statistics and methods for computing p-values. As is noted at the beginning of the page [http://www.conservapedia.com/Significance_of_E._Coli_Evolution_Experiments], the fundamental problem with the paper is that it used a flawed test statistic, not that it used Monte Carlo methods to find the p-value for that flawed statistic.

Every hypothesis test uses a test statistic to reduce the data to a single number. The p-value for the test statistic can be calculated analytically (as I’ve done for the chi-square test statistic) or by Monte Carlo methods. In the paper, Monte Carlo methods were used to compute the p-value of the “mutation generation” test statistic. The key problem with the analysis from the paper is that it doesn’t work to use a weighted average to test for variations in mutation rate. This is like trying to use the sample variance to test for an increase in the mean in Gaussian-distributed data. A statistic should be selected based on the null and alternate hypothesis distributions of the data. The chi-square test (unlike the weighted average from the paper) is a reasonable choice for data that mutates at a constant rate under the null hypothesis, but mutates at varying rates under the alternate hypothesis.

Able806, you made a good point about the contingency table cell frequencies being relatively low, but were wrong when you said ”the data set is actually too small to use the chi square method correctly”. In the low cell frequency case the chi-square test is still effective, but the null hypothesis distribution of the chi-square statistic starts to look less like the chi-square distribution. Thus, p-values calculated using the chi-square distribution may be a bit off. However, Monte Carlo p-values are always imperfect as well because it's impossible to generate an infinite number of random realizations. There are imperfections in p-values generated by analytic and Monte Carlo methods. However, low cell frequencies does not explain the >20x and >2.5x differences between chi-square p-values and p-values from the paper for experiments one and three. The reason for those huge differences was the use of the flawed test statistic (“mutation generation”) in the paper. [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

:SJohnson, the chi-squared test is a valuable statistical tool, but the limitations of the test must be acknowledged. The chi-squared test can only produce valid results if the assumptions that underly the test are not violated. As an analogy, Newtonian models of motion fail to produce accurate results as velocities approach the speed of light; under those circumstances one must switch to a theory that accounts for relativistic effects.

:It seems that you have simply dismissed the [http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm widely-acknowledged] [http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm fact] that the [http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html chi-squared test] is [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236 inappropriate] for use in [http://www.graphpad.com/www/Book/Choose.htm situations] where n in any cell is [http://mysite.du.edu/~jcalvert/econ/chisquar.htm less] less than a [http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1 threshold] [http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html number]. Different authors set different thresholds, but all are well above the numbers seen in your chi-squared analysis - even the most liberal guidelines advise against the chi-squared test when any expected cell frequency is less than one or more than 20% of the table cells are less than 5; others require that expected values in all cells must be more than 5. With smaller amounts of data, the test is insensitive and errs on the side of rejecting the hypothesis. If you attempt your chi-squared statistical analysis with a program that is more sophisticated than MS Excel (as I did), you get an error message indicating that the results are invalid due to low expected cell counts.

:That issue aside, there are other reasons that the chi-squared test is inappropriate here. As the links above point out, the categories tested must be truly independent; one example is that you can't use the chi-squared test to compare age and ability to kick a field goal by testing the same experimental group twice, one year apart; you have to test one group of age A and a different group of age B. In the case of the Blount paper, the categories are not independent. Even if there were adequate numbers to address the low-expected-frequency problem, this would make the chi-squared an invalid test in this case.

:There are other significant problems with the use of the chi-squared test in this circumstance, but they can wait until you address these first major problems.--[[User:ElyM|ElyM]] 12:18, 11 March 2009 (EDT)

::Wackerly et al. says in general it’s assumed that the cell frequencies are above five so that the chi-square statistic (under the null) is approximately chi-square distributed (see p. 703). That book does not say chi-square test results are invalid if frequencies are five or less. Your example of a chi-square test warning message (it said "warning" not "error" as you stated) in Minitab [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236] said “approximation probably invalid” referring to the chi-square distribution approximation to the chi-square test statistic’s distribution. Your example did not say “chi-square test invalid”. I agree that when cell frequencies are low, the chi-square test statistic’s distribution starts to deviate from the chi-square distribution. I maintain that this deviation is not enough to explain the >2.5x and >20x differences in the chi-square test p-values and the p-values from the paper.

::As the numerous links in your post proved, the chi-square test is widely-used by statisticians. Can you give examples of statisticians using mean mutation generation as a test statistic? Also, did your software agree with the chi-square test p-values I presented? Thanks. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::Thank you for giving page references for Wackerly; however it seems we have different editions, since page 703 in my copy (5th ed, 1996) does not deal with chi-squared issues at all. My copy does state the following, on page 622: "Although the mathematical proof is beyond the scope of this text, it can be shown that, when n is large [chi-squared] will possess approximately a chi-square probability distribution in repeated sampling." Then, on page 624: "Experience has shown that cell counts [n sub i] should not be too small in order that the chi-square distribution provide an accurate approximation to the distribution of [chi squared]. As a rule of thumb we require that all expected cell counts equal or exceed 5, although Cochran (1952) has noted that this value can be as low as 1 for some situations." Wackerly then goes on, in the problems sections, to describe the use of the chi-squared test as a "violation of good statistical practice" when "some expected counts [are] <5."

:::It seems that you are already aware that the [chi-square] statistic under the null is no longer chi-square distributed for small n; this is precisely why the test should not be used under those conditions. I can claim to be able to accelerate a 1-kg mass to 10 times the speed of light by applying 1 N of force for 95 years by using F=ma and t= (vf-vi)/a. Plugging the numbers into those equations will produce the same result every time, but the answer is illegitimate because those equations are only valid under certain assumptions, which are violated as velocities approach the speed of light. Similarly, having a statistical program calculate a chi-squared value given the Blount data will produce a number result, but since the assumptions of the test are violated the result is not legitimate. Yes, if I put the Blount data in SAS 9.2, I get the same numerical answer as you do, but I also get the following message: "WARNING: >89% of the cells have expected counts less than 5. Chi-square may not be a valid test." You may argue that that's a warning, not an error; that's a semantic distinction. The reason that the program says that it MAY not be valid is that the chi-squared test skews in the direction of being too conservative at low n values; the test has an acceptable rate of false positives but an unacceptably high rate of false negatives. Comparing the results of the Monte Carlo and chi-squared results in this case is like comparing the results of Newtonian and relativistic equations of motion: they can produce very different results from the same input data.

::::For a finite amount of data, the chi-square statistic is never chi-square distributed under the null. The p-values are always approximate regardless of cell frequencies. The approximation becomes more accurate as the amount of data increases, but I don’t believe that this inaccuracy will change p-values that are about 0.2 (for experiments 1 and 3) into statistically significant p-values. How much do you expect the p-values to change if an exact computation is used in place of the chi-square distribution approximation? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::Your last paragraph has a major non sequitur in it: yes, many statisticians use the chi-square test. As long as the assumptions of the test are not violated, it is a valuable tool. That has nothing to do with the validity of using mean mutation generation as a test statistic. 'Mean number of werewolf attacks in Mumbai in the week centered on the new moon, by month, from 1654 to 1798' is a valid test statistic. I am quite sure that it has never been used in a peer-reviewed paper before. That does not mean that I can't perform valid statistical tests on that statistic. If, however, the incorrect test is applied, the results of the analysis will be flawed. Papers apply a (relatively small) standard repertoire of valid tests to a (potentially infinite) number of test statistics. The particular test statistic used in a paper may never have been used before and may never be used again; that does not address the validity of the analysis. In Blount's case, the test is the Monte Carlo analysis, which is also "widely-used by statisticians".

::::There are an infinite number of ways to reduce a data set to a single number. However, it’s foolish to think every method would be effective. I gave an example of a flawed test statistic in an earlier post [http://www.conservapedia.com/index.php?title=Talk%3ASignificance_of_E._Coli_Evolution_Experiments&diff=635070&oldid=634987]. Another example of a flawed test statistic is the one used in the paper because it does not always detect deviations from the null hypothesis (see: [[Significance of E. Coli Evolution Experiments#Test Statistics]]).

::::Test statistics are typically derived. The likelihood ratio test is a common method used to derive them. The chi-square test for independence is an approximation to the LRT. Where is the derivation saying that mean mutation generation is an appropriate test statistic for this problem? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::We still haven't touched on the issue of the categories not being independent, which by itself is sufficient to invalidate the chi-squared technique. I'm new to this site, so I'm unsure as to the etiquette of making changes to the articles of another person - but the article here should at the very least mention that the chi-square test is being used here in a manner that violates its underlying assumptions in at least two fundamental ways, and the results are therefore suspect.--[[User:ElyM|ElyM]] 17:34, 12 March 2009 (EDT)

:::::When generating random realizations of experiment outcomes, the authors assumed that the total number of mutants was fixed. Thus the paper assumed the numbers of mutants per generation are statistically dependent. Does this seem like a realistic model, or do you think that if the experiments were recreated that the total number of mutants could vary? For example, if experiment one were recreated, would the total number of mutants always be exactly four? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::: It looks to me as though SJohnson has misinterpreted the application of the chi-squared test in quite a fundamental way. His/her analysis of Blount's data are therefore close to meaningless, regardless of whether the test used by Blount is appropriate or not. In my opinion, the entire page should therefore be deleted. [[User:FredFerguson|FredFerguson]] 08:18, 13 March 2009 (EDT)

::::: "Fred", perhaps you mistakenly think this is Wikipedia, where [[censorship]] and deletion of pages for ideological reasons are common. Not here.--[[User:Aschlafly|Andy Schlafly]] 10:23, 14 March 2009 (EDT)

:::::: Umm... I'm suggesting deletion for mathematical reasons, not ideological reasons. Using an argument filled with mathematical errors to try to support your case only detracts from your credibility. [[User:FredFerguson|FredFerguson]] 10:38, 14 March 2009 (EDT)

:::::: Actually, I think correction is better than deletion. So that's what I've done. [[User:FredFerguson|FredFerguson]] 11:01, 14 March 2009 (EDT)

::::::: I find no credibility in your denial of having ideological reasons.--[[User:Aschlafly|Andy Schlafly]] 11:04, 14 March 2009 (EDT)

== Misinterpretation of test ==

SJohnson, Your analysis misinterprets the test. You say the null hypothesis is that this mutation cannot happen. They saw a mutation (4 mutations, in fact, in the data set you show) so the null hypothesis (as you state is) is disproved. That's perfectly straightforward.

I don't know what the "mean mutation generation" test is but you're doing when you apply a chi-squared test to this dataset is to test if the mutations are evenly distributed throughout the generations. Your test says they are, so there's no strong evidence to suppose that mutations are likely to occur in one generation rather than another in the series of tests. Blount's test says thay aren't, so it's more likely that the mutation will occur later in the series of tests. I can't tell which test is right without knowing more about the test that Blount used.

But that point (the foregoing paragraph) has no bearing at all on the null hypothesis, as you describe it. The mutation appeared, so that means the hypothesis that the mutation can't happen is disproved. Very simple. [[User:FredFerguson|FredFerguson]] 21:10, 8 March 2009 (EDT)

:I never said that “the null hypothesis is that this mutation cannot happen”. The chi-square test statistic I'm using wouldn’t be defined if the null hypothesis mutation rate was zero because the <math>E\left[n_{i,j}\right]</math> term in the denominator of the statistic (see above equation) would be zero.

:The test statistic from the paper is the average of the generation numbers of observed mutations. For experiment one this number is
::<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)= 31750.
</math>
:The same number is shown in Table 2 of the paper. [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

:: SJohnson, the way you're calculating the chi-squared statistic implies that you're testing the null hypothesis of a constant mutation rate over time against an alternative hypothesis of a mutation rate which varies over time. [[User:FredFerguson|FredFerguson]] 11:02, 9 March 2009 (EDT)

As it currently stands, the article makes the following statement: "The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown." This misstates the null hypothesis of the paper, which is elaborated in the Introduction section of the paper, and repeated in the section '''Statistical Analysis of the Replay Experiments''':
<blockquote>
For each experiment, we compared the observed mean generation of those clones that yielded Cit+ variants to the mean expected under the null hypothesis that clones from all generations have equal likelihood. The null thus corresponds to the rare-mutation hypothesis laid out in the Introduction."
Block quote
</blockquote><ref>www.pnas.org/cgi/reprint/105/23/7899.pdf</ref>

The article also continues to describe 'mean mutation generation' as a ''test'' rather than a ''statistic'' to which the ''Monte Carlo test'' was applied.--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

==Unreferenced Claims==

I deleted the claim that mean mutation generation is an appropriate test statistic because no reference was produced that back that claim. No reference was provided to back the claim that the chi-square test p-values are always conservative, either. [[User:SJohnson|SJohnson]] 12:57, 14 March 2009 (EDT)

: The reference is Everitt. I'll check I put it in the right place. [[User:FredFerguson|FredFerguson]] 13:30, 14 March 2009 (EDT)

There was a typo in my edit summaries on the talk page and the main page. I meant to say "Removed unsupported claims" rather than "Removed supported claims". [[User:SJohnson|SJohnson]] 13:13, 14 March 2009 (EDT)

I do not believe that anyone has claimed that 'chi-square test p-values are ''always'' conservative'. The claim that has been made is that ''under certain circumstances'', namely low n and low individual cell values, the chi-square test is an invalid test; that under those circumstances the power of the test is low and it becomes impossible to reject the null hypothesis even when it is false. You may have missed the pertinent sections in my links above, so I will directly quote the relevant sections. All the quoted sections refer to chi-square testing in particular. Any bolding below is mine.

:This edit claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639379]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

<blockquote>
"Assumptions: 

Even though a nonparametric statistic does not require a normally distributed population, there still are some restrictions regarding its use. 

1. Representative sample (Random) 
2. The data must be in frequency form (nominal data) or greater. 
3. The individual observations must be independent of each other. 
4. '''Sample size must be adequate. In a 2 x 2 table, Chi Square should not be used if n is less than 20. In a larger table, no expected value should be less than 1, and not more than 20% of the variables can have expected values of less than 5'''. 
5. Distribution basis must be decided on before the data is collected. 
6. The sum of the observed frequencies must equal the sum of the expected frequencies."
</blockquote>
<ref>http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm
</ref>

<blockquote>
"Assumptions:
* Random sample data are assumed. As with all significance tests, if you have population data, then any table differences are real and therefore significant. If you have non-random sample data, significance cannot be established, though significance tests are nonetheless sometimes utilized as crude "rules of thumb" anyway.
* A sufficiently large sample size is assumed, as in all significance tests. '''Applying chi-square to small samples exposes the researcher to an unacceptable rate of Type II errors. There is no accepted cutoff. Some set the minimum sample size at 50, while others would allow as few as 20'''. Note chi-square must be calculated on actual count data, not substituting percentages, which would have the effect of pretending the sample size is 100.
* '''Adequate cell sizes are also assumed. Some require 5 or more, some require more than 5, and others require 10 or more. A common rule is 5 or more in all cells of a 2-by-2 table, and 5 or more in 80% of cells in larger tables, but no cells with zero count'''. When this assumption is not met, Yates' correction is applied.
* Independence. Observations must be independent. The same observation can only appear in one cell. '''This means chi-square cannot be used to test correlated data (ex., before-after, matched pairs, panel data)'''.
* Similar distribution. Observations must have the same underlying distribution.
* Known distribution. The hypothesized distribution is specified in advance, so that the number of observations that are expected to appear each cell in the table can be calculated without reference to the observed values. Normally this expected value is the crossproduct of the row and column marginals divided by the sample size.
* Non-directional hypotheses are assumed. Chi-square tests the hypothesis that two variables are related only by chance. If a significant relationship is found, this is not equivalent to establishing the researcher's hypothesis that A causes B, or that B causes A.
* Finite values. Observations must be grouped in categories.
* Normal distribution of deviations (observed minus expected values) is assumed. Note chi-square is a nonparametric test in the sense that is does not assume the parameter of normal distribution for the data -- only for the deviations.
* Data level. No assumption is made about level of data. Nominal, ordinal, or interval data may be used with chi-square tests."
</blockquote>
<ref>http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm</ref>

<blockquote>
"Assumptions: 
-None of the expected values may be less than 1 
-No more than 20% of the expected values may be less than 5"</blockquote> <ref>http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html</ref>

<blockquote>
"When performing a chi-square test, your data must satisfy important assumptions. Although these assumptions may be stated differently in different textbooks, they generally assert that: 
1)The sample must be randomly drawn from the population 
'''2)The sample size, n, must be large enough so that the expected cell count in each cell is greater than or equal to 5.''' 
Both assumptions must be met in the process of collecting your data, and violations of the second assumption will appear in the Minitab output when you run the analysis. 
... 
'''You may wonder why the second assumption is necessary for performing the chi-square test. The second assumption arises because the distribution of counts under the null hypothesis is multinomial, and the normal distribution can be used to approximate the multinomial distribution if the sample size is sufficiently large and the probability parameters aren't too small. It can be shown via the Central Limit Theorem that the multinomial distribution converges to the normal distribution as the sample size approaches infinity; however, there is no easy way to show mathematically how and when the convergence fails.'''"
</blockquote>
<ref>http://www.minitab.com/support/docs/Answers/Chi-Square%20Test%20Assumptions.pdf</ref>

<blockquote>
"'''The chi-square test is simpler to calculate but yields only an approximate P value. ... You should definitely avoid the chi-square test when the numbers in the contingency table are very small (any number less than about six)'''."
</blockquote> <ref>http://www.graphpad.com/www/Book/Choose.htm</ref>

<blockquote>
"The most important things to remember to get a valid χ2 test are that the expected values are not too small in any bin (certainly 5 or more), and that the degrees of freedom are properly evaluated. '''Unless you have a very large amount of data, the test is not very sensitive and errs on the side of safety. If you get a significant result, however, it is not likely to be wrong.'''"
</blockquote> <ref>http://mysite.du.edu/~jcalvert/econ/chisquar.htm</ref>

<blockquote>
"The critical assumptions of the chi-square test for k independent samples are similar to those for the chi-square test for two independent samples. ... 
'''4. No more than 20% of the cells may have expected frequencies of less than 5, and no cell should have an expected frequency of less than 1. 
The rule given in Assumption 4 is particularly important for a contingency table that is larger than 2X2'''"
</blockquote> <ref>http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1
</ref>

<blockquote>
"Special problems with small expected cell frequencies for the chi-square test: 
The chi-square test involves using the chi-square distribution to approximate the underlying exact distribution. The approximation becomes better as the expected cell frequencies grow larger, and '''may be inappropriate for tables with very small expected cell frequencies.''' 
'''For tables with expected cell frequencies less than 5, the chi-square approximation may not be reliable. A standard (and conservative) rule of thumb (due to Cochran) is to avoid using the chi-square test for tables with expected cell frequencies less than 1, or when more than 20% of the table cells have expected cell frequencies less than 5.''' 
Another rule of thumb (due to Roscoe and Byars) is that the average expected cell frequency should be at least 1 when the expected cell frequencies are close to equal, and 2 when they are not. (If the chosen significance level is 0.01 instead of 0.05, then double these numbers.) 
Koehler and Larntz suggest that if the total number of observations is at least 10, the number categories is at least 3, and the square of the total number of observations is at least 10 times the number of categories, then the chi-square approximation should be reasonable. 
Care should be taken when cell categories are combined (collapsed together) to fix problems of small expected cell frequencies. Collapsing can destroy evidence of non-independence, so a failure to reject the null hypothesis for the collapsed table does not rule out the possibility of non-independence in the original table. 
'''As with most statistical tests, the power of the chi-square test increases with a larger number of observations. If there are too few observations, it may be impossible to reject the null hypothesis even if it is false.'''"
</blockquote> <ref>http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html</ref>--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

: Thanks for this really excellent contribution, ElyM. The only thing I'd like to add is in relation to your initial statement, "I do not believe that anyone has claimed that 'chi-square test p-values are always conservative'". The question of whether a test is conservative in a particular situation is probabilistic. One can determine whether a test is likely to generate a p-value which is too high in a particular situation (e.g. for a chi-squared test, when there are lots of small expected values) but one needs an exact test (such as an appropriate Monte Carlo randomisation test) to determine whether the p-value in any ''particular'' test is in fact excessively high.

::This edit also claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639373]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

: I hope careful reading of your very clear description will put SJohnson's mind at rest on this subject. [[User:FredFerguson|FredFerguson]] 18:11, 14 March 2009 (EDT)

ElyM, you've provided nothing to address the basic flaw that "The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events." See [[Flaws in Lenski Study]]. Also, do not impose your view on the content page until after SJohnson has had an opportunity to respond to your posting. As to "Fred", his put-downs are getting tiresome and I'm going to review his edit pattern now to see if he's been contributing anything of value to this site.--[[User:Aschlafly|Andy Schlafly]] 14:06, 15 March 2009 (EDT)

::Mr. Schlafly, per your request I have not added anything to the content page as SJohnson has not yet responded to my posts. Since all of my comments have been in regards to SJohnson's use of the chi-square test in this particular article, I'm not sure why you expect me to address Blout's use of Monte Carlo - that issue seems to be addressed on the [[Flaws in Lenski Study]] page. SJohnson has added a reformulation of the chi-square test for two possible outcomes, and stated that the chi-square test is at a minimum when all success probabilities are equal. He then extrapolates from this to claim that the chi-square test is an effective test for the data from Blount.

::The reformulation of the equations for two possible outcomes does not address the underlying problem that the chi-square test has universally accepted parameters outside of which it is considered an invalid test; I have provided references for these parameters and shown that the data from Blount lies outside them. None of the expected cells in SJohnson's analysis have values above one, and the total n is four. SJohnson's own reference states that the application of the chi-square test in this circumstance is a "violation of good statistical practice". Analogously, combining F=ma and t=(vf-vi)/a into t=(vf-vi)m/F and showing that t is a minimum when m approaches zero does not address the fact that those Newtonian equations do not apply as velocities approach the speed of light. The legitimacy of Blount's arguments cannot be determined by the application of illegitimate counterarguments. If SJohnson or others can point to references from the statistical literature that show that Blount has made methodological errors - as I have been able to do with SJohnson's chi-square analysis - I would welcome their input, and no doubt Conservapedia's other readers would as well, and this page would be greatly improved.

::I have not seen a rebuttal from SJohnson in the four days since my last post, although he has added new material to the content page since then. In light of this, I would appreciate some guidelines as to when it is appropriate for me to add my information and references to the content page. I can add citations from the primary mathematical literature if necessary, but in general I find that these are less helpful as they are not easily accessible by readers without access to academic libraries.--[[User:ElyM|ElyM]] 18:07, 18 March 2009 (EDT)

== References ==
{{reflist}}

Talk:Significance of E. Coli Evolution Experiments

2009-03-19T00:28:18Z

SJohnson: Fixed typo in on of my earlier comments.

SJohnson, your assessment, while good in the utilization of the chi-squared test is unfortunately incorrect. The Monte Carlo resampling gives a more accurate p-value than the chi-squared. You may research the literature (i.e. publications in statistical mathematics, many pubs actualy compare Monte Carlo vs Chi Squared) to discover that this method is commonly used in advance statistical work and how it is more accurate than the chi-squared test.--[[User:Able806|Able806]] 17:00, 4 March 2009 (EST)

:It doesn’t make sense to compare the chi-square test, which is a specific statistical hypothesis test, to Monte Carlo methods, which can be used for anything from fluid motion modeling to p-value computations. You can use Monte Carlo methods to compute the p-values of the chi-square test!

:Monte Carlo methods involve the generation of random realizations. Your broad claim the Monte Carlo methods are “more accurate” than the chi-square test is obviously incorrect because the accuracy of Monte Carlo methods always depends on the number of random realizations generated. When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous.

:Which publications compare Monte Carlo to chi-square and show that the former is more accurate? Could you provide specific examples? Thanks. [[User:SJohnson|SJohnson]] 18:50, 4 March 2009 (EST)

:In furtherance of SJohnson's remarks with respect to rarely occurring events, the use of the basic Monte Carlo method is plainly incorrect for modeling a rarely occurring event, as the Lenski paper did. This has long been pointed out in [[Flaws in Richard Lenski Study]]. I know [[evolutionists]] will never admit a flaw in anything promoting their pet theory, but this (and other) flaws in that paper is undeniable.

:Watch how evolutionists defended obvious errors in the Lenski paper, and then realize why the [[Piltdown Man]] fraud was taught for 40 years without evolutionists admitting it was a hoax.--[[User:Aschlafly|Andy Schlafly]] 09:55, 5 March 2009 (EST)

::Andy, how exactly is the Monte Carlo method incorrect to use in this case? I have seen it used in publications with much smaller datasets.--[[User:Able806|Able806]] 10:29, 5 March 2009 (EST)

:::Able806, I'm interested in looking at the publications you mentioned that use Monte Carlo methods to analyze small data sets. Could you provide some examples? Thanks. [[User:SJohnson|SJohnson]] 16:41, 5 March 2009 (EST)

::::SJohnson, here are two papers, [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6WH8-45RFJ1J-19&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=1ad95954654bb97b17e474ce6b469f6e 1] and [http://cat.inist.fr/?aModele=afficheN&cpsidt=787963 2]. Most are in chemistry and genetics where you find the observed to be much smaller and have to use the MCM. You can search on the subject as well and find that how Lenski performed the test is the standard for microbiological genetic analysis.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

:::::Those papers have nothing to do with hypothesis testing. One is an archeology paper. To be blunt, it seems like you’re just doing internet searches on “Monte Carlo” to find these links. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

::::::SJohnson, actually they do, did you read the papers? If so you would see how they used the MCM for their data analysis of small data sets, which indeed was hypothesis testing and answers you inquiry about publications that use MCM for small data set analysis. If you wish I can try to track down some actual mathematical publications, however, I am not as familiar with mathematical journals as I am with science/medical journals (not knowing which mathematical journals are acceptable). I am assuming that you have a background in math and possibly access to mathematical journals, therefore if you know the reputable ones I can do the leg work.
::::::I believe the thing that needs to be looked at is there truly a problem with the choice of test and if so what is an alternative. Bayesian might be an option but seems to be difficult to employ for this situation.--[[User:Able806|Able806]] 12:36, 12 March 2009 (EDT)

:::Able806, you still seem to miss the point about how inappropriate the Monte Carlo method (as used in the Lenski paper) is for evaluating rarely occurring events. You need to open your mind to be productive. If you simply cling to a view that Lenski (who I don't think has any meaningful education in statistics) must somehow be right, then you're not going to make any progress in understanding the flaws.--[[User:Aschlafly|Andy Schlafly]] 17:07, 5 March 2009 (EST)

::::Andy, you still have not answered what you find inappropriate about his use of the Monte Carlo method? I am a reasonable person and with evidence I do have an open mind. I provided examples last week, with a working model, showing that Monte Carlo is better than the chi-square in this case. I have also shown where the Chi-Square was inappropriate due to the occurrence size as well. So if you have any evidence that Monte Carlo should not be used in the way that Lenski used please let it be shown.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

Sjohnson, I believe you just proved my point. In the literature of mean and covariance structure analysis, non-central chi-square distribution is commonly used to describe the behavior of the likelihood ratio statistic under alternative hypothesis; it is widely believed that the non-central chi-square distribution is justified by statistical theory. Actually, when the null hypothesis is not trivially violated, the non-central chi-square distribution cannot describe the LR statistic well even when data are normally distributed and the sample size is large. Monte Carlo results compare the strength of the normal distribution against that of the non-central chi-square distribution. In an association analysis comparing cases and controls with respect to allele frequencies at a highly polymorphic locus, a potential problem is that the conventional chi-squared test may not be valid for a large, sparse contingency table. Reliance on statistics with known asymptotic distribution is unnecessary, as Monte Carlo simulations can be performed to estimate the significance level of the test statistic.

Here is a [http://faculty.vassar.edu/lowry/chi_beta.html link] to a great page the provides an interactive example as to why the Chi Squared test would provide poor results compared to the Monte Carlo in relation to the Lenski data workup.

Something you may have overlooked was that the data set is actually too small to use the chi square method correctly. It is often accepted that is any of the analyzed data falls under 10 for a particular cell of the data set then the Yates correction needs to be applied; unfortunately the Yates correction can over correct thus skewing the p-value. Lenksi seemed to understand this by supporting his Monte Carlo p-value results with the Fisher z-transformation p-value.

I hope this helps.--[[User:Able806|Able806]] 10:27, 5 March 2009 (EST)

:I’m still waiting to hear which literature says that “Monte Carlo resampling” is “more accurate than the chi-squared test”. The page mentioned above [http://faculty.vassar.edu/lowry/chi_beta.html] is a discussion of why statisticians “fail to reject the null” rather than “accepting the null” when the p-value is above 0.05 or so. The page says nothing about superiority of Monte Carlo methods. Why were alternate hypothesis distributions mentioned? Only the null hypothesis distribution is used to calculate a p-value. Yates’s correction is for 2x2 contingency tables [http://en.wikipedia.org/wiki/Yates%27_correction_for_continuity]. It doesn’t apply in this case. Finally, what the heck do “covariance structure analysis” and “allele frequencies at a highly polymorphic locus” have to do with this problem? [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::SJohnson, I am looking for this paper for you, I cited it for one of my past publications dealing with allele frequencies (I believe it came from the Duke Biostatistics group). To answer your question about allele frequencies, that is the issue at hand, more about the genetics than the math, but it is the item being studied. So you stated that Yates can not be used and statistics says the number of occurrences is too small to evaluate using the Chi-Squared test so what would you recommend instead of the Monte-Carlo Method?

:Regarding the "Fisher z-transformation p-value" from the paper, garbage in garbage out. If the p-values were bad to begin with, then why would a combination of them be meaningful? [[User:SJohnson|SJohnson]] 10:49, 9 March 2009 (EDT)

::You are assuming that p-values are wrong based on a test that is inappropriate in this case due to data limitations. Did you perform a z-transformation on the chi-squared for the three data groups?--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

::You asked about the “Fisher z-transformation p-value”. The z-transformation test and Fisher’s method are actually two different things (see Whitlock's 2005 paper - Ref. 49 in Blount et al.). But no, I haven’t tried either. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::There's a large literature on various kinds of Monte Carlo test, a very short summary of which is that they're inevitably more accurate than parametric tests (e.g. F, t, chi-squared, etc) because they don't make assumptions about the distribution of the data under the null hypothesis. See for example ''Introduction to the Bootstrap'' by B. Efron and R. Tibshirani and ''The Jack-knife, the Bootstrap and Other Resampling Plans'', also by Efron. They're certainly applicable to small datasets and their accuracy is really only limited by the number of samples you care to take. E.g. 1000 M-C samples would give you a pretty accurate idea about significance at the alpha<1% level (That book should answer SJohnson's questions of 18:50 on 4/3/09 and 16:38 on 5/3/09 about accuracy and Aschalfly's comment of 17:07 on 5/3/09 about appropriateness of Monte Carlo tests.) [[User:FredFerguson|FredFerguson]] 16:53, 11 March 2009 (EDT)

::::Your claim that Monte Carlo methods are “inevitably more accurate” than other tests is obviously wrong because the accuracy of MC methods always depends on the number of realizations used. You should have written <math>\alpha=1\%</math>, not <math>\alpha<1\%</math>. If 1,000 random realizations are generated, the number of realizations above the true <math>\alpha=1\%</math> level is binomial with mean 10 and variance about 10. Thus, the standard deviation of the MC estimate is >0.003. In this example, a Monte Carlo p-value could be off by 30% and still be within a standard deviation. Is that really “pretty accurate”?

::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 3%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::::You're talking about miniscule differences in the accuracy of a test. 0.013 isn't very different from 0.007. In either case, it's very unlikely the experimenter would have obtained that result if the null hypothesis were true. If you're bothered about differences in P-values to the third decimals (which would make you unusual!), just run more MC realisations, that's all. Not really a problem. [[User:FredFerguson|FredFerguson]] 11:53, 12 March 2009 (EDT)

There’s still confusion about the difference between test statistics and Monte Carlo methods. Before you find a Monte Carlo estimate of a p-value, you need to select a test statistic to reduce the data set to a scalar. I am interested in hearing which test statistic you believe should be used in place of the chi-square test and why. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

----

Quick question for SJohnson: How many degrees of freedom did you choose when calculating the p-value? I'd like to know upon what condition you base that number. Thanks.--[[User:Argon|Argon]] 11:05, 5 March 2009 (EST)

:The degree of freedom for a contingency table is rows minus one times columns minus one. That is, <math> (r-1)(c-1) </math>. Here’s a pretty good tutorial I came across: [http://faculty.uncfsu.edu/dwallace/lesson%2020.pdf]. For the experiments from [http://www.pnas.org/content/105/23/7899.full.pdf], the DOFs are 11, 11, and 13. For experiment one, the chi-square test statistic is
::<math>
X^2
=\sum\limits_i\sum\limits_j
\frac{\left(n_{i,j}-E\left[n_{i,j}\right]\right)^2}
{E\left[n_{i,j}\right]}
</math>
::<math>
=\frac{\left(0-1/3\right)^2}{1/3}
+\frac{\left(6-17/3\right)^2}{17/3}
+\frac{\left(0-1/3\right)^2}{1/3}
+\ldots+
</math>
::<math>
+\frac{\left(2-1/3\right)^2}{1/3}
+\frac{\left(4-17/3\right)^2}{17/3}
</math>
::<math>
\approx
14.82
</math>
:where <math>n_{i,j}</math> is the observed value and <math>E\left[n_{i,j}\right]</math> is the expected null hypothesis value. So if you have MS Excel, another way to arrive at the p-value of 0.19 is to type “=CHIDIST(14.82,11)” into a cell. Cheers! [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::OK, thanks for the info. From what I'd calculated and looked up in tables, the numbers seemed close to a df=11 for a chi-square of ~14. (Aside: With terms having 17/3 in the denominator in the figures above, were you using the test of independence? I was using Pearson's test for [http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_for_fit_of_a_distribution fit of a distribution] which returns a chi-squared value of 14 and roughly matched the p-values you reported, assuming the df was 11).

::Also, the first sentence of the article reads: "Blount, Borland, and Lenski[1] claimed that a key evolutionary innovation was observed during a laboratory experiment. That claim is false." A small correction: There were several claims in the paper. The 'key evolutionary innovation' was acquiring the ability to utilize citrate as a food source. That claim was demonstrated multiple times. The claim, which pertains to this statistics discussion was that the Cit+ phenotype arose in a multi-step process, first requiring a rare, pre-adaptive mutation before additional mutation(s) lead to the subsequent development of citrate utilization.--[[User:Argon|Argon]] 20:46, 5 March 2009 (EST)

:::My biology-degreed wife assures me that mutation does not necessarily mean that evolution occurred. What the paper claimed is that evolution (a “key innovation”) occurred in the lab. The key innovation supposedly increased the mutation rate. In the experiments, the observed mutation rate increased after generation 31,000, but not enough to make a statistically significant claim that the rate is not constant. The analysis in the paper was similar to flipping a coin ten times, counting six heads and claiming that the coin must be biased against tails. In reality, there’s nothing surprising about a fair coin producing slightly more of one outcome than the other. Just like there's nothing surprising about there being slightly more mutations in later generations than early generations given the null hypothesis (constant mutation rate). [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

::::SJohnson, not to say anything about your wife, but has she had a 400 level molecular genetics course (most general biology degrees do not cover the detail unless they are specialized)? If so, she would have mentioned that if the mutation passes to the offspring and is selectively beneficial to the population then it is a step of evolution as along as the conditions continue through the sharing of the mutation with the population and the environment is such that reduces the growth rate of the non-transformed population. While not all mutations are signs that evolution occurred the mutations that pass to offspring and provide a benefit compared to other offspring are very strong indicators. In the case of this paper the population that evolved the cit+ was able to metabolize a chemical in their environment which allowed for an adaptation advantage compared to the non-transformed colonies.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

----

Let’s go back to the beginning. There appears to be confusion about the difference between test statistics and methods for computing p-values. As is noted at the beginning of the page [http://www.conservapedia.com/Significance_of_E._Coli_Evolution_Experiments], the fundamental problem with the paper is that it used a flawed test statistic, not that it used Monte Carlo methods to find the p-value for that flawed statistic.

Every hypothesis test uses a test statistic to reduce the data to a single number. The p-value for the test statistic can be calculated analytically (as I’ve done for the chi-square test statistic) or by Monte Carlo methods. In the paper, Monte Carlo methods were used to compute the p-value of the “mutation generation” test statistic. The key problem with the analysis from the paper is that it doesn’t work to use a weighted average to test for variations in mutation rate. This is like trying to use the sample variance to test for an increase in the mean in Gaussian-distributed data. A statistic should be selected based on the null and alternate hypothesis distributions of the data. The chi-square test (unlike the weighted average from the paper) is a reasonable choice for data that mutates at a constant rate under the null hypothesis, but mutates at varying rates under the alternate hypothesis.

Able806, you made a good point about the contingency table cell frequencies being relatively low, but were wrong when you said ”the data set is actually too small to use the chi square method correctly”. In the low cell frequency case the chi-square test is still effective, but the null hypothesis distribution of the chi-square statistic starts to look less like the chi-square distribution. Thus, p-values calculated using the chi-square distribution may be a bit off. However, Monte Carlo p-values are always imperfect as well because it's impossible to generate an infinite number of random realizations. There are imperfections in p-values generated by analytic and Monte Carlo methods. However, low cell frequencies does not explain the >20x and >2.5x differences between chi-square p-values and p-values from the paper for experiments one and three. The reason for those huge differences was the use of the flawed test statistic (“mutation generation”) in the paper. [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

:SJohnson, the chi-squared test is a valuable statistical tool, but the limitations of the test must be acknowledged. The chi-squared test can only produce valid results if the assumptions that underly the test are not violated. As an analogy, Newtonian models of motion fail to produce accurate results as velocities approach the speed of light; under those circumstances one must switch to a theory that accounts for relativistic effects.

:It seems that you have simply dismissed the [http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm widely-acknowledged] [http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm fact] that the [http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html chi-squared test] is [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236 inappropriate] for use in [http://www.graphpad.com/www/Book/Choose.htm situations] where n in any cell is [http://mysite.du.edu/~jcalvert/econ/chisquar.htm less] less than a [http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1 threshold] [http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html number]. Different authors set different thresholds, but all are well above the numbers seen in your chi-squared analysis - even the most liberal guidelines advise against the chi-squared test when any expected cell frequency is less than one or more than 20% of the table cells are less than 5; others require that expected values in all cells must be more than 5. With smaller amounts of data, the test is insensitive and errs on the side of rejecting the hypothesis. If you attempt your chi-squared statistical analysis with a program that is more sophisticated than MS Excel (as I did), you get an error message indicating that the results are invalid due to low expected cell counts.

:That issue aside, there are other reasons that the chi-squared test is inappropriate here. As the links above point out, the categories tested must be truly independent; one example is that you can't use the chi-squared test to compare age and ability to kick a field goal by testing the same experimental group twice, one year apart; you have to test one group of age A and a different group of age B. In the case of the Blount paper, the categories are not independent. Even if there were adequate numbers to address the low-expected-frequency problem, this would make the chi-squared an invalid test in this case.

:There are other significant problems with the use of the chi-squared test in this circumstance, but they can wait until you address these first major problems.--[[User:ElyM|ElyM]] 12:18, 11 March 2009 (EDT)

::Wackerly et al. says in general it’s assumed that the cell frequencies are above five so that the chi-square statistic (under the null) is approximately chi-square distributed (see p. 703). That book does not say chi-square test results are invalid if frequencies are five or less. Your example of a chi-square test warning message (it said "warning" not "error" as you stated) in Minitab [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236] said “approximation probably invalid” referring to the chi-square distribution approximation to the chi-square test statistic’s distribution. Your example did not say “chi-square test invalid”. I agree that when cell frequencies are low, the chi-square test statistic’s distribution starts to deviate from the chi-square distribution. I maintain that this deviation is not enough to explain the >2.5x and >20x differences in the chi-square test p-values and the p-values from the paper.

::As the numerous links in your post proved, the chi-square test is widely-used by statisticians. Can you give examples of statisticians using mean mutation generation as a test statistic? Also, did your software agree with the chi-square test p-values I presented? Thanks. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::Thank you for giving page references for Wackerly; however it seems we have different editions, since page 703 in my copy (5th ed, 1996) does not deal with chi-squared issues at all. My copy does state the following, on page 622: "Although the mathematical proof is beyond the scope of this text, it can be shown that, when n is large [chi-squared] will possess approximately a chi-square probability distribution in repeated sampling." Then, on page 624: "Experience has shown that cell counts [n sub i] should not be too small in order that the chi-square distribution provide an accurate approximation to the distribution of [chi squared]. As a rule of thumb we require that all expected cell counts equal or exceed 5, although Cochran (1952) has noted that this value can be as low as 1 for some situations." Wackerly then goes on, in the problems sections, to describe the use of the chi-squared test as a "violation of good statistical practice" when "some expected counts [are] <5."

:::It seems that you are already aware that the [chi-square] statistic under the null is no longer chi-square distributed for small n; this is precisely why the test should not be used under those conditions. I can claim to be able to accelerate a 1-kg mass to 10 times the speed of light by applying 1 N of force for 95 years by using F=ma and t= (vf-vi)/a. Plugging the numbers into those equations will produce the same result every time, but the answer is illegitimate because those equations are only valid under certain assumptions, which are violated as velocities approach the speed of light. Similarly, having a statistical program calculate a chi-squared value given the Blount data will produce a number result, but since the assumptions of the test are violated the result is not legitimate. Yes, if I put the Blount data in SAS 9.2, I get the same numerical answer as you do, but I also get the following message: "WARNING: >89% of the cells have expected counts less than 5. Chi-square may not be a valid test." You may argue that that's a warning, not an error; that's a semantic distinction. The reason that the program says that it MAY not be valid is that the chi-squared test skews in the direction of being too conservative at low n values; the test has an acceptable rate of false positives but an unacceptably high rate of false negatives. Comparing the results of the Monte Carlo and chi-squared results in this case is like comparing the results of Newtonian and relativistic equations of motion: they can produce very different results from the same input data.

:::Your last paragraph has a major non sequitur in it: yes, many statisticians use the chi-square test. As long as the assumptions of the test are not violated, it is a valuable tool. That has nothing to do with the validity of using mean mutation generation as a test statistic. 'Mean number of werewolf attacks in Mumbai in the week centered on the new moon, by month, from 1654 to 1798' is a valid test statistic. I am quite sure that it has never been used in a peer-reviewed paper before. That does not mean that I can't perform valid statistical tests on that statistic. If, however, the incorrect test is applied, the results of the analysis will be flawed. Papers apply a (relatively small) standard repertoire of valid tests to a (potentially infinite) number of test statistics. The particular test statistic used in a paper may never have been used before and may never be used again; that does not address the validity of the analysis. In Blount's case, the test is the Monte Carlo analysis, which is also "widely-used by statisticians".

:::We still haven't touched on the issue of the categories not being independent, which by itself is sufficient to invalidate the chi-squared technique. I'm new to this site, so I'm unsure as to the etiquette of making changes to the articles of another person - but the article here should at the very least mention that the chi-square test is being used here in a manner that violates its underlying assumptions in at least two fundamental ways, and the results are therefore suspect.--[[User:ElyM|ElyM]] 17:34, 12 March 2009 (EDT)

:::: It looks to me as though SJohnson has misinterpreted the application of the chi-squared test in quite a fundamental way. His/her analysis of Blount's data are therefore close to meaningless, regardless of whether the test used by Blount is appropriate or not. In my opinion, the entire page should therefore be deleted. [[User:FredFerguson|FredFerguson]] 08:18, 13 March 2009 (EDT)

::::: "Fred", perhaps you mistakenly think this is Wikipedia, where [[censorship]] and deletion of pages for ideological reasons are common. Not here.--[[User:Aschlafly|Andy Schlafly]] 10:23, 14 March 2009 (EDT)

:::::: Umm... I'm suggesting deletion for mathematical reasons, not ideological reasons. Using an argument filled with mathematical errors to try to support your case only detracts from your credibility. [[User:FredFerguson|FredFerguson]] 10:38, 14 March 2009 (EDT)

:::::: Actually, I think correction is better than deletion. So that's what I've done. [[User:FredFerguson|FredFerguson]] 11:01, 14 March 2009 (EDT)

::::::: I find no credibility in your denial of having ideological reasons.--[[User:Aschlafly|Andy Schlafly]] 11:04, 14 March 2009 (EDT)

== Misinterpretation of test ==

SJohnson, Your analysis misinterprets the test. You say the null hypothesis is that this mutation cannot happen. They saw a mutation (4 mutations, in fact, in the data set you show) so the null hypothesis (as you state is) is disproved. That's perfectly straightforward.

I don't know what the "mean mutation generation" test is but you're doing when you apply a chi-squared test to this dataset is to test if the mutations are evenly distributed throughout the generations. Your test says they are, so there's no strong evidence to suppose that mutations are likely to occur in one generation rather than another in the series of tests. Blount's test says thay aren't, so it's more likely that the mutation will occur later in the series of tests. I can't tell which test is right without knowing more about the test that Blount used.

But that point (the foregoing paragraph) has no bearing at all on the null hypothesis, as you describe it. The mutation appeared, so that means the hypothesis that the mutation can't happen is disproved. Very simple. [[User:FredFerguson|FredFerguson]] 21:10, 8 March 2009 (EDT)

:I never said that “the null hypothesis is that this mutation cannot happen”. The chi-square test statistic I'm using wouldn’t be defined if the null hypothesis mutation rate was zero because the <math>E\left[n_{i,j}\right]</math> term in the denominator of the statistic (see above equation) would be zero.

:The test statistic from the paper is the average of the generation numbers of observed mutations. For experiment one this number is
::<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)= 31750.
</math>
:The same number is shown in Table 2 of the paper. [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

:: SJohnson, the way you're calculating the chi-squared statistic implies that you're testing the null hypothesis of a constant mutation rate over time against an alternative hypothesis of a mutation rate which varies over time. [[User:FredFerguson|FredFerguson]] 11:02, 9 March 2009 (EDT)

As it currently stands, the article makes the following statement: "The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown." This misstates the null hypothesis of the paper, which is elaborated in the Introduction section of the paper, and repeated in the section '''Statistical Analysis of the Replay Experiments''':
<blockquote>
For each experiment, we compared the observed mean generation of those clones that yielded Cit+ variants to the mean expected under the null hypothesis that clones from all generations have equal likelihood. The null thus corresponds to the rare-mutation hypothesis laid out in the Introduction."
Block quote
</blockquote><ref>www.pnas.org/cgi/reprint/105/23/7899.pdf</ref>

The article also continues to describe 'mean mutation generation' as a ''test'' rather than a ''statistic'' to which the ''Monte Carlo test'' was applied.--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

==Unreferenced Claims==

I deleted the claim that mean mutation generation is an appropriate test statistic because no reference was produced that back that claim. No reference was provided to back the claim that the chi-square test p-values are always conservative, either. [[User:SJohnson|SJohnson]] 12:57, 14 March 2009 (EDT)

: The reference is Everitt. I'll check I put it in the right place. [[User:FredFerguson|FredFerguson]] 13:30, 14 March 2009 (EDT)

There was a typo in my edit summaries on the talk page and the main page. I meant to say "Removed unsupported claims" rather than "Removed supported claims". [[User:SJohnson|SJohnson]] 13:13, 14 March 2009 (EDT)

I do not believe that anyone has claimed that 'chi-square test p-values are ''always'' conservative'. The claim that has been made is that ''under certain circumstances'', namely low n and low individual cell values, the chi-square test is an invalid test; that under those circumstances the power of the test is low and it becomes impossible to reject the null hypothesis even when it is false. You may have missed the pertinent sections in my links above, so I will directly quote the relevant sections. All the quoted sections refer to chi-square testing in particular. Any bolding below is mine.
<blockquote>
"Assumptions: 

Even though a nonparametric statistic does not require a normally distributed population, there still are some restrictions regarding its use. 

1. Representative sample (Random) 
2. The data must be in frequency form (nominal data) or greater. 
3. The individual observations must be independent of each other. 
4. '''Sample size must be adequate. In a 2 x 2 table, Chi Square should not be used if n is less than 20. In a larger table, no expected value should be less than 1, and not more than 20% of the variables can have expected values of less than 5'''. 
5. Distribution basis must be decided on before the data is collected. 
6. The sum of the observed frequencies must equal the sum of the expected frequencies."
</blockquote>
<ref>http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm
</ref>

<blockquote>
"Assumptions:
* Random sample data are assumed. As with all significance tests, if you have population data, then any table differences are real and therefore significant. If you have non-random sample data, significance cannot be established, though significance tests are nonetheless sometimes utilized as crude "rules of thumb" anyway.
* A sufficiently large sample size is assumed, as in all significance tests. '''Applying chi-square to small samples exposes the researcher to an unacceptable rate of Type II errors. There is no accepted cutoff. Some set the minimum sample size at 50, while others would allow as few as 20'''. Note chi-square must be calculated on actual count data, not substituting percentages, which would have the effect of pretending the sample size is 100.
* '''Adequate cell sizes are also assumed. Some require 5 or more, some require more than 5, and others require 10 or more. A common rule is 5 or more in all cells of a 2-by-2 table, and 5 or more in 80% of cells in larger tables, but no cells with zero count'''. When this assumption is not met, Yates' correction is applied.
* Independence. Observations must be independent. The same observation can only appear in one cell. '''This means chi-square cannot be used to test correlated data (ex., before-after, matched pairs, panel data)'''.
* Similar distribution. Observations must have the same underlying distribution.
* Known distribution. The hypothesized distribution is specified in advance, so that the number of observations that are expected to appear each cell in the table can be calculated without reference to the observed values. Normally this expected value is the crossproduct of the row and column marginals divided by the sample size.
* Non-directional hypotheses are assumed. Chi-square tests the hypothesis that two variables are related only by chance. If a significant relationship is found, this is not equivalent to establishing the researcher's hypothesis that A causes B, or that B causes A.
* Finite values. Observations must be grouped in categories.
* Normal distribution of deviations (observed minus expected values) is assumed. Note chi-square is a nonparametric test in the sense that is does not assume the parameter of normal distribution for the data -- only for the deviations.
* Data level. No assumption is made about level of data. Nominal, ordinal, or interval data may be used with chi-square tests."
</blockquote>
<ref>http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm</ref>

<blockquote>
"Assumptions: 
-None of the expected values may be less than 1 
-No more than 20% of the expected values may be less than 5"</blockquote> <ref>http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html</ref>

<blockquote>
"When performing a chi-square test, your data must satisfy important assumptions. Although these assumptions may be stated differently in different textbooks, they generally assert that: 
1)The sample must be randomly drawn from the population 
'''2)The sample size, n, must be large enough so that the expected cell count in each cell is greater than or equal to 5.''' 
Both assumptions must be met in the process of collecting your data, and violations of the second assumption will appear in the Minitab output when you run the analysis. 
... 
'''You may wonder why the second assumption is necessary for performing the chi-square test. The second assumption arises because the distribution of counts under the null hypothesis is multinomial, and the normal distribution can be used to approximate the multinomial distribution if the sample size is sufficiently large and the probability parameters aren't too small. It can be shown via the Central Limit Theorem that the multinomial distribution converges to the normal distribution as the sample size approaches infinity; however, there is no easy way to show mathematically how and when the convergence fails.'''"
</blockquote>
<ref>http://www.minitab.com/support/docs/Answers/Chi-Square%20Test%20Assumptions.pdf</ref>

<blockquote>
"'''The chi-square test is simpler to calculate but yields only an approximate P value. ... You should definitely avoid the chi-square test when the numbers in the contingency table are very small (any number less than about six)'''."
</blockquote> <ref>http://www.graphpad.com/www/Book/Choose.htm</ref>

<blockquote>
"The most important things to remember to get a valid χ2 test are that the expected values are not too small in any bin (certainly 5 or more), and that the degrees of freedom are properly evaluated. '''Unless you have a very large amount of data, the test is not very sensitive and errs on the side of safety. If you get a significant result, however, it is not likely to be wrong.'''"
</blockquote> <ref>http://mysite.du.edu/~jcalvert/econ/chisquar.htm</ref>

<blockquote>
"The critical assumptions of the chi-square test for k independent samples are similar to those for the chi-square test for two independent samples. ... 
'''4. No more than 20% of the cells may have expected frequencies of less than 5, and no cell should have an expected frequency of less than 1. 
The rule given in Assumption 4 is particularly important for a contingency table that is larger than 2X2'''"
</blockquote> <ref>http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1
</ref>

<blockquote>
"Special problems with small expected cell frequencies for the chi-square test: 
The chi-square test involves using the chi-square distribution to approximate the underlying exact distribution. The approximation becomes better as the expected cell frequencies grow larger, and '''may be inappropriate for tables with very small expected cell frequencies.''' 
'''For tables with expected cell frequencies less than 5, the chi-square approximation may not be reliable. A standard (and conservative) rule of thumb (due to Cochran) is to avoid using the chi-square test for tables with expected cell frequencies less than 1, or when more than 20% of the table cells have expected cell frequencies less than 5.''' 
Another rule of thumb (due to Roscoe and Byars) is that the average expected cell frequency should be at least 1 when the expected cell frequencies are close to equal, and 2 when they are not. (If the chosen significance level is 0.01 instead of 0.05, then double these numbers.) 
Koehler and Larntz suggest that if the total number of observations is at least 10, the number categories is at least 3, and the square of the total number of observations is at least 10 times the number of categories, then the chi-square approximation should be reasonable. 
Care should be taken when cell categories are combined (collapsed together) to fix problems of small expected cell frequencies. Collapsing can destroy evidence of non-independence, so a failure to reject the null hypothesis for the collapsed table does not rule out the possibility of non-independence in the original table. 
'''As with most statistical tests, the power of the chi-square test increases with a larger number of observations. If there are too few observations, it may be impossible to reject the null hypothesis even if it is false.'''"
</blockquote> <ref>http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html</ref>--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

: Thanks for this really excellent contribution, ElyM. The only thing I'd like to add is in relation to your initial statement, "I do not believe that anyone has claimed that 'chi-square test p-values are always conservative'". The question of whether a test is conservative in a particular situation is probabilistic. One can determine whether a test is likely to generate a p-value which is too high in a particular situation (e.g. for a chi-squared test, when there are lots of small expected values) but one needs an exact test (such as an appropriate Monte Carlo randomisation test) to determine whether the p-value in any ''particular'' test is in fact excessively high.

: I hope careful reading of your very clear description will put SJohnson's mind at rest on this subject. [[User:FredFerguson|FredFerguson]] 18:11, 14 March 2009 (EDT)

ElyM, you've provided nothing to address the basic flaw that "The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events." See [[Flaws in Lenski Study]]. Also, do not impose your view on the content page until after SJohnson has had an opportunity to respond to your posting. As to "Fred", his put-downs are getting tiresome and I'm going to review his edit pattern now to see if he's been contributing anything of value to this site.--[[User:Aschlafly|Andy Schlafly]] 14:06, 15 March 2009 (EDT)

::Mr. Schlafly, per your request I have not added anything to the content page as SJohnson has not yet responded to my posts. Since all of my comments have been in regards to SJohnson's use of the chi-square test in this particular article, I'm not sure why you expect me to address Blout's use of Monte Carlo - that issue seems to be addressed on the [[Flaws in Lenski Study]] page. SJohnson has added a reformulation of the chi-square test for two possible outcomes, and stated that the chi-square test is at a minimum when all success probabilities are equal. He then extrapolates from this to claim that the chi-square test is an effective test for the data from Blount.

::The reformulation of the equations for two possible outcomes does not address the underlying problem that the chi-square test has universally accepted parameters outside of which it is considered an invalid test; I have provided references for these parameters and shown that the data from Blount lies outside them. None of the expected cells in SJohnson's analysis have values above one, and the total n is four. SJohnson's own reference states that the application of the chi-square test in this circumstance is a "violation of good statistical practice". Analogously, combining F=ma and t=(vf-vi)/a into t=(vf-vi)m/F and showing that t is a minimum when m approaches zero does not address the fact that those Newtonian equations do not apply as velocities approach the speed of light. The legitimacy of Blount's arguments cannot be determined by the application of illegitimate counterarguments. If SJohnson or others can point to references from the statistical literature that show that Blount has made methodological errors - as I have been able to do with SJohnson's chi-square analysis - I would welcome their input, and no doubt Conservapedia's other readers would as well, and this page would be greatly improved.

::I have not seen a rebuttal from SJohnson in the four days since my last post, although he has added new material to the content page since then. In light of this, I would appreciate some guidelines as to when it is appropriate for me to add my information and references to the content page. I can add citations from the primary mathematical literature if necessary, but in general I find that these are less helpful as they are not easily accessible by readers without access to academic libraries.--[[User:ElyM|ElyM]] 18:07, 18 March 2009 (EDT)

== References ==
{{reflist}}

Significance of E. Coli Evolution Experiments

2009-03-16T01:12:49Z

SJohnson:

Blount, Borland, and [[Lenski]]<ref> http://www.pnas.org/content/105/23/7899.full.pdf</ref> claimed that a key evolutionary innovation was observed during a [[laboratory]] [[experiment]]. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.

==Test Statistics==

The test statistic used in Blout, Borland, and Lenski was mean mutation generation. For example, the experiment one mean mutation generation is (see Tables 1 and 2 of the paper)
:<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)
=31750.
</math>

The null hypothesis from the paper was constant mutation rate over all generations. However, measurements of mean mutation
generation can fail to observe deviations from that null
hypothesis. Consider an experiment where the mutation
probabilities for generations 1, 2, and 3 are <math>p_1</math>, <math>p_2</math>, and
<math>p_3</math>, respectively. If the mutation probabilities per generation
are <math>p_1=p_2=p_3=p</math>, then the mean mutation generation is 2.
However, if the mutation probabilities are <math>p_1=p_3=p/2</math> and
<math>p_2=2p</math>, then the mean mutation generation is still 2. In the
latter example the experiment has deviated from the null
hypothesis, but the mean mutation
generation is insensitive to the
change. Because this test can fail to observe deviations from the null hypothesis, levels of statistical significance computed using this test (p-values) are meaningless.

If the trials from an experiment have two outcomes (e.g. success or failure), then the chi-square test for independence can be written
:<math>
X^2
=\sum\limits_{i=1}^M
\frac{\left(x_i-\hat{p}N_i\right)^2}
{\hat{p}(1-\hat{p})N_i}
</math>
where <math>x_i</math> is the number of successes (e.g. mutations) in the ''i''-th experiment, <math>N_i</math> is the number of trials in the ''i''-th experiment, and
:<math>
\hat{p}=\frac{\sum\limits_{i=1}^Mx_i}{\sum\limits_{i=1}^MN_i}
</math>
is the fraction of trials that are successful in the all experiments. The chi-square test is, on average, at a minimum when all success probabilities are equal (<math>p_i=p</math> for <math>i=1,2,\ldots,M</math>) and will, on average, increase whenever the data follows any other hypothesis (<math>\sum\limits_{i=1}^M
\left(p_i-\bar{p}\right)^2>0</math> where <math>\bar{p}</math> is the mean of the success probabilities). Thus the chi-square test is an effective hypothesis test for the data and hypotheses from Blount et al.

==Experiment One Data==

The data from experiment one of the paper is shown below (see Table 1 of the paper). The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown.

{|class="wikitable" style="text-align:center"
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|6
|0
|6
|0.333
|5.667
|-
|10000
|6
|0
|6
|0.333
|5.667
|-
|20000
|6
|0
|6
|0.333
|5.667
|-
|25000
|6
|0
|6
|0.333
|5.667
|-
|27500
|6
|0
|6
|0.333
|5.667
|-
|29000
|6
|0
|6
|0.333
|5.667
|-
|30000
|6
|0
|6
|0.333
|5.667
|-
|30500
|6
|1
|5
|0.333
|5.667
|-
|31000
|6
|0
|6
|0.333
|5.667
|-
|31500
|6
|1
|5
|0.333
|5.667
|-
|32000
|6
|0
|6
|0.333
|5.667
|-
|32500
|6
|2
|4
|0.333
|5.667
|-
|-
!Total
!72
!4
!68
!4
!68
|}

When the flawed test is used to compute the significance of this data, the p-value is 0.0085 (see Table 2 of the paper). This p-value is considered statistically significant. However, when the data is analyzed using a standard method (the chi-square test) the p-value is 0.19. This p-value is much larger than the one from the paper and indicates that there is no reason to reject the null hypothesis. The chi-square test p-value for experiment two is small (0.0004). However, experiment three is not statistically significant because its p-value is 0.22.

The chi-square test is a common statistical method.<ref>''Mathematical Statistics with Applications'' by Wackerly, Mendenhall, and Scheaffer, Section 14.4.</ref> It can be implemented in Microsoft Excel. If the numbers from the last four columns of the experiment one data table (excluding the “totals” row) are entered into Excel in rows 1-12 and columns A-D, then the p-value can be computed by entering “=CHITEST(A1:B12,C1:D12)” into any empty cell of the spreadsheet.

==Experiment Three Data==

The experiment three data from Blount et al. is shown in the table below. The expected numbers of mutants under the null hypothesis (constant mutation rate) is also shown.

{|class="wikitable" style="text-align:center"
|-
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|200
|0
|200
|0.571
|199.429
|-
|10000
|200
|0
|200
|0.571
|199.429
|-
|20000
|200
|0
|200
|0.571
|199.429
|-
|25000
|200
|0
|200
|0.571
|199.429
|-
|27500
|200
|2
|198
|0.571
|199.429
|-
|29000
|200
|0
|200
|0.571
|199.429
|-
|30000
|200
|2
|198
|0.571
|199.429
|-
|30500
|200
|0
|200
|0.571
|199.429
|-
|31000
|200
|0
|200
|0.571
|199.429
|-
|31500
|200
|0
|200
|0.571
|199.429
|-
|32000
|200
|1
|199
|0.571
|199.429
|-
|32500
|200
|1
|199
|0.571
|199.429
|-
|-
!Total
!2800
!8
!2792
!8
!2792
|}

==Comparison of p-Values==

The following table compares the p-values reported in Table 2 of Blount et al. to the chi-square p-values for the same experiments. For experiments one and three, the chi-square p-values are much larger than the "mean generation" test p-values from the paper.

{|class="wikitable" style="text-align:center"
|-
|
!Experiment 1
!Experiment 2
!Experiment 3
|-
!p-Value from Paper
|0.0085
|0.0007
|0.082
|-
!Chi-square p-value
|0.19
|0.0004
|0.22
|}

==References==
<references/>
[[Category:Conservapedia Dealings with PNAS and Lenski]]
[[Category:Statistics]]

==See Also==
http://www.sciencenews.org/index/feature/activity/view/id/40006/title/Molecular_Evolution

http://sciencenews.org/view/generic/id/40649/title/FOR_KIDS_Hitting_the_redo_button_on_evolution

Significance of E. Coli Evolution Experiments

2009-03-15T19:48:58Z

SJohnson: Added section on test statistics and described why the test from the paper doesn't work

Blount, Borland, and [[Lenski]]<ref> http://www.pnas.org/content/105/23/7899.full.pdf</ref> claimed that a key evolutionary innovation was observed during a [[laboratory]] [[experiment]]. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.

==Test Statistics==

The test statistic used in Blout, Borland, and Lenski was mean mutation generation. For example, the experiment one mean mutation generation is (see Tables 1 and 2 of the paper)
:<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)
=31750.
</math>

The null hypothesis from the paper was constant mutation rate over all generations. However, measurements of mean mutation
generation can fail to observe deviations from that null
hypothesis. Consider an experiment where the mutation
probabilities for generations 1, 2, and 3 are <math>p_1</math>, <math>p_2</math>, and
<math>p_3</math>, respectively. If the mutation probabilities per generation
are <math>p_1=p_2=p_3=p</math>, then the mean mutation generation is 2.
However, if the mutation probabilities are <math>p_1=p_3=p/2</math> and
<math>p_2=2p</math>, then the mean mutation generation is still 2. In the
latter example the experiment has deviated from the null
hypothesis, but the mean mutation
generation is insensitive to the
change. Because this test can fail to observe deviations from the null hypothesis, levels of statistical significance computed using this test (p-values) are meaningless.

If the trials from an experiment have two outcomes (e.g. success or failure), then the chi-square test for independence can be written
:<math>
X^2
=\sum\limits_{i=1}^M
\frac{\left(x_i-\hat{p}N_i\right)^2}
{\hat{p}(1-\hat{p})N_i}
</math>
where <math>x_i</math> is the number of successes (e.g. mutations) in the ''i''-th experiment, <math>N_i</math> is the number of trials in the ''i''-th experiment, and
:<math>
\hat{p}=\frac{\sum\limits_{i=1}^Mx_i}{\sum\limits_{i=1}^MN_i}
</math>
is the fraction of trials that are successful in the all experiments. The chi-square test is, on average, at a minimum when all success probabilities are equal (<math>p_i=p</math> for <math>i=1,2,\ldots,M</math>) and will, on average, increase whenever the data follows any other hypothesis (<math>\sum\limits_{i=1}^M
\left(p_i-\bar{p}\right)^2>0</math>) where <math>\bar{p}</math> is the mean of the success probabilities. Thus the chi-square test is an effective hypothesis test for the data and hypotheses from Blount et al.

==Experiment One Data==

The data from experiment one of the paper is shown below (see Table 1 of the paper). The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown.

{|class="wikitable" style="text-align:center"
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|6
|0
|6
|0.333
|5.667
|-
|10000
|6
|0
|6
|0.333
|5.667
|-
|20000
|6
|0
|6
|0.333
|5.667
|-
|25000
|6
|0
|6
|0.333
|5.667
|-
|27500
|6
|0
|6
|0.333
|5.667
|-
|29000
|6
|0
|6
|0.333
|5.667
|-
|30000
|6
|0
|6
|0.333
|5.667
|-
|30500
|6
|1
|5
|0.333
|5.667
|-
|31000
|6
|0
|6
|0.333
|5.667
|-
|31500
|6
|1
|5
|0.333
|5.667
|-
|32000
|6
|0
|6
|0.333
|5.667
|-
|32500
|6
|2
|4
|0.333
|5.667
|-
|-
!Total
!72
!4
!68
!4
!68
|}

When the flawed test is used to compute the significance of this data, the p-value is 0.0085 (see Table 2 of the paper). This p-value is considered statistically significant. However, when the data is analyzed using a standard method (the chi-square test) the p-value is 0.19. This p-value is much larger than the one from the paper and indicates that there is no reason to reject the null hypothesis. The chi-square test p-value for experiment two is small (0.0004). However, experiment three is not statistically significant because its p-value is 0.22.

The chi-square test is a common statistical method.<ref>''Mathematical Statistics with Applications'' by Wackerly, Mendenhall, and Scheaffer, Section 14.4.</ref> It can be implemented in Microsoft Excel. If the numbers from the last four columns of the experiment one data table (excluding the “totals” row) are entered into Excel in rows 1-12 and columns A-D, then the p-value can be computed by entering “=CHITEST(A1:B12,C1:D12)” into any empty cell of the spreadsheet.

==Experiment Three Data==

The experiment three data from Blount et al. is shown in the table below. The expected numbers of mutants under the null hypothesis (constant mutation rate) is also shown.

{|class="wikitable" style="text-align:center"
|-
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|200
|0
|200
|0.571
|199.429
|-
|10000
|200
|0
|200
|0.571
|199.429
|-
|20000
|200
|0
|200
|0.571
|199.429
|-
|25000
|200
|0
|200
|0.571
|199.429
|-
|27500
|200
|2
|198
|0.571
|199.429
|-
|29000
|200
|0
|200
|0.571
|199.429
|-
|30000
|200
|2
|198
|0.571
|199.429
|-
|30500
|200
|0
|200
|0.571
|199.429
|-
|31000
|200
|0
|200
|0.571
|199.429
|-
|31500
|200
|0
|200
|0.571
|199.429
|-
|32000
|200
|1
|199
|0.571
|199.429
|-
|32500
|200
|1
|199
|0.571
|199.429
|-
|-
!Total
!2800
!8
!2792
!8
!2792
|}

==Comparison of p-Values==

The following table compares the p-values reported in Table 2 of Blount et al. to the chi-square p-values for the same experiments. For experiments one and three, the chi-square p-values are much larger than the "mean generation" test p-values from the paper.

{|class="wikitable" style="text-align:center"
|-
|
!Experiment 1
!Experiment 2
!Experiment 3
|-
!p-Value from Paper
|0.0085
|0.0007
|0.082
|-
!Chi-square p-value
|0.19
|0.0004
|0.22
|}

==References==
<references/>
[[Category:Conservapedia Dealings with PNAS and Lenski]]
[[Category:Statistics]]

==See Also==
http://www.sciencenews.org/index/feature/activity/view/id/40006/title/Molecular_Evolution

http://sciencenews.org/view/generic/id/40649/title/FOR_KIDS_Hitting_the_redo_button_on_evolution

Significance of E. Coli Evolution Experiments

2009-03-14T17:18:09Z

SJohnson: Typo in my previous edit summary. It should have said, "Removed unsupported claims."

Talk:Significance of E. Coli Evolution Experiments

2009-03-14T17:13:55Z

SJohnson: /* Unreferenced Claims */ Removed unsupported claims

Talk:Significance of E. Coli Evolution Experiments

2009-03-14T16:57:11Z

SJohnson: Removed supported claims.

Significance of E. Coli Evolution Experiments

2009-03-14T16:57:06Z

SJohnson: Removed supported claims.

User talk:FredFerguson

2009-03-14T13:59:27Z

SJohnson:

{{welcome|sig=-[[User:Foxtrot|Foxtrot]] 09:15, 13 March 2009 (EDT)}}

==Warning==

Please do not throw around accusations lightly. Someone has taken the time to create a needed page and discuss it with you -- it is rather arrogant and disrespectful to then say the page needs to be deleted. -[[User:Foxtrot|Foxtrot]] 09:15, 13 March 2009 (EDT)

: The numerous mistakes in the page in question have been thoroughly discussed by me and others. It's not arrogant to say a page should be deleted when it's almost completely inaccurate. [[User:FredFerguson|FredFerguson]] 10:13, 13 March 2009 (EDT)

:: Can you enumerate the "numerous mistakes" on the page? And do you disagree with the chi-square test p-values on the page? [[User:SJohnson|SJohnson]] 09:59, 14 March 2009 (EDT)

Significance of E. Coli Evolution Experiments

2009-03-14T13:56:13Z

SJohnson: Some reorganization and added experiment 3

Talk:Significance of E. Coli Evolution Experiments

2009-03-12T14:10:19Z

SJohnson: Some responses

Talk:Significance of E. Coli Evolution Experiments

2009-03-09T14:49:14Z

SJohnson:

Talk:Significance of E. Coli Evolution Experiments

2009-03-09T14:46:28Z

SJohnson:

Flaws in Richard Lenski Study

2009-03-05T21:44:37Z

SJohnson: More of a typo than a flaw, but worth mentioning

[[Richard Lenski]] rejected a request to release his bacteria mutation data to the public,<ref>See [[Conservapedia:Lenski dialog]].</ref> but the following serious flaws are emerging about his work<ref>Blount et al., "Historical contingency and the [[evolution]] of a key innovation in an experimental population of ''Escherichia coli'', 105 PNAS 7899-7906 (June 10, 2008).</ref> even without a full disclosure of the data. Note that the peer review on Lenski's paper took somewhere between 0 (non-existent) and at most 14 days (including administrative time), and Lenski himself does not have any obvious expertise in statistics. In fact, Richard Lenski admits in his paper that he based his statistical conclusions on use of a website called "statistics101".

1. Lenski's "historical contingency" hypothesis, as specifically depicted in Figure 3, is contradicted by the data presented in the Third Experiment in Table 1 of his paper. Figure 3 proposes a step-up in mutation rate to Cit+ due to a historical contingency (potentiating mutation) occurring at about the 31,000th generation, yet the Third (and largest) Experiment in Table 1 shows Cit+ arising just as often before the 31,000th generation as after. The abstract, in further contradiction with Figure 3, suggests that the historical contingency (potentiating mutation) occurred prior to the 20,000th generation.

2. Lenski's two alternative hypotheses suggest a fixed mutation rate, but the failure of the mutations in his experiments to increase based on scale (number of samples) tends to disprove both of Lenski's alternative hypotheses. Yet Lenski's paper fails to address adequately this obvious flaw in the paper.

3. Richard Lenski incorrectly included generations of the ''E. coli'' already known to contain Cit+ variants in his experiments.<ref>Richard Lenski incorrectly included generations 31,500, 32,000 and 32,500.</ref> Once these generations are removed from the analysis, the data disprove Lenski's hypothesis.

4. The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events. The Third Experiment results are consistent with the null hypothesis, contrary to the paper's claim.

5. Lenski's largest experiment (Third Experiment) failed to support his hypothesis with statistical significance. Even though this largest experiment was nearly ten times the size of his other experiments, Richard Lenski did not weight this largest experiment correctly in combining his results.

6. It was error to include generations of the E. coli already known to contain trace Cit+ variants. The highly improbable occurrence of four Cit+ variants from the 32,000th generation in the Second Experiment suggests an origin from undetected, pre-existing Cit+ variants.

7. The Third Experiment was erroneously combined with the other two experiments based on outcome rather than sample size, thereby yielding a false claim of overall statistical significance.

8. Lenski's paper is not clear in explaining how the results of his largest experiment (Third Experiment) failed to confirm his hypothesis with statistical significance, even with the incorrect inclusion of the Cit+ variant generations. Instead, his paper refers to his largest experiment as "marginally ... significant," which serves to obscure its statistical insignificance. Other works published in PNAS are clear in defining statistical significance in the traditional way, which Lenski's Third Experiment (even with incorrect inclusion of the above-referenced generations) failed to satisfy.<ref>See, e.g., [http://www.pnas.org/cgi/content/full/0701990104 Cholera toxin induces malignant glioma cell differentiation]</ref>

9. The long lag time (over 12,000 generations) between the historical contingency (potentiating mutation) in the largest experiment disproves Lenski's implicit assumption that the potentiating mutation likely occurred in proximity with the occurrence of the Cit+ variant, and that the first occurrence of the Cit+ variant in the Third Experiment at the 20,000th generation somehow implies that a potentiating mutation occurred in its proximity.

10. Lenski's paper claims that "During [30,000 generations], each population experienced billions of mutations,<ref>Lenski cites one of his own prior articles for this.</ref> far more than the number of possible point mutations in the [approximately] 4.6-million-bp genome. This ratio implies, to a first approximation, that each population tried every typical one-step mutation many times." Lenski's conclusion is nonsensical because it assumes that the mutations are completely random '''and''' that each mutation has a roughly equal probability.

11. In Table 2 of [http://www.pnas.org/content/105/23/7899.full.pdf], the expected mean should be 26,382 generations, not 28,382.

== References ==

<references/>

== See also ==

*[[Letter to PNAS]]
*[[Significance of E. Coli Evolution Experiments]]
[[Category:Science]]

Talk:Significance of E. Coli Evolution Experiments

2009-03-05T21:41:18Z

SJohnson: Small data sets and MC analysis

Talk:Significance of E. Coli Evolution Experiments

2009-03-05T21:38:51Z

SJohnson: Responses to questions and comments

Significance of E. Coli Evolution Experiments

2009-03-05T14:10:37Z

SJohnson: Added table comparing p-values from paper to chi-square test p-values

Talk:Significance of E. Coli Evolution Experiments

2009-03-04T23:50:38Z

SJohnson: Comparing Monte Carlo to chi-square is an apples to oranges comparison

User talk:SJohnson

2009-03-04T17:22:48Z

SJohnson: More on p-values

{{welcome|sig=-[[User:Foxtrot|Foxtrot]] 10:49, 4 March 2009 (EST)}}

Thanks for your interesting entry on the p-values. I look forward to reviewing it further.--[[User:Aschlafly|Andy Schlafly]] 09:55, 4 March 2009 (EST)

:Anyone with Excel can calculate the p-values for the Blount el al. [http://myxo.css.msu.edu/lenski/pdf/2008,%20PNAS,%20Blount%20et%20al.pdf] experiments in minutes. The following table shows what the experiment one data should look like once entered into the spreadsheet. The Excel row numbers and column letters are shown. Once the data is entered into Excel, just type “=CHITEST(A1:B12,C1:D12)” into any empty cell of the spreadsheet to find the p-value of the experiment.

{|class="wikitable" style="text-align:center"
!
!A
!B
!C
!D
|-
!1
|0
|6
|0.333
|5.667
|-
!2
|0
|6
|0.333
|5.667
|-
!3
|0
|6
|0.333
|5.667
|-
!4
|0
|6
|0.333
|5.667
|-
!5
|0
|6
|0.333
|5.667
|-
!6
|0
|6
|0.333
|5.667
|-
!7
|0
|6
|0.333
|5.667
|-
!8
|1
|5
|0.333
|5.667
|-
!9
|0
|6
|0.333
|5.667
|-
!10
|1
|5
|0.333
|5.667
|-
!11
|0
|6
|0.333
|5.667
|-
!12
|2
|4
|0.333
|5.667
|-

:The first column is the number of mutations observed for each generation. The second column is the number that didn’t mutate. The third column is the expected number of mutants per generation under the null hypothesis. The null hypothesis from the paper is that the mutation rate is constant for all generations. The estimate of the mutation rate under the null is simply the total number of mutants divided by the total number of trials (4/72 for experiment one). Thus, column three is the number of trials per generation times the expected null hypothesis mutation rate (6*4/72 = 1/3 for experiment one). Column four is the expected number of trials that don’t mutate, which is also the number of trials minus the expected number of mutations.

:The best reference I’ve seen on chi-square tests is ''Mathematical Statistics with Applications'' by Wackerly et al. Chapter 14 of that book discusses the chi-square test in general and Section 14.4 shows specifically how to analyze contingency tables such as this one. If you don’t have access to that book, Wikipedia gives an alright explanation of the process on the page for “Pearson's chi-square test” under the heading “Test of independence”. [[User:SJohnson|SJohnson]] 12:22, 4 March 2009 (EST)

Flaws in Richard Lenski Study

2009-03-04T14:28:55Z

SJohnson: Link to page about the incorrect p-values in the E. Coli evolution paper

[[Richard Lenski]] rejected a request to release his bacteria mutation data to the public,<ref>See [[Conservapedia:Lenski dialog]].</ref> but the following serious flaws are emerging about his work<ref>Blount et al., "Historical contingency and the [[evolution]] of a key innovation in an experimental population of ''Escherichia coli'', 105 PNAS 7899-7906 (June 10, 2008).</ref> even without a full disclosure of the data. Note that the peer review on Lenski's paper took somewhere between 0 (non-existent) and at most 14 days (including administrative time), and Lenski himself does not have any obvious expertise in statistics. In fact, Richard Lenski admits in his paper that he based his statistical conclusions on use of a website called "statistics101".

1. Lenski's "historical contingency" hypothesis, as specifically depicted in Figure 3, is contradicted by the data presented in the Third Experiment in Table 1 of his paper. Figure 3 proposes a step-up in mutation rate to Cit+ due to a historical contingency (potentiating mutation) occurring at about the 31,000th generation, yet the Third (and largest) Experiment in Table 1 shows Cit+ arising just as often before the 31,000th generation as after. The abstract, in further contradiction with Figure 3, suggests that the historical contingency (potentiating mutation) occurred prior to the 20,000th generation.

2. Lenski's two alternative hypotheses suggest a fixed mutation rate, but the failure of the mutations in his experiments to increase based on scale (number of samples) tends to disprove both of Lenski's alternative hypotheses. Yet Lenski's paper fails to address adequately this obvious flaw in the paper.

2. Richard Lenski incorrectly included generations of the ''E. coli'' already known to contain Cit+ variants in his experiments.<ref>Richard Lenski incorrectly included generations 31,500, 32,000 and 32,500.</ref> Once these generations are removed from the analysis, the data disprove Lenski's hypothesis.

3. The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events. The Third Experiment results are consistent with the null hypothesis, contrary to the paper's claim.

3. Lenski's largest experiment (Third Experiment) failed to support his hypothesis with statistical significance. Even though this largest experiment was nearly ten times the size of his other experiments, Richard Lenski did not weight this largest experiment correctly in combining his results.

4. It was error to include generations of the E. coli already known to contain trace Cit+ variants. The highly improbable occurrence of four Cit+ variants from the 32,000th generation in the Second Experiment suggests an origin from undetected, pre-existing Cit+ variants.

5. The Third Experiment was erroneously combined with the other two experiments based on outcome rather than sample size, thereby yielding a false claim of overall statistical significance.

6. Lenski's paper is not clear in explaining how the results of his largest experiment (Third Experiment) failed to confirm his hypothesis with statistical significance, even with the incorrect inclusion of the Cit+ variant generations. Instead, his paper refers to his largest experiment as "marginally ... significant," which serves to obscure its statistical insignificance. Other works published in PNAS are clear in defining statistical significance in the traditional way, which Lenski's Third Experiment (even with incorrect inclusion of the above-referenced generations) failed to satisfy.<ref>See, e.g., [http://www.pnas.org/cgi/content/full/0701990104 Cholera toxin induces malignant glioma cell differentiation]</ref>

7. The long lag time (over 12,000 generations) between the historical contingency (potentiating mutation) in the largest experiment disproves Lenski's implicit assumption that the potentiating mutation likely occurred in proximity with the occurrence of the Cit+ variant, and that the first occurrence of the Cit+ variant in the Third Experiment at the 20,000th generation somehow implies that a potentiating mutation occurred in its proximity.

8. Lenski's paper claims that "During [30,000 generations], each population experienced billions of mutations,<ref>Lenski cites one of his own prior articles for this.</ref> far more than the number of possible point mutations in the [approximately] 4.6-million-bp genome. This ratio implies, to a first approximation, that each population tried every typical one-step mutation many times." Lenski's conclusion is nonsensical because it assumes that the mutations are completely random '''and''' that each mutation has a roughly equal probability.

== References ==

<references/>

== See also ==

*[[Letter to PNAS]]
*[[Significance of E. Coli Evolution Experiments]]
[[Category:Science]]

Richard Lenski

2009-03-04T14:27:46Z

SJohnson: Link to page about the incorrect p-values in the E. Coli evolution paper

'''Richard Lenski''' is a professor of microbial [[ecology]] at Michigan State [[University]].<ref>Richard Lenski Biographical Sketch [http://myxo.css.msu.edu/BioSketch.html]</ref><ref>Richard Lenski Homepage [https://www.msu.edu/~lenski/]</ref> He holds a B.A. from Oberlin College (but does not disclose his field of study in his biographical sketch), and a [[doctorate]] in zoology<ref>Purpose in Evolution symposium Participants [http://www.templeton.org/humble_approach_initiative/Purpose_in_Evolution/part6.html]</ref> from the [[University of North Carolina]].<ref>[https://www.msu.edu/~lenski/]</ref>
When Lenski started graduate school at UNC in 1977, his father, professor Gerhard Lenski, was the Chair of the Division of Social Sciences at the same school. <ref>http://hirr.hartsem.edu/ency/lenski.htm</ref>

Richard Lenski is best known for his 20-year old E. coli experiment in which he claims to have captured evolution in progress, saying that bacteria made minor changes in the long-term laboratory study while insisting that it was not due to contamination.

The 2008 paper he co-authored was [[peer review]]ed in 14 days, sparking obvious questions regarding potential problems <ref>See [[Conservapedia talk:Lenski dialog]].</ref> about the thoroughness of the review. The statistical analysis in Lenski's paper has been criticized for having serious flaws.<ref>See [[Flaws in Richard Lenski Study]].</ref>

When Richard Lenski received a public request for the [[data]] underlying for his published claims, he did not provide the actual data even though his study was taxpayer-funded and even though the request was made in part to enable review of the data by students of the requestor.<ref>[[Conservapedia:Lenski dialog]].</ref>

Undisclosed or obscured data for Lenski's 2008 paper are noted below (pp. 2-3 from paper, superscripts omitted):<ref>For those wishing to review the paper personally it may be found at [http://myxo.css.msu.edu/lenski/pdf/2008,%20PNAS,%20Blount%20et%20al.pdf this link] from Lenski's website.</ref><ref>A Lenski defender has asserted that this table format is copyrighted under [[GFDL]], which imposes overly complex rules for reuse.</ref>

<includeonly>
{{DEFAULTSORT:{{PAGENAME}}}}
</includeonly>
__NOTOC__
<TABLE width=100% cellspacing=2 cellpadding=2>
<TR><TD width=50% valign=top bgcolor=#003366 align=center>Undisclosed Data
</TD><TD width=50% valign=top bgcolor=#000000 align=center>Some Questions about the missing data</TD></TR></table>

====Visual inspection====
<TABLE width=100% cellspacing=10>
<TR><TD width=50% valign=top>
Evolution of Cit Function in Population Ara-3. The LTEE populations are transferred daily into fresh medium, and the turbidity of each is checked visually at that time. ...
</TD><TD width=50% valign=top>
Data on these observations?

</TD></TR></table>

====Turbidity====
<TABLE width=100% cellspacing=10>
<TR><TD width=50% valign=top>
...the cultures are only slightly turbid when transferred. Occasional contaminants that grow on citrate have been seen over the 20 years of this experiment.
</TD><TD width=50% valign=top>
Data? When and how many?
</TD></TR></table>

====Contaminants???====
<TABLE width=100% cellspacing=10>
<TR><TD width=50% valign=top>
These contaminated cultures reach much higher turbidity owing to the high concentration of [[citrate]] in the medium, which allows the contaminants to reach high density. (When contamination occurs, the affected population is restarted from the latest frozen sample.)
</TD><TD width=50% valign=top>
Data for when that occurred, and how often?

A Richard Lenski defender says, "If interested in a thorough review, contact the group directly, with a legitimate request for data. Otherwise, the disclaimer is a more than adequate description of the sample handling process."

In fact, two requests were made directly to Lenski for data, without success. His [[Conservapedia:Lenski dialog#Second_Reply|second response]] was rude and insulting.
</TD></TR></table>

====The Numbers Please?====
<TABLE width=100% cellspacing=10>
<TR><TD width=50% valign=top>
After 33,127 generations, one population, designated Ara-3, displayed significantly elevated turbidity that continued to rise for several days (Fig. 1).
</TD><TD width=50% valign=top>
Higher resolution data underlying figure not provided despite request.
</TD></TR></table>

====More missing Numbers====
<TABLE width=100% cellspacing=10>
<TR><TD width=50% valign=top>
A number...
</TD><TD width=50% valign=top>
Data? How many? Statistically significant?
</TD></TR></table>

====Trust us====
<TABLE width=100% cellspacing=10>
<TR><TD width=50% valign=top>
of Cit clones were isolated from the population and checked for phenotypic markers characteristic of the ancestral E. coli strain used to start the LTEE: all
</TD><TD width=50% valign=top>
Data? How many? Statistically significant?
</TD></TR></table>

====Missing Characteristics====
<TABLE width=100% cellspacing=10>
<TR><TD width=50% valign=top>
...were Ara, T5-sensitive, and T6-resistant, as expected (2)
</TD><TD width=50% valign=top>
Data about these and other characteristics?

A Richard Lenski defender says, "Interested parties should consult the literature on these topics. It is not within the scope of this paper to address the significance of these markers."

What is missing from disclosure are the data that the markers were actually observed in a reliable and conclusive manner in this study.
</TD></TR></table>

====DNA====
<TABLE width=100% cellspacing=10>
<TR><TD width=50% valign=top>
DNA sequencing also showed...
</TD><TD width=50% valign=top>
DATA??

A Richard Lenski defender says, "The inclusion of this data would be considerably more extensive than appropriate for the inclusion in this paper."

No one asked for the data to be printed in the paper. What is missing is disclosure of the data on the website, or upon public request.
</TD></TR></table>

====More missing data====
<TABLE width=100% cellspacing=10>
<TR><TD width=50% valign=top>
...that Cit clones have the same mutations in the pykF and nadR genes as do clones from earlier generations of the Ara-3 population, ...
</TD><TD width=50% valign=top>
Data about these and other characteristics?

A Lenski defender cites to general information unrelated to this study: Available in other publications,
pykF <ref>[http://www.ihop-net.org/UniPub/iHOP/gs/519955.html "IHOP-pykF"]</ref> and
nadR <ref>[http://www.ihop-net.org/UniPub/iHOP/mgi/447129.html "IHOP-nadR"]</ref>.

A Richard Lenski supporter also cites an earlier claim (perhaps also unsupported by public release of the data) by Lenski's group: See also the reference for the sequence identification which Blount ''et al.'' list as reference #30 in their paper: Woods, R., D. Schneider, C. L. Winkworth, M. A. Riley, and R. E. Lenski. 2006. Tests of parallel molecular evolution in a long-term experiment with Escherichia coli. PNAS, USA 103:9107-9112.<ref>http://www.pnas.org/cgi/content/abstract/0602917103v1</ref>.
</TD></TR></table>

====More missing data====
<TABLE width=100% cellspacing=10>
<TR><TD width=50% valign=top>
...and each of these mutations distinguishes this population from all the others (30).
</TD><TD width=50% valign=top>
Data distinguishing "This population from all the others"?
</TD></TR></table>

====Fast Tracked====
<TABLE width=100% cellspacing=10>
<TR><TD width=50% valign=top>
Therefore, the Cit variant arose within the LTEE and is not a contaminant.
</TD><TD width=50% valign=top>
The astoundingly short 14-day [[peer review | PEER REVIEW]] period, measured from the day the paper was sent out for review to the day of formal contribution by Lenski after acceptance, raises questions as to whether there was any meaningful peer review of this at all.
</TD></TR></table>

====Sample Sizes?====
<TABLE width=100% cellspacing=10>
<TR><TD width=50% valign=top>
Cit+ cells constituted [about] 0.5% of the population at generation 31,500, then 15% and 19% in the next two samples, but only [about] 1.1% at generation 33,000.
</TD><TD width=50% valign=top>
What were the sample sizes and where are the actual data for this sampling?
</TD></TR></table>

Lenski's paper was published in the Proceedings of the [[National Academy of Sciences]], a professional society of scientists with a strong bias towards [[atheism]] and [[evolutionism]]. <ref>http://www.stephenjaygould.org/ctrl/news/file002.html</ref> <ref>http://www.answersingenesis.org/docs/3506.asp</ref>

Lenski's work has been criticized in the past also. In a paper Lenski published in ''Nature'' in May 8, 2003 entitled "The Evolutionary Origin of Complex Features," he described a computer simulation that contained no actual biology. In fact, the discussion section of the paper stated:

:"Some readers might suggest that we 'stacked the deck' by studying the evolution of a complex feature that could be built on simpler functions that were also useful. However, that is precisely what evolutionary theory requires ...."

Dr. William Dembski explained, "In other words, the computer programmers built into the simulation what they thought evolution needed to make it work. The validity of this study therefore depends on whether the simulation faithfully models biological reality."<ref>http://www.designinference.com/documents/2003.09.UncDiss_Intro_Contribs.pdf</ref>

Others have commented on Lenski's work while taking it at face value.<ref>[[Michael Behe]] wrote, in ''[[The Edge of Evolution]]'', "I had argued that the extreme rarity of the development of chloroquine resistance in malaria was likely the result of the need for several mutations to occur before the trait appeared. Even though the evolutionary literature contains discussions of multiple mutations (5), Darwinian reviewers drew back in horror, acted as if I had blasphemed, and argued desperately that a series of single beneficial mutations certainly could do the trick. Now here we have Richard Lenski affirming that the evolution of some pretty simple cellular features likely requires multiple mutations." Behe also said, "If the development of many of the features of the cell required multiple mutations during the course of evolution, then the cell is beyond Darwinian explanation. I show in The Edge of Evolution that it is very reasonable to conclude they did."[http://www.amazon.com/gp/blog/post/PLNK3U696N278Z93O]</ref>

==See Also==

*[[Conservapedia:Lenski dialog]] (letters requesting data, and Lenski's responses)
*[[Conservapedia talk:Lenski dialog]] (discussion of Lenski controversy<ref>See [[Liberal style]] point 1</ref>)
*[[Letter to PNAS]]
*[[Significance of E. Coli Evolution Experiments]]
[[Category:Biology]]
[[Category:Scientists]]
[[Category:Academics]]
[[Category:Conservapedia Dealings with PNAS and Lenski]]
[[Category:Evolution]]
[[Category:Deceit]]

{{evolution}}

==References==
<references/>

{{DEFAULTSORT:Lenski, Richard}}

Significance of E. Coli Evolution Experiments

2009-03-04T14:26:03Z

SJohnson: Created page about the incorrect p-values in the E. Coli evolution paper

Blount, Borland, and Lenski<ref> http://www.pnas.org/content/105/23/7899.full.pdf</ref> claimed that a key evolutionary innovation was observed during a laboratory experiment. That claim is false. The claim was based on incorrect measurements of statistical significance. Rather than using a test from the statistics literature, a flawed test was contrived and used to measure significance. The flawed test (“mean mutation generation”) produced artificially low p-values.

The data from experiment one of the paper is shown below (see Table 1 of the paper). The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown.

{|class="wikitable" style="text-align:center"
!Generation
!Trials
!Mutants
!Statics
!Expected Mutants
!Expected Statics
|-
|0
|6
|0
|6
|0.333
|5.667
|-
|10000
|6
|0
|6
|0.333
|5.667
|-
|20000
|6
|0
|6
|0.333
|5.667
|-
|25000
|6
|0
|6
|0.333
|5.667
|-
|27500
|6
|0
|6
|0.333
|5.667
|-
|29000
|6
|0
|6
|0.333
|5.667
|-
|30000
|6
|0
|6
|0.333
|5.667
|-
|30500
|6
|1
|5
|0.333
|5.667
|-
|31000
|6
|0
|6
|0.333
|5.667
|-
|31500
|6
|1
|5
|0.333
|5.667
|-
|32000
|6
|0
|6
|0.333
|5.667
|-
|32500
|6
|2
|4
|0.333
|5.667
|-
|-
!Total
!72
!4
!68
!4
!68
|}

When the flawed test is used to compute the significance of this data, the p-value is 0.0085 (see Table 2 of the paper). This p-value is considered statistically significant. However, when the data is analyzed using a standard method (the chi-square test) the p-value is 0.19. This p-value is much larger than the one from the paper and indicates that there is no reason to reject the null hypothesis. The chi-square test p-value for experiment two is small (0.0004). However, experiment three is not statistically significant because its p-value is 0.22.

The chi-square test is a common statistical method.<ref>''Mathematical Statistics with Applications'' by Wackerly, Mendenhall, and Scheaffer, Section 14.4.</ref> It can be implemented in Microsoft Excel. If the numbers from the last four columns of the table above (excluding the “totals” row) are entered into Excel in rows 1-12 and columns A-D, then the p-value can be computed by entering “=CHITEST(A1:B12,C1:D12)” into any empty cell of the spreadsheet.

==References==
<references/>