Conservapedia - User contributions [en]

Talk:Significance of E. Coli Evolution Experiments

2009-03-26T01:01:57Z

Argon: /* Unreferenced Claims */

SJohnson, your assessment, while good in the utilization of the chi-squared test is unfortunately incorrect. The Monte Carlo resampling gives a more accurate p-value than the chi-squared. You may research the literature (i.e. publications in statistical mathematics, many pubs actualy compare Monte Carlo vs Chi Squared) to discover that this method is commonly used in advance statistical work and how it is more accurate than the chi-squared test.--[[User:Able806|Able806]] 17:00, 4 March 2009 (EST)

:It doesn’t make sense to compare the chi-square test, which is a specific statistical hypothesis test, to Monte Carlo methods, which can be used for anything from fluid motion modeling to p-value computations. You can use Monte Carlo methods to compute the p-values of the chi-square test!

:Monte Carlo methods involve the generation of random realizations. Your broad claim the Monte Carlo methods are “more accurate” than the chi-square test is obviously incorrect because the accuracy of Monte Carlo methods always depends on the number of random realizations generated. When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous.

:Which publications compare Monte Carlo to chi-square and show that the former is more accurate? Could you provide specific examples? Thanks. [[User:SJohnson|SJohnson]] 18:50, 4 March 2009 (EST)

:In furtherance of SJohnson's remarks with respect to rarely occurring events, the use of the basic Monte Carlo method is plainly incorrect for modeling a rarely occurring event, as the Lenski paper did. This has long been pointed out in [[Flaws in Richard Lenski Study]]. I know [[evolutionists]] will never admit a flaw in anything promoting their pet theory, but this (and other) flaws in that paper is undeniable.

:Watch how evolutionists defended obvious errors in the Lenski paper, and then realize why the [[Piltdown Man]] fraud was taught for 40 years without evolutionists admitting it was a hoax.--[[User:Aschlafly|Andy Schlafly]] 09:55, 5 March 2009 (EST)

::Andy, how exactly is the Monte Carlo method incorrect to use in this case? I have seen it used in publications with much smaller datasets.--[[User:Able806|Able806]] 10:29, 5 March 2009 (EST)

:::Able806, I'm interested in looking at the publications you mentioned that use Monte Carlo methods to analyze small data sets. Could you provide some examples? Thanks. [[User:SJohnson|SJohnson]] 16:41, 5 March 2009 (EST)

::::SJohnson, here are two papers, [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6WH8-45RFJ1J-19&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=1ad95954654bb97b17e474ce6b469f6e 1] and [http://cat.inist.fr/?aModele=afficheN&cpsidt=787963 2]. Most are in chemistry and genetics where you find the observed to be much smaller and have to use the MCM. You can search on the subject as well and find that how Lenski performed the test is the standard for microbiological genetic analysis.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

:::::Those papers have nothing to do with hypothesis testing. One is an archeology paper. To be blunt, it seems like you’re just doing internet searches on “Monte Carlo” to find these links. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

::::::SJohnson, actually they do, did you read the papers? If so you would see how they used the MCM for their data analysis of small data sets, which indeed was hypothesis testing and answers you inquiry about publications that use MCM for small data set analysis. If you wish I can try to track down some actual mathematical publications, however, I am not as familiar with mathematical journals as I am with science/medical journals (not knowing which mathematical journals are acceptable). I am assuming that you have a background in math and possibly access to mathematical journals, therefore if you know the reputable ones I can do the leg work.
::::::I believe the thing that needs to be looked at is there truly a problem with the choice of test and if so what is an alternative. Bayesian might be an option but seems to be difficult to employ for this situation.--[[User:Able806|Able806]] 12:36, 12 March 2009 (EDT)

:::Able806, you still seem to miss the point about how inappropriate the Monte Carlo method (as used in the Lenski paper) is for evaluating rarely occurring events. You need to open your mind to be productive. If you simply cling to a view that Lenski (who I don't think has any meaningful education in statistics) must somehow be right, then you're not going to make any progress in understanding the flaws.--[[User:Aschlafly|Andy Schlafly]] 17:07, 5 March 2009 (EST)

::::Andy, you still have not answered what you find inappropriate about his use of the Monte Carlo method? I am a reasonable person and with evidence I do have an open mind. I provided examples last week, with a working model, showing that Monte Carlo is better than the chi-square in this case. I have also shown where the Chi-Square was inappropriate due to the occurrence size as well. So if you have any evidence that Monte Carlo should not be used in the way that Lenski used please let it be shown.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

Sjohnson, I believe you just proved my point. In the literature of mean and covariance structure analysis, non-central chi-square distribution is commonly used to describe the behavior of the likelihood ratio statistic under alternative hypothesis; it is widely believed that the non-central chi-square distribution is justified by statistical theory. Actually, when the null hypothesis is not trivially violated, the non-central chi-square distribution cannot describe the LR statistic well even when data are normally distributed and the sample size is large. Monte Carlo results compare the strength of the normal distribution against that of the non-central chi-square distribution. In an association analysis comparing cases and controls with respect to allele frequencies at a highly polymorphic locus, a potential problem is that the conventional chi-squared test may not be valid for a large, sparse contingency table. Reliance on statistics with known asymptotic distribution is unnecessary, as Monte Carlo simulations can be performed to estimate the significance level of the test statistic.

Here is a [http://faculty.vassar.edu/lowry/chi_beta.html link] to a great page the provides an interactive example as to why the Chi Squared test would provide poor results compared to the Monte Carlo in relation to the Lenski data workup.

Something you may have overlooked was that the data set is actually too small to use the chi square method correctly. It is often accepted that is any of the analyzed data falls under 10 for a particular cell of the data set then the Yates correction needs to be applied; unfortunately the Yates correction can over correct thus skewing the p-value. Lenksi seemed to understand this by supporting his Monte Carlo p-value results with the Fisher z-transformation p-value.

I hope this helps.--[[User:Able806|Able806]] 10:27, 5 March 2009 (EST)

:I’m still waiting to hear which literature says that “Monte Carlo resampling” is “more accurate than the chi-squared test”. The page mentioned above [http://faculty.vassar.edu/lowry/chi_beta.html] is a discussion of why statisticians “fail to reject the null” rather than “accepting the null” when the p-value is above 0.05 or so. The page says nothing about superiority of Monte Carlo methods. Why were alternate hypothesis distributions mentioned? Only the null hypothesis distribution is used to calculate a p-value. Yates’s correction is for 2x2 contingency tables [http://en.wikipedia.org/wiki/Yates%27_correction_for_continuity]. It doesn’t apply in this case. Finally, what the heck do “covariance structure analysis” and “allele frequencies at a highly polymorphic locus” have to do with this problem? [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::SJohnson, I am looking for this paper for you, I cited it for one of my past publications dealing with allele frequencies (I believe it came from the Duke Biostatistics group). To answer your question about allele frequencies, that is the issue at hand, more about the genetics than the math, but it is the item being studied. So you stated that Yates can not be used and statistics says the number of occurrences is too small to evaluate using the Chi-Squared test so what would you recommend instead of the Monte-Carlo Method?

:Regarding the "Fisher z-transformation p-value" from the paper, garbage in garbage out. If the p-values were bad to begin with, then why would a combination of them be meaningful? [[User:SJohnson|SJohnson]] 10:49, 9 March 2009 (EDT)

::You are assuming that p-values are wrong based on a test that is inappropriate in this case due to data limitations. Did you perform a z-transformation on the chi-squared for the three data groups?--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

::You asked about the “Fisher z-transformation p-value”. The z-transformation test and Fisher’s method are actually two different things (see Whitlock's 2005 paper - Ref. 49 in Blount et al.). But no, I haven’t tried either. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::There's a large literature on various kinds of Monte Carlo test, a very short summary of which is that they're inevitably more accurate than parametric tests (e.g. F, t, chi-squared, etc) because they don't make assumptions about the distribution of the data under the null hypothesis. See for example ''Introduction to the Bootstrap'' by B. Efron and R. Tibshirani and ''The Jack-knife, the Bootstrap and Other Resampling Plans'', also by Efron. They're certainly applicable to small datasets and their accuracy is really only limited by the number of samples you care to take. E.g. 1000 M-C samples would give you a pretty accurate idea about significance at the alpha<1% level (That book should answer SJohnson's questions of 18:50 on 4/3/09 and 16:38 on 5/3/09 about accuracy and Aschalfly's comment of 17:07 on 5/3/09 about appropriateness of Monte Carlo tests.) [[User:FredFerguson|FredFerguson]] 16:53, 11 March 2009 (EDT)

::::Your claim that Monte Carlo methods are “inevitably more accurate” than other tests is obviously wrong because the accuracy of MC methods always depends on the number of realizations used. You should have written <math>\alpha=1\%</math>, not <math>\alpha<1\%</math>. If 1,000 random realizations are generated, the number of realizations above the true <math>\alpha=1\%</math> level is binomial with mean 10 and variance about 10. Thus, the standard deviation of the MC estimate is >0.003. In this example, a Monte Carlo p-value could be off by 30% and still be within a standard deviation. Is that really “pretty accurate”?

::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 3%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::::You're talking about miniscule differences in the accuracy of a test. 0.013 isn't very different from 0.007. In either case, it's very unlikely the experimenter would have obtained that result if the null hypothesis were true. If you're bothered about differences in P-values to the third decimals (which would make you unusual!), just run more MC realisations, that's all. Not really a problem. [[User:FredFerguson|FredFerguson]] 11:53, 12 March 2009 (EDT)

There’s still confusion about the difference between test statistics and Monte Carlo methods. Before you find a Monte Carlo estimate of a p-value, you need to select a test statistic to reduce the data set to a scalar. I am interested in hearing which test statistic you believe should be used in place of the chi-square test and why. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

----

Quick question for SJohnson: How many degrees of freedom did you choose when calculating the p-value? I'd like to know upon what condition you base that number. Thanks.--[[User:Argon|Argon]] 11:05, 5 March 2009 (EST)

:The degree of freedom for a contingency table is rows minus one times columns minus one. That is, <math> (r-1)(c-1) </math>. Here’s a pretty good tutorial I came across: [http://faculty.uncfsu.edu/dwallace/lesson%2020.pdf]. For the experiments from [http://www.pnas.org/content/105/23/7899.full.pdf], the DOFs are 11, 11, and 13. For experiment one, the chi-square test statistic is
::<math>
X^2
=\sum\limits_i\sum\limits_j
\frac{\left(n_{i,j}-E\left[n_{i,j}\right]\right)^2}
{E\left[n_{i,j}\right]}
</math>
::<math>
=\frac{\left(0-1/3\right)^2}{1/3}
+\frac{\left(6-17/3\right)^2}{17/3}
+\frac{\left(0-1/3\right)^2}{1/3}
+\ldots+
</math>
::<math>
+\frac{\left(2-1/3\right)^2}{1/3}
+\frac{\left(4-17/3\right)^2}{17/3}
</math>
::<math>
\approx
14.82
</math>
:where <math>n_{i,j}</math> is the observed value and <math>E\left[n_{i,j}\right]</math> is the expected null hypothesis value. So if you have MS Excel, another way to arrive at the p-value of 0.19 is to type “=CHIDIST(14.82,11)” into a cell. Cheers! [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::OK, thanks for the info. From what I'd calculated and looked up in tables, the numbers seemed close to a df=11 for a chi-square of ~14. (Aside: With terms having 17/3 in the denominator in the figures above, were you using the test of independence? I was using Pearson's test for [http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_for_fit_of_a_distribution fit of a distribution] which returns a chi-squared value of 14 and roughly matched the p-values you reported, assuming the df was 11).

::Also, the first sentence of the article reads: "Blount, Borland, and Lenski[1] claimed that a key evolutionary innovation was observed during a laboratory experiment. That claim is false." A small correction: There were several claims in the paper. The 'key evolutionary innovation' was acquiring the ability to utilize citrate as a food source. That claim was demonstrated multiple times. The claim, which pertains to this statistics discussion was that the Cit+ phenotype arose in a multi-step process, first requiring a rare, pre-adaptive mutation before additional mutation(s) lead to the subsequent development of citrate utilization.--[[User:Argon|Argon]] 20:46, 5 March 2009 (EST)

:::My biology-degreed wife assures me that mutation does not necessarily mean that evolution occurred. What the paper claimed is that evolution (a “key innovation”) occurred in the lab. The key innovation supposedly increased the mutation rate. In the experiments, the observed mutation rate increased after generation 31,000, but not enough to make a statistically significant claim that the rate is not constant. The analysis in the paper was similar to flipping a coin ten times, counting six heads and claiming that the coin must be biased against tails. In reality, there’s nothing surprising about a fair coin producing slightly more of one outcome than the other. Just like there's nothing surprising about there being slightly more mutations in later generations than early generations given the null hypothesis (constant mutation rate). [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

>>Inserting a later comment first<<
SJohnson, the paper's title is: "Historical contingency '''and the evolution of a key innovation''' in an experimental population of ''Escherichia coli''" As I mentioned earlier, the key innovation is the evolution of the Cit+ phenotype and not the timing or rate of its acquisition. And yes, it *is* evolution (call it microevolution, if you wish). Blount et al went on further to speculate how this evolutionary innovation arose and they proposed the historical contingency hypothesis in which 'pre-adaptive' mutations were required before the Cit+ phenotype developed. It is only this latter hypothesis that you are attempting to address with your chi-square analysis, not the fact that Cit+ mutants arose (which is the evolutionary innovation).--[[User:Argon|Argon]] 21:57, 18 March 2009 (EDT)

::::SJohnson, not to say anything about your wife, but has she had a 400 level molecular genetics course (most general biology degrees do not cover the detail unless they are specialized)? If so, she would have mentioned that if the mutation passes to the offspring and is selectively beneficial to the population then it is a step of evolution as along as the conditions continue through the sharing of the mutation with the population and the environment is such that reduces the growth rate of the non-transformed population. While not all mutations are signs that evolution occurred the mutations that pass to offspring and provide a benefit compared to other offspring are very strong indicators. In the case of this paper the population that evolved the cit+ was able to metabolize a chemical in their environment which allowed for an adaptation advantage compared to the non-transformed colonies.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

----

Let’s go back to the beginning. There appears to be confusion about the difference between test statistics and methods for computing p-values. As is noted at the beginning of the page [http://www.conservapedia.com/Significance_of_E._Coli_Evolution_Experiments], the fundamental problem with the paper is that it used a flawed test statistic, not that it used Monte Carlo methods to find the p-value for that flawed statistic.

Every hypothesis test uses a test statistic to reduce the data to a single number. The p-value for the test statistic can be calculated analytically (as I’ve done for the chi-square test statistic) or by Monte Carlo methods. In the paper, Monte Carlo methods were used to compute the p-value of the “mutation generation” test statistic. The key problem with the analysis from the paper is that it doesn’t work to use a weighted average to test for variations in mutation rate. This is like trying to use the sample variance to test for an increase in the mean in Gaussian-distributed data. A statistic should be selected based on the null and alternate hypothesis distributions of the data. The chi-square test (unlike the weighted average from the paper) is a reasonable choice for data that mutates at a constant rate under the null hypothesis, but mutates at varying rates under the alternate hypothesis.

Able806, you made a good point about the contingency table cell frequencies being relatively low, but were wrong when you said ”the data set is actually too small to use the chi square method correctly”. In the low cell frequency case the chi-square test is still effective, but the null hypothesis distribution of the chi-square statistic starts to look less like the chi-square distribution. Thus, p-values calculated using the chi-square distribution may be a bit off. However, Monte Carlo p-values are always imperfect as well because it's impossible to generate an infinite number of random realizations. There are imperfections in p-values generated by analytic and Monte Carlo methods. However, low cell frequencies does not explain the >20x and >2.5x differences between chi-square p-values and p-values from the paper for experiments one and three. The reason for those huge differences was the use of the flawed test statistic (“mutation generation”) in the paper. [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

:SJohnson, the chi-squared test is a valuable statistical tool, but the limitations of the test must be acknowledged. The chi-squared test can only produce valid results if the assumptions that underly the test are not violated. As an analogy, Newtonian models of motion fail to produce accurate results as velocities approach the speed of light; under those circumstances one must switch to a theory that accounts for relativistic effects.

:It seems that you have simply dismissed the [http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm widely-acknowledged] [http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm fact] that the [http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html chi-squared test] is [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236 inappropriate] for use in [http://www.graphpad.com/www/Book/Choose.htm situations] where n in any cell is [http://mysite.du.edu/~jcalvert/econ/chisquar.htm less] less than a [http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1 threshold] [http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html number]. Different authors set different thresholds, but all are well above the numbers seen in your chi-squared analysis - even the most liberal guidelines advise against the chi-squared test when any expected cell frequency is less than one or more than 20% of the table cells are less than 5; others require that expected values in all cells must be more than 5. With smaller amounts of data, the test is insensitive and errs on the side of rejecting the hypothesis. If you attempt your chi-squared statistical analysis with a program that is more sophisticated than MS Excel (as I did), you get an error message indicating that the results are invalid due to low expected cell counts.

:That issue aside, there are other reasons that the chi-squared test is inappropriate here. As the links above point out, the categories tested must be truly independent; one example is that you can't use the chi-squared test to compare age and ability to kick a field goal by testing the same experimental group twice, one year apart; you have to test one group of age A and a different group of age B. In the case of the Blount paper, the categories are not independent. Even if there were adequate numbers to address the low-expected-frequency problem, this would make the chi-squared an invalid test in this case.

:There are other significant problems with the use of the chi-squared test in this circumstance, but they can wait until you address these first major problems.--[[User:ElyM|ElyM]] 12:18, 11 March 2009 (EDT)

::Wackerly et al. says in general it’s assumed that the cell frequencies are above five so that the chi-square statistic (under the null) is approximately chi-square distributed (see p. 703). That book does not say chi-square test results are invalid if frequencies are five or less. Your example of a chi-square test warning message (it said "warning" not "error" as you stated) in Minitab [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236] said “approximation probably invalid” referring to the chi-square distribution approximation to the chi-square test statistic’s distribution. Your example did not say “chi-square test invalid”. I agree that when cell frequencies are low, the chi-square test statistic’s distribution starts to deviate from the chi-square distribution. I maintain that this deviation is not enough to explain the >2.5x and >20x differences in the chi-square test p-values and the p-values from the paper.

::As the numerous links in your post proved, the chi-square test is widely-used by statisticians. Can you give examples of statisticians using mean mutation generation as a test statistic? Also, did your software agree with the chi-square test p-values I presented? Thanks. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::Thank you for giving page references for Wackerly; however it seems we have different editions, since page 703 in my copy (5th ed, 1996) does not deal with chi-squared issues at all. My copy does state the following, on page 622: "Although the mathematical proof is beyond the scope of this text, it can be shown that, when n is large [chi-squared] will possess approximately a chi-square probability distribution in repeated sampling." Then, on page 624: "Experience has shown that cell counts [n sub i] should not be too small in order that the chi-square distribution provide an accurate approximation to the distribution of [chi squared]. As a rule of thumb we require that all expected cell counts equal or exceed 5, although Cochran (1952) has noted that this value can be as low as 1 for some situations." Wackerly then goes on, in the problems sections, to describe the use of the chi-squared test as a "violation of good statistical practice" when "some expected counts [are] <5."

:::It seems that you are already aware that the [chi-square] statistic under the null is no longer chi-square distributed for small n; this is precisely why the test should not be used under those conditions. I can claim to be able to accelerate a 1-kg mass to 10 times the speed of light by applying 1 N of force for 95 years by using F=ma and t= (vf-vi)/a. Plugging the numbers into those equations will produce the same result every time, but the answer is illegitimate because those equations are only valid under certain assumptions, which are violated as velocities approach the speed of light. Similarly, having a statistical program calculate a chi-squared value given the Blount data will produce a number result, but since the assumptions of the test are violated the result is not legitimate. Yes, if I put the Blount data in SAS 9.2, I get the same numerical answer as you do, but I also get the following message: "WARNING: >89% of the cells have expected counts less than 5. Chi-square may not be a valid test." You may argue that that's a warning, not an error; that's a semantic distinction. The reason that the program says that it MAY not be valid is that the chi-squared test skews in the direction of being too conservative at low n values; the test has an acceptable rate of false positives but an unacceptably high rate of false negatives. Comparing the results of the Monte Carlo and chi-squared results in this case is like comparing the results of Newtonian and relativistic equations of motion: they can produce very different results from the same input data.

::::For a finite amount of data, the chi-square statistic is never chi-square distributed under the null. The p-values are always approximate regardless of cell frequencies. The approximation becomes more accurate as the amount of data increases, but I don’t believe that this inaccuracy will change p-values that are about 0.2 (for experiments 1 and 3) into statistically significant p-values. How much do you expect the p-values to change if an exact computation is used in place of the chi-square distribution approximation? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::Your last paragraph has a major non sequitur in it: yes, many statisticians use the chi-square test. As long as the assumptions of the test are not violated, it is a valuable tool. That has nothing to do with the validity of using mean mutation generation as a test statistic. 'Mean number of werewolf attacks in Mumbai in the week centered on the new moon, by month, from 1654 to 1798' is a valid test statistic. I am quite sure that it has never been used in a peer-reviewed paper before. That does not mean that I can't perform valid statistical tests on that statistic. If, however, the incorrect test is applied, the results of the analysis will be flawed. Papers apply a (relatively small) standard repertoire of valid tests to a (potentially infinite) number of test statistics. The particular test statistic used in a paper may never have been used before and may never be used again; that does not address the validity of the analysis. In Blount's case, the test is the Monte Carlo analysis, which is also "widely-used by statisticians".

::::There are an infinite number of ways to reduce a data set to a single number. However, it’s foolish to think every method would be effective. I gave an example of a flawed test statistic in an earlier post [http://www.conservapedia.com/index.php?title=Talk%3ASignificance_of_E._Coli_Evolution_Experiments&diff=635070&oldid=634987]. Another example of a flawed test statistic is the one used in the paper because it does not always detect deviations from the null hypothesis (see: [[Significance of E. Coli Evolution Experiments#Test Statistics]]).

::::Test statistics are typically derived. The likelihood ratio test is a common method used to derive them. The chi-square test for independence is an approximation to the LRT. Where is the derivation saying that mean mutation generation is an appropriate test statistic for this problem? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::We still haven't touched on the issue of the categories not being independent, which by itself is sufficient to invalidate the chi-squared technique. I'm new to this site, so I'm unsure as to the etiquette of making changes to the articles of another person - but the article here should at the very least mention that the chi-square test is being used here in a manner that violates its underlying assumptions in at least two fundamental ways, and the results are therefore suspect.--[[User:ElyM|ElyM]] 17:34, 12 March 2009 (EDT)

:::::When generating random realizations of experiment outcomes, the authors assumed that the total number of mutants was fixed. Thus the paper assumed the numbers of mutants per generation are statistically dependent. Does this seem like a realistic model, or do you think that if the experiments were recreated that the total number of mutants could vary? For example, if experiment one were recreated, would the total number of mutants always be exactly four? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::: It looks to me as though SJohnson has misinterpreted the application of the chi-squared test in quite a fundamental way. His/her analysis of Blount's data are therefore close to meaningless, regardless of whether the test used by Blount is appropriate or not. In my opinion, the entire page should therefore be deleted. [[User:FredFerguson|FredFerguson]] 08:18, 13 March 2009 (EDT)

::::: "Fred", perhaps you mistakenly think this is Wikipedia, where [[censorship]] and deletion of pages for ideological reasons are common. Not here.--[[User:Aschlafly|Andy Schlafly]] 10:23, 14 March 2009 (EDT)

:::::: Umm... I'm suggesting deletion for mathematical reasons, not ideological reasons. Using an argument filled with mathematical errors to try to support your case only detracts from your credibility. [[User:FredFerguson|FredFerguson]] 10:38, 14 March 2009 (EDT)

:::::: Actually, I think correction is better than deletion. So that's what I've done. [[User:FredFerguson|FredFerguson]] 11:01, 14 March 2009 (EDT)

::::::: I find no credibility in your denial of having ideological reasons.--[[User:Aschlafly|Andy Schlafly]] 11:04, 14 March 2009 (EDT)

== Misinterpretation of test ==

SJohnson, Your analysis misinterprets the test. You say the null hypothesis is that this mutation cannot happen. They saw a mutation (4 mutations, in fact, in the data set you show) so the null hypothesis (as you state is) is disproved. That's perfectly straightforward.

I don't know what the "mean mutation generation" test is but you're doing when you apply a chi-squared test to this dataset is to test if the mutations are evenly distributed throughout the generations. Your test says they are, so there's no strong evidence to suppose that mutations are likely to occur in one generation rather than another in the series of tests. Blount's test says thay aren't, so it's more likely that the mutation will occur later in the series of tests. I can't tell which test is right without knowing more about the test that Blount used.

But that point (the foregoing paragraph) has no bearing at all on the null hypothesis, as you describe it. The mutation appeared, so that means the hypothesis that the mutation can't happen is disproved. Very simple. [[User:FredFerguson|FredFerguson]] 21:10, 8 March 2009 (EDT)

:I never said that “the null hypothesis is that this mutation cannot happen”. The chi-square test statistic I'm using wouldn’t be defined if the null hypothesis mutation rate was zero because the <math>E\left[n_{i,j}\right]</math> term in the denominator of the statistic (see above equation) would be zero.

:The test statistic from the paper is the average of the generation numbers of observed mutations. For experiment one this number is
::<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)= 31750.
</math>
:The same number is shown in Table 2 of the paper. [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

:: SJohnson, the way you're calculating the chi-squared statistic implies that you're testing the null hypothesis of a constant mutation rate over time against an alternative hypothesis of a mutation rate which varies over time. [[User:FredFerguson|FredFerguson]] 11:02, 9 March 2009 (EDT)

As it currently stands, the article makes the following statement: "The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown." This misstates the null hypothesis of the paper, which is elaborated in the Introduction section of the paper, and repeated in the section '''Statistical Analysis of the Replay Experiments''':
<blockquote>
For each experiment, we compared the observed mean generation of those clones that yielded Cit+ variants to the mean expected under the null hypothesis that clones from all generations have equal likelihood. The null thus corresponds to the rare-mutation hypothesis laid out in the Introduction."
Block quote
</blockquote><ref>www.pnas.org/cgi/reprint/105/23/7899.pdf</ref>

The article also continues to describe 'mean mutation generation' as a ''test'' rather than a ''statistic'' to which the ''Monte Carlo test'' was applied.--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

I'm a bit confused about why the Chi-squared test, which we're told compares the results to a null hypothesis of a constant mutation rate, seems insensitive to which generations the Cit+ mutations are found. Instead, the chi-square test seems only to be evaluating whether the frequencies of Cit+ mutations in any particular generation are 'expected'. Thus the test is asking whether finding a distribution (e.g. in the first experiment) across nine periods that have no mutations, two periods that have one mutation and one period with two mutations is a statistically significant deviation from what you'd expect of the mutations were randomly distributed. The number returned from the function is the same regardless of the order of Cit+ results. The number of mutations per bin is not the only question being asked. Instead it's the '''order and temporal distribution''' of Cit+ mutants that the analyses probably need to confront. It's not whether one can get nine no-mutants, two single mutants and one double-mutant result, it's a matter of '''when''' they occur and whether that distribution affects the significance of the results. Blount's hypothesis is that mutations should appear later in the experiment. When formulating a suitable null hypothesis, wouldn't one want to take the timing of Cit+ mutants into consideration too?--[[User:Argon|Argon]] 22:25, 18 March 2009 (EDT)

==Unreferenced Claims==

I deleted the claim that mean mutation generation is an appropriate test statistic because no reference was produced that back that claim. No reference was provided to back the claim that the chi-square test p-values are always conservative, either. [[User:SJohnson|SJohnson]] 12:57, 14 March 2009 (EDT)

: The reference is Everitt. I'll check I put it in the right place. [[User:FredFerguson|FredFerguson]] 13:30, 14 March 2009 (EDT)

There was a typo in my edit summaries on the talk page and the main page. I meant to say "Removed unsupported claims" rather than "Removed supported claims". [[User:SJohnson|SJohnson]] 13:13, 14 March 2009 (EDT)

I do not believe that anyone has claimed that 'chi-square test p-values are ''always'' conservative'. The claim that has been made is that ''under certain circumstances'', namely low n and low individual cell values, the chi-square test is an invalid test; that under those circumstances the power of the test is low and it becomes impossible to reject the null hypothesis even when it is false. You may have missed the pertinent sections in my links above, so I will directly quote the relevant sections. All the quoted sections refer to chi-square testing in particular. Any bolding below is mine.

:This edit claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639379]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

<blockquote>
"Assumptions: 

Even though a nonparametric statistic does not require a normally distributed population, there still are some restrictions regarding its use. 

1. Representative sample (Random) 
2. The data must be in frequency form (nominal data) or greater. 
3. The individual observations must be independent of each other. 
4. '''Sample size must be adequate. In a 2 x 2 table, Chi Square should not be used if n is less than 20. In a larger table, no expected value should be less than 1, and not more than 20% of the variables can have expected values of less than 5'''. 
5. Distribution basis must be decided on before the data is collected. 
6. The sum of the observed frequencies must equal the sum of the expected frequencies."
</blockquote>
<ref>http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm
</ref>

<blockquote>
"Assumptions:
* Random sample data are assumed. As with all significance tests, if you have population data, then any table differences are real and therefore significant. If you have non-random sample data, significance cannot be established, though significance tests are nonetheless sometimes utilized as crude "rules of thumb" anyway.
* A sufficiently large sample size is assumed, as in all significance tests. '''Applying chi-square to small samples exposes the researcher to an unacceptable rate of Type II errors. There is no accepted cutoff. Some set the minimum sample size at 50, while others would allow as few as 20'''. Note chi-square must be calculated on actual count data, not substituting percentages, which would have the effect of pretending the sample size is 100.
* '''Adequate cell sizes are also assumed. Some require 5 or more, some require more than 5, and others require 10 or more. A common rule is 5 or more in all cells of a 2-by-2 table, and 5 or more in 80% of cells in larger tables, but no cells with zero count'''. When this assumption is not met, Yates' correction is applied.
* Independence. Observations must be independent. The same observation can only appear in one cell. '''This means chi-square cannot be used to test correlated data (ex., before-after, matched pairs, panel data)'''.
* Similar distribution. Observations must have the same underlying distribution.
* Known distribution. The hypothesized distribution is specified in advance, so that the number of observations that are expected to appear each cell in the table can be calculated without reference to the observed values. Normally this expected value is the crossproduct of the row and column marginals divided by the sample size.
* Non-directional hypotheses are assumed. Chi-square tests the hypothesis that two variables are related only by chance. If a significant relationship is found, this is not equivalent to establishing the researcher's hypothesis that A causes B, or that B causes A.
* Finite values. Observations must be grouped in categories.
* Normal distribution of deviations (observed minus expected values) is assumed. Note chi-square is a nonparametric test in the sense that is does not assume the parameter of normal distribution for the data -- only for the deviations.
* Data level. No assumption is made about level of data. Nominal, ordinal, or interval data may be used with chi-square tests."
</blockquote>
<ref>http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm</ref>

<blockquote>
"Assumptions: 
-None of the expected values may be less than 1 
-No more than 20% of the expected values may be less than 5"</blockquote> <ref>http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html</ref>

<blockquote>
"When performing a chi-square test, your data must satisfy important assumptions. Although these assumptions may be stated differently in different textbooks, they generally assert that: 
1)The sample must be randomly drawn from the population 
'''2)The sample size, n, must be large enough so that the expected cell count in each cell is greater than or equal to 5.''' 
Both assumptions must be met in the process of collecting your data, and violations of the second assumption will appear in the Minitab output when you run the analysis. 
... 
'''You may wonder why the second assumption is necessary for performing the chi-square test. The second assumption arises because the distribution of counts under the null hypothesis is multinomial, and the normal distribution can be used to approximate the multinomial distribution if the sample size is sufficiently large and the probability parameters aren't too small. It can be shown via the Central Limit Theorem that the multinomial distribution converges to the normal distribution as the sample size approaches infinity; however, there is no easy way to show mathematically how and when the convergence fails.'''"
</blockquote>
<ref>http://www.minitab.com/support/docs/Answers/Chi-Square%20Test%20Assumptions.pdf</ref>

<blockquote>
"'''The chi-square test is simpler to calculate but yields only an approximate P value. ... You should definitely avoid the chi-square test when the numbers in the contingency table are very small (any number less than about six)'''."
</blockquote> <ref>http://www.graphpad.com/www/Book/Choose.htm</ref>

<blockquote>
"The most important things to remember to get a valid χ2 test are that the expected values are not too small in any bin (certainly 5 or more), and that the degrees of freedom are properly evaluated. '''Unless you have a very large amount of data, the test is not very sensitive and errs on the side of safety. If you get a significant result, however, it is not likely to be wrong.'''"
</blockquote> <ref>http://mysite.du.edu/~jcalvert/econ/chisquar.htm</ref>

<blockquote>
"The critical assumptions of the chi-square test for k independent samples are similar to those for the chi-square test for two independent samples. ... 
'''4. No more than 20% of the cells may have expected frequencies of less than 5, and no cell should have an expected frequency of less than 1. 
The rule given in Assumption 4 is particularly important for a contingency table that is larger than 2X2'''"
</blockquote> <ref>http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1
</ref>

<blockquote>
"Special problems with small expected cell frequencies for the chi-square test: 
The chi-square test involves using the chi-square distribution to approximate the underlying exact distribution. The approximation becomes better as the expected cell frequencies grow larger, and '''may be inappropriate for tables with very small expected cell frequencies.''' 
'''For tables with expected cell frequencies less than 5, the chi-square approximation may not be reliable. A standard (and conservative) rule of thumb (due to Cochran) is to avoid using the chi-square test for tables with expected cell frequencies less than 1, or when more than 20% of the table cells have expected cell frequencies less than 5.''' 
Another rule of thumb (due to Roscoe and Byars) is that the average expected cell frequency should be at least 1 when the expected cell frequencies are close to equal, and 2 when they are not. (If the chosen significance level is 0.01 instead of 0.05, then double these numbers.) 
Koehler and Larntz suggest that if the total number of observations is at least 10, the number categories is at least 3, and the square of the total number of observations is at least 10 times the number of categories, then the chi-square approximation should be reasonable. 
Care should be taken when cell categories are combined (collapsed together) to fix problems of small expected cell frequencies. Collapsing can destroy evidence of non-independence, so a failure to reject the null hypothesis for the collapsed table does not rule out the possibility of non-independence in the original table. 
'''As with most statistical tests, the power of the chi-square test increases with a larger number of observations. If there are too few observations, it may be impossible to reject the null hypothesis even if it is false.'''"
</blockquote> <ref>http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html</ref>--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

: Thanks for this really excellent contribution, ElyM. The only thing I'd like to add is in relation to your initial statement, "I do not believe that anyone has claimed that 'chi-square test p-values are always conservative'". The question of whether a test is conservative in a particular situation is probabilistic. One can determine whether a test is likely to generate a p-value which is too high in a particular situation (e.g. for a chi-squared test, when there are lots of small expected values) but one needs an exact test (such as an appropriate Monte Carlo randomisation test) to determine whether the p-value in any ''particular'' test is in fact excessively high.

::This edit also claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639373]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

: I hope careful reading of your very clear description will put SJohnson's mind at rest on this subject. [[User:FredFerguson|FredFerguson]] 18:11, 14 March 2009 (EDT)

ElyM, you've provided nothing to address the basic flaw that "The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events." See [[Flaws in Lenski Study]]. Also, do not impose your view on the content page until after SJohnson has had an opportunity to respond to your posting. As to "Fred", his put-downs are getting tiresome and I'm going to review his edit pattern now to see if he's been contributing anything of value to this site.--[[User:Aschlafly|Andy Schlafly]] 14:06, 15 March 2009 (EDT)

::Mr. Schlafly, per your request I have not added anything to the content page as SJohnson has not yet responded to my posts. Since all of my comments have been in regards to SJohnson's use of the chi-square test in this particular article, I'm not sure why you expect me to address Blout's use of Monte Carlo - that issue seems to be addressed on the [[Flaws in Lenski Study]] page. SJohnson has added a reformulation of the chi-square test for two possible outcomes, and stated that the chi-square test is at a minimum when all success probabilities are equal. He then extrapolates from this to claim that the chi-square test is an effective test for the data from Blount.

::The reformulation of the equations for two possible outcomes does not address the underlying problem that the chi-square test has universally accepted parameters outside of which it is considered an invalid test; I have provided references for these parameters and shown that the data from Blount lies outside them. None of the expected cells in SJohnson's analysis have values above one, and the total n is four. SJohnson's own reference states that the application of the chi-square test in this circumstance is a "violation of good statistical practice". Analogously, combining F=ma and t=(vf-vi)/a into t=(vf-vi)m/F and showing that t is a minimum when m approaches zero does not address the fact that those Newtonian equations do not apply as velocities approach the speed of light. The legitimacy of Blount's arguments cannot be determined by the application of illegitimate counterarguments. If SJohnson or others can point to references from the statistical literature that show that Blount has made methodological errors - as I have been able to do with SJohnson's chi-square analysis - I would welcome their input, and no doubt Conservapedia's other readers would as well, and this page would be greatly improved.

::I have not seen a rebuttal from SJohnson in the four days since my last post, although he has added new material to the content page since then. In light of this, I would appreciate some guidelines as to when it is appropriate for me to add my information and references to the content page. I can add citations from the primary mathematical literature if necessary, but in general I find that these are less helpful as they are not easily accessible by readers without access to academic libraries.--[[User:ElyM|ElyM]] 18:07, 18 March 2009 (EDT)

::: You say, "I'm not sure why you expect me to address Blout's use of Monte Carlo." The reason is obvious: the title of the content page is the "Significance of E. Coli Evolution Experiments." You haven't addressed the inappropriateness of using Monte Carlo simulations for assessing the significance rarely occurring events, which was central to Lenski's statistical claims. I suggest you address this flaw if you want to be taken seriously.--[[User:Aschlafly|Andy Schlafly]] 23:12, 18 March 2009 (EDT)

::::ASchlafly, again per your request, and based on your statement regarding the "inappropriateness of using Monte Carlo simulations for assessing the significance of rarely occurring events", I have spent the last several days reviewing the literature available to me on Monte Carlo and other resampling techniques, looking for ways in which Blount may have made a methodological error of the sort that SJohnson has made. I have been unable to find any examples of authors suggesting that Monte Carlo be avoided for low ''n'', or for events with low probability regardless of'' n'', much less providing specific cutoff numbers as are seen in the references that I provided for the chi-square test. Similarly, the technique that Blount used does not require/assume that categories are unrelated, as the chi-square test does. Of course, the absence of evidence is not evidence of absence, and I may have misinterpreted the basis of your objection. At this point I'll need you to explain your objection in more detail if you wish me to find the appropriate literature addressing your concerns. Do you believe that the number of resamplings was too low in Blount's paper? That the analysis should have been performed with a software package other than Statistics101? Some other procedural issue? Some issue of interpretation?

::::The statistical problem that Blount must address is straightforward: given a distribution of mutant cultures that ''appears'' to be skewed toward the higher generations, what is the probability that this same amount of skew (or a greater degree) could arise by chance, given the null hypothesis that every generation is equally likely to produce a mutant? Interestingly, in the case of the first replay experiment, the total number of ways to randomly select (equal probability, no replacement) four cultures from seventy-two is 72x71x70x69, or 24,690,960. This number is small enough that a program can brute-force-calculate the 'mean generation number' of ''all possible'' combinations of four cultures in a reasonable amount of time. An experimentally-derived 'mean generation number' can be checked against this exhaustive list, and the number of means equal to or larger than the experimental mean can be found exactly. Converting this number to a percentage of 24,690,960 provides an exact p-value for any given experimental 'mean generation number'. This exhaustive approach is different than the Monte Carlo technique, in that ''all possible'' outcomes are examined, rather than a ''random subset'' of all possible outcomes. For the first replay experiment, it provides a way to independently check Blount's Monte Carlo results. This approach is not possible for the second and third replay experiments, in which the total number of possible combinations becomes impractically large: 340!/335! = 4.41 x10^12 and 2800!/2792! = 3.74 x 10^27, respectively.

::::I asked a colleague to run just such a brute-force program for me on the first replay data. I also ran several Monte Carlo simulations ('''not''' using Statsistics101) with Blount's data, using twenty-five million, one hundred million, and 493,819,200 resamplings - note that this last is twenty times the number of all possible combinations of 4 samples drawn without replacement from 72. The p-values from the 25M, 100M, and 493M Monte Carlo resamplings (0.00844, 0.00846, and 0.00846, respectively) compare favorably with Blount's 1M value of 0.0085 and the non-Monte-Carlo brute-force exact calculation, which provides a p-value of 0.008457. Thus it appears that Blount's statistical results are confirmed by a ''non-Monte Carlo'' technique, at least for the first replay experiment.

:::::What do you get for the experiment two p-value using the method from the paper and at least ten million realizations? [[User:SJohnson|SJohnson]] 08:48, 25 March 2009 (EDT)

::::::For the second replay experiment, Blount reports that one million resamplings gives a p-value of 0.0007. When I run the Monte Carlo simulations, ten million resamplings give a p-value of 0.00060; one hundred million resamplings give a p-value of 0.00062, and one ''billion'' resamplings give a p of 0.00061.

::::::As to the brute-force method for the second replay: the 4.41x10^12 combinations of five cultures picked from 340 actually represents 'only' 36.8 billion unique combinations, since for the purposes of calculating a mean generation value, the ordering of the cultures does not matter: 0, 0, 0, 0, 10 gives the same mean as 10, 0, 0, 0, 0 and 0, 10, 0, 0, 0. With brute force, it turns out that out of the 36,760,655,568 unique combinations possible in the second replay, 22,536,306 have means that are greater than or equal to 32,100.

::::::22,536,306 / 36,760,655,568 = 0.000613 = the ''exact'' p-value derived from exhaustive evaluation rather than Monte Carlo.

::::::The third replay has 9.27 x 10^22 unique combinations; at a billion comparisons a minute it would take over 170,000,000 years to check them all.--[[User:ElyM|ElyM]] 12:36, 25 March 2009 (EDT)

::::::----------
::::::Here are pointers to the freely available Statistics 101 package <ref>http://www.statistics101.net/statistics101web_000003.htm</ref> and the actual programs run through the package by Blount ''et al.'' <ref>http://myxo.css.msu.edu/ecoli/citrate2008/MCprograms.html</ref>. The stats package is written in Java and should run under many operating systems. A 10 million trial run of the second experiment took a bit of time and yielded a p-value of 0.00061. Ten separate, one-million trial runs produced an average p-value of 0.00061 (std.dev=0.00002, n=10). Even with trial sizes of 5K, the numbers averaged about 0.0006 (std.dev=0.0004 n=10).--[[User:Argon|Argon]] 20:52, 25 March 2009 (EDT)

::::My intention is not to get caught up in a digression about Monte Carlo, though - I'd rather keep the focus on the fact that the main article should acknowledge that SJohnson is using chi-square in a way that violates accepted guidelines; this remains true whether Blount's analysis is valid or not.--[[User:ElyM|ElyM]] 12:13, 23 March 2009 (EDT)

== References ==
{{reflist}}

Talk:Significance of E. Coli Evolution Experiments

2009-03-26T00:53:15Z

Argon: Fixed link

SJohnson, your assessment, while good in the utilization of the chi-squared test is unfortunately incorrect. The Monte Carlo resampling gives a more accurate p-value than the chi-squared. You may research the literature (i.e. publications in statistical mathematics, many pubs actualy compare Monte Carlo vs Chi Squared) to discover that this method is commonly used in advance statistical work and how it is more accurate than the chi-squared test.--[[User:Able806|Able806]] 17:00, 4 March 2009 (EST)

:It doesn’t make sense to compare the chi-square test, which is a specific statistical hypothesis test, to Monte Carlo methods, which can be used for anything from fluid motion modeling to p-value computations. You can use Monte Carlo methods to compute the p-values of the chi-square test!

:Monte Carlo methods involve the generation of random realizations. Your broad claim the Monte Carlo methods are “more accurate” than the chi-square test is obviously incorrect because the accuracy of Monte Carlo methods always depends on the number of random realizations generated. When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous.

:Which publications compare Monte Carlo to chi-square and show that the former is more accurate? Could you provide specific examples? Thanks. [[User:SJohnson|SJohnson]] 18:50, 4 March 2009 (EST)

:In furtherance of SJohnson's remarks with respect to rarely occurring events, the use of the basic Monte Carlo method is plainly incorrect for modeling a rarely occurring event, as the Lenski paper did. This has long been pointed out in [[Flaws in Richard Lenski Study]]. I know [[evolutionists]] will never admit a flaw in anything promoting their pet theory, but this (and other) flaws in that paper is undeniable.

:Watch how evolutionists defended obvious errors in the Lenski paper, and then realize why the [[Piltdown Man]] fraud was taught for 40 years without evolutionists admitting it was a hoax.--[[User:Aschlafly|Andy Schlafly]] 09:55, 5 March 2009 (EST)

::Andy, how exactly is the Monte Carlo method incorrect to use in this case? I have seen it used in publications with much smaller datasets.--[[User:Able806|Able806]] 10:29, 5 March 2009 (EST)

:::Able806, I'm interested in looking at the publications you mentioned that use Monte Carlo methods to analyze small data sets. Could you provide some examples? Thanks. [[User:SJohnson|SJohnson]] 16:41, 5 March 2009 (EST)

::::SJohnson, here are two papers, [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6WH8-45RFJ1J-19&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=1ad95954654bb97b17e474ce6b469f6e 1] and [http://cat.inist.fr/?aModele=afficheN&cpsidt=787963 2]. Most are in chemistry and genetics where you find the observed to be much smaller and have to use the MCM. You can search on the subject as well and find that how Lenski performed the test is the standard for microbiological genetic analysis.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

:::::Those papers have nothing to do with hypothesis testing. One is an archeology paper. To be blunt, it seems like you’re just doing internet searches on “Monte Carlo” to find these links. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

::::::SJohnson, actually they do, did you read the papers? If so you would see how they used the MCM for their data analysis of small data sets, which indeed was hypothesis testing and answers you inquiry about publications that use MCM for small data set analysis. If you wish I can try to track down some actual mathematical publications, however, I am not as familiar with mathematical journals as I am with science/medical journals (not knowing which mathematical journals are acceptable). I am assuming that you have a background in math and possibly access to mathematical journals, therefore if you know the reputable ones I can do the leg work.
::::::I believe the thing that needs to be looked at is there truly a problem with the choice of test and if so what is an alternative. Bayesian might be an option but seems to be difficult to employ for this situation.--[[User:Able806|Able806]] 12:36, 12 March 2009 (EDT)

:::Able806, you still seem to miss the point about how inappropriate the Monte Carlo method (as used in the Lenski paper) is for evaluating rarely occurring events. You need to open your mind to be productive. If you simply cling to a view that Lenski (who I don't think has any meaningful education in statistics) must somehow be right, then you're not going to make any progress in understanding the flaws.--[[User:Aschlafly|Andy Schlafly]] 17:07, 5 March 2009 (EST)

::::Andy, you still have not answered what you find inappropriate about his use of the Monte Carlo method? I am a reasonable person and with evidence I do have an open mind. I provided examples last week, with a working model, showing that Monte Carlo is better than the chi-square in this case. I have also shown where the Chi-Square was inappropriate due to the occurrence size as well. So if you have any evidence that Monte Carlo should not be used in the way that Lenski used please let it be shown.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

Sjohnson, I believe you just proved my point. In the literature of mean and covariance structure analysis, non-central chi-square distribution is commonly used to describe the behavior of the likelihood ratio statistic under alternative hypothesis; it is widely believed that the non-central chi-square distribution is justified by statistical theory. Actually, when the null hypothesis is not trivially violated, the non-central chi-square distribution cannot describe the LR statistic well even when data are normally distributed and the sample size is large. Monte Carlo results compare the strength of the normal distribution against that of the non-central chi-square distribution. In an association analysis comparing cases and controls with respect to allele frequencies at a highly polymorphic locus, a potential problem is that the conventional chi-squared test may not be valid for a large, sparse contingency table. Reliance on statistics with known asymptotic distribution is unnecessary, as Monte Carlo simulations can be performed to estimate the significance level of the test statistic.

Here is a [http://faculty.vassar.edu/lowry/chi_beta.html link] to a great page the provides an interactive example as to why the Chi Squared test would provide poor results compared to the Monte Carlo in relation to the Lenski data workup.

Something you may have overlooked was that the data set is actually too small to use the chi square method correctly. It is often accepted that is any of the analyzed data falls under 10 for a particular cell of the data set then the Yates correction needs to be applied; unfortunately the Yates correction can over correct thus skewing the p-value. Lenksi seemed to understand this by supporting his Monte Carlo p-value results with the Fisher z-transformation p-value.

I hope this helps.--[[User:Able806|Able806]] 10:27, 5 March 2009 (EST)

:I’m still waiting to hear which literature says that “Monte Carlo resampling” is “more accurate than the chi-squared test”. The page mentioned above [http://faculty.vassar.edu/lowry/chi_beta.html] is a discussion of why statisticians “fail to reject the null” rather than “accepting the null” when the p-value is above 0.05 or so. The page says nothing about superiority of Monte Carlo methods. Why were alternate hypothesis distributions mentioned? Only the null hypothesis distribution is used to calculate a p-value. Yates’s correction is for 2x2 contingency tables [http://en.wikipedia.org/wiki/Yates%27_correction_for_continuity]. It doesn’t apply in this case. Finally, what the heck do “covariance structure analysis” and “allele frequencies at a highly polymorphic locus” have to do with this problem? [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::SJohnson, I am looking for this paper for you, I cited it for one of my past publications dealing with allele frequencies (I believe it came from the Duke Biostatistics group). To answer your question about allele frequencies, that is the issue at hand, more about the genetics than the math, but it is the item being studied. So you stated that Yates can not be used and statistics says the number of occurrences is too small to evaluate using the Chi-Squared test so what would you recommend instead of the Monte-Carlo Method?

:Regarding the "Fisher z-transformation p-value" from the paper, garbage in garbage out. If the p-values were bad to begin with, then why would a combination of them be meaningful? [[User:SJohnson|SJohnson]] 10:49, 9 March 2009 (EDT)

::You are assuming that p-values are wrong based on a test that is inappropriate in this case due to data limitations. Did you perform a z-transformation on the chi-squared for the three data groups?--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

::You asked about the “Fisher z-transformation p-value”. The z-transformation test and Fisher’s method are actually two different things (see Whitlock's 2005 paper - Ref. 49 in Blount et al.). But no, I haven’t tried either. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::There's a large literature on various kinds of Monte Carlo test, a very short summary of which is that they're inevitably more accurate than parametric tests (e.g. F, t, chi-squared, etc) because they don't make assumptions about the distribution of the data under the null hypothesis. See for example ''Introduction to the Bootstrap'' by B. Efron and R. Tibshirani and ''The Jack-knife, the Bootstrap and Other Resampling Plans'', also by Efron. They're certainly applicable to small datasets and their accuracy is really only limited by the number of samples you care to take. E.g. 1000 M-C samples would give you a pretty accurate idea about significance at the alpha<1% level (That book should answer SJohnson's questions of 18:50 on 4/3/09 and 16:38 on 5/3/09 about accuracy and Aschalfly's comment of 17:07 on 5/3/09 about appropriateness of Monte Carlo tests.) [[User:FredFerguson|FredFerguson]] 16:53, 11 March 2009 (EDT)

::::Your claim that Monte Carlo methods are “inevitably more accurate” than other tests is obviously wrong because the accuracy of MC methods always depends on the number of realizations used. You should have written <math>\alpha=1\%</math>, not <math>\alpha<1\%</math>. If 1,000 random realizations are generated, the number of realizations above the true <math>\alpha=1\%</math> level is binomial with mean 10 and variance about 10. Thus, the standard deviation of the MC estimate is >0.003. In this example, a Monte Carlo p-value could be off by 30% and still be within a standard deviation. Is that really “pretty accurate”?

::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 3%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::::You're talking about miniscule differences in the accuracy of a test. 0.013 isn't very different from 0.007. In either case, it's very unlikely the experimenter would have obtained that result if the null hypothesis were true. If you're bothered about differences in P-values to the third decimals (which would make you unusual!), just run more MC realisations, that's all. Not really a problem. [[User:FredFerguson|FredFerguson]] 11:53, 12 March 2009 (EDT)

There’s still confusion about the difference between test statistics and Monte Carlo methods. Before you find a Monte Carlo estimate of a p-value, you need to select a test statistic to reduce the data set to a scalar. I am interested in hearing which test statistic you believe should be used in place of the chi-square test and why. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

----

Quick question for SJohnson: How many degrees of freedom did you choose when calculating the p-value? I'd like to know upon what condition you base that number. Thanks.--[[User:Argon|Argon]] 11:05, 5 March 2009 (EST)

:The degree of freedom for a contingency table is rows minus one times columns minus one. That is, <math> (r-1)(c-1) </math>. Here’s a pretty good tutorial I came across: [http://faculty.uncfsu.edu/dwallace/lesson%2020.pdf]. For the experiments from [http://www.pnas.org/content/105/23/7899.full.pdf], the DOFs are 11, 11, and 13. For experiment one, the chi-square test statistic is
::<math>
X^2
=\sum\limits_i\sum\limits_j
\frac{\left(n_{i,j}-E\left[n_{i,j}\right]\right)^2}
{E\left[n_{i,j}\right]}
</math>
::<math>
=\frac{\left(0-1/3\right)^2}{1/3}
+\frac{\left(6-17/3\right)^2}{17/3}
+\frac{\left(0-1/3\right)^2}{1/3}
+\ldots+
</math>
::<math>
+\frac{\left(2-1/3\right)^2}{1/3}
+\frac{\left(4-17/3\right)^2}{17/3}
</math>
::<math>
\approx
14.82
</math>
:where <math>n_{i,j}</math> is the observed value and <math>E\left[n_{i,j}\right]</math> is the expected null hypothesis value. So if you have MS Excel, another way to arrive at the p-value of 0.19 is to type “=CHIDIST(14.82,11)” into a cell. Cheers! [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::OK, thanks for the info. From what I'd calculated and looked up in tables, the numbers seemed close to a df=11 for a chi-square of ~14. (Aside: With terms having 17/3 in the denominator in the figures above, were you using the test of independence? I was using Pearson's test for [http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_for_fit_of_a_distribution fit of a distribution] which returns a chi-squared value of 14 and roughly matched the p-values you reported, assuming the df was 11).

::Also, the first sentence of the article reads: "Blount, Borland, and Lenski[1] claimed that a key evolutionary innovation was observed during a laboratory experiment. That claim is false." A small correction: There were several claims in the paper. The 'key evolutionary innovation' was acquiring the ability to utilize citrate as a food source. That claim was demonstrated multiple times. The claim, which pertains to this statistics discussion was that the Cit+ phenotype arose in a multi-step process, first requiring a rare, pre-adaptive mutation before additional mutation(s) lead to the subsequent development of citrate utilization.--[[User:Argon|Argon]] 20:46, 5 March 2009 (EST)

:::My biology-degreed wife assures me that mutation does not necessarily mean that evolution occurred. What the paper claimed is that evolution (a “key innovation”) occurred in the lab. The key innovation supposedly increased the mutation rate. In the experiments, the observed mutation rate increased after generation 31,000, but not enough to make a statistically significant claim that the rate is not constant. The analysis in the paper was similar to flipping a coin ten times, counting six heads and claiming that the coin must be biased against tails. In reality, there’s nothing surprising about a fair coin producing slightly more of one outcome than the other. Just like there's nothing surprising about there being slightly more mutations in later generations than early generations given the null hypothesis (constant mutation rate). [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

>>Inserting a later comment first<<
SJohnson, the paper's title is: "Historical contingency '''and the evolution of a key innovation''' in an experimental population of ''Escherichia coli''" As I mentioned earlier, the key innovation is the evolution of the Cit+ phenotype and not the timing or rate of its acquisition. And yes, it *is* evolution (call it microevolution, if you wish). Blount et al went on further to speculate how this evolutionary innovation arose and they proposed the historical contingency hypothesis in which 'pre-adaptive' mutations were required before the Cit+ phenotype developed. It is only this latter hypothesis that you are attempting to address with your chi-square analysis, not the fact that Cit+ mutants arose (which is the evolutionary innovation).--[[User:Argon|Argon]] 21:57, 18 March 2009 (EDT)

::::SJohnson, not to say anything about your wife, but has she had a 400 level molecular genetics course (most general biology degrees do not cover the detail unless they are specialized)? If so, she would have mentioned that if the mutation passes to the offspring and is selectively beneficial to the population then it is a step of evolution as along as the conditions continue through the sharing of the mutation with the population and the environment is such that reduces the growth rate of the non-transformed population. While not all mutations are signs that evolution occurred the mutations that pass to offspring and provide a benefit compared to other offspring are very strong indicators. In the case of this paper the population that evolved the cit+ was able to metabolize a chemical in their environment which allowed for an adaptation advantage compared to the non-transformed colonies.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

----

Let’s go back to the beginning. There appears to be confusion about the difference between test statistics and methods for computing p-values. As is noted at the beginning of the page [http://www.conservapedia.com/Significance_of_E._Coli_Evolution_Experiments], the fundamental problem with the paper is that it used a flawed test statistic, not that it used Monte Carlo methods to find the p-value for that flawed statistic.

Every hypothesis test uses a test statistic to reduce the data to a single number. The p-value for the test statistic can be calculated analytically (as I’ve done for the chi-square test statistic) or by Monte Carlo methods. In the paper, Monte Carlo methods were used to compute the p-value of the “mutation generation” test statistic. The key problem with the analysis from the paper is that it doesn’t work to use a weighted average to test for variations in mutation rate. This is like trying to use the sample variance to test for an increase in the mean in Gaussian-distributed data. A statistic should be selected based on the null and alternate hypothesis distributions of the data. The chi-square test (unlike the weighted average from the paper) is a reasonable choice for data that mutates at a constant rate under the null hypothesis, but mutates at varying rates under the alternate hypothesis.

Able806, you made a good point about the contingency table cell frequencies being relatively low, but were wrong when you said ”the data set is actually too small to use the chi square method correctly”. In the low cell frequency case the chi-square test is still effective, but the null hypothesis distribution of the chi-square statistic starts to look less like the chi-square distribution. Thus, p-values calculated using the chi-square distribution may be a bit off. However, Monte Carlo p-values are always imperfect as well because it's impossible to generate an infinite number of random realizations. There are imperfections in p-values generated by analytic and Monte Carlo methods. However, low cell frequencies does not explain the >20x and >2.5x differences between chi-square p-values and p-values from the paper for experiments one and three. The reason for those huge differences was the use of the flawed test statistic (“mutation generation”) in the paper. [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

:SJohnson, the chi-squared test is a valuable statistical tool, but the limitations of the test must be acknowledged. The chi-squared test can only produce valid results if the assumptions that underly the test are not violated. As an analogy, Newtonian models of motion fail to produce accurate results as velocities approach the speed of light; under those circumstances one must switch to a theory that accounts for relativistic effects.

:It seems that you have simply dismissed the [http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm widely-acknowledged] [http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm fact] that the [http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html chi-squared test] is [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236 inappropriate] for use in [http://www.graphpad.com/www/Book/Choose.htm situations] where n in any cell is [http://mysite.du.edu/~jcalvert/econ/chisquar.htm less] less than a [http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1 threshold] [http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html number]. Different authors set different thresholds, but all are well above the numbers seen in your chi-squared analysis - even the most liberal guidelines advise against the chi-squared test when any expected cell frequency is less than one or more than 20% of the table cells are less than 5; others require that expected values in all cells must be more than 5. With smaller amounts of data, the test is insensitive and errs on the side of rejecting the hypothesis. If you attempt your chi-squared statistical analysis with a program that is more sophisticated than MS Excel (as I did), you get an error message indicating that the results are invalid due to low expected cell counts.

:That issue aside, there are other reasons that the chi-squared test is inappropriate here. As the links above point out, the categories tested must be truly independent; one example is that you can't use the chi-squared test to compare age and ability to kick a field goal by testing the same experimental group twice, one year apart; you have to test one group of age A and a different group of age B. In the case of the Blount paper, the categories are not independent. Even if there were adequate numbers to address the low-expected-frequency problem, this would make the chi-squared an invalid test in this case.

:There are other significant problems with the use of the chi-squared test in this circumstance, but they can wait until you address these first major problems.--[[User:ElyM|ElyM]] 12:18, 11 March 2009 (EDT)

::Wackerly et al. says in general it’s assumed that the cell frequencies are above five so that the chi-square statistic (under the null) is approximately chi-square distributed (see p. 703). That book does not say chi-square test results are invalid if frequencies are five or less. Your example of a chi-square test warning message (it said "warning" not "error" as you stated) in Minitab [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236] said “approximation probably invalid” referring to the chi-square distribution approximation to the chi-square test statistic’s distribution. Your example did not say “chi-square test invalid”. I agree that when cell frequencies are low, the chi-square test statistic’s distribution starts to deviate from the chi-square distribution. I maintain that this deviation is not enough to explain the >2.5x and >20x differences in the chi-square test p-values and the p-values from the paper.

::As the numerous links in your post proved, the chi-square test is widely-used by statisticians. Can you give examples of statisticians using mean mutation generation as a test statistic? Also, did your software agree with the chi-square test p-values I presented? Thanks. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::Thank you for giving page references for Wackerly; however it seems we have different editions, since page 703 in my copy (5th ed, 1996) does not deal with chi-squared issues at all. My copy does state the following, on page 622: "Although the mathematical proof is beyond the scope of this text, it can be shown that, when n is large [chi-squared] will possess approximately a chi-square probability distribution in repeated sampling." Then, on page 624: "Experience has shown that cell counts [n sub i] should not be too small in order that the chi-square distribution provide an accurate approximation to the distribution of [chi squared]. As a rule of thumb we require that all expected cell counts equal or exceed 5, although Cochran (1952) has noted that this value can be as low as 1 for some situations." Wackerly then goes on, in the problems sections, to describe the use of the chi-squared test as a "violation of good statistical practice" when "some expected counts [are] <5."

:::It seems that you are already aware that the [chi-square] statistic under the null is no longer chi-square distributed for small n; this is precisely why the test should not be used under those conditions. I can claim to be able to accelerate a 1-kg mass to 10 times the speed of light by applying 1 N of force for 95 years by using F=ma and t= (vf-vi)/a. Plugging the numbers into those equations will produce the same result every time, but the answer is illegitimate because those equations are only valid under certain assumptions, which are violated as velocities approach the speed of light. Similarly, having a statistical program calculate a chi-squared value given the Blount data will produce a number result, but since the assumptions of the test are violated the result is not legitimate. Yes, if I put the Blount data in SAS 9.2, I get the same numerical answer as you do, but I also get the following message: "WARNING: >89% of the cells have expected counts less than 5. Chi-square may not be a valid test." You may argue that that's a warning, not an error; that's a semantic distinction. The reason that the program says that it MAY not be valid is that the chi-squared test skews in the direction of being too conservative at low n values; the test has an acceptable rate of false positives but an unacceptably high rate of false negatives. Comparing the results of the Monte Carlo and chi-squared results in this case is like comparing the results of Newtonian and relativistic equations of motion: they can produce very different results from the same input data.

::::For a finite amount of data, the chi-square statistic is never chi-square distributed under the null. The p-values are always approximate regardless of cell frequencies. The approximation becomes more accurate as the amount of data increases, but I don’t believe that this inaccuracy will change p-values that are about 0.2 (for experiments 1 and 3) into statistically significant p-values. How much do you expect the p-values to change if an exact computation is used in place of the chi-square distribution approximation? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::Your last paragraph has a major non sequitur in it: yes, many statisticians use the chi-square test. As long as the assumptions of the test are not violated, it is a valuable tool. That has nothing to do with the validity of using mean mutation generation as a test statistic. 'Mean number of werewolf attacks in Mumbai in the week centered on the new moon, by month, from 1654 to 1798' is a valid test statistic. I am quite sure that it has never been used in a peer-reviewed paper before. That does not mean that I can't perform valid statistical tests on that statistic. If, however, the incorrect test is applied, the results of the analysis will be flawed. Papers apply a (relatively small) standard repertoire of valid tests to a (potentially infinite) number of test statistics. The particular test statistic used in a paper may never have been used before and may never be used again; that does not address the validity of the analysis. In Blount's case, the test is the Monte Carlo analysis, which is also "widely-used by statisticians".

::::There are an infinite number of ways to reduce a data set to a single number. However, it’s foolish to think every method would be effective. I gave an example of a flawed test statistic in an earlier post [http://www.conservapedia.com/index.php?title=Talk%3ASignificance_of_E._Coli_Evolution_Experiments&diff=635070&oldid=634987]. Another example of a flawed test statistic is the one used in the paper because it does not always detect deviations from the null hypothesis (see: [[Significance of E. Coli Evolution Experiments#Test Statistics]]).

::::Test statistics are typically derived. The likelihood ratio test is a common method used to derive them. The chi-square test for independence is an approximation to the LRT. Where is the derivation saying that mean mutation generation is an appropriate test statistic for this problem? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::We still haven't touched on the issue of the categories not being independent, which by itself is sufficient to invalidate the chi-squared technique. I'm new to this site, so I'm unsure as to the etiquette of making changes to the articles of another person - but the article here should at the very least mention that the chi-square test is being used here in a manner that violates its underlying assumptions in at least two fundamental ways, and the results are therefore suspect.--[[User:ElyM|ElyM]] 17:34, 12 March 2009 (EDT)

:::::When generating random realizations of experiment outcomes, the authors assumed that the total number of mutants was fixed. Thus the paper assumed the numbers of mutants per generation are statistically dependent. Does this seem like a realistic model, or do you think that if the experiments were recreated that the total number of mutants could vary? For example, if experiment one were recreated, would the total number of mutants always be exactly four? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::: It looks to me as though SJohnson has misinterpreted the application of the chi-squared test in quite a fundamental way. His/her analysis of Blount's data are therefore close to meaningless, regardless of whether the test used by Blount is appropriate or not. In my opinion, the entire page should therefore be deleted. [[User:FredFerguson|FredFerguson]] 08:18, 13 March 2009 (EDT)

::::: "Fred", perhaps you mistakenly think this is Wikipedia, where [[censorship]] and deletion of pages for ideological reasons are common. Not here.--[[User:Aschlafly|Andy Schlafly]] 10:23, 14 March 2009 (EDT)

:::::: Umm... I'm suggesting deletion for mathematical reasons, not ideological reasons. Using an argument filled with mathematical errors to try to support your case only detracts from your credibility. [[User:FredFerguson|FredFerguson]] 10:38, 14 March 2009 (EDT)

:::::: Actually, I think correction is better than deletion. So that's what I've done. [[User:FredFerguson|FredFerguson]] 11:01, 14 March 2009 (EDT)

::::::: I find no credibility in your denial of having ideological reasons.--[[User:Aschlafly|Andy Schlafly]] 11:04, 14 March 2009 (EDT)

== Misinterpretation of test ==

SJohnson, Your analysis misinterprets the test. You say the null hypothesis is that this mutation cannot happen. They saw a mutation (4 mutations, in fact, in the data set you show) so the null hypothesis (as you state is) is disproved. That's perfectly straightforward.

I don't know what the "mean mutation generation" test is but you're doing when you apply a chi-squared test to this dataset is to test if the mutations are evenly distributed throughout the generations. Your test says they are, so there's no strong evidence to suppose that mutations are likely to occur in one generation rather than another in the series of tests. Blount's test says thay aren't, so it's more likely that the mutation will occur later in the series of tests. I can't tell which test is right without knowing more about the test that Blount used.

But that point (the foregoing paragraph) has no bearing at all on the null hypothesis, as you describe it. The mutation appeared, so that means the hypothesis that the mutation can't happen is disproved. Very simple. [[User:FredFerguson|FredFerguson]] 21:10, 8 March 2009 (EDT)

:I never said that “the null hypothesis is that this mutation cannot happen”. The chi-square test statistic I'm using wouldn’t be defined if the null hypothesis mutation rate was zero because the <math>E\left[n_{i,j}\right]</math> term in the denominator of the statistic (see above equation) would be zero.

:The test statistic from the paper is the average of the generation numbers of observed mutations. For experiment one this number is
::<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)= 31750.
</math>
:The same number is shown in Table 2 of the paper. [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

:: SJohnson, the way you're calculating the chi-squared statistic implies that you're testing the null hypothesis of a constant mutation rate over time against an alternative hypothesis of a mutation rate which varies over time. [[User:FredFerguson|FredFerguson]] 11:02, 9 March 2009 (EDT)

As it currently stands, the article makes the following statement: "The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown." This misstates the null hypothesis of the paper, which is elaborated in the Introduction section of the paper, and repeated in the section '''Statistical Analysis of the Replay Experiments''':
<blockquote>
For each experiment, we compared the observed mean generation of those clones that yielded Cit+ variants to the mean expected under the null hypothesis that clones from all generations have equal likelihood. The null thus corresponds to the rare-mutation hypothesis laid out in the Introduction."
Block quote
</blockquote><ref>www.pnas.org/cgi/reprint/105/23/7899.pdf</ref>

The article also continues to describe 'mean mutation generation' as a ''test'' rather than a ''statistic'' to which the ''Monte Carlo test'' was applied.--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

I'm a bit confused about why the Chi-squared test, which we're told compares the results to a null hypothesis of a constant mutation rate, seems insensitive to which generations the Cit+ mutations are found. Instead, the chi-square test seems only to be evaluating whether the frequencies of Cit+ mutations in any particular generation are 'expected'. Thus the test is asking whether finding a distribution (e.g. in the first experiment) across nine periods that have no mutations, two periods that have one mutation and one period with two mutations is a statistically significant deviation from what you'd expect of the mutations were randomly distributed. The number returned from the function is the same regardless of the order of Cit+ results. The number of mutations per bin is not the only question being asked. Instead it's the '''order and temporal distribution''' of Cit+ mutants that the analyses probably need to confront. It's not whether one can get nine no-mutants, two single mutants and one double-mutant result, it's a matter of '''when''' they occur and whether that distribution affects the significance of the results. Blount's hypothesis is that mutations should appear later in the experiment. When formulating a suitable null hypothesis, wouldn't one want to take the timing of Cit+ mutants into consideration too?--[[User:Argon|Argon]] 22:25, 18 March 2009 (EDT)

==Unreferenced Claims==

I deleted the claim that mean mutation generation is an appropriate test statistic because no reference was produced that back that claim. No reference was provided to back the claim that the chi-square test p-values are always conservative, either. [[User:SJohnson|SJohnson]] 12:57, 14 March 2009 (EDT)

: The reference is Everitt. I'll check I put it in the right place. [[User:FredFerguson|FredFerguson]] 13:30, 14 March 2009 (EDT)

There was a typo in my edit summaries on the talk page and the main page. I meant to say "Removed unsupported claims" rather than "Removed supported claims". [[User:SJohnson|SJohnson]] 13:13, 14 March 2009 (EDT)

I do not believe that anyone has claimed that 'chi-square test p-values are ''always'' conservative'. The claim that has been made is that ''under certain circumstances'', namely low n and low individual cell values, the chi-square test is an invalid test; that under those circumstances the power of the test is low and it becomes impossible to reject the null hypothesis even when it is false. You may have missed the pertinent sections in my links above, so I will directly quote the relevant sections. All the quoted sections refer to chi-square testing in particular. Any bolding below is mine.

:This edit claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639379]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

<blockquote>
"Assumptions: 

Even though a nonparametric statistic does not require a normally distributed population, there still are some restrictions regarding its use. 

1. Representative sample (Random) 
2. The data must be in frequency form (nominal data) or greater. 
3. The individual observations must be independent of each other. 
4. '''Sample size must be adequate. In a 2 x 2 table, Chi Square should not be used if n is less than 20. In a larger table, no expected value should be less than 1, and not more than 20% of the variables can have expected values of less than 5'''. 
5. Distribution basis must be decided on before the data is collected. 
6. The sum of the observed frequencies must equal the sum of the expected frequencies."
</blockquote>
<ref>http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm
</ref>

<blockquote>
"Assumptions:
* Random sample data are assumed. As with all significance tests, if you have population data, then any table differences are real and therefore significant. If you have non-random sample data, significance cannot be established, though significance tests are nonetheless sometimes utilized as crude "rules of thumb" anyway.
* A sufficiently large sample size is assumed, as in all significance tests. '''Applying chi-square to small samples exposes the researcher to an unacceptable rate of Type II errors. There is no accepted cutoff. Some set the minimum sample size at 50, while others would allow as few as 20'''. Note chi-square must be calculated on actual count data, not substituting percentages, which would have the effect of pretending the sample size is 100.
* '''Adequate cell sizes are also assumed. Some require 5 or more, some require more than 5, and others require 10 or more. A common rule is 5 or more in all cells of a 2-by-2 table, and 5 or more in 80% of cells in larger tables, but no cells with zero count'''. When this assumption is not met, Yates' correction is applied.
* Independence. Observations must be independent. The same observation can only appear in one cell. '''This means chi-square cannot be used to test correlated data (ex., before-after, matched pairs, panel data)'''.
* Similar distribution. Observations must have the same underlying distribution.
* Known distribution. The hypothesized distribution is specified in advance, so that the number of observations that are expected to appear each cell in the table can be calculated without reference to the observed values. Normally this expected value is the crossproduct of the row and column marginals divided by the sample size.
* Non-directional hypotheses are assumed. Chi-square tests the hypothesis that two variables are related only by chance. If a significant relationship is found, this is not equivalent to establishing the researcher's hypothesis that A causes B, or that B causes A.
* Finite values. Observations must be grouped in categories.
* Normal distribution of deviations (observed minus expected values) is assumed. Note chi-square is a nonparametric test in the sense that is does not assume the parameter of normal distribution for the data -- only for the deviations.
* Data level. No assumption is made about level of data. Nominal, ordinal, or interval data may be used with chi-square tests."
</blockquote>
<ref>http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm</ref>

<blockquote>
"Assumptions: 
-None of the expected values may be less than 1 
-No more than 20% of the expected values may be less than 5"</blockquote> <ref>http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html</ref>

<blockquote>
"When performing a chi-square test, your data must satisfy important assumptions. Although these assumptions may be stated differently in different textbooks, they generally assert that: 
1)The sample must be randomly drawn from the population 
'''2)The sample size, n, must be large enough so that the expected cell count in each cell is greater than or equal to 5.''' 
Both assumptions must be met in the process of collecting your data, and violations of the second assumption will appear in the Minitab output when you run the analysis. 
... 
'''You may wonder why the second assumption is necessary for performing the chi-square test. The second assumption arises because the distribution of counts under the null hypothesis is multinomial, and the normal distribution can be used to approximate the multinomial distribution if the sample size is sufficiently large and the probability parameters aren't too small. It can be shown via the Central Limit Theorem that the multinomial distribution converges to the normal distribution as the sample size approaches infinity; however, there is no easy way to show mathematically how and when the convergence fails.'''"
</blockquote>
<ref>http://www.minitab.com/support/docs/Answers/Chi-Square%20Test%20Assumptions.pdf</ref>

<blockquote>
"'''The chi-square test is simpler to calculate but yields only an approximate P value. ... You should definitely avoid the chi-square test when the numbers in the contingency table are very small (any number less than about six)'''."
</blockquote> <ref>http://www.graphpad.com/www/Book/Choose.htm</ref>

<blockquote>
"The most important things to remember to get a valid χ2 test are that the expected values are not too small in any bin (certainly 5 or more), and that the degrees of freedom are properly evaluated. '''Unless you have a very large amount of data, the test is not very sensitive and errs on the side of safety. If you get a significant result, however, it is not likely to be wrong.'''"
</blockquote> <ref>http://mysite.du.edu/~jcalvert/econ/chisquar.htm</ref>

<blockquote>
"The critical assumptions of the chi-square test for k independent samples are similar to those for the chi-square test for two independent samples. ... 
'''4. No more than 20% of the cells may have expected frequencies of less than 5, and no cell should have an expected frequency of less than 1. 
The rule given in Assumption 4 is particularly important for a contingency table that is larger than 2X2'''"
</blockquote> <ref>http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1
</ref>

<blockquote>
"Special problems with small expected cell frequencies for the chi-square test: 
The chi-square test involves using the chi-square distribution to approximate the underlying exact distribution. The approximation becomes better as the expected cell frequencies grow larger, and '''may be inappropriate for tables with very small expected cell frequencies.''' 
'''For tables with expected cell frequencies less than 5, the chi-square approximation may not be reliable. A standard (and conservative) rule of thumb (due to Cochran) is to avoid using the chi-square test for tables with expected cell frequencies less than 1, or when more than 20% of the table cells have expected cell frequencies less than 5.''' 
Another rule of thumb (due to Roscoe and Byars) is that the average expected cell frequency should be at least 1 when the expected cell frequencies are close to equal, and 2 when they are not. (If the chosen significance level is 0.01 instead of 0.05, then double these numbers.) 
Koehler and Larntz suggest that if the total number of observations is at least 10, the number categories is at least 3, and the square of the total number of observations is at least 10 times the number of categories, then the chi-square approximation should be reasonable. 
Care should be taken when cell categories are combined (collapsed together) to fix problems of small expected cell frequencies. Collapsing can destroy evidence of non-independence, so a failure to reject the null hypothesis for the collapsed table does not rule out the possibility of non-independence in the original table. 
'''As with most statistical tests, the power of the chi-square test increases with a larger number of observations. If there are too few observations, it may be impossible to reject the null hypothesis even if it is false.'''"
</blockquote> <ref>http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html</ref>--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

: Thanks for this really excellent contribution, ElyM. The only thing I'd like to add is in relation to your initial statement, "I do not believe that anyone has claimed that 'chi-square test p-values are always conservative'". The question of whether a test is conservative in a particular situation is probabilistic. One can determine whether a test is likely to generate a p-value which is too high in a particular situation (e.g. for a chi-squared test, when there are lots of small expected values) but one needs an exact test (such as an appropriate Monte Carlo randomisation test) to determine whether the p-value in any ''particular'' test is in fact excessively high.

::This edit also claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639373]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

: I hope careful reading of your very clear description will put SJohnson's mind at rest on this subject. [[User:FredFerguson|FredFerguson]] 18:11, 14 March 2009 (EDT)

ElyM, you've provided nothing to address the basic flaw that "The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events." See [[Flaws in Lenski Study]]. Also, do not impose your view on the content page until after SJohnson has had an opportunity to respond to your posting. As to "Fred", his put-downs are getting tiresome and I'm going to review his edit pattern now to see if he's been contributing anything of value to this site.--[[User:Aschlafly|Andy Schlafly]] 14:06, 15 March 2009 (EDT)

::Mr. Schlafly, per your request I have not added anything to the content page as SJohnson has not yet responded to my posts. Since all of my comments have been in regards to SJohnson's use of the chi-square test in this particular article, I'm not sure why you expect me to address Blout's use of Monte Carlo - that issue seems to be addressed on the [[Flaws in Lenski Study]] page. SJohnson has added a reformulation of the chi-square test for two possible outcomes, and stated that the chi-square test is at a minimum when all success probabilities are equal. He then extrapolates from this to claim that the chi-square test is an effective test for the data from Blount.

::The reformulation of the equations for two possible outcomes does not address the underlying problem that the chi-square test has universally accepted parameters outside of which it is considered an invalid test; I have provided references for these parameters and shown that the data from Blount lies outside them. None of the expected cells in SJohnson's analysis have values above one, and the total n is four. SJohnson's own reference states that the application of the chi-square test in this circumstance is a "violation of good statistical practice". Analogously, combining F=ma and t=(vf-vi)/a into t=(vf-vi)m/F and showing that t is a minimum when m approaches zero does not address the fact that those Newtonian equations do not apply as velocities approach the speed of light. The legitimacy of Blount's arguments cannot be determined by the application of illegitimate counterarguments. If SJohnson or others can point to references from the statistical literature that show that Blount has made methodological errors - as I have been able to do with SJohnson's chi-square analysis - I would welcome their input, and no doubt Conservapedia's other readers would as well, and this page would be greatly improved.

::I have not seen a rebuttal from SJohnson in the four days since my last post, although he has added new material to the content page since then. In light of this, I would appreciate some guidelines as to when it is appropriate for me to add my information and references to the content page. I can add citations from the primary mathematical literature if necessary, but in general I find that these are less helpful as they are not easily accessible by readers without access to academic libraries.--[[User:ElyM|ElyM]] 18:07, 18 March 2009 (EDT)

::: You say, "I'm not sure why you expect me to address Blout's use of Monte Carlo." The reason is obvious: the title of the content page is the "Significance of E. Coli Evolution Experiments." You haven't addressed the inappropriateness of using Monte Carlo simulations for assessing the significance rarely occurring events, which was central to Lenski's statistical claims. I suggest you address this flaw if you want to be taken seriously.--[[User:Aschlafly|Andy Schlafly]] 23:12, 18 March 2009 (EDT)

::::ASchlafly, again per your request, and based on your statement regarding the "inappropriateness of using Monte Carlo simulations for assessing the significance of rarely occurring events", I have spent the last several days reviewing the literature available to me on Monte Carlo and other resampling techniques, looking for ways in which Blount may have made a methodological error of the sort that SJohnson has made. I have been unable to find any examples of authors suggesting that Monte Carlo be avoided for low ''n'', or for events with low probability regardless of'' n'', much less providing specific cutoff numbers as are seen in the references that I provided for the chi-square test. Similarly, the technique that Blount used does not require/assume that categories are unrelated, as the chi-square test does. Of course, the absence of evidence is not evidence of absence, and I may have misinterpreted the basis of your objection. At this point I'll need you to explain your objection in more detail if you wish me to find the appropriate literature addressing your concerns. Do you believe that the number of resamplings was too low in Blount's paper? That the analysis should have been performed with a software package other than Statistics101? Some other procedural issue? Some issue of interpretation?

::::The statistical problem that Blount must address is straightforward: given a distribution of mutant cultures that ''appears'' to be skewed toward the higher generations, what is the probability that this same amount of skew (or a greater degree) could arise by chance, given the null hypothesis that every generation is equally likely to produce a mutant? Interestingly, in the case of the first replay experiment, the total number of ways to randomly select (equal probability, no replacement) four cultures from seventy-two is 72x71x70x69, or 24,690,960. This number is small enough that a program can brute-force-calculate the 'mean generation number' of ''all possible'' combinations of four cultures in a reasonable amount of time. An experimentally-derived 'mean generation number' can be checked against this exhaustive list, and the number of means equal to or larger than the experimental mean can be found exactly. Converting this number to a percentage of 24,690,960 provides an exact p-value for any given experimental 'mean generation number'. This exhaustive approach is different than the Monte Carlo technique, in that ''all possible'' outcomes are examined, rather than a ''random subset'' of all possible outcomes. For the first replay experiment, it provides a way to independently check Blount's Monte Carlo results. This approach is not possible for the second and third replay experiments, in which the total number of possible combinations becomes impractically large: 340!/335! = 4.41 x10^12 and 2800!/2792! = 3.74 x 10^27, respectively.

::::I asked a colleague to run just such a brute-force program for me on the first replay data. I also ran several Monte Carlo simulations ('''not''' using Statsistics101) with Blount's data, using twenty-five million, one hundred million, and 493,819,200 resamplings - note that this last is twenty times the number of all possible combinations of 4 samples drawn without replacement from 72. The p-values from the 25M, 100M, and 493M Monte Carlo resamplings (0.00844, 0.00846, and 0.00846, respectively) compare favorably with Blount's 1M value of 0.0085 and the non-Monte-Carlo brute-force exact calculation, which provides a p-value of 0.008457. Thus it appears that Blount's statistical results are confirmed by a ''non-Monte Carlo'' technique, at least for the first replay experiment.

:::::What do you get for the experiment two p-value using the method from the paper and at least ten million realizations? [[User:SJohnson|SJohnson]] 08:48, 25 March 2009 (EDT)

::::::For the second replay experiment, Blount reports that one million resamplings gives a p-value of 0.0007. When I run the Monte Carlo simulations, ten million resamplings give a p-value of 0.00060; one hundred million resamplings give a p-value of 0.00062, and one ''billion'' resamplings give a p of 0.00061.

::::::As to the brute-force method for the second replay: the 4.41x10^12 combinations of five cultures picked from 340 actually represents 'only' 36.8 billion unique combinations, since for the purposes of calculating a mean generation value, the ordering of the cultures does not matter: 0, 0, 0, 0, 10 gives the same mean as 10, 0, 0, 0, 0 and 0, 10, 0, 0, 0. With brute force, it turns out that out of the 36,760,655,568 unique combinations possible in the second replay, 22,536,306 have means that are greater than or equal to 32,100.

::::::22,536,306 / 36,760,655,568 = 0.000613 = the ''exact'' p-value derived from exhaustive evaluation rather than Monte Carlo.

::::::The third replay has 9.27 x 10^22 unique combinations; at a billion comparisons a minute it would take over 170,000,000 years to check them all.--[[User:ElyM|ElyM]] 12:36, 25 March 2009 (EDT)

::::::----------
::::::Here are pointers to the Statistics 101 package <ref>http://www.statistics101.net/statistics101web_000003.htm</ref> and the actual programs run through the package by Blount ''et al.'' <ref>http://myxo.css.msu.edu/ecoli/citrate2008/MCprograms.html</ref>. The stats package is written in Java and should run under most major operating systems. A 10 million trial run of the second experiment took a bit of time and yielded a p-value of 0.00061. Ten separate, one-million trial runs produced an average p-value of 0.00061 (std.dev=0.00002, n=10). The numbers averaged about 0.0006 (std.dev=0.0004 n=10) 5K size trials.--[[User:Argon|Argon]] 20:52, 25 March 2009 (EDT)

::::My intention is not to get caught up in a digression about Monte Carlo, though - I'd rather keep the focus on the fact that the main article should acknowledge that SJohnson is using chi-square in a way that violates accepted guidelines; this remains true whether Blount's analysis is valid or not.--[[User:ElyM|ElyM]] 12:13, 23 March 2009 (EDT)

== References ==
{{reflist}}

Talk:Significance of E. Coli Evolution Experiments

2009-03-26T00:52:06Z

Argon: /* Unreferenced Claims */

SJohnson, your assessment, while good in the utilization of the chi-squared test is unfortunately incorrect. The Monte Carlo resampling gives a more accurate p-value than the chi-squared. You may research the literature (i.e. publications in statistical mathematics, many pubs actualy compare Monte Carlo vs Chi Squared) to discover that this method is commonly used in advance statistical work and how it is more accurate than the chi-squared test.--[[User:Able806|Able806]] 17:00, 4 March 2009 (EST)

:It doesn’t make sense to compare the chi-square test, which is a specific statistical hypothesis test, to Monte Carlo methods, which can be used for anything from fluid motion modeling to p-value computations. You can use Monte Carlo methods to compute the p-values of the chi-square test!

:Monte Carlo methods involve the generation of random realizations. Your broad claim the Monte Carlo methods are “more accurate” than the chi-square test is obviously incorrect because the accuracy of Monte Carlo methods always depends on the number of random realizations generated. When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous.

:Which publications compare Monte Carlo to chi-square and show that the former is more accurate? Could you provide specific examples? Thanks. [[User:SJohnson|SJohnson]] 18:50, 4 March 2009 (EST)

:In furtherance of SJohnson's remarks with respect to rarely occurring events, the use of the basic Monte Carlo method is plainly incorrect for modeling a rarely occurring event, as the Lenski paper did. This has long been pointed out in [[Flaws in Richard Lenski Study]]. I know [[evolutionists]] will never admit a flaw in anything promoting their pet theory, but this (and other) flaws in that paper is undeniable.

:Watch how evolutionists defended obvious errors in the Lenski paper, and then realize why the [[Piltdown Man]] fraud was taught for 40 years without evolutionists admitting it was a hoax.--[[User:Aschlafly|Andy Schlafly]] 09:55, 5 March 2009 (EST)

::Andy, how exactly is the Monte Carlo method incorrect to use in this case? I have seen it used in publications with much smaller datasets.--[[User:Able806|Able806]] 10:29, 5 March 2009 (EST)

:::Able806, I'm interested in looking at the publications you mentioned that use Monte Carlo methods to analyze small data sets. Could you provide some examples? Thanks. [[User:SJohnson|SJohnson]] 16:41, 5 March 2009 (EST)

::::SJohnson, here are two papers, [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6WH8-45RFJ1J-19&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=1ad95954654bb97b17e474ce6b469f6e 1] and [http://cat.inist.fr/?aModele=afficheN&cpsidt=787963 2]. Most are in chemistry and genetics where you find the observed to be much smaller and have to use the MCM. You can search on the subject as well and find that how Lenski performed the test is the standard for microbiological genetic analysis.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

:::::Those papers have nothing to do with hypothesis testing. One is an archeology paper. To be blunt, it seems like you’re just doing internet searches on “Monte Carlo” to find these links. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

::::::SJohnson, actually they do, did you read the papers? If so you would see how they used the MCM for their data analysis of small data sets, which indeed was hypothesis testing and answers you inquiry about publications that use MCM for small data set analysis. If you wish I can try to track down some actual mathematical publications, however, I am not as familiar with mathematical journals as I am with science/medical journals (not knowing which mathematical journals are acceptable). I am assuming that you have a background in math and possibly access to mathematical journals, therefore if you know the reputable ones I can do the leg work.
::::::I believe the thing that needs to be looked at is there truly a problem with the choice of test and if so what is an alternative. Bayesian might be an option but seems to be difficult to employ for this situation.--[[User:Able806|Able806]] 12:36, 12 March 2009 (EDT)

:::Able806, you still seem to miss the point about how inappropriate the Monte Carlo method (as used in the Lenski paper) is for evaluating rarely occurring events. You need to open your mind to be productive. If you simply cling to a view that Lenski (who I don't think has any meaningful education in statistics) must somehow be right, then you're not going to make any progress in understanding the flaws.--[[User:Aschlafly|Andy Schlafly]] 17:07, 5 March 2009 (EST)

::::Andy, you still have not answered what you find inappropriate about his use of the Monte Carlo method? I am a reasonable person and with evidence I do have an open mind. I provided examples last week, with a working model, showing that Monte Carlo is better than the chi-square in this case. I have also shown where the Chi-Square was inappropriate due to the occurrence size as well. So if you have any evidence that Monte Carlo should not be used in the way that Lenski used please let it be shown.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

Sjohnson, I believe you just proved my point. In the literature of mean and covariance structure analysis, non-central chi-square distribution is commonly used to describe the behavior of the likelihood ratio statistic under alternative hypothesis; it is widely believed that the non-central chi-square distribution is justified by statistical theory. Actually, when the null hypothesis is not trivially violated, the non-central chi-square distribution cannot describe the LR statistic well even when data are normally distributed and the sample size is large. Monte Carlo results compare the strength of the normal distribution against that of the non-central chi-square distribution. In an association analysis comparing cases and controls with respect to allele frequencies at a highly polymorphic locus, a potential problem is that the conventional chi-squared test may not be valid for a large, sparse contingency table. Reliance on statistics with known asymptotic distribution is unnecessary, as Monte Carlo simulations can be performed to estimate the significance level of the test statistic.

Here is a [http://faculty.vassar.edu/lowry/chi_beta.html link] to a great page the provides an interactive example as to why the Chi Squared test would provide poor results compared to the Monte Carlo in relation to the Lenski data workup.

Something you may have overlooked was that the data set is actually too small to use the chi square method correctly. It is often accepted that is any of the analyzed data falls under 10 for a particular cell of the data set then the Yates correction needs to be applied; unfortunately the Yates correction can over correct thus skewing the p-value. Lenksi seemed to understand this by supporting his Monte Carlo p-value results with the Fisher z-transformation p-value.

I hope this helps.--[[User:Able806|Able806]] 10:27, 5 March 2009 (EST)

:I’m still waiting to hear which literature says that “Monte Carlo resampling” is “more accurate than the chi-squared test”. The page mentioned above [http://faculty.vassar.edu/lowry/chi_beta.html] is a discussion of why statisticians “fail to reject the null” rather than “accepting the null” when the p-value is above 0.05 or so. The page says nothing about superiority of Monte Carlo methods. Why were alternate hypothesis distributions mentioned? Only the null hypothesis distribution is used to calculate a p-value. Yates’s correction is for 2x2 contingency tables [http://en.wikipedia.org/wiki/Yates%27_correction_for_continuity]. It doesn’t apply in this case. Finally, what the heck do “covariance structure analysis” and “allele frequencies at a highly polymorphic locus” have to do with this problem? [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::SJohnson, I am looking for this paper for you, I cited it for one of my past publications dealing with allele frequencies (I believe it came from the Duke Biostatistics group). To answer your question about allele frequencies, that is the issue at hand, more about the genetics than the math, but it is the item being studied. So you stated that Yates can not be used and statistics says the number of occurrences is too small to evaluate using the Chi-Squared test so what would you recommend instead of the Monte-Carlo Method?

:Regarding the "Fisher z-transformation p-value" from the paper, garbage in garbage out. If the p-values were bad to begin with, then why would a combination of them be meaningful? [[User:SJohnson|SJohnson]] 10:49, 9 March 2009 (EDT)

::You are assuming that p-values are wrong based on a test that is inappropriate in this case due to data limitations. Did you perform a z-transformation on the chi-squared for the three data groups?--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

::You asked about the “Fisher z-transformation p-value”. The z-transformation test and Fisher’s method are actually two different things (see Whitlock's 2005 paper - Ref. 49 in Blount et al.). But no, I haven’t tried either. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::There's a large literature on various kinds of Monte Carlo test, a very short summary of which is that they're inevitably more accurate than parametric tests (e.g. F, t, chi-squared, etc) because they don't make assumptions about the distribution of the data under the null hypothesis. See for example ''Introduction to the Bootstrap'' by B. Efron and R. Tibshirani and ''The Jack-knife, the Bootstrap and Other Resampling Plans'', also by Efron. They're certainly applicable to small datasets and their accuracy is really only limited by the number of samples you care to take. E.g. 1000 M-C samples would give you a pretty accurate idea about significance at the alpha<1% level (That book should answer SJohnson's questions of 18:50 on 4/3/09 and 16:38 on 5/3/09 about accuracy and Aschalfly's comment of 17:07 on 5/3/09 about appropriateness of Monte Carlo tests.) [[User:FredFerguson|FredFerguson]] 16:53, 11 March 2009 (EDT)

::::Your claim that Monte Carlo methods are “inevitably more accurate” than other tests is obviously wrong because the accuracy of MC methods always depends on the number of realizations used. You should have written <math>\alpha=1\%</math>, not <math>\alpha<1\%</math>. If 1,000 random realizations are generated, the number of realizations above the true <math>\alpha=1\%</math> level is binomial with mean 10 and variance about 10. Thus, the standard deviation of the MC estimate is >0.003. In this example, a Monte Carlo p-value could be off by 30% and still be within a standard deviation. Is that really “pretty accurate”?

::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 3%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::::You're talking about miniscule differences in the accuracy of a test. 0.013 isn't very different from 0.007. In either case, it's very unlikely the experimenter would have obtained that result if the null hypothesis were true. If you're bothered about differences in P-values to the third decimals (which would make you unusual!), just run more MC realisations, that's all. Not really a problem. [[User:FredFerguson|FredFerguson]] 11:53, 12 March 2009 (EDT)

There’s still confusion about the difference between test statistics and Monte Carlo methods. Before you find a Monte Carlo estimate of a p-value, you need to select a test statistic to reduce the data set to a scalar. I am interested in hearing which test statistic you believe should be used in place of the chi-square test and why. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

----

Quick question for SJohnson: How many degrees of freedom did you choose when calculating the p-value? I'd like to know upon what condition you base that number. Thanks.--[[User:Argon|Argon]] 11:05, 5 March 2009 (EST)

:The degree of freedom for a contingency table is rows minus one times columns minus one. That is, <math> (r-1)(c-1) </math>. Here’s a pretty good tutorial I came across: [http://faculty.uncfsu.edu/dwallace/lesson%2020.pdf]. For the experiments from [http://www.pnas.org/content/105/23/7899.full.pdf], the DOFs are 11, 11, and 13. For experiment one, the chi-square test statistic is
::<math>
X^2
=\sum\limits_i\sum\limits_j
\frac{\left(n_{i,j}-E\left[n_{i,j}\right]\right)^2}
{E\left[n_{i,j}\right]}
</math>
::<math>
=\frac{\left(0-1/3\right)^2}{1/3}
+\frac{\left(6-17/3\right)^2}{17/3}
+\frac{\left(0-1/3\right)^2}{1/3}
+\ldots+
</math>
::<math>
+\frac{\left(2-1/3\right)^2}{1/3}
+\frac{\left(4-17/3\right)^2}{17/3}
</math>
::<math>
\approx
14.82
</math>
:where <math>n_{i,j}</math> is the observed value and <math>E\left[n_{i,j}\right]</math> is the expected null hypothesis value. So if you have MS Excel, another way to arrive at the p-value of 0.19 is to type “=CHIDIST(14.82,11)” into a cell. Cheers! [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::OK, thanks for the info. From what I'd calculated and looked up in tables, the numbers seemed close to a df=11 for a chi-square of ~14. (Aside: With terms having 17/3 in the denominator in the figures above, were you using the test of independence? I was using Pearson's test for [http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_for_fit_of_a_distribution fit of a distribution] which returns a chi-squared value of 14 and roughly matched the p-values you reported, assuming the df was 11).

::Also, the first sentence of the article reads: "Blount, Borland, and Lenski[1] claimed that a key evolutionary innovation was observed during a laboratory experiment. That claim is false." A small correction: There were several claims in the paper. The 'key evolutionary innovation' was acquiring the ability to utilize citrate as a food source. That claim was demonstrated multiple times. The claim, which pertains to this statistics discussion was that the Cit+ phenotype arose in a multi-step process, first requiring a rare, pre-adaptive mutation before additional mutation(s) lead to the subsequent development of citrate utilization.--[[User:Argon|Argon]] 20:46, 5 March 2009 (EST)

:::My biology-degreed wife assures me that mutation does not necessarily mean that evolution occurred. What the paper claimed is that evolution (a “key innovation”) occurred in the lab. The key innovation supposedly increased the mutation rate. In the experiments, the observed mutation rate increased after generation 31,000, but not enough to make a statistically significant claim that the rate is not constant. The analysis in the paper was similar to flipping a coin ten times, counting six heads and claiming that the coin must be biased against tails. In reality, there’s nothing surprising about a fair coin producing slightly more of one outcome than the other. Just like there's nothing surprising about there being slightly more mutations in later generations than early generations given the null hypothesis (constant mutation rate). [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

>>Inserting a later comment first<<
SJohnson, the paper's title is: "Historical contingency '''and the evolution of a key innovation''' in an experimental population of ''Escherichia coli''" As I mentioned earlier, the key innovation is the evolution of the Cit+ phenotype and not the timing or rate of its acquisition. And yes, it *is* evolution (call it microevolution, if you wish). Blount et al went on further to speculate how this evolutionary innovation arose and they proposed the historical contingency hypothesis in which 'pre-adaptive' mutations were required before the Cit+ phenotype developed. It is only this latter hypothesis that you are attempting to address with your chi-square analysis, not the fact that Cit+ mutants arose (which is the evolutionary innovation).--[[User:Argon|Argon]] 21:57, 18 March 2009 (EDT)

::::SJohnson, not to say anything about your wife, but has she had a 400 level molecular genetics course (most general biology degrees do not cover the detail unless they are specialized)? If so, she would have mentioned that if the mutation passes to the offspring and is selectively beneficial to the population then it is a step of evolution as along as the conditions continue through the sharing of the mutation with the population and the environment is such that reduces the growth rate of the non-transformed population. While not all mutations are signs that evolution occurred the mutations that pass to offspring and provide a benefit compared to other offspring are very strong indicators. In the case of this paper the population that evolved the cit+ was able to metabolize a chemical in their environment which allowed for an adaptation advantage compared to the non-transformed colonies.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

----

Let’s go back to the beginning. There appears to be confusion about the difference between test statistics and methods for computing p-values. As is noted at the beginning of the page [http://www.conservapedia.com/Significance_of_E._Coli_Evolution_Experiments], the fundamental problem with the paper is that it used a flawed test statistic, not that it used Monte Carlo methods to find the p-value for that flawed statistic.

Every hypothesis test uses a test statistic to reduce the data to a single number. The p-value for the test statistic can be calculated analytically (as I’ve done for the chi-square test statistic) or by Monte Carlo methods. In the paper, Monte Carlo methods were used to compute the p-value of the “mutation generation” test statistic. The key problem with the analysis from the paper is that it doesn’t work to use a weighted average to test for variations in mutation rate. This is like trying to use the sample variance to test for an increase in the mean in Gaussian-distributed data. A statistic should be selected based on the null and alternate hypothesis distributions of the data. The chi-square test (unlike the weighted average from the paper) is a reasonable choice for data that mutates at a constant rate under the null hypothesis, but mutates at varying rates under the alternate hypothesis.

Able806, you made a good point about the contingency table cell frequencies being relatively low, but were wrong when you said ”the data set is actually too small to use the chi square method correctly”. In the low cell frequency case the chi-square test is still effective, but the null hypothesis distribution of the chi-square statistic starts to look less like the chi-square distribution. Thus, p-values calculated using the chi-square distribution may be a bit off. However, Monte Carlo p-values are always imperfect as well because it's impossible to generate an infinite number of random realizations. There are imperfections in p-values generated by analytic and Monte Carlo methods. However, low cell frequencies does not explain the >20x and >2.5x differences between chi-square p-values and p-values from the paper for experiments one and three. The reason for those huge differences was the use of the flawed test statistic (“mutation generation”) in the paper. [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

:SJohnson, the chi-squared test is a valuable statistical tool, but the limitations of the test must be acknowledged. The chi-squared test can only produce valid results if the assumptions that underly the test are not violated. As an analogy, Newtonian models of motion fail to produce accurate results as velocities approach the speed of light; under those circumstances one must switch to a theory that accounts for relativistic effects.

:It seems that you have simply dismissed the [http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm widely-acknowledged] [http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm fact] that the [http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html chi-squared test] is [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236 inappropriate] for use in [http://www.graphpad.com/www/Book/Choose.htm situations] where n in any cell is [http://mysite.du.edu/~jcalvert/econ/chisquar.htm less] less than a [http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1 threshold] [http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html number]. Different authors set different thresholds, but all are well above the numbers seen in your chi-squared analysis - even the most liberal guidelines advise against the chi-squared test when any expected cell frequency is less than one or more than 20% of the table cells are less than 5; others require that expected values in all cells must be more than 5. With smaller amounts of data, the test is insensitive and errs on the side of rejecting the hypothesis. If you attempt your chi-squared statistical analysis with a program that is more sophisticated than MS Excel (as I did), you get an error message indicating that the results are invalid due to low expected cell counts.

:That issue aside, there are other reasons that the chi-squared test is inappropriate here. As the links above point out, the categories tested must be truly independent; one example is that you can't use the chi-squared test to compare age and ability to kick a field goal by testing the same experimental group twice, one year apart; you have to test one group of age A and a different group of age B. In the case of the Blount paper, the categories are not independent. Even if there were adequate numbers to address the low-expected-frequency problem, this would make the chi-squared an invalid test in this case.

:There are other significant problems with the use of the chi-squared test in this circumstance, but they can wait until you address these first major problems.--[[User:ElyM|ElyM]] 12:18, 11 March 2009 (EDT)

::Wackerly et al. says in general it’s assumed that the cell frequencies are above five so that the chi-square statistic (under the null) is approximately chi-square distributed (see p. 703). That book does not say chi-square test results are invalid if frequencies are five or less. Your example of a chi-square test warning message (it said "warning" not "error" as you stated) in Minitab [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236] said “approximation probably invalid” referring to the chi-square distribution approximation to the chi-square test statistic’s distribution. Your example did not say “chi-square test invalid”. I agree that when cell frequencies are low, the chi-square test statistic’s distribution starts to deviate from the chi-square distribution. I maintain that this deviation is not enough to explain the >2.5x and >20x differences in the chi-square test p-values and the p-values from the paper.

::As the numerous links in your post proved, the chi-square test is widely-used by statisticians. Can you give examples of statisticians using mean mutation generation as a test statistic? Also, did your software agree with the chi-square test p-values I presented? Thanks. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::Thank you for giving page references for Wackerly; however it seems we have different editions, since page 703 in my copy (5th ed, 1996) does not deal with chi-squared issues at all. My copy does state the following, on page 622: "Although the mathematical proof is beyond the scope of this text, it can be shown that, when n is large [chi-squared] will possess approximately a chi-square probability distribution in repeated sampling." Then, on page 624: "Experience has shown that cell counts [n sub i] should not be too small in order that the chi-square distribution provide an accurate approximation to the distribution of [chi squared]. As a rule of thumb we require that all expected cell counts equal or exceed 5, although Cochran (1952) has noted that this value can be as low as 1 for some situations." Wackerly then goes on, in the problems sections, to describe the use of the chi-squared test as a "violation of good statistical practice" when "some expected counts [are] <5."

:::It seems that you are already aware that the [chi-square] statistic under the null is no longer chi-square distributed for small n; this is precisely why the test should not be used under those conditions. I can claim to be able to accelerate a 1-kg mass to 10 times the speed of light by applying 1 N of force for 95 years by using F=ma and t= (vf-vi)/a. Plugging the numbers into those equations will produce the same result every time, but the answer is illegitimate because those equations are only valid under certain assumptions, which are violated as velocities approach the speed of light. Similarly, having a statistical program calculate a chi-squared value given the Blount data will produce a number result, but since the assumptions of the test are violated the result is not legitimate. Yes, if I put the Blount data in SAS 9.2, I get the same numerical answer as you do, but I also get the following message: "WARNING: >89% of the cells have expected counts less than 5. Chi-square may not be a valid test." You may argue that that's a warning, not an error; that's a semantic distinction. The reason that the program says that it MAY not be valid is that the chi-squared test skews in the direction of being too conservative at low n values; the test has an acceptable rate of false positives but an unacceptably high rate of false negatives. Comparing the results of the Monte Carlo and chi-squared results in this case is like comparing the results of Newtonian and relativistic equations of motion: they can produce very different results from the same input data.

::::For a finite amount of data, the chi-square statistic is never chi-square distributed under the null. The p-values are always approximate regardless of cell frequencies. The approximation becomes more accurate as the amount of data increases, but I don’t believe that this inaccuracy will change p-values that are about 0.2 (for experiments 1 and 3) into statistically significant p-values. How much do you expect the p-values to change if an exact computation is used in place of the chi-square distribution approximation? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::Your last paragraph has a major non sequitur in it: yes, many statisticians use the chi-square test. As long as the assumptions of the test are not violated, it is a valuable tool. That has nothing to do with the validity of using mean mutation generation as a test statistic. 'Mean number of werewolf attacks in Mumbai in the week centered on the new moon, by month, from 1654 to 1798' is a valid test statistic. I am quite sure that it has never been used in a peer-reviewed paper before. That does not mean that I can't perform valid statistical tests on that statistic. If, however, the incorrect test is applied, the results of the analysis will be flawed. Papers apply a (relatively small) standard repertoire of valid tests to a (potentially infinite) number of test statistics. The particular test statistic used in a paper may never have been used before and may never be used again; that does not address the validity of the analysis. In Blount's case, the test is the Monte Carlo analysis, which is also "widely-used by statisticians".

::::There are an infinite number of ways to reduce a data set to a single number. However, it’s foolish to think every method would be effective. I gave an example of a flawed test statistic in an earlier post [http://www.conservapedia.com/index.php?title=Talk%3ASignificance_of_E._Coli_Evolution_Experiments&diff=635070&oldid=634987]. Another example of a flawed test statistic is the one used in the paper because it does not always detect deviations from the null hypothesis (see: [[Significance of E. Coli Evolution Experiments#Test Statistics]]).

::::Test statistics are typically derived. The likelihood ratio test is a common method used to derive them. The chi-square test for independence is an approximation to the LRT. Where is the derivation saying that mean mutation generation is an appropriate test statistic for this problem? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::We still haven't touched on the issue of the categories not being independent, which by itself is sufficient to invalidate the chi-squared technique. I'm new to this site, so I'm unsure as to the etiquette of making changes to the articles of another person - but the article here should at the very least mention that the chi-square test is being used here in a manner that violates its underlying assumptions in at least two fundamental ways, and the results are therefore suspect.--[[User:ElyM|ElyM]] 17:34, 12 March 2009 (EDT)

:::::When generating random realizations of experiment outcomes, the authors assumed that the total number of mutants was fixed. Thus the paper assumed the numbers of mutants per generation are statistically dependent. Does this seem like a realistic model, or do you think that if the experiments were recreated that the total number of mutants could vary? For example, if experiment one were recreated, would the total number of mutants always be exactly four? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::: It looks to me as though SJohnson has misinterpreted the application of the chi-squared test in quite a fundamental way. His/her analysis of Blount's data are therefore close to meaningless, regardless of whether the test used by Blount is appropriate or not. In my opinion, the entire page should therefore be deleted. [[User:FredFerguson|FredFerguson]] 08:18, 13 March 2009 (EDT)

::::: "Fred", perhaps you mistakenly think this is Wikipedia, where [[censorship]] and deletion of pages for ideological reasons are common. Not here.--[[User:Aschlafly|Andy Schlafly]] 10:23, 14 March 2009 (EDT)

:::::: Umm... I'm suggesting deletion for mathematical reasons, not ideological reasons. Using an argument filled with mathematical errors to try to support your case only detracts from your credibility. [[User:FredFerguson|FredFerguson]] 10:38, 14 March 2009 (EDT)

:::::: Actually, I think correction is better than deletion. So that's what I've done. [[User:FredFerguson|FredFerguson]] 11:01, 14 March 2009 (EDT)

::::::: I find no credibility in your denial of having ideological reasons.--[[User:Aschlafly|Andy Schlafly]] 11:04, 14 March 2009 (EDT)

== Misinterpretation of test ==

SJohnson, Your analysis misinterprets the test. You say the null hypothesis is that this mutation cannot happen. They saw a mutation (4 mutations, in fact, in the data set you show) so the null hypothesis (as you state is) is disproved. That's perfectly straightforward.

I don't know what the "mean mutation generation" test is but you're doing when you apply a chi-squared test to this dataset is to test if the mutations are evenly distributed throughout the generations. Your test says they are, so there's no strong evidence to suppose that mutations are likely to occur in one generation rather than another in the series of tests. Blount's test says thay aren't, so it's more likely that the mutation will occur later in the series of tests. I can't tell which test is right without knowing more about the test that Blount used.

But that point (the foregoing paragraph) has no bearing at all on the null hypothesis, as you describe it. The mutation appeared, so that means the hypothesis that the mutation can't happen is disproved. Very simple. [[User:FredFerguson|FredFerguson]] 21:10, 8 March 2009 (EDT)

:I never said that “the null hypothesis is that this mutation cannot happen”. The chi-square test statistic I'm using wouldn’t be defined if the null hypothesis mutation rate was zero because the <math>E\left[n_{i,j}\right]</math> term in the denominator of the statistic (see above equation) would be zero.

:The test statistic from the paper is the average of the generation numbers of observed mutations. For experiment one this number is
::<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)= 31750.
</math>
:The same number is shown in Table 2 of the paper. [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

:: SJohnson, the way you're calculating the chi-squared statistic implies that you're testing the null hypothesis of a constant mutation rate over time against an alternative hypothesis of a mutation rate which varies over time. [[User:FredFerguson|FredFerguson]] 11:02, 9 March 2009 (EDT)

As it currently stands, the article makes the following statement: "The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown." This misstates the null hypothesis of the paper, which is elaborated in the Introduction section of the paper, and repeated in the section '''Statistical Analysis of the Replay Experiments''':
<blockquote>
For each experiment, we compared the observed mean generation of those clones that yielded Cit+ variants to the mean expected under the null hypothesis that clones from all generations have equal likelihood. The null thus corresponds to the rare-mutation hypothesis laid out in the Introduction."
Block quote
</blockquote><ref>www.pnas.org/cgi/reprint/105/23/7899.pdf</ref>

The article also continues to describe 'mean mutation generation' as a ''test'' rather than a ''statistic'' to which the ''Monte Carlo test'' was applied.--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

I'm a bit confused about why the Chi-squared test, which we're told compares the results to a null hypothesis of a constant mutation rate, seems insensitive to which generations the Cit+ mutations are found. Instead, the chi-square test seems only to be evaluating whether the frequencies of Cit+ mutations in any particular generation are 'expected'. Thus the test is asking whether finding a distribution (e.g. in the first experiment) across nine periods that have no mutations, two periods that have one mutation and one period with two mutations is a statistically significant deviation from what you'd expect of the mutations were randomly distributed. The number returned from the function is the same regardless of the order of Cit+ results. The number of mutations per bin is not the only question being asked. Instead it's the '''order and temporal distribution''' of Cit+ mutants that the analyses probably need to confront. It's not whether one can get nine no-mutants, two single mutants and one double-mutant result, it's a matter of '''when''' they occur and whether that distribution affects the significance of the results. Blount's hypothesis is that mutations should appear later in the experiment. When formulating a suitable null hypothesis, wouldn't one want to take the timing of Cit+ mutants into consideration too?--[[User:Argon|Argon]] 22:25, 18 March 2009 (EDT)

==Unreferenced Claims==

I deleted the claim that mean mutation generation is an appropriate test statistic because no reference was produced that back that claim. No reference was provided to back the claim that the chi-square test p-values are always conservative, either. [[User:SJohnson|SJohnson]] 12:57, 14 March 2009 (EDT)

: The reference is Everitt. I'll check I put it in the right place. [[User:FredFerguson|FredFerguson]] 13:30, 14 March 2009 (EDT)

There was a typo in my edit summaries on the talk page and the main page. I meant to say "Removed unsupported claims" rather than "Removed supported claims". [[User:SJohnson|SJohnson]] 13:13, 14 March 2009 (EDT)

I do not believe that anyone has claimed that 'chi-square test p-values are ''always'' conservative'. The claim that has been made is that ''under certain circumstances'', namely low n and low individual cell values, the chi-square test is an invalid test; that under those circumstances the power of the test is low and it becomes impossible to reject the null hypothesis even when it is false. You may have missed the pertinent sections in my links above, so I will directly quote the relevant sections. All the quoted sections refer to chi-square testing in particular. Any bolding below is mine.

:This edit claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639379]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

<blockquote>
"Assumptions: 

Even though a nonparametric statistic does not require a normally distributed population, there still are some restrictions regarding its use. 

1. Representative sample (Random) 
2. The data must be in frequency form (nominal data) or greater. 
3. The individual observations must be independent of each other. 
4. '''Sample size must be adequate. In a 2 x 2 table, Chi Square should not be used if n is less than 20. In a larger table, no expected value should be less than 1, and not more than 20% of the variables can have expected values of less than 5'''. 
5. Distribution basis must be decided on before the data is collected. 
6. The sum of the observed frequencies must equal the sum of the expected frequencies."
</blockquote>
<ref>http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm
</ref>

<blockquote>
"Assumptions:
* Random sample data are assumed. As with all significance tests, if you have population data, then any table differences are real and therefore significant. If you have non-random sample data, significance cannot be established, though significance tests are nonetheless sometimes utilized as crude "rules of thumb" anyway.
* A sufficiently large sample size is assumed, as in all significance tests. '''Applying chi-square to small samples exposes the researcher to an unacceptable rate of Type II errors. There is no accepted cutoff. Some set the minimum sample size at 50, while others would allow as few as 20'''. Note chi-square must be calculated on actual count data, not substituting percentages, which would have the effect of pretending the sample size is 100.
* '''Adequate cell sizes are also assumed. Some require 5 or more, some require more than 5, and others require 10 or more. A common rule is 5 or more in all cells of a 2-by-2 table, and 5 or more in 80% of cells in larger tables, but no cells with zero count'''. When this assumption is not met, Yates' correction is applied.
* Independence. Observations must be independent. The same observation can only appear in one cell. '''This means chi-square cannot be used to test correlated data (ex., before-after, matched pairs, panel data)'''.
* Similar distribution. Observations must have the same underlying distribution.
* Known distribution. The hypothesized distribution is specified in advance, so that the number of observations that are expected to appear each cell in the table can be calculated without reference to the observed values. Normally this expected value is the crossproduct of the row and column marginals divided by the sample size.
* Non-directional hypotheses are assumed. Chi-square tests the hypothesis that two variables are related only by chance. If a significant relationship is found, this is not equivalent to establishing the researcher's hypothesis that A causes B, or that B causes A.
* Finite values. Observations must be grouped in categories.
* Normal distribution of deviations (observed minus expected values) is assumed. Note chi-square is a nonparametric test in the sense that is does not assume the parameter of normal distribution for the data -- only for the deviations.
* Data level. No assumption is made about level of data. Nominal, ordinal, or interval data may be used with chi-square tests."
</blockquote>
<ref>http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm</ref>

<blockquote>
"Assumptions: 
-None of the expected values may be less than 1 
-No more than 20% of the expected values may be less than 5"</blockquote> <ref>http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html</ref>

<blockquote>
"When performing a chi-square test, your data must satisfy important assumptions. Although these assumptions may be stated differently in different textbooks, they generally assert that: 
1)The sample must be randomly drawn from the population 
'''2)The sample size, n, must be large enough so that the expected cell count in each cell is greater than or equal to 5.''' 
Both assumptions must be met in the process of collecting your data, and violations of the second assumption will appear in the Minitab output when you run the analysis. 
... 
'''You may wonder why the second assumption is necessary for performing the chi-square test. The second assumption arises because the distribution of counts under the null hypothesis is multinomial, and the normal distribution can be used to approximate the multinomial distribution if the sample size is sufficiently large and the probability parameters aren't too small. It can be shown via the Central Limit Theorem that the multinomial distribution converges to the normal distribution as the sample size approaches infinity; however, there is no easy way to show mathematically how and when the convergence fails.'''"
</blockquote>
<ref>http://www.minitab.com/support/docs/Answers/Chi-Square%20Test%20Assumptions.pdf</ref>

<blockquote>
"'''The chi-square test is simpler to calculate but yields only an approximate P value. ... You should definitely avoid the chi-square test when the numbers in the contingency table are very small (any number less than about six)'''."
</blockquote> <ref>http://www.graphpad.com/www/Book/Choose.htm</ref>

<blockquote>
"The most important things to remember to get a valid χ2 test are that the expected values are not too small in any bin (certainly 5 or more), and that the degrees of freedom are properly evaluated. '''Unless you have a very large amount of data, the test is not very sensitive and errs on the side of safety. If you get a significant result, however, it is not likely to be wrong.'''"
</blockquote> <ref>http://mysite.du.edu/~jcalvert/econ/chisquar.htm</ref>

<blockquote>
"The critical assumptions of the chi-square test for k independent samples are similar to those for the chi-square test for two independent samples. ... 
'''4. No more than 20% of the cells may have expected frequencies of less than 5, and no cell should have an expected frequency of less than 1. 
The rule given in Assumption 4 is particularly important for a contingency table that is larger than 2X2'''"
</blockquote> <ref>http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1
</ref>

<blockquote>
"Special problems with small expected cell frequencies for the chi-square test: 
The chi-square test involves using the chi-square distribution to approximate the underlying exact distribution. The approximation becomes better as the expected cell frequencies grow larger, and '''may be inappropriate for tables with very small expected cell frequencies.''' 
'''For tables with expected cell frequencies less than 5, the chi-square approximation may not be reliable. A standard (and conservative) rule of thumb (due to Cochran) is to avoid using the chi-square test for tables with expected cell frequencies less than 1, or when more than 20% of the table cells have expected cell frequencies less than 5.''' 
Another rule of thumb (due to Roscoe and Byars) is that the average expected cell frequency should be at least 1 when the expected cell frequencies are close to equal, and 2 when they are not. (If the chosen significance level is 0.01 instead of 0.05, then double these numbers.) 
Koehler and Larntz suggest that if the total number of observations is at least 10, the number categories is at least 3, and the square of the total number of observations is at least 10 times the number of categories, then the chi-square approximation should be reasonable. 
Care should be taken when cell categories are combined (collapsed together) to fix problems of small expected cell frequencies. Collapsing can destroy evidence of non-independence, so a failure to reject the null hypothesis for the collapsed table does not rule out the possibility of non-independence in the original table. 
'''As with most statistical tests, the power of the chi-square test increases with a larger number of observations. If there are too few observations, it may be impossible to reject the null hypothesis even if it is false.'''"
</blockquote> <ref>http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html</ref>--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

: Thanks for this really excellent contribution, ElyM. The only thing I'd like to add is in relation to your initial statement, "I do not believe that anyone has claimed that 'chi-square test p-values are always conservative'". The question of whether a test is conservative in a particular situation is probabilistic. One can determine whether a test is likely to generate a p-value which is too high in a particular situation (e.g. for a chi-squared test, when there are lots of small expected values) but one needs an exact test (such as an appropriate Monte Carlo randomisation test) to determine whether the p-value in any ''particular'' test is in fact excessively high.

::This edit also claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639373]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

: I hope careful reading of your very clear description will put SJohnson's mind at rest on this subject. [[User:FredFerguson|FredFerguson]] 18:11, 14 March 2009 (EDT)

ElyM, you've provided nothing to address the basic flaw that "The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events." See [[Flaws in Lenski Study]]. Also, do not impose your view on the content page until after SJohnson has had an opportunity to respond to your posting. As to "Fred", his put-downs are getting tiresome and I'm going to review his edit pattern now to see if he's been contributing anything of value to this site.--[[User:Aschlafly|Andy Schlafly]] 14:06, 15 March 2009 (EDT)

::Mr. Schlafly, per your request I have not added anything to the content page as SJohnson has not yet responded to my posts. Since all of my comments have been in regards to SJohnson's use of the chi-square test in this particular article, I'm not sure why you expect me to address Blout's use of Monte Carlo - that issue seems to be addressed on the [[Flaws in Lenski Study]] page. SJohnson has added a reformulation of the chi-square test for two possible outcomes, and stated that the chi-square test is at a minimum when all success probabilities are equal. He then extrapolates from this to claim that the chi-square test is an effective test for the data from Blount.

::The reformulation of the equations for two possible outcomes does not address the underlying problem that the chi-square test has universally accepted parameters outside of which it is considered an invalid test; I have provided references for these parameters and shown that the data from Blount lies outside them. None of the expected cells in SJohnson's analysis have values above one, and the total n is four. SJohnson's own reference states that the application of the chi-square test in this circumstance is a "violation of good statistical practice". Analogously, combining F=ma and t=(vf-vi)/a into t=(vf-vi)m/F and showing that t is a minimum when m approaches zero does not address the fact that those Newtonian equations do not apply as velocities approach the speed of light. The legitimacy of Blount's arguments cannot be determined by the application of illegitimate counterarguments. If SJohnson or others can point to references from the statistical literature that show that Blount has made methodological errors - as I have been able to do with SJohnson's chi-square analysis - I would welcome their input, and no doubt Conservapedia's other readers would as well, and this page would be greatly improved.

::I have not seen a rebuttal from SJohnson in the four days since my last post, although he has added new material to the content page since then. In light of this, I would appreciate some guidelines as to when it is appropriate for me to add my information and references to the content page. I can add citations from the primary mathematical literature if necessary, but in general I find that these are less helpful as they are not easily accessible by readers without access to academic libraries.--[[User:ElyM|ElyM]] 18:07, 18 March 2009 (EDT)

::: You say, "I'm not sure why you expect me to address Blout's use of Monte Carlo." The reason is obvious: the title of the content page is the "Significance of E. Coli Evolution Experiments." You haven't addressed the inappropriateness of using Monte Carlo simulations for assessing the significance rarely occurring events, which was central to Lenski's statistical claims. I suggest you address this flaw if you want to be taken seriously.--[[User:Aschlafly|Andy Schlafly]] 23:12, 18 March 2009 (EDT)

::::ASchlafly, again per your request, and based on your statement regarding the "inappropriateness of using Monte Carlo simulations for assessing the significance of rarely occurring events", I have spent the last several days reviewing the literature available to me on Monte Carlo and other resampling techniques, looking for ways in which Blount may have made a methodological error of the sort that SJohnson has made. I have been unable to find any examples of authors suggesting that Monte Carlo be avoided for low ''n'', or for events with low probability regardless of'' n'', much less providing specific cutoff numbers as are seen in the references that I provided for the chi-square test. Similarly, the technique that Blount used does not require/assume that categories are unrelated, as the chi-square test does. Of course, the absence of evidence is not evidence of absence, and I may have misinterpreted the basis of your objection. At this point I'll need you to explain your objection in more detail if you wish me to find the appropriate literature addressing your concerns. Do you believe that the number of resamplings was too low in Blount's paper? That the analysis should have been performed with a software package other than Statistics101? Some other procedural issue? Some issue of interpretation?

::::The statistical problem that Blount must address is straightforward: given a distribution of mutant cultures that ''appears'' to be skewed toward the higher generations, what is the probability that this same amount of skew (or a greater degree) could arise by chance, given the null hypothesis that every generation is equally likely to produce a mutant? Interestingly, in the case of the first replay experiment, the total number of ways to randomly select (equal probability, no replacement) four cultures from seventy-two is 72x71x70x69, or 24,690,960. This number is small enough that a program can brute-force-calculate the 'mean generation number' of ''all possible'' combinations of four cultures in a reasonable amount of time. An experimentally-derived 'mean generation number' can be checked against this exhaustive list, and the number of means equal to or larger than the experimental mean can be found exactly. Converting this number to a percentage of 24,690,960 provides an exact p-value for any given experimental 'mean generation number'. This exhaustive approach is different than the Monte Carlo technique, in that ''all possible'' outcomes are examined, rather than a ''random subset'' of all possible outcomes. For the first replay experiment, it provides a way to independently check Blount's Monte Carlo results. This approach is not possible for the second and third replay experiments, in which the total number of possible combinations becomes impractically large: 340!/335! = 4.41 x10^12 and 2800!/2792! = 3.74 x 10^27, respectively.

::::I asked a colleague to run just such a brute-force program for me on the first replay data. I also ran several Monte Carlo simulations ('''not''' using Statsistics101) with Blount's data, using twenty-five million, one hundred million, and 493,819,200 resamplings - note that this last is twenty times the number of all possible combinations of 4 samples drawn without replacement from 72. The p-values from the 25M, 100M, and 493M Monte Carlo resamplings (0.00844, 0.00846, and 0.00846, respectively) compare favorably with Blount's 1M value of 0.0085 and the non-Monte-Carlo brute-force exact calculation, which provides a p-value of 0.008457. Thus it appears that Blount's statistical results are confirmed by a ''non-Monte Carlo'' technique, at least for the first replay experiment.

:::::What do you get for the experiment two p-value using the method from the paper and at least ten million realizations? [[User:SJohnson|SJohnson]] 08:48, 25 March 2009 (EDT)

::::::For the second replay experiment, Blount reports that one million resamplings gives a p-value of 0.0007. When I run the Monte Carlo simulations, ten million resamplings give a p-value of 0.00060; one hundred million resamplings give a p-value of 0.00062, and one ''billion'' resamplings give a p of 0.00061.

::::::As to the brute-force method for the second replay: the 4.41x10^12 combinations of five cultures picked from 340 actually represents 'only' 36.8 billion unique combinations, since for the purposes of calculating a mean generation value, the ordering of the cultures does not matter: 0, 0, 0, 0, 10 gives the same mean as 10, 0, 0, 0, 0 and 0, 10, 0, 0, 0. With brute force, it turns out that out of the 36,760,655,568 unique combinations possible in the second replay, 22,536,306 have means that are greater than or equal to 32,100.

::::::22,536,306 / 36,760,655,568 = 0.000613 = the ''exact'' p-value derived from exhaustive evaluation rather than Monte Carlo.

::::::The third replay has 9.27 x 10^22 unique combinations; at a billion comparisons a minute it would take over 170,000,000 years to check them all.--[[User:ElyM|ElyM]] 12:36, 25 March 2009 (EDT)

::::::----------
::::::Here are pointers to the Statistics 101 package <ref>http://www.statistics101.net/statistics101web_000003.htm</ref> and the actual programs run through the package by Blount ''et al.'' <ref>https://myxo.css.msu.edu/ecoli/citrate2008/MCprograms.html</ref>. The stats package is written in Java and should run under most major operating systems. A 10 million trial run of the second experiment took a bit of time and yielded a p-value of 0.00061. Ten separate, one-million trial runs produced an average p-value of 0.00061 (std.dev=0.00002, n=10). The numbers averaged about 0.0006 (std.dev=0.0004 n=10) 5K size trials.--[[User:Argon|Argon]] 20:52, 25 March 2009 (EDT)

::::My intention is not to get caught up in a digression about Monte Carlo, though - I'd rather keep the focus on the fact that the main article should acknowledge that SJohnson is using chi-square in a way that violates accepted guidelines; this remains true whether Blount's analysis is valid or not.--[[User:ElyM|ElyM]] 12:13, 23 March 2009 (EDT)

== References ==
{{reflist}}

Talk:Significance of E. Coli Evolution Experiments

2009-03-19T02:25:06Z

Argon: /* Misinterpretation of test */

Talk:Significance of E. Coli Evolution Experiments

2009-03-19T01:58:23Z

Argon:

SJohnson, your assessment, while good in the utilization of the chi-squared test is unfortunately incorrect. The Monte Carlo resampling gives a more accurate p-value than the chi-squared. You may research the literature (i.e. publications in statistical mathematics, many pubs actualy compare Monte Carlo vs Chi Squared) to discover that this method is commonly used in advance statistical work and how it is more accurate than the chi-squared test.--[[User:Able806|Able806]] 17:00, 4 March 2009 (EST)

:It doesn’t make sense to compare the chi-square test, which is a specific statistical hypothesis test, to Monte Carlo methods, which can be used for anything from fluid motion modeling to p-value computations. You can use Monte Carlo methods to compute the p-values of the chi-square test!

:Monte Carlo methods involve the generation of random realizations. Your broad claim the Monte Carlo methods are “more accurate” than the chi-square test is obviously incorrect because the accuracy of Monte Carlo methods always depends on the number of random realizations generated. When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous.

:Which publications compare Monte Carlo to chi-square and show that the former is more accurate? Could you provide specific examples? Thanks. [[User:SJohnson|SJohnson]] 18:50, 4 March 2009 (EST)

:In furtherance of SJohnson's remarks with respect to rarely occurring events, the use of the basic Monte Carlo method is plainly incorrect for modeling a rarely occurring event, as the Lenski paper did. This has long been pointed out in [[Flaws in Richard Lenski Study]]. I know [[evolutionists]] will never admit a flaw in anything promoting their pet theory, but this (and other) flaws in that paper is undeniable.

:Watch how evolutionists defended obvious errors in the Lenski paper, and then realize why the [[Piltdown Man]] fraud was taught for 40 years without evolutionists admitting it was a hoax.--[[User:Aschlafly|Andy Schlafly]] 09:55, 5 March 2009 (EST)

::Andy, how exactly is the Monte Carlo method incorrect to use in this case? I have seen it used in publications with much smaller datasets.--[[User:Able806|Able806]] 10:29, 5 March 2009 (EST)

:::Able806, I'm interested in looking at the publications you mentioned that use Monte Carlo methods to analyze small data sets. Could you provide some examples? Thanks. [[User:SJohnson|SJohnson]] 16:41, 5 March 2009 (EST)

::::SJohnson, here are two papers, [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6WH8-45RFJ1J-19&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=1ad95954654bb97b17e474ce6b469f6e 1] and [http://cat.inist.fr/?aModele=afficheN&cpsidt=787963 2]. Most are in chemistry and genetics where you find the observed to be much smaller and have to use the MCM. You can search on the subject as well and find that how Lenski performed the test is the standard for microbiological genetic analysis.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

:::::Those papers have nothing to do with hypothesis testing. One is an archeology paper. To be blunt, it seems like you’re just doing internet searches on “Monte Carlo” to find these links. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

::::::SJohnson, actually they do, did you read the papers? If so you would see how they used the MCM for their data analysis of small data sets, which indeed was hypothesis testing and answers you inquiry about publications that use MCM for small data set analysis. If you wish I can try to track down some actual mathematical publications, however, I am not as familiar with mathematical journals as I am with science/medical journals (not knowing which mathematical journals are acceptable). I am assuming that you have a background in math and possibly access to mathematical journals, therefore if you know the reputable ones I can do the leg work.
::::::I believe the thing that needs to be looked at is there truly a problem with the choice of test and if so what is an alternative. Bayesian might be an option but seems to be difficult to employ for this situation.--[[User:Able806|Able806]] 12:36, 12 March 2009 (EDT)

:::Able806, you still seem to miss the point about how inappropriate the Monte Carlo method (as used in the Lenski paper) is for evaluating rarely occurring events. You need to open your mind to be productive. If you simply cling to a view that Lenski (who I don't think has any meaningful education in statistics) must somehow be right, then you're not going to make any progress in understanding the flaws.--[[User:Aschlafly|Andy Schlafly]] 17:07, 5 March 2009 (EST)

::::Andy, you still have not answered what you find inappropriate about his use of the Monte Carlo method? I am a reasonable person and with evidence I do have an open mind. I provided examples last week, with a working model, showing that Monte Carlo is better than the chi-square in this case. I have also shown where the Chi-Square was inappropriate due to the occurrence size as well. So if you have any evidence that Monte Carlo should not be used in the way that Lenski used please let it be shown.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

Sjohnson, I believe you just proved my point. In the literature of mean and covariance structure analysis, non-central chi-square distribution is commonly used to describe the behavior of the likelihood ratio statistic under alternative hypothesis; it is widely believed that the non-central chi-square distribution is justified by statistical theory. Actually, when the null hypothesis is not trivially violated, the non-central chi-square distribution cannot describe the LR statistic well even when data are normally distributed and the sample size is large. Monte Carlo results compare the strength of the normal distribution against that of the non-central chi-square distribution. In an association analysis comparing cases and controls with respect to allele frequencies at a highly polymorphic locus, a potential problem is that the conventional chi-squared test may not be valid for a large, sparse contingency table. Reliance on statistics with known asymptotic distribution is unnecessary, as Monte Carlo simulations can be performed to estimate the significance level of the test statistic.

Here is a [http://faculty.vassar.edu/lowry/chi_beta.html link] to a great page the provides an interactive example as to why the Chi Squared test would provide poor results compared to the Monte Carlo in relation to the Lenski data workup.

Something you may have overlooked was that the data set is actually too small to use the chi square method correctly. It is often accepted that is any of the analyzed data falls under 10 for a particular cell of the data set then the Yates correction needs to be applied; unfortunately the Yates correction can over correct thus skewing the p-value. Lenksi seemed to understand this by supporting his Monte Carlo p-value results with the Fisher z-transformation p-value.

I hope this helps.--[[User:Able806|Able806]] 10:27, 5 March 2009 (EST)

:I’m still waiting to hear which literature says that “Monte Carlo resampling” is “more accurate than the chi-squared test”. The page mentioned above [http://faculty.vassar.edu/lowry/chi_beta.html] is a discussion of why statisticians “fail to reject the null” rather than “accepting the null” when the p-value is above 0.05 or so. The page says nothing about superiority of Monte Carlo methods. Why were alternate hypothesis distributions mentioned? Only the null hypothesis distribution is used to calculate a p-value. Yates’s correction is for 2x2 contingency tables [http://en.wikipedia.org/wiki/Yates%27_correction_for_continuity]. It doesn’t apply in this case. Finally, what the heck do “covariance structure analysis” and “allele frequencies at a highly polymorphic locus” have to do with this problem? [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::SJohnson, I am looking for this paper for you, I cited it for one of my past publications dealing with allele frequencies (I believe it came from the Duke Biostatistics group). To answer your question about allele frequencies, that is the issue at hand, more about the genetics than the math, but it is the item being studied. So you stated that Yates can not be used and statistics says the number of occurrences is too small to evaluate using the Chi-Squared test so what would you recommend instead of the Monte-Carlo Method?

:Regarding the "Fisher z-transformation p-value" from the paper, garbage in garbage out. If the p-values were bad to begin with, then why would a combination of them be meaningful? [[User:SJohnson|SJohnson]] 10:49, 9 March 2009 (EDT)

::You are assuming that p-values are wrong based on a test that is inappropriate in this case due to data limitations. Did you perform a z-transformation on the chi-squared for the three data groups?--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

::You asked about the “Fisher z-transformation p-value”. The z-transformation test and Fisher’s method are actually two different things (see Whitlock's 2005 paper - Ref. 49 in Blount et al.). But no, I haven’t tried either. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::There's a large literature on various kinds of Monte Carlo test, a very short summary of which is that they're inevitably more accurate than parametric tests (e.g. F, t, chi-squared, etc) because they don't make assumptions about the distribution of the data under the null hypothesis. See for example ''Introduction to the Bootstrap'' by B. Efron and R. Tibshirani and ''The Jack-knife, the Bootstrap and Other Resampling Plans'', also by Efron. They're certainly applicable to small datasets and their accuracy is really only limited by the number of samples you care to take. E.g. 1000 M-C samples would give you a pretty accurate idea about significance at the alpha<1% level (That book should answer SJohnson's questions of 18:50 on 4/3/09 and 16:38 on 5/3/09 about accuracy and Aschalfly's comment of 17:07 on 5/3/09 about appropriateness of Monte Carlo tests.) [[User:FredFerguson|FredFerguson]] 16:53, 11 March 2009 (EDT)

::::Your claim that Monte Carlo methods are “inevitably more accurate” than other tests is obviously wrong because the accuracy of MC methods always depends on the number of realizations used. You should have written <math>\alpha=1\%</math>, not <math>\alpha<1\%</math>. If 1,000 random realizations are generated, the number of realizations above the true <math>\alpha=1\%</math> level is binomial with mean 10 and variance about 10. Thus, the standard deviation of the MC estimate is >0.003. In this example, a Monte Carlo p-value could be off by 30% and still be within a standard deviation. Is that really “pretty accurate”?

::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 3%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::::You're talking about miniscule differences in the accuracy of a test. 0.013 isn't very different from 0.007. In either case, it's very unlikely the experimenter would have obtained that result if the null hypothesis were true. If you're bothered about differences in P-values to the third decimals (which would make you unusual!), just run more MC realisations, that's all. Not really a problem. [[User:FredFerguson|FredFerguson]] 11:53, 12 March 2009 (EDT)

There’s still confusion about the difference between test statistics and Monte Carlo methods. Before you find a Monte Carlo estimate of a p-value, you need to select a test statistic to reduce the data set to a scalar. I am interested in hearing which test statistic you believe should be used in place of the chi-square test and why. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

----

Quick question for SJohnson: How many degrees of freedom did you choose when calculating the p-value? I'd like to know upon what condition you base that number. Thanks.--[[User:Argon|Argon]] 11:05, 5 March 2009 (EST)

:The degree of freedom for a contingency table is rows minus one times columns minus one. That is, <math> (r-1)(c-1) </math>. Here’s a pretty good tutorial I came across: [http://faculty.uncfsu.edu/dwallace/lesson%2020.pdf]. For the experiments from [http://www.pnas.org/content/105/23/7899.full.pdf], the DOFs are 11, 11, and 13. For experiment one, the chi-square test statistic is
::<math>
X^2
=\sum\limits_i\sum\limits_j
\frac{\left(n_{i,j}-E\left[n_{i,j}\right]\right)^2}
{E\left[n_{i,j}\right]}
</math>
::<math>
=\frac{\left(0-1/3\right)^2}{1/3}
+\frac{\left(6-17/3\right)^2}{17/3}
+\frac{\left(0-1/3\right)^2}{1/3}
+\ldots+
</math>
::<math>
+\frac{\left(2-1/3\right)^2}{1/3}
+\frac{\left(4-17/3\right)^2}{17/3}
</math>
::<math>
\approx
14.82
</math>
:where <math>n_{i,j}</math> is the observed value and <math>E\left[n_{i,j}\right]</math> is the expected null hypothesis value. So if you have MS Excel, another way to arrive at the p-value of 0.19 is to type “=CHIDIST(14.82,11)” into a cell. Cheers! [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::OK, thanks for the info. From what I'd calculated and looked up in tables, the numbers seemed close to a df=11 for a chi-square of ~14. (Aside: With terms having 17/3 in the denominator in the figures above, were you using the test of independence? I was using Pearson's test for [http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_for_fit_of_a_distribution fit of a distribution] which returns a chi-squared value of 14 and roughly matched the p-values you reported, assuming the df was 11).

::Also, the first sentence of the article reads: "Blount, Borland, and Lenski[1] claimed that a key evolutionary innovation was observed during a laboratory experiment. That claim is false." A small correction: There were several claims in the paper. The 'key evolutionary innovation' was acquiring the ability to utilize citrate as a food source. That claim was demonstrated multiple times. The claim, which pertains to this statistics discussion was that the Cit+ phenotype arose in a multi-step process, first requiring a rare, pre-adaptive mutation before additional mutation(s) lead to the subsequent development of citrate utilization.--[[User:Argon|Argon]] 20:46, 5 March 2009 (EST)

:::My biology-degreed wife assures me that mutation does not necessarily mean that evolution occurred. What the paper claimed is that evolution (a “key innovation”) occurred in the lab. The key innovation supposedly increased the mutation rate. In the experiments, the observed mutation rate increased after generation 31,000, but not enough to make a statistically significant claim that the rate is not constant. The analysis in the paper was similar to flipping a coin ten times, counting six heads and claiming that the coin must be biased against tails. In reality, there’s nothing surprising about a fair coin producing slightly more of one outcome than the other. Just like there's nothing surprising about there being slightly more mutations in later generations than early generations given the null hypothesis (constant mutation rate). [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

>>Inserting a later comment first<<
SJohnson, the paper's title is: "Historical contingency '''and the evolution of a key innovation''' in an experimental population of ''Escherichia coli''" As I mentioned earlier, the key innovation is the evolution of the Cit+ phenotype and not the timing or rate of its acquisition. And yes, it *is* evolution (call it microevolution, if you wish). Blount et al went on further to speculate how this evolutionary innovation arose and they proposed the historical contingency hypothesis in which 'pre-adaptive' mutations were required before the Cit+ phenotype developed. It is only this latter hypothesis that you are attempting to address with your chi-square analysis, not the fact that Cit+ mutants arose (which is the evolutionary innovation).--[[User:Argon|Argon]] 21:57, 18 March 2009 (EDT)

::::SJohnson, not to say anything about your wife, but has she had a 400 level molecular genetics course (most general biology degrees do not cover the detail unless they are specialized)? If so, she would have mentioned that if the mutation passes to the offspring and is selectively beneficial to the population then it is a step of evolution as along as the conditions continue through the sharing of the mutation with the population and the environment is such that reduces the growth rate of the non-transformed population. While not all mutations are signs that evolution occurred the mutations that pass to offspring and provide a benefit compared to other offspring are very strong indicators. In the case of this paper the population that evolved the cit+ was able to metabolize a chemical in their environment which allowed for an adaptation advantage compared to the non-transformed colonies.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

----

Let’s go back to the beginning. There appears to be confusion about the difference between test statistics and methods for computing p-values. As is noted at the beginning of the page [http://www.conservapedia.com/Significance_of_E._Coli_Evolution_Experiments], the fundamental problem with the paper is that it used a flawed test statistic, not that it used Monte Carlo methods to find the p-value for that flawed statistic.

Every hypothesis test uses a test statistic to reduce the data to a single number. The p-value for the test statistic can be calculated analytically (as I’ve done for the chi-square test statistic) or by Monte Carlo methods. In the paper, Monte Carlo methods were used to compute the p-value of the “mutation generation” test statistic. The key problem with the analysis from the paper is that it doesn’t work to use a weighted average to test for variations in mutation rate. This is like trying to use the sample variance to test for an increase in the mean in Gaussian-distributed data. A statistic should be selected based on the null and alternate hypothesis distributions of the data. The chi-square test (unlike the weighted average from the paper) is a reasonable choice for data that mutates at a constant rate under the null hypothesis, but mutates at varying rates under the alternate hypothesis.

Able806, you made a good point about the contingency table cell frequencies being relatively low, but were wrong when you said ”the data set is actually too small to use the chi square method correctly”. In the low cell frequency case the chi-square test is still effective, but the null hypothesis distribution of the chi-square statistic starts to look less like the chi-square distribution. Thus, p-values calculated using the chi-square distribution may be a bit off. However, Monte Carlo p-values are always imperfect as well because it's impossible to generate an infinite number of random realizations. There are imperfections in p-values generated by analytic and Monte Carlo methods. However, low cell frequencies does not explain the >20x and >2.5x differences between chi-square p-values and p-values from the paper for experiments one and three. The reason for those huge differences was the use of the flawed test statistic (“mutation generation”) in the paper. [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

:SJohnson, the chi-squared test is a valuable statistical tool, but the limitations of the test must be acknowledged. The chi-squared test can only produce valid results if the assumptions that underly the test are not violated. As an analogy, Newtonian models of motion fail to produce accurate results as velocities approach the speed of light; under those circumstances one must switch to a theory that accounts for relativistic effects.

:It seems that you have simply dismissed the [http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm widely-acknowledged] [http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm fact] that the [http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html chi-squared test] is [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236 inappropriate] for use in [http://www.graphpad.com/www/Book/Choose.htm situations] where n in any cell is [http://mysite.du.edu/~jcalvert/econ/chisquar.htm less] less than a [http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1 threshold] [http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html number]. Different authors set different thresholds, but all are well above the numbers seen in your chi-squared analysis - even the most liberal guidelines advise against the chi-squared test when any expected cell frequency is less than one or more than 20% of the table cells are less than 5; others require that expected values in all cells must be more than 5. With smaller amounts of data, the test is insensitive and errs on the side of rejecting the hypothesis. If you attempt your chi-squared statistical analysis with a program that is more sophisticated than MS Excel (as I did), you get an error message indicating that the results are invalid due to low expected cell counts.

:That issue aside, there are other reasons that the chi-squared test is inappropriate here. As the links above point out, the categories tested must be truly independent; one example is that you can't use the chi-squared test to compare age and ability to kick a field goal by testing the same experimental group twice, one year apart; you have to test one group of age A and a different group of age B. In the case of the Blount paper, the categories are not independent. Even if there were adequate numbers to address the low-expected-frequency problem, this would make the chi-squared an invalid test in this case.

:There are other significant problems with the use of the chi-squared test in this circumstance, but they can wait until you address these first major problems.--[[User:ElyM|ElyM]] 12:18, 11 March 2009 (EDT)

::Wackerly et al. says in general it’s assumed that the cell frequencies are above five so that the chi-square statistic (under the null) is approximately chi-square distributed (see p. 703). That book does not say chi-square test results are invalid if frequencies are five or less. Your example of a chi-square test warning message (it said "warning" not "error" as you stated) in Minitab [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236] said “approximation probably invalid” referring to the chi-square distribution approximation to the chi-square test statistic’s distribution. Your example did not say “chi-square test invalid”. I agree that when cell frequencies are low, the chi-square test statistic’s distribution starts to deviate from the chi-square distribution. I maintain that this deviation is not enough to explain the >2.5x and >20x differences in the chi-square test p-values and the p-values from the paper.

::As the numerous links in your post proved, the chi-square test is widely-used by statisticians. Can you give examples of statisticians using mean mutation generation as a test statistic? Also, did your software agree with the chi-square test p-values I presented? Thanks. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::Thank you for giving page references for Wackerly; however it seems we have different editions, since page 703 in my copy (5th ed, 1996) does not deal with chi-squared issues at all. My copy does state the following, on page 622: "Although the mathematical proof is beyond the scope of this text, it can be shown that, when n is large [chi-squared] will possess approximately a chi-square probability distribution in repeated sampling." Then, on page 624: "Experience has shown that cell counts [n sub i] should not be too small in order that the chi-square distribution provide an accurate approximation to the distribution of [chi squared]. As a rule of thumb we require that all expected cell counts equal or exceed 5, although Cochran (1952) has noted that this value can be as low as 1 for some situations." Wackerly then goes on, in the problems sections, to describe the use of the chi-squared test as a "violation of good statistical practice" when "some expected counts [are] <5."

:::It seems that you are already aware that the [chi-square] statistic under the null is no longer chi-square distributed for small n; this is precisely why the test should not be used under those conditions. I can claim to be able to accelerate a 1-kg mass to 10 times the speed of light by applying 1 N of force for 95 years by using F=ma and t= (vf-vi)/a. Plugging the numbers into those equations will produce the same result every time, but the answer is illegitimate because those equations are only valid under certain assumptions, which are violated as velocities approach the speed of light. Similarly, having a statistical program calculate a chi-squared value given the Blount data will produce a number result, but since the assumptions of the test are violated the result is not legitimate. Yes, if I put the Blount data in SAS 9.2, I get the same numerical answer as you do, but I also get the following message: "WARNING: >89% of the cells have expected counts less than 5. Chi-square may not be a valid test." You may argue that that's a warning, not an error; that's a semantic distinction. The reason that the program says that it MAY not be valid is that the chi-squared test skews in the direction of being too conservative at low n values; the test has an acceptable rate of false positives but an unacceptably high rate of false negatives. Comparing the results of the Monte Carlo and chi-squared results in this case is like comparing the results of Newtonian and relativistic equations of motion: they can produce very different results from the same input data.

::::For a finite amount of data, the chi-square statistic is never chi-square distributed under the null. The p-values are always approximate regardless of cell frequencies. The approximation becomes more accurate as the amount of data increases, but I don’t believe that this inaccuracy will change p-values that are about 0.2 (for experiments 1 and 3) into statistically significant p-values. How much do you expect the p-values to change if an exact computation is used in place of the chi-square distribution approximation? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::Your last paragraph has a major non sequitur in it: yes, many statisticians use the chi-square test. As long as the assumptions of the test are not violated, it is a valuable tool. That has nothing to do with the validity of using mean mutation generation as a test statistic. 'Mean number of werewolf attacks in Mumbai in the week centered on the new moon, by month, from 1654 to 1798' is a valid test statistic. I am quite sure that it has never been used in a peer-reviewed paper before. That does not mean that I can't perform valid statistical tests on that statistic. If, however, the incorrect test is applied, the results of the analysis will be flawed. Papers apply a (relatively small) standard repertoire of valid tests to a (potentially infinite) number of test statistics. The particular test statistic used in a paper may never have been used before and may never be used again; that does not address the validity of the analysis. In Blount's case, the test is the Monte Carlo analysis, which is also "widely-used by statisticians".

::::There are an infinite number of ways to reduce a data set to a single number. However, it’s foolish to think every method would be effective. I gave an example of a flawed test statistic in an earlier post [http://www.conservapedia.com/index.php?title=Talk%3ASignificance_of_E._Coli_Evolution_Experiments&diff=635070&oldid=634987]. Another example of a flawed test statistic is the one used in the paper because it does not always detect deviations from the null hypothesis (see: [[Significance of E. Coli Evolution Experiments#Test Statistics]]).

::::Test statistics are typically derived. The likelihood ratio test is a common method used to derive them. The chi-square test for independence is an approximation to the LRT. Where is the derivation saying that mean mutation generation is an appropriate test statistic for this problem? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::We still haven't touched on the issue of the categories not being independent, which by itself is sufficient to invalidate the chi-squared technique. I'm new to this site, so I'm unsure as to the etiquette of making changes to the articles of another person - but the article here should at the very least mention that the chi-square test is being used here in a manner that violates its underlying assumptions in at least two fundamental ways, and the results are therefore suspect.--[[User:ElyM|ElyM]] 17:34, 12 March 2009 (EDT)

:::::When generating random realizations of experiment outcomes, the authors assumed that the total number of mutants was fixed. Thus the paper assumed the numbers of mutants per generation are statistically dependent. Does this seem like a realistic model, or do you think that if the experiments were recreated that the total number of mutants could vary? For example, if experiment one were recreated, would the total number of mutants always be exactly four? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::: It looks to me as though SJohnson has misinterpreted the application of the chi-squared test in quite a fundamental way. His/her analysis of Blount's data are therefore close to meaningless, regardless of whether the test used by Blount is appropriate or not. In my opinion, the entire page should therefore be deleted. [[User:FredFerguson|FredFerguson]] 08:18, 13 March 2009 (EDT)

::::: "Fred", perhaps you mistakenly think this is Wikipedia, where [[censorship]] and deletion of pages for ideological reasons are common. Not here.--[[User:Aschlafly|Andy Schlafly]] 10:23, 14 March 2009 (EDT)

:::::: Umm... I'm suggesting deletion for mathematical reasons, not ideological reasons. Using an argument filled with mathematical errors to try to support your case only detracts from your credibility. [[User:FredFerguson|FredFerguson]] 10:38, 14 March 2009 (EDT)

:::::: Actually, I think correction is better than deletion. So that's what I've done. [[User:FredFerguson|FredFerguson]] 11:01, 14 March 2009 (EDT)

::::::: I find no credibility in your denial of having ideological reasons.--[[User:Aschlafly|Andy Schlafly]] 11:04, 14 March 2009 (EDT)

== Misinterpretation of test ==

SJohnson, Your analysis misinterprets the test. You say the null hypothesis is that this mutation cannot happen. They saw a mutation (4 mutations, in fact, in the data set you show) so the null hypothesis (as you state is) is disproved. That's perfectly straightforward.

I don't know what the "mean mutation generation" test is but you're doing when you apply a chi-squared test to this dataset is to test if the mutations are evenly distributed throughout the generations. Your test says they are, so there's no strong evidence to suppose that mutations are likely to occur in one generation rather than another in the series of tests. Blount's test says thay aren't, so it's more likely that the mutation will occur later in the series of tests. I can't tell which test is right without knowing more about the test that Blount used.

But that point (the foregoing paragraph) has no bearing at all on the null hypothesis, as you describe it. The mutation appeared, so that means the hypothesis that the mutation can't happen is disproved. Very simple. [[User:FredFerguson|FredFerguson]] 21:10, 8 March 2009 (EDT)

:I never said that “the null hypothesis is that this mutation cannot happen”. The chi-square test statistic I'm using wouldn’t be defined if the null hypothesis mutation rate was zero because the <math>E\left[n_{i,j}\right]</math> term in the denominator of the statistic (see above equation) would be zero.

:The test statistic from the paper is the average of the generation numbers of observed mutations. For experiment one this number is
::<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)= 31750.
</math>
:The same number is shown in Table 2 of the paper. [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

:: SJohnson, the way you're calculating the chi-squared statistic implies that you're testing the null hypothesis of a constant mutation rate over time against an alternative hypothesis of a mutation rate which varies over time. [[User:FredFerguson|FredFerguson]] 11:02, 9 March 2009 (EDT)

As it currently stands, the article makes the following statement: "The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown." This misstates the null hypothesis of the paper, which is elaborated in the Introduction section of the paper, and repeated in the section '''Statistical Analysis of the Replay Experiments''':
<blockquote>
For each experiment, we compared the observed mean generation of those clones that yielded Cit+ variants to the mean expected under the null hypothesis that clones from all generations have equal likelihood. The null thus corresponds to the rare-mutation hypothesis laid out in the Introduction."
Block quote
</blockquote><ref>www.pnas.org/cgi/reprint/105/23/7899.pdf</ref>

The article also continues to describe 'mean mutation generation' as a ''test'' rather than a ''statistic'' to which the ''Monte Carlo test'' was applied.--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

==Unreferenced Claims==

I deleted the claim that mean mutation generation is an appropriate test statistic because no reference was produced that back that claim. No reference was provided to back the claim that the chi-square test p-values are always conservative, either. [[User:SJohnson|SJohnson]] 12:57, 14 March 2009 (EDT)

: The reference is Everitt. I'll check I put it in the right place. [[User:FredFerguson|FredFerguson]] 13:30, 14 March 2009 (EDT)

There was a typo in my edit summaries on the talk page and the main page. I meant to say "Removed unsupported claims" rather than "Removed supported claims". [[User:SJohnson|SJohnson]] 13:13, 14 March 2009 (EDT)

I do not believe that anyone has claimed that 'chi-square test p-values are ''always'' conservative'. The claim that has been made is that ''under certain circumstances'', namely low n and low individual cell values, the chi-square test is an invalid test; that under those circumstances the power of the test is low and it becomes impossible to reject the null hypothesis even when it is false. You may have missed the pertinent sections in my links above, so I will directly quote the relevant sections. All the quoted sections refer to chi-square testing in particular. Any bolding below is mine.

:This edit claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639379]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

<blockquote>
"Assumptions: 

Even though a nonparametric statistic does not require a normally distributed population, there still are some restrictions regarding its use. 

1. Representative sample (Random) 
2. The data must be in frequency form (nominal data) or greater. 
3. The individual observations must be independent of each other. 
4. '''Sample size must be adequate. In a 2 x 2 table, Chi Square should not be used if n is less than 20. In a larger table, no expected value should be less than 1, and not more than 20% of the variables can have expected values of less than 5'''. 
5. Distribution basis must be decided on before the data is collected. 
6. The sum of the observed frequencies must equal the sum of the expected frequencies."
</blockquote>
<ref>http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm
</ref>

<blockquote>
"Assumptions:
* Random sample data are assumed. As with all significance tests, if you have population data, then any table differences are real and therefore significant. If you have non-random sample data, significance cannot be established, though significance tests are nonetheless sometimes utilized as crude "rules of thumb" anyway.
* A sufficiently large sample size is assumed, as in all significance tests. '''Applying chi-square to small samples exposes the researcher to an unacceptable rate of Type II errors. There is no accepted cutoff. Some set the minimum sample size at 50, while others would allow as few as 20'''. Note chi-square must be calculated on actual count data, not substituting percentages, which would have the effect of pretending the sample size is 100.
* '''Adequate cell sizes are also assumed. Some require 5 or more, some require more than 5, and others require 10 or more. A common rule is 5 or more in all cells of a 2-by-2 table, and 5 or more in 80% of cells in larger tables, but no cells with zero count'''. When this assumption is not met, Yates' correction is applied.
* Independence. Observations must be independent. The same observation can only appear in one cell. '''This means chi-square cannot be used to test correlated data (ex., before-after, matched pairs, panel data)'''.
* Similar distribution. Observations must have the same underlying distribution.
* Known distribution. The hypothesized distribution is specified in advance, so that the number of observations that are expected to appear each cell in the table can be calculated without reference to the observed values. Normally this expected value is the crossproduct of the row and column marginals divided by the sample size.
* Non-directional hypotheses are assumed. Chi-square tests the hypothesis that two variables are related only by chance. If a significant relationship is found, this is not equivalent to establishing the researcher's hypothesis that A causes B, or that B causes A.
* Finite values. Observations must be grouped in categories.
* Normal distribution of deviations (observed minus expected values) is assumed. Note chi-square is a nonparametric test in the sense that is does not assume the parameter of normal distribution for the data -- only for the deviations.
* Data level. No assumption is made about level of data. Nominal, ordinal, or interval data may be used with chi-square tests."
</blockquote>
<ref>http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm</ref>

<blockquote>
"Assumptions: 
-None of the expected values may be less than 1 
-No more than 20% of the expected values may be less than 5"</blockquote> <ref>http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html</ref>

<blockquote>
"When performing a chi-square test, your data must satisfy important assumptions. Although these assumptions may be stated differently in different textbooks, they generally assert that: 
1)The sample must be randomly drawn from the population 
'''2)The sample size, n, must be large enough so that the expected cell count in each cell is greater than or equal to 5.''' 
Both assumptions must be met in the process of collecting your data, and violations of the second assumption will appear in the Minitab output when you run the analysis. 
... 
'''You may wonder why the second assumption is necessary for performing the chi-square test. The second assumption arises because the distribution of counts under the null hypothesis is multinomial, and the normal distribution can be used to approximate the multinomial distribution if the sample size is sufficiently large and the probability parameters aren't too small. It can be shown via the Central Limit Theorem that the multinomial distribution converges to the normal distribution as the sample size approaches infinity; however, there is no easy way to show mathematically how and when the convergence fails.'''"
</blockquote>
<ref>http://www.minitab.com/support/docs/Answers/Chi-Square%20Test%20Assumptions.pdf</ref>

<blockquote>
"'''The chi-square test is simpler to calculate but yields only an approximate P value. ... You should definitely avoid the chi-square test when the numbers in the contingency table are very small (any number less than about six)'''."
</blockquote> <ref>http://www.graphpad.com/www/Book/Choose.htm</ref>

<blockquote>
"The most important things to remember to get a valid χ2 test are that the expected values are not too small in any bin (certainly 5 or more), and that the degrees of freedom are properly evaluated. '''Unless you have a very large amount of data, the test is not very sensitive and errs on the side of safety. If you get a significant result, however, it is not likely to be wrong.'''"
</blockquote> <ref>http://mysite.du.edu/~jcalvert/econ/chisquar.htm</ref>

<blockquote>
"The critical assumptions of the chi-square test for k independent samples are similar to those for the chi-square test for two independent samples. ... 
'''4. No more than 20% of the cells may have expected frequencies of less than 5, and no cell should have an expected frequency of less than 1. 
The rule given in Assumption 4 is particularly important for a contingency table that is larger than 2X2'''"
</blockquote> <ref>http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1
</ref>

<blockquote>
"Special problems with small expected cell frequencies for the chi-square test: 
The chi-square test involves using the chi-square distribution to approximate the underlying exact distribution. The approximation becomes better as the expected cell frequencies grow larger, and '''may be inappropriate for tables with very small expected cell frequencies.''' 
'''For tables with expected cell frequencies less than 5, the chi-square approximation may not be reliable. A standard (and conservative) rule of thumb (due to Cochran) is to avoid using the chi-square test for tables with expected cell frequencies less than 1, or when more than 20% of the table cells have expected cell frequencies less than 5.''' 
Another rule of thumb (due to Roscoe and Byars) is that the average expected cell frequency should be at least 1 when the expected cell frequencies are close to equal, and 2 when they are not. (If the chosen significance level is 0.01 instead of 0.05, then double these numbers.) 
Koehler and Larntz suggest that if the total number of observations is at least 10, the number categories is at least 3, and the square of the total number of observations is at least 10 times the number of categories, then the chi-square approximation should be reasonable. 
Care should be taken when cell categories are combined (collapsed together) to fix problems of small expected cell frequencies. Collapsing can destroy evidence of non-independence, so a failure to reject the null hypothesis for the collapsed table does not rule out the possibility of non-independence in the original table. 
'''As with most statistical tests, the power of the chi-square test increases with a larger number of observations. If there are too few observations, it may be impossible to reject the null hypothesis even if it is false.'''"
</blockquote> <ref>http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html</ref>--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

: Thanks for this really excellent contribution, ElyM. The only thing I'd like to add is in relation to your initial statement, "I do not believe that anyone has claimed that 'chi-square test p-values are always conservative'". The question of whether a test is conservative in a particular situation is probabilistic. One can determine whether a test is likely to generate a p-value which is too high in a particular situation (e.g. for a chi-squared test, when there are lots of small expected values) but one needs an exact test (such as an appropriate Monte Carlo randomisation test) to determine whether the p-value in any ''particular'' test is in fact excessively high.

::This edit also claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639373]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

: I hope careful reading of your very clear description will put SJohnson's mind at rest on this subject. [[User:FredFerguson|FredFerguson]] 18:11, 14 March 2009 (EDT)

ElyM, you've provided nothing to address the basic flaw that "The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events." See [[Flaws in Lenski Study]]. Also, do not impose your view on the content page until after SJohnson has had an opportunity to respond to your posting. As to "Fred", his put-downs are getting tiresome and I'm going to review his edit pattern now to see if he's been contributing anything of value to this site.--[[User:Aschlafly|Andy Schlafly]] 14:06, 15 March 2009 (EDT)

::Mr. Schlafly, per your request I have not added anything to the content page as SJohnson has not yet responded to my posts. Since all of my comments have been in regards to SJohnson's use of the chi-square test in this particular article, I'm not sure why you expect me to address Blout's use of Monte Carlo - that issue seems to be addressed on the [[Flaws in Lenski Study]] page. SJohnson has added a reformulation of the chi-square test for two possible outcomes, and stated that the chi-square test is at a minimum when all success probabilities are equal. He then extrapolates from this to claim that the chi-square test is an effective test for the data from Blount.

::The reformulation of the equations for two possible outcomes does not address the underlying problem that the chi-square test has universally accepted parameters outside of which it is considered an invalid test; I have provided references for these parameters and shown that the data from Blount lies outside them. None of the expected cells in SJohnson's analysis have values above one, and the total n is four. SJohnson's own reference states that the application of the chi-square test in this circumstance is a "violation of good statistical practice". Analogously, combining F=ma and t=(vf-vi)/a into t=(vf-vi)m/F and showing that t is a minimum when m approaches zero does not address the fact that those Newtonian equations do not apply as velocities approach the speed of light. The legitimacy of Blount's arguments cannot be determined by the application of illegitimate counterarguments. If SJohnson or others can point to references from the statistical literature that show that Blount has made methodological errors - as I have been able to do with SJohnson's chi-square analysis - I would welcome their input, and no doubt Conservapedia's other readers would as well, and this page would be greatly improved.

::I have not seen a rebuttal from SJohnson in the four days since my last post, although he has added new material to the content page since then. In light of this, I would appreciate some guidelines as to when it is appropriate for me to add my information and references to the content page. I can add citations from the primary mathematical literature if necessary, but in general I find that these are less helpful as they are not easily accessible by readers without access to academic libraries.--[[User:ElyM|ElyM]] 18:07, 18 March 2009 (EDT)

== References ==
{{reflist}}

Talk:Significance of E. Coli Evolution Experiments

2009-03-19T01:57:44Z

Argon:

SJohnson, your assessment, while good in the utilization of the chi-squared test is unfortunately incorrect. The Monte Carlo resampling gives a more accurate p-value than the chi-squared. You may research the literature (i.e. publications in statistical mathematics, many pubs actualy compare Monte Carlo vs Chi Squared) to discover that this method is commonly used in advance statistical work and how it is more accurate than the chi-squared test.--[[User:Able806|Able806]] 17:00, 4 March 2009 (EST)

:It doesn’t make sense to compare the chi-square test, which is a specific statistical hypothesis test, to Monte Carlo methods, which can be used for anything from fluid motion modeling to p-value computations. You can use Monte Carlo methods to compute the p-values of the chi-square test!

:Monte Carlo methods involve the generation of random realizations. Your broad claim the Monte Carlo methods are “more accurate” than the chi-square test is obviously incorrect because the accuracy of Monte Carlo methods always depends on the number of random realizations generated. When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous.

:Which publications compare Monte Carlo to chi-square and show that the former is more accurate? Could you provide specific examples? Thanks. [[User:SJohnson|SJohnson]] 18:50, 4 March 2009 (EST)

:In furtherance of SJohnson's remarks with respect to rarely occurring events, the use of the basic Monte Carlo method is plainly incorrect for modeling a rarely occurring event, as the Lenski paper did. This has long been pointed out in [[Flaws in Richard Lenski Study]]. I know [[evolutionists]] will never admit a flaw in anything promoting their pet theory, but this (and other) flaws in that paper is undeniable.

:Watch how evolutionists defended obvious errors in the Lenski paper, and then realize why the [[Piltdown Man]] fraud was taught for 40 years without evolutionists admitting it was a hoax.--[[User:Aschlafly|Andy Schlafly]] 09:55, 5 March 2009 (EST)

::Andy, how exactly is the Monte Carlo method incorrect to use in this case? I have seen it used in publications with much smaller datasets.--[[User:Able806|Able806]] 10:29, 5 March 2009 (EST)

:::Able806, I'm interested in looking at the publications you mentioned that use Monte Carlo methods to analyze small data sets. Could you provide some examples? Thanks. [[User:SJohnson|SJohnson]] 16:41, 5 March 2009 (EST)

::::SJohnson, here are two papers, [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6WH8-45RFJ1J-19&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=1ad95954654bb97b17e474ce6b469f6e 1] and [http://cat.inist.fr/?aModele=afficheN&cpsidt=787963 2]. Most are in chemistry and genetics where you find the observed to be much smaller and have to use the MCM. You can search on the subject as well and find that how Lenski performed the test is the standard for microbiological genetic analysis.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

:::::Those papers have nothing to do with hypothesis testing. One is an archeology paper. To be blunt, it seems like you’re just doing internet searches on “Monte Carlo” to find these links. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

::::::SJohnson, actually they do, did you read the papers? If so you would see how they used the MCM for their data analysis of small data sets, which indeed was hypothesis testing and answers you inquiry about publications that use MCM for small data set analysis. If you wish I can try to track down some actual mathematical publications, however, I am not as familiar with mathematical journals as I am with science/medical journals (not knowing which mathematical journals are acceptable). I am assuming that you have a background in math and possibly access to mathematical journals, therefore if you know the reputable ones I can do the leg work.
::::::I believe the thing that needs to be looked at is there truly a problem with the choice of test and if so what is an alternative. Bayesian might be an option but seems to be difficult to employ for this situation.--[[User:Able806|Able806]] 12:36, 12 March 2009 (EDT)

:::Able806, you still seem to miss the point about how inappropriate the Monte Carlo method (as used in the Lenski paper) is for evaluating rarely occurring events. You need to open your mind to be productive. If you simply cling to a view that Lenski (who I don't think has any meaningful education in statistics) must somehow be right, then you're not going to make any progress in understanding the flaws.--[[User:Aschlafly|Andy Schlafly]] 17:07, 5 March 2009 (EST)

::::Andy, you still have not answered what you find inappropriate about his use of the Monte Carlo method? I am a reasonable person and with evidence I do have an open mind. I provided examples last week, with a working model, showing that Monte Carlo is better than the chi-square in this case. I have also shown where the Chi-Square was inappropriate due to the occurrence size as well. So if you have any evidence that Monte Carlo should not be used in the way that Lenski used please let it be shown.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

Sjohnson, I believe you just proved my point. In the literature of mean and covariance structure analysis, non-central chi-square distribution is commonly used to describe the behavior of the likelihood ratio statistic under alternative hypothesis; it is widely believed that the non-central chi-square distribution is justified by statistical theory. Actually, when the null hypothesis is not trivially violated, the non-central chi-square distribution cannot describe the LR statistic well even when data are normally distributed and the sample size is large. Monte Carlo results compare the strength of the normal distribution against that of the non-central chi-square distribution. In an association analysis comparing cases and controls with respect to allele frequencies at a highly polymorphic locus, a potential problem is that the conventional chi-squared test may not be valid for a large, sparse contingency table. Reliance on statistics with known asymptotic distribution is unnecessary, as Monte Carlo simulations can be performed to estimate the significance level of the test statistic.

Here is a [http://faculty.vassar.edu/lowry/chi_beta.html link] to a great page the provides an interactive example as to why the Chi Squared test would provide poor results compared to the Monte Carlo in relation to the Lenski data workup.

Something you may have overlooked was that the data set is actually too small to use the chi square method correctly. It is often accepted that is any of the analyzed data falls under 10 for a particular cell of the data set then the Yates correction needs to be applied; unfortunately the Yates correction can over correct thus skewing the p-value. Lenksi seemed to understand this by supporting his Monte Carlo p-value results with the Fisher z-transformation p-value.

I hope this helps.--[[User:Able806|Able806]] 10:27, 5 March 2009 (EST)

:I’m still waiting to hear which literature says that “Monte Carlo resampling” is “more accurate than the chi-squared test”. The page mentioned above [http://faculty.vassar.edu/lowry/chi_beta.html] is a discussion of why statisticians “fail to reject the null” rather than “accepting the null” when the p-value is above 0.05 or so. The page says nothing about superiority of Monte Carlo methods. Why were alternate hypothesis distributions mentioned? Only the null hypothesis distribution is used to calculate a p-value. Yates’s correction is for 2x2 contingency tables [http://en.wikipedia.org/wiki/Yates%27_correction_for_continuity]. It doesn’t apply in this case. Finally, what the heck do “covariance structure analysis” and “allele frequencies at a highly polymorphic locus” have to do with this problem? [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::SJohnson, I am looking for this paper for you, I cited it for one of my past publications dealing with allele frequencies (I believe it came from the Duke Biostatistics group). To answer your question about allele frequencies, that is the issue at hand, more about the genetics than the math, but it is the item being studied. So you stated that Yates can not be used and statistics says the number of occurrences is too small to evaluate using the Chi-Squared test so what would you recommend instead of the Monte-Carlo Method?

:Regarding the "Fisher z-transformation p-value" from the paper, garbage in garbage out. If the p-values were bad to begin with, then why would a combination of them be meaningful? [[User:SJohnson|SJohnson]] 10:49, 9 March 2009 (EDT)

::You are assuming that p-values are wrong based on a test that is inappropriate in this case due to data limitations. Did you perform a z-transformation on the chi-squared for the three data groups?--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

::You asked about the “Fisher z-transformation p-value”. The z-transformation test and Fisher’s method are actually two different things (see Whitlock's 2005 paper - Ref. 49 in Blount et al.). But no, I haven’t tried either. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::There's a large literature on various kinds of Monte Carlo test, a very short summary of which is that they're inevitably more accurate than parametric tests (e.g. F, t, chi-squared, etc) because they don't make assumptions about the distribution of the data under the null hypothesis. See for example ''Introduction to the Bootstrap'' by B. Efron and R. Tibshirani and ''The Jack-knife, the Bootstrap and Other Resampling Plans'', also by Efron. They're certainly applicable to small datasets and their accuracy is really only limited by the number of samples you care to take. E.g. 1000 M-C samples would give you a pretty accurate idea about significance at the alpha<1% level (That book should answer SJohnson's questions of 18:50 on 4/3/09 and 16:38 on 5/3/09 about accuracy and Aschalfly's comment of 17:07 on 5/3/09 about appropriateness of Monte Carlo tests.) [[User:FredFerguson|FredFerguson]] 16:53, 11 March 2009 (EDT)

::::Your claim that Monte Carlo methods are “inevitably more accurate” than other tests is obviously wrong because the accuracy of MC methods always depends on the number of realizations used. You should have written <math>\alpha=1\%</math>, not <math>\alpha<1\%</math>. If 1,000 random realizations are generated, the number of realizations above the true <math>\alpha=1\%</math> level is binomial with mean 10 and variance about 10. Thus, the standard deviation of the MC estimate is >0.003. In this example, a Monte Carlo p-value could be off by 30% and still be within a standard deviation. Is that really “pretty accurate”?

::::Using one million MC realizations (as done in the paper) at the <math>\alpha=0.001</math> level means the standard deviation is about 3%. The paper reported a p-value of less than 0.001 (experiment two). It wouldn’t surprise me to find out that the experiment two p-value for the flawed test is off because only one million realizations were used. My original statement, “When p-values are small, Monte Carlo methods are notoriously inaccurate unless the number of realizations generated is enormous” is correct. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::::You're talking about miniscule differences in the accuracy of a test. 0.013 isn't very different from 0.007. In either case, it's very unlikely the experimenter would have obtained that result if the null hypothesis were true. If you're bothered about differences in P-values to the third decimals (which would make you unusual!), just run more MC realisations, that's all. Not really a problem. [[User:FredFerguson|FredFerguson]] 11:53, 12 March 2009 (EDT)

There’s still confusion about the difference between test statistics and Monte Carlo methods. Before you find a Monte Carlo estimate of a p-value, you need to select a test statistic to reduce the data set to a scalar. I am interested in hearing which test statistic you believe should be used in place of the chi-square test and why. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

----

Quick question for SJohnson: How many degrees of freedom did you choose when calculating the p-value? I'd like to know upon what condition you base that number. Thanks.--[[User:Argon|Argon]] 11:05, 5 March 2009 (EST)

:The degree of freedom for a contingency table is rows minus one times columns minus one. That is, <math> (r-1)(c-1) </math>. Here’s a pretty good tutorial I came across: [http://faculty.uncfsu.edu/dwallace/lesson%2020.pdf]. For the experiments from [http://www.pnas.org/content/105/23/7899.full.pdf], the DOFs are 11, 11, and 13. For experiment one, the chi-square test statistic is
::<math>
X^2
=\sum\limits_i\sum\limits_j
\frac{\left(n_{i,j}-E\left[n_{i,j}\right]\right)^2}
{E\left[n_{i,j}\right]}
</math>
::<math>
=\frac{\left(0-1/3\right)^2}{1/3}
+\frac{\left(6-17/3\right)^2}{17/3}
+\frac{\left(0-1/3\right)^2}{1/3}
+\ldots+
</math>
::<math>
+\frac{\left(2-1/3\right)^2}{1/3}
+\frac{\left(4-17/3\right)^2}{17/3}
</math>
::<math>
\approx
14.82
</math>
:where <math>n_{i,j}</math> is the observed value and <math>E\left[n_{i,j}\right]</math> is the expected null hypothesis value. So if you have MS Excel, another way to arrive at the p-value of 0.19 is to type “=CHIDIST(14.82,11)” into a cell. Cheers! [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

::OK, thanks for the info. From what I'd calculated and looked up in tables, the numbers seemed close to a df=11 for a chi-square of ~14. (Aside: With terms having 17/3 in the denominator in the figures above, were you using the test of independence? I was using Pearson's test for [http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_for_fit_of_a_distribution fit of a distribution] which returns a chi-squared value of 14 and roughly matched the p-values you reported, assuming the df was 11).

::Also, the first sentence of the article reads: "Blount, Borland, and Lenski[1] claimed that a key evolutionary innovation was observed during a laboratory experiment. That claim is false." A small correction: There were several claims in the paper. The 'key evolutionary innovation' was acquiring the ability to utilize citrate as a food source. That claim was demonstrated multiple times. The claim, which pertains to this statistics discussion was that the Cit+ phenotype arose in a multi-step process, first requiring a rare, pre-adaptive mutation before additional mutation(s) lead to the subsequent development of citrate utilization.--[[User:Argon|Argon]] 20:46, 5 March 2009 (EST)

:::My biology-degreed wife assures me that mutation does not necessarily mean that evolution occurred. What the paper claimed is that evolution (a “key innovation”) occurred in the lab. The key innovation supposedly increased the mutation rate. In the experiments, the observed mutation rate increased after generation 31,000, but not enough to make a statistically significant claim that the rate is not constant. The analysis in the paper was similar to flipping a coin ten times, counting six heads and claiming that the coin must be biased against tails. In reality, there’s nothing surprising about a fair coin producing slightly more of one outcome than the other. Just like there's nothing surprising about there being slightly more mutations in later generations than early generations given the null hypothesis (constant mutation rate). [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

>>Insert later comment first<<
SJohnson, the paper's title is: "Historical contingency '''and the evolution of a key innovation''' in an experimental population of ''Escherichia coli''" As I mentioned earlier, the key innovation is the evolution of the Cit+ phenotype and not the timing or rate of its acquisition. And yes, it *is* evolution (call it microevolution, if you wish). Blount et al went on further to speculate how this evolutionary innovation arose and they proposed the historical contingency hypothesis in which 'pre-adaptive' mutations were required before the Cit+ phenotype developed. It is only this latter hypothesis that you are attempting to address with your chi-square analysis, not the fact that Cit+ mutants arose (which is the evolutionary innovation).--[[User:Argon|Argon]] 21:57, 18 March 2009 (EDT)

::::SJohnson, not to say anything about your wife, but has she had a 400 level molecular genetics course (most general biology degrees do not cover the detail unless they are specialized)? If so, she would have mentioned that if the mutation passes to the offspring and is selectively beneficial to the population then it is a step of evolution as along as the conditions continue through the sharing of the mutation with the population and the environment is such that reduces the growth rate of the non-transformed population. While not all mutations are signs that evolution occurred the mutations that pass to offspring and provide a benefit compared to other offspring are very strong indicators. In the case of this paper the population that evolved the cit+ was able to metabolize a chemical in their environment which allowed for an adaptation advantage compared to the non-transformed colonies.--[[User:Able806|Able806]] 10:19, 11 March 2009 (EDT)

----

Let’s go back to the beginning. There appears to be confusion about the difference between test statistics and methods for computing p-values. As is noted at the beginning of the page [http://www.conservapedia.com/Significance_of_E._Coli_Evolution_Experiments], the fundamental problem with the paper is that it used a flawed test statistic, not that it used Monte Carlo methods to find the p-value for that flawed statistic.

Every hypothesis test uses a test statistic to reduce the data to a single number. The p-value for the test statistic can be calculated analytically (as I’ve done for the chi-square test statistic) or by Monte Carlo methods. In the paper, Monte Carlo methods were used to compute the p-value of the “mutation generation” test statistic. The key problem with the analysis from the paper is that it doesn’t work to use a weighted average to test for variations in mutation rate. This is like trying to use the sample variance to test for an increase in the mean in Gaussian-distributed data. A statistic should be selected based on the null and alternate hypothesis distributions of the data. The chi-square test (unlike the weighted average from the paper) is a reasonable choice for data that mutates at a constant rate under the null hypothesis, but mutates at varying rates under the alternate hypothesis.

Able806, you made a good point about the contingency table cell frequencies being relatively low, but were wrong when you said ”the data set is actually too small to use the chi square method correctly”. In the low cell frequency case the chi-square test is still effective, but the null hypothesis distribution of the chi-square statistic starts to look less like the chi-square distribution. Thus, p-values calculated using the chi-square distribution may be a bit off. However, Monte Carlo p-values are always imperfect as well because it's impossible to generate an infinite number of random realizations. There are imperfections in p-values generated by analytic and Monte Carlo methods. However, low cell frequencies does not explain the >20x and >2.5x differences between chi-square p-values and p-values from the paper for experiments one and three. The reason for those huge differences was the use of the flawed test statistic (“mutation generation”) in the paper. [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)

:SJohnson, the chi-squared test is a valuable statistical tool, but the limitations of the test must be acknowledged. The chi-squared test can only produce valid results if the assumptions that underly the test are not violated. As an analogy, Newtonian models of motion fail to produce accurate results as velocities approach the speed of light; under those circumstances one must switch to a theory that accounts for relativistic effects.

:It seems that you have simply dismissed the [http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm widely-acknowledged] [http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm fact] that the [http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html chi-squared test] is [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236 inappropriate] for use in [http://www.graphpad.com/www/Book/Choose.htm situations] where n in any cell is [http://mysite.du.edu/~jcalvert/econ/chisquar.htm less] less than a [http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1 threshold] [http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html number]. Different authors set different thresholds, but all are well above the numbers seen in your chi-squared analysis - even the most liberal guidelines advise against the chi-squared test when any expected cell frequency is less than one or more than 20% of the table cells are less than 5; others require that expected values in all cells must be more than 5. With smaller amounts of data, the test is insensitive and errs on the side of rejecting the hypothesis. If you attempt your chi-squared statistical analysis with a program that is more sophisticated than MS Excel (as I did), you get an error message indicating that the results are invalid due to low expected cell counts.

:That issue aside, there are other reasons that the chi-squared test is inappropriate here. As the links above point out, the categories tested must be truly independent; one example is that you can't use the chi-squared test to compare age and ability to kick a field goal by testing the same experimental group twice, one year apart; you have to test one group of age A and a different group of age B. In the case of the Blount paper, the categories are not independent. Even if there were adequate numbers to address the low-expected-frequency problem, this would make the chi-squared an invalid test in this case.

:There are other significant problems with the use of the chi-squared test in this circumstance, but they can wait until you address these first major problems.--[[User:ElyM|ElyM]] 12:18, 11 March 2009 (EDT)

::Wackerly et al. says in general it’s assumed that the cell frequencies are above five so that the chi-square statistic (under the null) is approximately chi-square distributed (see p. 703). That book does not say chi-square test results are invalid if frequencies are five or less. Your example of a chi-square test warning message (it said "warning" not "error" as you stated) in Minitab [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236] said “approximation probably invalid” referring to the chi-square distribution approximation to the chi-square test statistic’s distribution. Your example did not say “chi-square test invalid”. I agree that when cell frequencies are low, the chi-square test statistic’s distribution starts to deviate from the chi-square distribution. I maintain that this deviation is not enough to explain the >2.5x and >20x differences in the chi-square test p-values and the p-values from the paper.

::As the numerous links in your post proved, the chi-square test is widely-used by statisticians. Can you give examples of statisticians using mean mutation generation as a test statistic? Also, did your software agree with the chi-square test p-values I presented? Thanks. [[User:SJohnson|SJohnson]] 10:10, 12 March 2009 (EDT)

:::Thank you for giving page references for Wackerly; however it seems we have different editions, since page 703 in my copy (5th ed, 1996) does not deal with chi-squared issues at all. My copy does state the following, on page 622: "Although the mathematical proof is beyond the scope of this text, it can be shown that, when n is large [chi-squared] will possess approximately a chi-square probability distribution in repeated sampling." Then, on page 624: "Experience has shown that cell counts [n sub i] should not be too small in order that the chi-square distribution provide an accurate approximation to the distribution of [chi squared]. As a rule of thumb we require that all expected cell counts equal or exceed 5, although Cochran (1952) has noted that this value can be as low as 1 for some situations." Wackerly then goes on, in the problems sections, to describe the use of the chi-squared test as a "violation of good statistical practice" when "some expected counts [are] <5."

:::It seems that you are already aware that the [chi-square] statistic under the null is no longer chi-square distributed for small n; this is precisely why the test should not be used under those conditions. I can claim to be able to accelerate a 1-kg mass to 10 times the speed of light by applying 1 N of force for 95 years by using F=ma and t= (vf-vi)/a. Plugging the numbers into those equations will produce the same result every time, but the answer is illegitimate because those equations are only valid under certain assumptions, which are violated as velocities approach the speed of light. Similarly, having a statistical program calculate a chi-squared value given the Blount data will produce a number result, but since the assumptions of the test are violated the result is not legitimate. Yes, if I put the Blount data in SAS 9.2, I get the same numerical answer as you do, but I also get the following message: "WARNING: >89% of the cells have expected counts less than 5. Chi-square may not be a valid test." You may argue that that's a warning, not an error; that's a semantic distinction. The reason that the program says that it MAY not be valid is that the chi-squared test skews in the direction of being too conservative at low n values; the test has an acceptable rate of false positives but an unacceptably high rate of false negatives. Comparing the results of the Monte Carlo and chi-squared results in this case is like comparing the results of Newtonian and relativistic equations of motion: they can produce very different results from the same input data.

::::For a finite amount of data, the chi-square statistic is never chi-square distributed under the null. The p-values are always approximate regardless of cell frequencies. The approximation becomes more accurate as the amount of data increases, but I don’t believe that this inaccuracy will change p-values that are about 0.2 (for experiments 1 and 3) into statistically significant p-values. How much do you expect the p-values to change if an exact computation is used in place of the chi-square distribution approximation? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::Your last paragraph has a major non sequitur in it: yes, many statisticians use the chi-square test. As long as the assumptions of the test are not violated, it is a valuable tool. That has nothing to do with the validity of using mean mutation generation as a test statistic. 'Mean number of werewolf attacks in Mumbai in the week centered on the new moon, by month, from 1654 to 1798' is a valid test statistic. I am quite sure that it has never been used in a peer-reviewed paper before. That does not mean that I can't perform valid statistical tests on that statistic. If, however, the incorrect test is applied, the results of the analysis will be flawed. Papers apply a (relatively small) standard repertoire of valid tests to a (potentially infinite) number of test statistics. The particular test statistic used in a paper may never have been used before and may never be used again; that does not address the validity of the analysis. In Blount's case, the test is the Monte Carlo analysis, which is also "widely-used by statisticians".

::::There are an infinite number of ways to reduce a data set to a single number. However, it’s foolish to think every method would be effective. I gave an example of a flawed test statistic in an earlier post [http://www.conservapedia.com/index.php?title=Talk%3ASignificance_of_E._Coli_Evolution_Experiments&diff=635070&oldid=634987]. Another example of a flawed test statistic is the one used in the paper because it does not always detect deviations from the null hypothesis (see: [[Significance of E. Coli Evolution Experiments#Test Statistics]]).

::::Test statistics are typically derived. The likelihood ratio test is a common method used to derive them. The chi-square test for independence is an approximation to the LRT. Where is the derivation saying that mean mutation generation is an appropriate test statistic for this problem? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::We still haven't touched on the issue of the categories not being independent, which by itself is sufficient to invalidate the chi-squared technique. I'm new to this site, so I'm unsure as to the etiquette of making changes to the articles of another person - but the article here should at the very least mention that the chi-square test is being used here in a manner that violates its underlying assumptions in at least two fundamental ways, and the results are therefore suspect.--[[User:ElyM|ElyM]] 17:34, 12 March 2009 (EDT)

:::::When generating random realizations of experiment outcomes, the authors assumed that the total number of mutants was fixed. Thus the paper assumed the numbers of mutants per generation are statistically dependent. Does this seem like a realistic model, or do you think that if the experiments were recreated that the total number of mutants could vary? For example, if experiment one were recreated, would the total number of mutants always be exactly four? [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

:::: It looks to me as though SJohnson has misinterpreted the application of the chi-squared test in quite a fundamental way. His/her analysis of Blount's data are therefore close to meaningless, regardless of whether the test used by Blount is appropriate or not. In my opinion, the entire page should therefore be deleted. [[User:FredFerguson|FredFerguson]] 08:18, 13 March 2009 (EDT)

::::: "Fred", perhaps you mistakenly think this is Wikipedia, where [[censorship]] and deletion of pages for ideological reasons are common. Not here.--[[User:Aschlafly|Andy Schlafly]] 10:23, 14 March 2009 (EDT)

:::::: Umm... I'm suggesting deletion for mathematical reasons, not ideological reasons. Using an argument filled with mathematical errors to try to support your case only detracts from your credibility. [[User:FredFerguson|FredFerguson]] 10:38, 14 March 2009 (EDT)

:::::: Actually, I think correction is better than deletion. So that's what I've done. [[User:FredFerguson|FredFerguson]] 11:01, 14 March 2009 (EDT)

::::::: I find no credibility in your denial of having ideological reasons.--[[User:Aschlafly|Andy Schlafly]] 11:04, 14 March 2009 (EDT)

== Misinterpretation of test ==

SJohnson, Your analysis misinterprets the test. You say the null hypothesis is that this mutation cannot happen. They saw a mutation (4 mutations, in fact, in the data set you show) so the null hypothesis (as you state is) is disproved. That's perfectly straightforward.

I don't know what the "mean mutation generation" test is but you're doing when you apply a chi-squared test to this dataset is to test if the mutations are evenly distributed throughout the generations. Your test says they are, so there's no strong evidence to suppose that mutations are likely to occur in one generation rather than another in the series of tests. Blount's test says thay aren't, so it's more likely that the mutation will occur later in the series of tests. I can't tell which test is right without knowing more about the test that Blount used.

But that point (the foregoing paragraph) has no bearing at all on the null hypothesis, as you describe it. The mutation appeared, so that means the hypothesis that the mutation can't happen is disproved. Very simple. [[User:FredFerguson|FredFerguson]] 21:10, 8 March 2009 (EDT)

:I never said that “the null hypothesis is that this mutation cannot happen”. The chi-square test statistic I'm using wouldn’t be defined if the null hypothesis mutation rate was zero because the <math>E\left[n_{i,j}\right]</math> term in the denominator of the statistic (see above equation) would be zero.

:The test statistic from the paper is the average of the generation numbers of observed mutations. For experiment one this number is
::<math>
\frac{1}{4}\left(30500+31500+2\times32500\right)= 31750.
</math>
:The same number is shown in Table 2 of the paper. [[User:SJohnson|SJohnson]] 10:46, 9 March 2009 (EDT)

:: SJohnson, the way you're calculating the chi-squared statistic implies that you're testing the null hypothesis of a constant mutation rate over time against an alternative hypothesis of a mutation rate which varies over time. [[User:FredFerguson|FredFerguson]] 11:02, 9 March 2009 (EDT)

As it currently stands, the article makes the following statement: "The expected outcomes under the null hypothesis (no evolutionary innovation occurs) are also shown." This misstates the null hypothesis of the paper, which is elaborated in the Introduction section of the paper, and repeated in the section '''Statistical Analysis of the Replay Experiments''':
<blockquote>
For each experiment, we compared the observed mean generation of those clones that yielded Cit+ variants to the mean expected under the null hypothesis that clones from all generations have equal likelihood. The null thus corresponds to the rare-mutation hypothesis laid out in the Introduction."
Block quote
</blockquote><ref>www.pnas.org/cgi/reprint/105/23/7899.pdf</ref>

The article also continues to describe 'mean mutation generation' as a ''test'' rather than a ''statistic'' to which the ''Monte Carlo test'' was applied.--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

==Unreferenced Claims==

I deleted the claim that mean mutation generation is an appropriate test statistic because no reference was produced that back that claim. No reference was provided to back the claim that the chi-square test p-values are always conservative, either. [[User:SJohnson|SJohnson]] 12:57, 14 March 2009 (EDT)

: The reference is Everitt. I'll check I put it in the right place. [[User:FredFerguson|FredFerguson]] 13:30, 14 March 2009 (EDT)

There was a typo in my edit summaries on the talk page and the main page. I meant to say "Removed unsupported claims" rather than "Removed supported claims". [[User:SJohnson|SJohnson]] 13:13, 14 March 2009 (EDT)

I do not believe that anyone has claimed that 'chi-square test p-values are ''always'' conservative'. The claim that has been made is that ''under certain circumstances'', namely low n and low individual cell values, the chi-square test is an invalid test; that under those circumstances the power of the test is low and it becomes impossible to reject the null hypothesis even when it is false. You may have missed the pertinent sections in my links above, so I will directly quote the relevant sections. All the quoted sections refer to chi-square testing in particular. Any bolding below is mine.

:This edit claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639379]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

<blockquote>
"Assumptions: 

Even though a nonparametric statistic does not require a normally distributed population, there still are some restrictions regarding its use. 

1. Representative sample (Random) 
2. The data must be in frequency form (nominal data) or greater. 
3. The individual observations must be independent of each other. 
4. '''Sample size must be adequate. In a 2 x 2 table, Chi Square should not be used if n is less than 20. In a larger table, no expected value should be less than 1, and not more than 20% of the variables can have expected values of less than 5'''. 
5. Distribution basis must be decided on before the data is collected. 
6. The sum of the observed frequencies must equal the sum of the expected frequencies."
</blockquote>
<ref>http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm
</ref>

<blockquote>
"Assumptions:
* Random sample data are assumed. As with all significance tests, if you have population data, then any table differences are real and therefore significant. If you have non-random sample data, significance cannot be established, though significance tests are nonetheless sometimes utilized as crude "rules of thumb" anyway.
* A sufficiently large sample size is assumed, as in all significance tests. '''Applying chi-square to small samples exposes the researcher to an unacceptable rate of Type II errors. There is no accepted cutoff. Some set the minimum sample size at 50, while others would allow as few as 20'''. Note chi-square must be calculated on actual count data, not substituting percentages, which would have the effect of pretending the sample size is 100.
* '''Adequate cell sizes are also assumed. Some require 5 or more, some require more than 5, and others require 10 or more. A common rule is 5 or more in all cells of a 2-by-2 table, and 5 or more in 80% of cells in larger tables, but no cells with zero count'''. When this assumption is not met, Yates' correction is applied.
* Independence. Observations must be independent. The same observation can only appear in one cell. '''This means chi-square cannot be used to test correlated data (ex., before-after, matched pairs, panel data)'''.
* Similar distribution. Observations must have the same underlying distribution.
* Known distribution. The hypothesized distribution is specified in advance, so that the number of observations that are expected to appear each cell in the table can be calculated without reference to the observed values. Normally this expected value is the crossproduct of the row and column marginals divided by the sample size.
* Non-directional hypotheses are assumed. Chi-square tests the hypothesis that two variables are related only by chance. If a significant relationship is found, this is not equivalent to establishing the researcher's hypothesis that A causes B, or that B causes A.
* Finite values. Observations must be grouped in categories.
* Normal distribution of deviations (observed minus expected values) is assumed. Note chi-square is a nonparametric test in the sense that is does not assume the parameter of normal distribution for the data -- only for the deviations.
* Data level. No assumption is made about level of data. Nominal, ordinal, or interval data may be used with chi-square tests."
</blockquote>
<ref>http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm</ref>

<blockquote>
"Assumptions: 
-None of the expected values may be less than 1 
-No more than 20% of the expected values may be less than 5"</blockquote> <ref>http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html</ref>

<blockquote>
"When performing a chi-square test, your data must satisfy important assumptions. Although these assumptions may be stated differently in different textbooks, they generally assert that: 
1)The sample must be randomly drawn from the population 
'''2)The sample size, n, must be large enough so that the expected cell count in each cell is greater than or equal to 5.''' 
Both assumptions must be met in the process of collecting your data, and violations of the second assumption will appear in the Minitab output when you run the analysis. 
... 
'''You may wonder why the second assumption is necessary for performing the chi-square test. The second assumption arises because the distribution of counts under the null hypothesis is multinomial, and the normal distribution can be used to approximate the multinomial distribution if the sample size is sufficiently large and the probability parameters aren't too small. It can be shown via the Central Limit Theorem that the multinomial distribution converges to the normal distribution as the sample size approaches infinity; however, there is no easy way to show mathematically how and when the convergence fails.'''"
</blockquote>
<ref>http://www.minitab.com/support/docs/Answers/Chi-Square%20Test%20Assumptions.pdf</ref>

<blockquote>
"'''The chi-square test is simpler to calculate but yields only an approximate P value. ... You should definitely avoid the chi-square test when the numbers in the contingency table are very small (any number less than about six)'''."
</blockquote> <ref>http://www.graphpad.com/www/Book/Choose.htm</ref>

<blockquote>
"The most important things to remember to get a valid χ2 test are that the expected values are not too small in any bin (certainly 5 or more), and that the degrees of freedom are properly evaluated. '''Unless you have a very large amount of data, the test is not very sensitive and errs on the side of safety. If you get a significant result, however, it is not likely to be wrong.'''"
</blockquote> <ref>http://mysite.du.edu/~jcalvert/econ/chisquar.htm</ref>

<blockquote>
"The critical assumptions of the chi-square test for k independent samples are similar to those for the chi-square test for two independent samples. ... 
'''4. No more than 20% of the cells may have expected frequencies of less than 5, and no cell should have an expected frequency of less than 1. 
The rule given in Assumption 4 is particularly important for a contingency table that is larger than 2X2'''"
</blockquote> <ref>http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1
</ref>

<blockquote>
"Special problems with small expected cell frequencies for the chi-square test: 
The chi-square test involves using the chi-square distribution to approximate the underlying exact distribution. The approximation becomes better as the expected cell frequencies grow larger, and '''may be inappropriate for tables with very small expected cell frequencies.''' 
'''For tables with expected cell frequencies less than 5, the chi-square approximation may not be reliable. A standard (and conservative) rule of thumb (due to Cochran) is to avoid using the chi-square test for tables with expected cell frequencies less than 1, or when more than 20% of the table cells have expected cell frequencies less than 5.''' 
Another rule of thumb (due to Roscoe and Byars) is that the average expected cell frequency should be at least 1 when the expected cell frequencies are close to equal, and 2 when they are not. (If the chosen significance level is 0.01 instead of 0.05, then double these numbers.) 
Koehler and Larntz suggest that if the total number of observations is at least 10, the number categories is at least 3, and the square of the total number of observations is at least 10 times the number of categories, then the chi-square approximation should be reasonable. 
Care should be taken when cell categories are combined (collapsed together) to fix problems of small expected cell frequencies. Collapsing can destroy evidence of non-independence, so a failure to reject the null hypothesis for the collapsed table does not rule out the possibility of non-independence in the original table. 
'''As with most statistical tests, the power of the chi-square test increases with a larger number of observations. If there are too few observations, it may be impossible to reject the null hypothesis even if it is false.'''"
</blockquote> <ref>http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html</ref>--[[User:ElyM|ElyM]] 17:24, 14 March 2009 (EDT)

: Thanks for this really excellent contribution, ElyM. The only thing I'd like to add is in relation to your initial statement, "I do not believe that anyone has claimed that 'chi-square test p-values are always conservative'". The question of whether a test is conservative in a particular situation is probabilistic. One can determine whether a test is likely to generate a p-value which is too high in a particular situation (e.g. for a chi-squared test, when there are lots of small expected values) but one needs an exact test (such as an appropriate Monte Carlo randomisation test) to determine whether the p-value in any ''particular'' test is in fact excessively high.

::This edit also claimed that chi-square test p-values are conservative, but didn't back that claim with a reference: [http://www.conservapedia.com/index.php?title=Significance_of_E._Coli_Evolution_Experiments&diff=next&oldid=639373]. [[User:SJohnson|SJohnson]] 20:49, 18 March 2009 (EDT)

: I hope careful reading of your very clear description will put SJohnson's mind at rest on this subject. [[User:FredFerguson|FredFerguson]] 18:11, 14 March 2009 (EDT)

ElyM, you've provided nothing to address the basic flaw that "The paper incorrectly applied a Monte Carlo resampling test to exclude the null hypothesis for rarely occurring events." See [[Flaws in Lenski Study]]. Also, do not impose your view on the content page until after SJohnson has had an opportunity to respond to your posting. As to "Fred", his put-downs are getting tiresome and I'm going to review his edit pattern now to see if he's been contributing anything of value to this site.--[[User:Aschlafly|Andy Schlafly]] 14:06, 15 March 2009 (EDT)

::Mr. Schlafly, per your request I have not added anything to the content page as SJohnson has not yet responded to my posts. Since all of my comments have been in regards to SJohnson's use of the chi-square test in this particular article, I'm not sure why you expect me to address Blout's use of Monte Carlo - that issue seems to be addressed on the [[Flaws in Lenski Study]] page. SJohnson has added a reformulation of the chi-square test for two possible outcomes, and stated that the chi-square test is at a minimum when all success probabilities are equal. He then extrapolates from this to claim that the chi-square test is an effective test for the data from Blount.

::The reformulation of the equations for two possible outcomes does not address the underlying problem that the chi-square test has universally accepted parameters outside of which it is considered an invalid test; I have provided references for these parameters and shown that the data from Blount lies outside them. None of the expected cells in SJohnson's analysis have values above one, and the total n is four. SJohnson's own reference states that the application of the chi-square test in this circumstance is a "violation of good statistical practice". Analogously, combining F=ma and t=(vf-vi)/a into t=(vf-vi)m/F and showing that t is a minimum when m approaches zero does not address the fact that those Newtonian equations do not apply as velocities approach the speed of light. The legitimacy of Blount's arguments cannot be determined by the application of illegitimate counterarguments. If SJohnson or others can point to references from the statistical literature that show that Blount has made methodological errors - as I have been able to do with SJohnson's chi-square analysis - I would welcome their input, and no doubt Conservapedia's other readers would as well, and this page would be greatly improved.

::I have not seen a rebuttal from SJohnson in the four days since my last post, although he has added new material to the content page since then. In light of this, I would appreciate some guidelines as to when it is appropriate for me to add my information and references to the content page. I can add citations from the primary mathematical literature if necessary, but in general I find that these are less helpful as they are not easily accessible by readers without access to academic libraries.--[[User:ElyM|ElyM]] 18:07, 18 March 2009 (EDT)

== References ==
{{reflist}}

World History Lecture Eight

2009-03-19T01:40:41Z

Argon: Living in plush quarters is not like relocation to derelict horse stables and tar paper shacks (ask a Japanese-American sometime)

[[World History Lecture One|1]]-[[World History Lecture Two|2]]-[[World History Lecture Three|3]]-[[World History Lecture Four|4]]-[[World History Lecture Five|5]]-[[World History Lecture Six|6]]-[[World History Lecture Seven|7]]-[[World History Lecture Eight|8]]-[[World History Lecture Nine|9]]-[[World History Lecture Ten|10]]-[[World History Lecture Eleven|11]]-[[World History Lecture Twelve|12]]-[[World History Lecture Thirteen|13]]-[[World History Lecture Fourteen|14]]

Our midterm exam will be next week. It will be multiple-choice, perhaps 30 minutes in length, and will cover lectures one through eight (that includes this one). Items and topics mentioned on the homework will be given greater priority in designing the questions.

This exam will be less detailed than prior exams. Try to understand the "big picture" of world history. Do not expect to be asked about specific dates, specific Roman emperors, or other detailed matters. Focus more on trends and broader concepts.

Remember and use the following test-taking tips for multiple-choice exams. You can improve your scores enormously by following and mastering these tips.

== Test-taking tips ==

'''Understand''': Understand the question before you try to answer it. Read the question twice if you have doubts. You want an answer that fits the question best, like a key fitting a lock.

'''Eliminate''': Eliminate the wrong answers before picking the right one. This can bump your score up another point or two.

'''First Impression''': Go with your first impression unless you have a good reason to change it. More often than not, your first impression will be right.

'''Stay on Track''': Stay on track during the test. Expect to miss a few, and do not waste too much time on a difficult question.

'''Try Different Approaches''': Try different approaches to a difficult question until you figure out the right answer. There are many ways to find the correct answer for a question. Don't give up too easily on figuring out a question.

'''Avoid Broad Language in Answers''': Broad or sweeping terms like "only", "never", "solely", "everyone", "always", "purely", "all" and "every" are often signs of '''incorrect''' answers because they are rarely true.

'''Try to Answer Every Question''': You can't increase your score by refusing to answer, just as you can't score a basket in basketball without tossing the ball towards the net. Trying is better than not trying. Try hard on every question, and if no points are deducted for wrong answers, be sure to answer every question. Even if you don't know the answer, try to make an educated guess.

Test-taking on multiple-choice exams is a skill that can be mastered with practice, determination and cleverness.

== Introduction ==

This lecture will be briefer than usual, with no homework questions, in order to give you time to prepare for the exam.

On September 11, 1683, Muslims made a second major attempt to conquer Europe (recall that Charles the Hammer turned back their first attempt). A massive Islamic army arrived at the Gates of Vienna. An alliance of Christian armies, led by the Polish King Jan III Sobieski, defeated the Muslim force.<ref>http://gatesofvienna.blogspot.com/2006/09/other-september-11th.html</ref> Some have observed that the "9/11" massacre in New York City was on the anniversary of this day.

Two other momentous events happened in the 1600s which would also affect the world forever. In 1607, Englishmen established a permanent settlement at Jamestown, Virginia. In 1611, the King James Bible was published with this introduction:<ref name="KJV">http://www.av1611.org/kjv/kjvhist.html</ref>

:THE HOLY BIBLE, Conteyning the Old Testament, and the New: Newly Translated out of the Originall tongues: & with the former Translations diligently compared and revised, by his Majesties Special Commandment. Appointed to be read in Churches. Imprinted at London by Robert Barker, Printer to the Kings most Excellent Majestie. ANNO DOM. 1611.

Some might say that the publication of the King James Bible was more significant than the settlement at Jamestown. This version of the Bible, which became known as the "Authorized Version," was a spectacular work that transformed and elevated the entire English language to new heights. Even a Catholic scholar expressed the highest praise for this Protestant work, as all Christians recognized it to be an absolute masterpiece:<ref name="KJV"/>

:The highest eulogiums have been made on the translation of James the First, both by our own writers and by foreigners. And, indeed, if accuracy, fidelity, and the strictest attention to the letter of the text, be supposed to constitute the qualities of an excellent version, this of all versions, must, in general, be accounted the most excellent. Every sentence, every work, every syllable, every letter and point, seem to have been weighed with the nicest exactitude; and expressed, either in the text, or margin, with the greatest precision.

To this day many feel that the King James Bible is the finest English translation available, despite the antiquity of some of its terminology. It is free from modern translation biases that hide the existence and wrath of Hell, for example. But for the purposes of this course, we are interested in how this was a defining moment in the development of the English language, and how the language has grown since then.

The King James Bible established a framework of knowledge of the English language, which began to develop rapidly afterward. English is unique in its ability to develop new words, at a rate of about 1000 new words per year.<ref>See [[Essay:Best New Conservative Words]].</ref> This enables English to constantly improve its ability to express concepts. English easily incorporates and adapts words from other languages, such as "detente" (French for a lessening in tensions between nations) and "tsunami" (Japanese for a massive tidal wave).

English has the smallest alphabet of widely used languages;<ref>Some describe the Hebrew alphabet, without vowels, as having only 22 characters, but modern Hebrew (spoken by a few million people) has more characters than English. See http://www.omniglot.com/writing/hebrew.htm . The obscure [[Elder Futhark]] runic writing has an alphabet of 24.</ref> this compact simplicity gives English an advantage, particularly in using computers. For example, modern English lacks accented letters (letters bearing a diacritic mark such as the "e" in "noël"), hybrid letters (such as the archaic digraph ligature æ used in British English in words such as encyclopædia or the ampersand character - a contraction of "et"), and pictograph characters.

It helps that English uses a writing system that has some relation between sounds and symbols as opposed to a purely ideographic system. This also applies to all other languages using alphabets, abjads, syllabaries, and so on - such as ancient Greek and Latin (though English developed from German, not from Greek or Latin).

English features easy interchangeability of nouns, verbs, and adjectives, without much variance in form for pronouns and verbs. That promotes easy communication through brief, cryptic messages if necessary, the style preferred by electronic media. English enjoys a powerful pipe-like quality, such that one phrase can be cut and pasted to another phrase with ease. Computer-based cutting and pasting text works more efficiently in English than in many other languages, such as those using pictorial characters. This gives English an advantage in the internet medium, and about 80% of the internet uses English. English is now the second most widely spoken language in the world, with only Chinese dialects spoken by more people. English is overwhelmingly the second language of choice for non-English-speaking people.<ref>See Barbara Wallraff, “What Global Language?” 286 Atlantic Monthly No. 5, at 52 (Nov. 2000) (noting, ''inter alia'', that “English is the working language of the Asian trade group ASEAN” and is also “the official language of the European Central Bank”).</ref>

==The Age of Exploration==

The “Age of Exploration” consisted of Christians from Europe searching the rest of our globe to spread the faith and seek wealth, particularly between 1450 and 1650. This began the “Global Age” in world history, which continues to this day.

As Europe was emerging from the Middle Ages and experiencing the Renaissance, people had the curiosity and the courage to look beyond their own continent to others. Scientific discoveries and technological improvements made it easier to travel across the vast oceans. Christians sought to spread their faith far and wide, and merchants sought new goods and business opportunities in both the East and West. North and South America awaited visits by the Europeans.
[[Image:Caravel.jpg|right|thumb|A caravel]]
The Portuguese were the first great explorers. They developed the “caravel”, a small, lightweight ship with three lateen-sailed masts that were faster and more maneuverable in shallow water than previous designs (see right). They could also hold a fair amount of spices, and the more cargo that a ship could hold, the greater its profitability would be. European demand for exotic Indian goods, especially spices, was strong, and this drove explorers to discover new routes to India different from the Eastern Mediterranean trade controlled by the Ottoman Muslims.

The discovery of new western routes accidentally led to the discovery of lands in the New World (Western Hemisphere). Portugal and Spain then emerged as the two major players in what was to become an intense European competition for exploration, conversion and colonization.

===Portuguese Exploration===

The Portuguese initiated the age of exploration in Europe as early as about 1419, when Prince Henry the Navigator established his School of Navigation and made it his life’s work to explore the West African coast and reach the Indian Ocean. His ships reached as far as Sierra Leone, but he failed to reach India. Another explorer named Bartholomew Diaz likewise failed to reach India, but he did manage to sail around the tip of Africa (the Cape of Good Hope) in 1487 and thereby establish the route to India.

In 1492, with funding from Spain (see below), Christopher Columbus embarked on his famous voyage with the dream of reaching India, and he failed in that goal also. It was Vasco da Gama who finally reached India by sailing eastward around Africa and up the East African coast between 1497 and 1499, with the help of the powerful monsoon winds. Shortly thereafter, regular trade was established with India under the Portuguese King Manuel, and conflict began with the Muslims over dominance of Indian Ocean trade.

Later, with funding and men from Spain (see below), the Portuguese navigator Ferdinand Magellan became the first to lead an expedition around the globe.

 
===Spanish Exploration===

The Spanish entered the scene in the year 1492 — during the reign of King Ferdinand and Queen Isabella, who were working hard to eradicate Islam and establish Christianity as the only religion in Spain. Known as the “Reconquista” (or Reconquest) of Spain, it ultimately succeeded in the reconquest of Granada, the last Muslim city in Spain. As explained in the prior Lecture, Queen Isabella ordered all Muslims and Jewish residents to convert or leave Spain. Initially, this resulted in an economic and academic setback for Spain, as Jewish residents had comprised a large number of scholars and merchants.

It was during this time that a young Italian by the name of Christopher Columbus first approached the Portuguese King John II to ask him for financial support. Columbus, like others from the Italian city of Genoa, was an expert mariner, having started as early as the age of 14. He had studied the works of Ptolemy on his own; Columbus did not attend formal school, just as homeschoolers do not today. Columbus at all times was a devout Christian, writing frequently about his faith in his diary. His desire to sail to India was motivated in large part by his attempt to spread the Christian faith. He repeatedly cited a desire to convert pagans to Christianity when he sought support for his voyage. Late in life, even when treated as a prisoner, he sought to devote all his money and efforts towards winning back Jesus Christ’s Holy Sepulchre (burial cave) from Muslims in the Holy Land.

King John II of Portugal denied Columbus’s request to fund his voyage across the Atlantic. Indeed, Columbus was ridiculed for seeking to reach India by sailing west rather than east.

The funding for Columbus came from Spain instead. Though Columbus was Italian, it was Spain that financed his voyages as the strain of fighting the Muslims subsided. Isabella and Ferdinand felt secure in granting the Genoese upstart Christopher Columbus his long-awaited funds. Beginning on August 3, 1492, Columbus embarked on four voyages, becoming perhaps the first European to discover the New World, although at first he believed he had reached India (hence his name for the inhabitants of the New World: “Indians”). The discoveries by Columbus included the Bahamas, Cuba, Haiti, San Salvador, Puerto Rico, Jamaica, Trinidad and Honduras. Columbus, the master mariner that he was, subsequently sailed back to Europe and then again to the Caribbean, and was able to find the exact same locations on his return voyage. Columbus’ astounding success presented an increasing challenge to Spain’s Portuguese rivals.

The pope finally resolved this rivalry between Spain and Portugal in the Treaty of Tordesillas in 1494, which established an 1100-mile long “Line of Demarcation,” which was 50 degrees longitude (north-to-south) in the Western Hemisphere. This division of world territories gave Portugal trading rights in India, China, the East Indies, East Brazil and the Spanish Americas. Spain was given control of the remaining, vast majority of the Americas. Ferdinand Magellan then led the first successful voyage to circumnavigate the globe in 1519. Although Magellan was killed by Philippine natives before completing the last leg of the journey, he was the first to sail around the tip of South America to enter the Pacific Ocean. Only 18 of Magellan’s approximately 250 sailors made it back to Spain three years later, in 1522.

As a result of Magellan’s journey, Spain gained control of many South American territories, including Peru. Spain also obtained control over the Philippines in the Far East. Spanish warrior Hernando Cortes led the conquest of Mexico, with the initial help of the Aztecs, many of whom believed Cortes was their god Quetzalcoatl (who, conveniently for the Spanish, was expected to appear that very year). Cortes captured the Aztec ruler Moctezuma (popularly known as “Montezuma”) and conquered their empire. Cortes was also aided by an Aztec woman named Malinche, who served as a guide and translator. Cortes burned the Aztec capital Tenochtitlan and built Mexico City in its place.

Francisco Pizzaro led the conquest of Peru, eventually killing their emperor, Atahualpa. The ancient Peruvian city of Cuzco was completely destroyed and replaced with the city of Lima. While in Peru, Pizzaro also discovered the greatest silver mine in the Americas. The vast majority of silver obtained was traded with China, whose economy was based on silver. Both Cortes’ and Pizzaro’s conquests were facilitated by the use of horses and superior weapons, and the South Americans were also devastated by European diseases like smallpox. The native population had no immunity to such disease, and some historians claim that the Spanish intentionally spread the deadly disease by giving the natives blankets that carried the germ. There is one defect to that anti-Christian theory: the germ theory was not discovered until the 19th century! No one knew that diseases were spread by germs in the 1500s, so the story is pure fiction.

Using the Spanish system of “encomiendas”, many of the South American natives became slaves to their Spanish conquerors. A Dominican monk and missionary, [[Bartholomew de Las Casas]], sent sensational descriptions to the Spanish government of the supposed atrocities committed against the “patient, meek and peaceful” natives by the Spanish conquistadors, whom he described as “cruel tigers, wolves and lions.” Although some of what Las Casas said was true, historical research has proved most of it to be false and exaggerated. However dishonest his methods may have been, Las Casas’ reports eventually led to the king of Spain issuing “New Laws,” by which exploitation of the natives was ended by the replacement of the old “encomiendas” with the new “repartimientos”. Both systems are now criticized as having been unfair. The system of “encomiendas” was based on royal grants of land, while the system of “repartimientos” was based on gifts of Indian labor. Either way, the Indians ended up working for the Spanish, but it was typical throughout history for conquerors to put the conquered people to work, as in "to the victors go the spoils."

===Exploration by Other Nations, and the Seven Years War===

Other European nations participated in exploration as well. King Henry VII of England sent John Cabot, a resident of London born in Genoa (like Columbus), to explore across the Northern Atlantic Ocean. Cabot discovered Newfoundland and the New England coast in 1497 and 1498, respectively, and the land became the property of England. From France, Jacques Cartier discovered Canada and the Mississippi Basin, including Louisiana in 1534 and 1541. These colonies were initially less significant than those in South America, and there was a lack of initial interest in North America because it did not have the gold and silver of South America. Whereas intermarriage and cultural blending occurred heavily among the Spanish and the Indians in South America, it barely existed between the English or French and the Indians in North America.

Of course, eventually the interest and development in North America increased and surpassed that of South America. Commercial competition between European nations culminated in the Seven Years’ War (1756-1763) — a conflict brought to America in the form of the French and Indian War. This War marked the end of French (and Spanish) influence in North America, and signaled the rise of Britain as the primary world superpower for the next 150 years.

The Seven Years War in Europe, of which the French and Indian War was a small part in North America, consisted of 20 major battles and ended up taking the lives of perhaps a million people. Britain and France fought in both Europe and North America; Spain and Portugal fought in South America. The war reached every inhabited continent except Australia, which had been discovered in only the 1600s and was not yet well-settled by Europeans. France, Russia, Sweden, Poland and Austria were allied against Britain and Prussia in the struggle for control of Silesia, a region of Eastern Europe that is mostly part of Poland today.

Britain won the war, partly due to the genius of William Pitt, after whom Pittsburgh is named. Britain gained both Canada and India as a result of the Seven Years War, but lost the American colonies two decades later in the Revolutionary War. Pitt himself had left power before the war concluded (though he returned to power later); he opposed the peace treaty on the grounds that it did not give enough to Britain. France was humiliated by this war, which may have contributed to the French Revolution a few decades later.

===The Slave Trade===

In what is known as the “Columbian Exchange,” plants and animals were brought from America to Europe and vice-versa. Columbus brought from Europe to America wheat, melons, onions, grapes, sugar cane and horses. Horses proved to be influential not only in the European conquest of native peoples but also in the natives’ abilities to avoid white settlers, particularly among the Plains Indians in North America.

America had new fruits and vegetables that were then exported to Europe and Africa. America introduced the white potato to Europe and the manioc or cassava to Africa. Chili peppers, pumpkins, squash and peanuts were also brought back from America and became popular in countries such as India. These new foods helped increase population, and in Africa the population growth almost canceled out population losses caused by the slave trade.

Slavery, though not considered part of the Columbian Exchange, became a hurtful form of trade. In 1455, a “papal bull” (formal letter by the pope) justified a “right” of Christian nations to enslave any non-Christians in the name of exploration. The Spanish had already been enslaving South American natives on a limited basis, but with the rise of sugar plantations the need for a larger slave force arose. Millions of African slaves were brought by the Spanish and Portuguese to Mexico, Peru, the Caribbean and especially Brazil. The growth of sugar — which had first been introduced to Europe when the Muslims ruled Spain — was exploding in popularity throughout the entire western world. Soon France, the Netherlands and Great Britain were also establishing profitable sugar plantations in the new world. The plantation system began in Brazil, where rich white plantation owners were the highest rung in the social hierarchy and the enslaved Africans were at the bottom. Obviously life on a sugar plantation was very hard work for a slave.

[[Slavery]] was widespread within Africa itself, and the richest in Africa were not those owning the most land, but those who owned the most slaves. In the Sahara Desert, slaves worked in caravans and were used in gold and salt mining. Slaves were usually prisoners of war from other areas of Africa, or debtors, or enemies of the king, but many women outside of those three categories were also enslaved in African societies. Polygamy — the practice of having more than one wife — was common in Africa, as was the existence of harems, from which African women were often sold to join Arabian or Middle Eastern harems.

The trading of slaves with other countries was encouraged in Africa, and was considered an important component of the African economy. Slave trade across the Atlantic (the Trans-Atlantic slave trade) became a booming business for Europeans and Africans alike, by which African rulers sold their people to Europeans for goods such as iron, alcohol, tobacco and guns. Trans-Atlantic trade led to the degrading use of “chattel” slaves, whereby the slaves were treated purely as property of the owner. The slaves served as sailors, skilled craftsmen or farmers. The journey across the Atlantic, known as the Middle Passage, led to the death of 10-20% of the African slaves. But an even higher percentage lost their lives in the journey from their homes in Africa to the African coast, where they were to board the slave ships.

The Trans-Atlantic slave trade was one component in a system of routes known as the “Triangular Trade” between the Caribbean islands or South America, and New England, and the West Coast of Africa. The three main items that were exchanged were sugar, rum and slaves. European goods, mainly guns, were used to buy slaves from Africa. The slaves were then shipped to the Americas. Then, from America, sugar, rum and tobacco were brought back to Europe, completing the “triangle” of trade. No specific “triangle” of trade has ever been identified and the term was not even used at the time to describe the trade routes; this concept may simply be the result of the imagination of modern historians.

The end of slavery in Europe did not come until the beginning of the 1800s, when Christian reformers such as William Wilberforce and John Wesley began speaking out against the evils of the system. Although the United States banned the slave trade in the early 1800s, slavery in the United States was not abolished until decades later in 1865 when the 13th Amendment was passed, and in South America it was finally ended in Brazil in 1888.

===China’s Role in the Age of Exploration===

In the East, under the Ming dynasty, Emperor Yongle sent a band of explorers led by “Zheng He” throughout parts of southern Asia, Persia, Arabia and Africa. The crew of over 28,000 men embarked on seven naval expeditions in the Indian Ocean from 1405 to 1423 and, as a result, trade with East Africa began. However, by the 1430s, Mongol raids caused the Chinese to withdraw from exploration for the time being, and instead spend their time and money on defenses such as repairing the Great Wall.

The Ming dynasty thereby retreated from the world stage, mostly in order to strengthen Chinese defenses against Mongol threats. Beijing became the capital of China, and eventually the Ming rulers began to retreat even more completely from the outside world, living in an elaborate Beijing palace known as the “Forbidden City.” When a devastating famine occurred in China, the rulers were living extravagantly in their secluded palace and cared little for the plight of the peasants. The people of China revolted, and in 1644, the Ming Dynasty was replaced with the Qing, made up of the Manchu peoples.

Under the Qing dynasty, marriage between members of the Chinese and Manchurians was forbidden, and the official spoken language remained Chinese. Female infanticide occurred, as well as the practice of binding the feet of rich girls and, increasingly, of middle class girls also. The primary philosophy was Neo-Confucianism, a mixture of Buddhism and traditional Confucianism. Jesuits attempted to evangelize China, but were not very successful. At this time all Catholic liturgical services were required to be in Latin, a language that did not fit well with the very different Chinese culture. Nevertheless, a Jesuit presence in China contributed to the exchange of ideas and technology between the East and the West.

China did not withdraw completely from world trade during the Ming and Qing dynasties. China continued to trade with the Spanish, Portuguese, Japanese and Dutch, mainly for silver, which had become the Chinese currency, and with the Russians for furs. Under the Ming dynasty, beautiful blue and white porcelain was produced and exported. The influence of the Ming vase spread throughout the Muslim and Christian worlds, as seen in Holland through the production of Delft pottery; in England, Wedgwood China became the finest producer of china. Other Chinese exports included silk and tea, and imports consisted of spices, textiles and animal skins from Europe. Maize, sweet potatoes and peanuts were brought over from the Americas and became popular crops in China.

==Growth of Nation-States and Monarchs==

The development of European nation-states continued throughout the Renaissance and Age of Exploration. In the 1400s and 1500s, the most powerful nation-states were England, France and Spain. The Holy Roman Empire, which consisted of parts of Italy and Germany, was a powerful but declining force. Enemies that weakened the Holy Roman Empire included the Ottoman Muslims and the French, who resisted an expansion by the Holy Roman Empire. The Reformation contributed to the downfall of the Holy Roman Empire, as did the fact that the cultures of Germany and Italy were too different to remain easily within the same empire.

The governments of the European nation-states grew in power by expanding their central authority. They increased bureaucracy, the size of their armies and the amounts of their taxation. The rise of absolutist rulers contributed to the centralization of France, Spain, Austria, Russia and Prussia. The nature of an absolute monarch was expressed well in this famous quote attributed to Louis XIV (1527-1598), “L’etat c’est moi!” (“I am the state”). Absolute monarchs controlled all aspects of government and society including the church, the bureaucracy and the military. Absolute monarchs claimed “divine right” to rule, stating that they had been appointed by God Himself and were therefore responsible to God alone. They enjoyed “sovereign immunity” under the view that “the king can do no wrong,” a principle that continues to exist in English and American law today (unless government “waives” its immunity with a law granting limited rights to sue it).

Religious conflict in England between Catholics and Anglicans caused absolutism to fail there. The “Glorious Revolution” (so named by supporters of the Church of England) brought down the Catholic King James II and the idea of divine right along with him, placing William and Mary on the throne in 1688. In France, however, absolutism triumphed until the French Revolution, especially with the reign of Louis XIV, the leading absolute monarch in Europe. The French court at Versailles during the reign of Louis XIV, who called himself the “Sun King,” was considered rich in culture and an example to other societies. Truth be told, it was also extravagant and immoral.

In Spain, Philip II (1556-1598), who was the great-grandson of Isabella and Ferdinand, worked to keep Spain Catholic, and viewed Elizabeth I of England as his main rival. But his mighty Spanish Armada was defeated by a quicker English fleet in 1588. In 1579, William of Orange led the Netherlands in a revolt against their Spanish masters. These two losses contributed to the weakening of Spanish power.

Meanwhile, artistic pursuits flourished during this period in Spain. El Greco (1541-1614), a painter, architect and sculptor best known for his paintings of saints and martyrs. as well as a stunning drawn image of a storm over a Spanish town (“A View of Toledo”). Diego Velázquez painted portraits of the royal family and scenes from daily life at court. In 1605 Miguel de Cervantes wrote the novel Don Quixote, probably the most influential piece of Spanish literature, and Lope de Vega wrote an estimated 2,000 plays.

In Russia, absolute emperors known as “czars” arose. The first czar of Russia was Ivan IV “the Terrible,” who executed many innocent people during his fearful reign from 1547 to 1584. Ivan was succeeded by his son Feodor, whose repeated failures led Russia into a “Time of Troubles.” In 1613, the first of a long line of Romanov czars, Mikhail Romanov, took the throne. Under Romanov rule, serfs lost almost all of their freedoms. Peter the Great, another Romanov who ruled Russia from 1682 to 1696, instituted sweeping reforms in Russian government and society. He built St. Petersburg and tried to make European traditions a part of Russia. In the Great Northern War against Sweden, new territories were gained. Other absolutist rulers arose, such as the Hapsburg Empress Maria Theresa in Austria, and the Hohenzollern Frederick the Great from Prussia. These two rulers clashed in the War of the Austrian Succession, during which Frederick won the territory of Silesia.

Europeans during this era desired a “balance of power” that would prevent any single nation from becoming most powerful. The expression of this idea was seen in the intense competition and warfare between the various European nations for control in Europe, especially with the Thirty Years’ War. The war initially began as a religious conflict in Germany between Bohemian Protestants and Hapsburg Catholics in 1618. However, it soon became a political conflict for power that involved Spain, Holland, Denmark, France, Switzerland and The Holy Roman Empire. The Treaty of Westphalia finally marked the end of the Thirty Years’ War in 1648. The war significantly weakened Germany and the Holy Roman Empire, as well as Austria and Spain. Although France was largely a Catholic country, it had chosen the side of the winning Protestants in order to oppose the Holy Roman Empire, and France emerged strengthened and more unified as a result.

==The Scientific Revolution==

A remarkable Polish scientist named [[Nicolaus Copernicus]] (1473-1543) made a stunning claim that rocked the religious and scientific world: he asserted that the earth revolved around the sun, not the sun around the earth. “Finally we shall place the Sun himself at the center of the Universe. All this is suggested by the systematic procession of events and the harmony of the whole Universe, if only we face the facts, as they say, ‘with both eyes open.’” With this bold declaration, Copernicus began the “Scientific Revolution,” which rejected geocentric (earth-centered) theory of the universe dating back over a thousand years to Ptolemy.
[[Image:Copernican model from book.jpg|right]]
Copernicus worked for years on his book describing his theory, and did not publish it until near the very end of his life. He entitled it, “De revolutionibus orbium coelestium” (“On the Revolutions of the Heavenly Spheres”), and published it in 1543. The work was far from perfect: Copernicus assumed that the planets moved in circular revolutions (they actually revolve in ellipses, though the orbit of the earth is nearly circular), and Copernicus also thought that the sun stood motionless as some kind of stationery fixture in the universe (the sun actually moves rapidly too). He also thought the sun was at the center of the universe. A diagram from his book displaying circular planetary orbits around the sun is to the right.

The reception to his work was initially positive within the Catholic Church, which Copernicus served, but to Protestants it conflicted with literal interpretations of the Bible, such as the account of how Joshua benefited from the sun standing still as it passed over the earth. “And the sun stood still, and the moon stayed, until the people had avenged themselves upon their enemies. Is not this written in the book of Jasher? So the sun stood still in the midst of heaven, and hasted not to go down about a whole day.”<ref>Joshua 10:13.</ref> There were few Protestants in Poland then (or now), and thus Copernicus died without much controversy.

[[Johannes Kepler]] (1571-1630), a Protestant in Germany, built on Copernicus’s work and discovered that planets orbited the sun in ellipses rather than circular orbits. Kepler was a brilliant mathematician, astronomer and devout Christian who cited God many times in all of his writings. He felt it was his Christian duty to understand the creation of God, the universe. He also felt that man, being in the image of God, was fully capable of understanding the universe. Like Plato and Pythagoras, Kepler felt that God must have created the universe according to a mathematical plan.

[[Galileo]] (1564-1642) was a contemporary of Kepler who lived in Catholic Italy. Galileo was not as bright as Kepler or Copernicus, and was a bit of an entertaining showman who made a lively party guest. He taught geometry to medical students so that they could use astronomy and astrology in the practice of medicine! Galileo’s primary scientific contribution was his improvement on the telescope in order to view celestial bodies.

But by this time the Catholic Church was in a deadly struggle with Protestantism, which was critical of Copernicus’s theory of the earth revolving around the sun. Wars were being fought and soldiers were dying over religious differences. In 1616, the Catholic Church officially prohibited teaching Copernicus’s theory as fact, although there was no objection to learning it as a mathematical theory. Galileo, however, felt Copernicus was correct and wanted to teach his theory. He had met with Pope Urban VIII on numerous occasions and felt that the Church would not object to Galileo’s publishing the Copernican theory about the universe.

There was no right of free speech in Italy at that time, and in 1632 Galileo published a work called “Dialogue Concerning the Two Chief Systems of the World - Ptolemaic and Copernican” containing Galileo's claim that he had approval of the Catholic Church (an “imprimatur”). Buyers and readers of the book were led to believe that Galileo’s views reflected the official views of the Church, when they did not. Galileo had received permission from a local cleric in Florence, but not from Church authorities in Rome, which would have been required for a work of that significance. Moreover, the Church authorities in Rome felt that Galileo knew that he was prohibited from claiming that the Church supported the Copernican view. This would be like a government official releasing a key military secret with the approval of his boss, but without the higher-level approval that would obviously be required for something extremely significant.

Galileo also angered Church officials in the way that his book mocked opponents of the Copernican system. His “dialogue” was between a fictional Salviati, who supported the Copernican system, and an Aristotelian philosopher given the unflattering name “Simplicio”. The simpled-minded “Simplicio” was made to look like the pope himself, and was mocked in the book for his skepticism about the Copernican system.

In the book’s climax, Salviati supposedly proves the Copernican system by citing the movement of the tides. This theory of Galileo’s was completely false and had already been disproved by Kepler.

Catholic Church officials were furious about this book and the Inquisition banned its sale. Galileo was summoned to Rome. Unlike Luther, Galileo remained a Catholic and fully accepted whatever judgment the Church would render against him. His “trial” was a bit of a farce, probably put on for show more than anything, with Galileo living in plush quarters at all times. At the end of the trial Galileo was required to live in a house where the Church could keep an eye on him during the controversy. Galileo was not imprisoned. Galileo then continued his scientific work but was kept from embarrassing the Church further. Galileo never split from the Church and completely accepted his punishment.

There were other noteworthy scientists during this era. [[Vesalius]] advanced the understanding of anatomy for medical doctors. [[John Harvey]] explained the circulatory system in human anatomy. [[Rene Descartes]] was a French mathematician and philosopher who discovered [[analytical geometry]] and famously declared, “I think, therefore I am.” [[Francis Bacon]] promoted research based on experimentation.

In England there was Sir [[Isaac Newton]] (1643-1727), who after Jesus Christ was perhaps the most influential man who ever lived. If Newton were alive today, then he would be described as a Christian fundamentalist. Everything Newton did was inspired and motivated by his powerful faith in the Word of God in the Bible. Abandoned by his own mother, Newton had an [[inferiority complex]] that drove him to achieve more and more. In one summer he discovered the laws of [[gravity]] and also [[calculus]], and the world has never been the same since.

Like most other men who made great advances in knowledge in world history, Newton discovered and explained something that was unseen: the invisible force of gravity. Newton proposed action-at-a-distance, whereby one object attracts another through an invisible force of gravity. Newton was ridiculed for proposing that something unseen could act at a distance and influence something else millions of miles away from it. Yet Newton was right. Speaking of the world in which we live, Newton once said, “Why not try to understand it?” Newton understood it and explained it to the rest of us, and we are forever grateful.

A German contemporary of Newton, Gottfried [[Leibniz]], deserves mention. Like Newton, Leibniz was a devout Christian. In fact, Leibniz was a Lutheran who dreamed of reuniting that denomination with the Catholic Church. Leibniz was also a brilliant mathematician who developed calculus independent of Newton. It was Leibniz, not Newton, who first published the notation for the integral:
<math>\int_a^b f(x)dx</math> But an ugly dispute developed between Newton and Leibniz over who discovered calculus first. Most English-speaking historians give the credit to Newton but bias may play a role in that. Regardless, the world benefited immensely from Leibniz as well as Newton.

==The Enlightenment==

The “Enlightenment” was a philosophical movement in the 1700s that emphasized an intellectual approach rejecting traditional social, political and (sometimes) religious views. Philosophers in the Enlightenment felt that breakthroughs in science, such as Isaac Newton’s discoveries, could be duplicated in other fields through a systematic and logical approach. However, Newton’s breakthroughs were inspired by his Christian faith, while some of the Enlightenment thinkers rejected and even criticized Christianity. There were Enlightenment philosophers in England, France and even the American colonies. There were stunning new works in classical music, and the arts flourished in what is known as the neo-classical period.

Note how the term “Enlightenment” is biased and self-serving. It makes it appear that people became enlightened compared to their ignorant ancestors. In fact, the achievements of the so-called “Enlightenment” movement were not any greater than other movements and periods in history, and there are valid criticisms of certain aspects of the Enlightenment.

Frenchman Voltaire (1694-1778) was a leading philosopher in the Enlightenment, advocating freedom everywhere and emphasizing his form of reason. He wrote “The Candide,” in which Voltaire described many bad things that happen to Candide in order to make the point that the world could be a better place. Increased freedom was Voltaire’s way of improving things. His ideas later formed the basis of the French Revolution, when many innocent people (including the French royalty and much of the aristocracy) were murdered and churches destroyed amid a reign of terror.

Englishman John Locke (1632-1704) was the leading political philosopher, whose ideas helped the American colonists form a new government. Locke described society as a contract between an individual and society. This was a radical concept at a time when monarchs and the divine right of kings was the controlling theory. Locke’s view helped lay the foundation for the constitutional government that we use in the United States. Locke had built on the prior work of Englishmen Francis Bacon and Thomas Hobbes.

Sir William Blackstone (1723-1780) was the leading legal authority on English law, upon which much American law is based. Law students in America hear about him frequently in law school. In “The Rights of Englishmen,” Blackstone described the source of the rights of the people.

Frenchman Baron de Montesquieu (1689-1755) proposed the concept of separation of powers, and checks and balances, in government. This inspired the United States [[Constitution]]. His book, “The Spirit of the Laws,” explained essential aspects of good government that became enormously influential.

Another Frenchman, Denis Diderot (1713-1784), contributed to literature, speculated on free will and attachment to material things, and edited an encyclopedia of scientific and social knowledge known simply as “Encyclopedia”.

Swiss-born political philosopher Jean Jacques Rousseau (1712-1778) wrote several controversial works. He felt that politics and morality could not be separated, and that the will of the majority was not always correct. Many Americans today would agree with that view. However, Rousseau also attacked private property, and laid the groundwork for future [[communist]] writers such as [[Karl Marx]]. Rousseau declared that government’s goal should be to provide freedom, equality and justice. But note that freedom often results in inequality.

Irishman Edmund Burke (1729-1797) was in the British parliament when the conflict with the American colonies occurred, and he sided with the colonies, making him a hero to many in America. He is also considered to be the world’s first political conservative, and many conservatives today praise him for that reason. His works included “Inquiries into the Sublime and Beautiful” (defending the American colonies and examining how people interpret what they see) and “Reflections on the Revolution in France” (warning, even while everyone else praised the French Revolution, how they were all wrong and how bad the French Revolution was because a few would gain control and abuse their power). What was striking about Burke was that he was almost always right in his political assessments and predictions.

The Scotsman David Hume (1711-1776) was a philosopher and historian who promoted materialism and naturalism rather than spirituality. He was a “skeptic” towards religion, and he penned “A Treatise on Human Nature.” He quipped, “You can tell what is inside a person’s soul by what comes out if it.” Hume is considered the greatest philosopher to write in English, but that is only because there were so few good English philosophers. Hume has been criticized for his atheistic approach and his chief claim to fame is that Charles Darwin declared Hume to have been his central influence, as did “Darwin’s bulldog,” Thomas Henry Huxley. Hume believed in relativism rather than absolute truth, and that not even what God creates has absolute beauty: “Beauty in things exists merely in the mind which contemplates them,” Hume declared.<ref>David Hume’s ''Essays, Moral and Political'' (1742).</ref>

The German Immanuel Kant (1724-1804) was one of the most famous philosophers ever. In his “A Critique of Pure Reason,” Kant criticizes pure reason as a guide to life. Kant may not have been a Christian himself, but he considered Christian values to be the best values in the world. Kant also established a systematic basis for critical philosophy and suggested a material origin for the solar system. Kant’s own suggestion for a moral daily life was this: don’t do something which, if everyone did it, then the outcome would be bad. Expressed another way, an act is moral only if it works as a rule for everyone. For example, littering would be wrong because if everyone did it, then there would be an ugly mess. Kant is taught in all college philosophy departments to this day, but not because he praised Christianity!

Finally, let’s recognize American contributions during the Enlightenment. The greatest works about government were created by the Founding Fathers. These works include the Declaration of Independence (1776) with its statement of inalienable rights and a right to break the social contract with a ruler when he (the king) violates natural rights. The United States Constitution (1787) gave the world a masterful design for government. The Federalist Papers (1788), and particularly articles by Alexander Hamilton and James Madison (most notably No. 10), were also significant.

==Capitalism==

Just as Isaac Newton’s discovery and explanation of the unseen force of gravity was a brilliant insight in science, Adam Smith of Scotland discovered a powerful unseen force in economics known as the “invisible hand.” If government allows free enterprise to flourish on its own, without interference by government, then an “invisible hand” harnesses the power of self-interest for the overall good of society. Companies are guided by this “invisible hand” to work in a way that is beneficial to others, and the overall wealth and progress of society will increase. This is what Adam Smith wrote in the influential ''The Wealth of Nations'', published in the same year as the Declaration of Independence in America: 1776. Adam Smith’s book is still considered the world’s greatest economics masterpiece. He explained how the basis of wealth is found in a free economy with an unregulated exchange of goods, whereby the supply of goods and services responds in an efficient way to the demand of the public without government intervention. Smith advocated the concept of laissez faire, which means “allow to do” or, more simply, “hands off” by government. Ironically, homeschooling in New Jersey (and a few other states) is an activity that most resembles Adam Smith’s ideal of an unregulated market. Let’s defend this lack of regulation of homeschooling while we can!

“[[Capitalism]]” became the prevailing economic theory in Britain as a result of the insights of Adam Smith and others. The Merriam-Webster online dictionary defines capitalism as follows: “an economic system characterized by private or corporate ownership of capital goods, by investments that are determined by private decision, and by prices, production, and the distribution of goods that are determined mainly by competition in a free market.” Britain quickly became the largest and greatest empire in the history of the world as a result of its adherence to this superior economic system. The British empire was larger than even the Mongol empire, except that the British empire was not "contiguous" on land, but included faraway places like Australia and India.

A related economic theory known as “mercantilism” also became popular. Mercantilism was a policy for a nation (such as Britain) to increase its own national wealth based on trade with other nations and territories. Under mercantilism, a nation should accumulate gold by exporting more goods than it imported, and by using colonies (such as the English colonies in America) to ship raw materials to the mother country that could be manufactured and exported to other peoples. Another key component of mercantilism was establishing foreign trading monopolies that would have unfair advantages over competitors in other countries. One such British monopoly was the East Indian Tea Company, and its unfair business advantage caused the colonists in Boston to revolt in the form of the Boston Tea Party.

While Adam Smith’s “invisible hand” was positive in almost every way, “mercantilism” did have a dark side of exploitation of colonies for the benefit of the mother country.

Adam Smith did not invent capitalism, but gave it a powerful intellectual justification. In 1607 -- over 150 years before Adam Smith -- capitalism had already played a prominent role in the first settlement of North America at Jamestown, which was funded by a joint-stock company (a privately owned company having a structure similar to the corporations of today). Joint-stock companies or corporations were (and are) a form of capitalism; decisions are made privately by those acting on behalf of the owners, without any direct control by government.

== References ==

<references/>
[[Category:World History lectures]]
{{DEFAULTSORT:World History Lecture 08}}

Talk:Significance of E. Coli Evolution Experiments

2009-03-06T01:48:42Z

Argon: typos

Talk:Significance of E. Coli Evolution Experiments

2009-03-06T01:46:36Z

Argon:

Talk:Significance of E. Coli Evolution Experiments

2009-03-05T16:06:05Z

Argon:

Talk:Significance of E. Coli Evolution Experiments

2009-03-05T16:05:37Z

Argon: Deg of freedom question