17
edits
Changes
Jump to navigation
Jump to search
Line 78:
Line 78: + + + + + + + +
Talk:Significance of E. Coli Evolution Experiments (view source)
Revision as of 16:18, March 11, 2009
, 16:18, March 11, 2009Problems with chi-squared test when assumptions are violated
Able806, you made a good point about the contingency table cell frequencies being relatively low, but were wrong when you said ”the data set is actually too small to use the chi square method correctly”. In the low cell frequency case the chi-square test is still effective, but the null hypothesis distribution of the chi-square statistic starts to look less like the chi-square distribution. Thus, p-values calculated using the chi-square distribution may be a bit off. However, Monte Carlo p-values are always imperfect as well because it's impossible to generate an infinite number of random realizations. There are imperfections in p-values generated by analytic and Monte Carlo methods. However, low cell frequencies does not explain the >20x and >2.5x differences between chi-square p-values and p-values from the paper for experiments one and three. The reason for those huge differences was the use of the flawed test statistic (“mutation generation”) in the paper. [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)
Able806, you made a good point about the contingency table cell frequencies being relatively low, but were wrong when you said ”the data set is actually too small to use the chi square method correctly”. In the low cell frequency case the chi-square test is still effective, but the null hypothesis distribution of the chi-square statistic starts to look less like the chi-square distribution. Thus, p-values calculated using the chi-square distribution may be a bit off. However, Monte Carlo p-values are always imperfect as well because it's impossible to generate an infinite number of random realizations. There are imperfections in p-values generated by analytic and Monte Carlo methods. However, low cell frequencies does not explain the >20x and >2.5x differences between chi-square p-values and p-values from the paper for experiments one and three. The reason for those huge differences was the use of the flawed test statistic (“mutation generation”) in the paper. [[User:SJohnson|SJohnson]] 16:38, 5 March 2009 (EST)
:SJohnson, the chi-squared test is a valuable statistical tool, but the limitations of the test must be acknowledged. The chi-squared test can only produce valid results if the assumptions that underly the test are not violated. As an analogy, Newtonian models of motion fail to produce accurate results as velocities approach the speed of light; under those circumstances one must switch to a theory that accounts for relativistic effects.
:It seems that you have simply dismissed the [http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage28.htm widely-acknowledged] [http://faculty.chass.ncsu.edu/garson/PA765/chisq.htm fact] that the [http://www.wellesley.edu/Psychology/Psych205/chisquareindep.html chi-squared test] is [http://www.minitab.com/support/answers/answer.aspx?log=0&id=2236 inappropriate] for use in [http://www.graphpad.com/www/Book/Choose.htm situations] where n in any cell is [http://mysite.du.edu/~jcalvert/econ/chisquar.htm less] less than a [http://books.google.com/books?id=yU15rUiLRI8C&pg=PA201&lpg=PA201&dq=chi-square+test+assumptions&source=bl&ots=FRY0LwQ3z_&sig=FyIvzJx3hjQ8nWlu2cpmZj3pwXY&hl=en&ei=fm-1SayaNI_MMKX5tO4E&sa=X&oi=book_result&ct=result#PPA185,M1 threshold] [http://www.basic.northwestern.edu/statguidefiles/gf-dist_ass_viol.html number]. Different authors set different thresholds, but all are well above the numbers seen in your chi-squared analysis - even the most liberal guidelines advise against the chi-squared test when any expected cell frequency is less than one or more than 20% of the table cells are less than 5; others require that expected values in all cells must be more than 5. With smaller amounts of data, the test is insensitive and errs on the side of rejecting the hypothesis. If you attempt your chi-squared statistical analysis with a program that is more sophisticated than MS Excel (as I did), you get an error message indicating that the results are invalid due to low expected cell counts.
:That issue aside, there are other reasons that the chi-squared test is inappropriate here. As the links above point out, the categories tested must be truly independent; one example is that you can't use the chi-squared test to compare age and ability to kick a field goal by testing the same experimental group twice, one year apart; you have to test one group of age A and a different group of age B. In the case of the Blount paper, the categories are not independent. Even if there were adequate numbers to address the low-expected-frequency problem, this would make the chi-squared an invalid test in this case.
:There are other significant problems with the use of the chi-squared test in this circumstance, but they can wait until you address these first major problems.--[[User:ElyM|ElyM]] 12:18, 11 March 2009 (EDT)
== Misinterpretation of test ==
== Misinterpretation of test ==