14
edits
Changes
Jump to navigation
Jump to search
Line 547:
Line 547: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Talk:Essay:Quantifying Openmindedness (view source)
Revision as of 14:33, June 18, 2012
, 14:33, June 18, 2012→Extremely poor design: new section
::If I am missing the point of all this (maybe a conservative would answer "no" to those questions?), then I'm happy to learn, but what this questionnaire seems to be is a list of things conservatives believe in, and if you do not agree it says you are closed-minded, rather than merely, "not a conservative". --[[User:Thal|Thal]] 11:54, 17 June 2012 (EDT)
::If I am missing the point of all this (maybe a conservative would answer "no" to those questions?), then I'm happy to learn, but what this questionnaire seems to be is a list of things conservatives believe in, and if you do not agree it says you are closed-minded, rather than merely, "not a conservative". --[[User:Thal|Thal]] 11:54, 17 June 2012 (EDT)
== Extremely poor design ==
As someone who actually has training and education in research methodology, survey design, statistics, and data analysis I find this test of "quantifying" open-mindedness insulting to the profession. The author of this test obviously has little to no understanding of proper survey design. At best, I would say that if this test quantifies anything, it's the open-mindedness towards Conservative ideals of Conservapedia's regular visitors. Question #2 is the only question I would accept as a decent measure of actual open-mindedness.
(My use of the terms "error" and "bias" are used strictly in the statistical/research design sense)
Just a sample of the errors include:
A) Biased questions: This test, as another person has mentioned, is designed to make Conservatives appear more open-minded than liberals. If I were to develop a test in direct opposition to this test I would have questions such as, "Do you think it's impossible that Jesus never existed?" or "Do you resist accepting that God doesn't exist?" and so on.....using the same "yes/No" format we can assume that questions such as these would make "Liberals" appear more open-minded (as long as answers are honest).
B) The fact that the author has told us in advance what he considers to be the "correct answers" (I'll use this term since it's obvious that the measure of open-mindedness reflects only conservative ideals). This fact increases greatly the response bias....which means additional survey error to muddle through to find any truth in the analysis of the data.
C)Closed-ended questions that require respondents to provide answers as though they were responding to open-ended questions. Basically, the individual questions are so poorly designed, convoluted, and imbued with biased assumptions that the test becomes even less quantitative and more qualitative as respondents feel necessary to explain their yes/no answers which fail to address the questions sufficiently.
D)Even if the questions truly were quantitative in nature, the obvious sampling bias pretty much makes this test useless outside of Conservapedia.
E)The use of absolute terms such as "Impossible", despite attempts to define it, lead to even more error and bias. Even grade school children are taught to watch out for these types of terms on multiple choice exams ("always" and "never").
F) I'll finish with this although I could go on: This measurement does a very poor job of defining actual open-mindedness. IF you look at the questions and ask yourself, "Do these questions accurately reflect true open-mindedness in its most general sense?" the honest answer is no. Open-mindedness, in this test, reflects only an openness to specific conservative/literal creationist ideals.
Using some of the questions provided, a better design would include questions such as (and these are a bit off the cuff so I will not say these are perfect, perfecting questions like this requires trial and error):
#3) Do you find it difficult to admit when a long held belief is proven wrong? (which can replace a majority of the questions)
#14) Do you find it difficult to admit that what you are taught in school may not be accurate?
#17A) Do you find it difficult to question the validity of expert opinion on subjects you are not familiar with?
#17B) Do you find it difficult to accept the insights of non-experts/laymen on subjects you are not familiar with?
Towards the end of the following research are a set of questions which I feel do a far better job at measuring true open-mindedness. Notice the variation in how questions are asked, enabling a wider interpretation of what open-mindedness means. Also notice how much more sophisticated the statistical analysis is over the simplistic method used on the test provided by Conservapedia which relies on simply adding up yes and no answers without even the most basic weighting method.
http://www.psyc.jmu.edu/assessment/research/pdfs/APS_AOT_Final_Marsh&Pastor.pdf
[[User:Willingham|Willingham]] 10:33, 18 June 2012 (EDT)