Block, SkipCaptcha, Automoderated users, Bots, edit
57,719
edits
Changes
Jump to navigation
Jump to search
Line 1:
Line 1: − + + + + + + + + + + + + + Line 12:
Line 24: − + Line 20:
Line 32: − + Line 36:
Line 48: − +
clean up
'''Hot-Deck Imputation''' replaces missing data with comparable data from the same set. "Hot-deck imputation is a means of imputing data, using the data from other observations in the sample at hand." <ref>http://analytics.ncsu.edu/sesug/1999/075.pdf</ref> For example, suppose census officials were unable to count the number of people in a given house and decided to fill in the missing data using hot-deck imputation. They would use the data from a similar house in the same area, and substitute the number of people in that house for the missing data.
'''Hot-Deck Imputation''' replaces missing data with comparable data from the same set. "Hot-deck imputation is a means of imputing data, using the data from other observations in the sample at hand." <ref>http://analytics.ncsu.edu/sesug/1999/075.pdf</ref> For example, suppose census officials were unable to count the number of people in a given house and decided to fill in the missing data using hot-deck imputation. They would use the data from a similar house in the same area, and substitute the number of people in that house for the missing data. Hot-deck imputation is one of the most widely used imputation methods.<ref>http://nces.ed.gov/StatProg/2002/appendixb3.asp</ref>
This paper discusses various types of imputation as well as their benefits and problems: http://nces.ed.gov/StatProg/2002/appendixb3.asp
==The Flaws of Imputation==
All methods of imputation are less-than-ideal. "Kalton and Kasprzyk, 1982...cautioned that imputation methods do not necessarily lead to a reduction in bias, relative to the incomplete data set. And, they warned against the danger of analysts treating the "complete" cases as actual responses, thus overstating the precision of the survey estimates." <ref>http://nces.ed.gov/StatProg/2002/appendixb3.asp</ref>
"One problem occurs with [hot-deck imputation] when several records with missing values occur together on the file. This results in the current donor value being assigned to multiple records, thus leading to a lack of precision in the survey estimates (Kalton and Kasprzyk, 1986)."<ref>http://nces.ed.gov/StatProg/2002/appendixb3.asp</ref> The method by which the donor value is selected could easily skew results, particularly in areas where there are typically many blank values, such as densely populated cities. If the population of these areas (which tend to be democratic) were exaggerated, redistricting could assign more Representatives to their state. Ultimately, this would result in more liberal politicians in Congress.
Finally, the results of hot-deck imputations must be treated carefully. These data sets appear to be complete, but in reality, they are not. "Regardless of the specifics, all hot-deck procedures take imputed values from a respondent in the same data file, thus yielding imputations that are valid, although not necessarily internally consistent for the respondent values. In order to evaluate the hot-deck imputation used for any specific data collection, detailed information is required." <ref>http://nces.ed.gov/StatProg/2002/appendixb3.asp</ref>
==Imputation in the United States Census==
"Although imputation was used in the 1940 and 1950 censuses to determine characteristics of the population, it was not used to determine the actual population count for apportionment purposes until 1960." <ref>''Utah v. Evans'', 182 F. Supp. 2d 1165 (2001)</ref>
"Although imputation was used in the 1940 and 1950 censuses to determine characteristics of the population, it was not used to determine the actual population count for apportionment purposes until 1960." <ref>''Utah v. Evans'', 182 F. Supp. 2d 1165 (2001)</ref>
1. We impute age, race and gender independently.<br>
1. We impute age, race and gender independently.<br>
2. For race, we first try to base imputation on other data (e.g., ethnicity) for the same person.<br>
2. For race, we first try to base imputation on other data (e.g., ethnicity) for the same person.<br>
3. We impute all remaining missing values using only "good" unimputed) data for another person in the same group, or ctrlnum.<br>
3. We impute all remaining missing values using only "good" (unimputed) data for another person in the same group, or ctrlnum.<br>
4. We use each "good" data value only once for imputation.<br>
4. We use each "good" data value only once for imputation.<br>
5. We work backwards over the ctrlnum; then, if necessary, we work forwards.<br>
5. We work backwards over the ctrlnum; then, if necessary, we work forwards.<br>
==''Utah v. Evans''==
==''Utah v. Evans''==
''Utah v. Evans'', 182 F. Supp. 2d 1165 (2001) is the leading court case about the use of Hot-Deck Imputation in the census. After the use of Hot-Deck techniques in the 2000 census, one of Utah's congressional seats was appointed to North Carolina. Utah challenged this, on the grounds that Hot-Deck Imputation is a form of sampling, and "violated various statutory provisions and the Constitution."
''[[Utah v. Evans]]'', 182 F. Supp. 2d 1165 (2001) is the leading court case about the use of Hot-Deck Imputation in the census. After the use of Hot-Deck techniques in the 2000 census, one of Utah's congressional seats was appointed to North Carolina. Utah challenged this, on the grounds that Hot-Deck Imputation is a form of sampling, and "violated various statutory provisions and the Constitution."
The District Court for the District of Utah, Central Division stated "We begin by noting that section 195 does not preclude the Census Bureau from the use of every type of statistical methodology in arriving at apportionment figures during a decennial census. Instead, it prohibits only "the use of the statistical method known as 'sampling.'"
The District Court for the District of Utah, Central Division stated "We begin by noting that section 195 does not preclude the Census Bureau from the use of every type of statistical methodology in arriving at apportionment figures during a decennial census. Instead, it prohibits only "the use of the statistical method known as 'sampling.'"
In addressing the potential of manipulation of Hot-Deck Imputation, the court said, "The Court need not decide here the precise methodological limits foreseen by the Census Clause. It need say only that in this instance, where all efforts have been made to reach every household, where the methods used consist not of statistical sampling but of inference, where that inference involves a tiny percent of the population, where the alternative is to make a far less accurate assessment of the population, and where consequently manipulation of the method is highly unlikely, those limits are not exceeded."
In addressing the potential of manipulation of Hot-Deck Imputation, the court said, "The Court need not decide here the precise methodological limits foreseen by the Census Clause. It need say only that in this instance, where all efforts have been made to reach every household, where the methods used consist not of statistical sampling but of inference, where that inference involves a tiny percent of the population, where the alternative is to make a far less accurate assessment of the population, and where consequently manipulation of the method is highly unlikely, those limits are not exceeded."
This statement seems to contradict itself. If hot-deck imputation is used for such a "tiny percent of the population", how can it's omission cause a "far less accurate assessment of the population"? In order to skew the results that much, hot-deck imputation would have to be used for much more than a "tiny percent". The court is exaggerating the impact of hot-deck imputation. If used only when necessary, the data collected from hot-deck imputation may be negligible.
This statement seems to contradict itself. If hot-deck imputation is used for such a "tiny percent of the population", how can its omission cause a "far less accurate assessment of the population"? In order to skew the results that much, hot-deck imputation would have to be used for much more than a "tiny percent". The court is exaggerating the impact of hot-deck imputation. If used only when necessary, the data collected from hot-deck imputation may be negligible.
As the court says above, hot-deck imputation is valuable only when used out of absolute necessity. The problem is, there is no way to make sure that its use is saved for situations which truly require it. The lack of laws about the use of hot-deck imputation opens the opportunity for its misuse and manipluation.
As the court says above, hot-deck imputation is valuable only when used out of absolute necessity. The problem is, there is no way to make sure that its use is saved for situations which truly require it. The lack of laws about the use of hot-deck imputation opens the opportunity for its misuse and manipluation.