<p>Wow. This reminds me of the story (which might even be true) of a man whose Social Security checks stopped arriving. He called the Social Security Administration to ask why. "Well, it's because you're deceased." Unable to convince the SSA over the phone that he wasn't dead, he was granted an appeal. Following a 45 minute meeting in which he displayed his birth certificate, driver's license, Social Security card, etc. he was told to go home and wait for the appeal board's decision. It arrived two weeks later. "The Appeals Board has concluded that you are indeed dead, and therefore not entitled to further Social Security benefits." </p>
<p>So help me out here everyone. The College Board has admitted that several thousand SATs were improperly scored, and today Booz Allen says the scoring method is reliable. Hmmm.</p>
<p>Well, just a thought... it depends on what you mean by "reliable." </p>
<p>If I remember correctly, some 20,000 SATs were incorrectly scored. The SAT is administered to a few million students per year. If this is the only big error in, say, 10 years, do the (rough) math: 10 years * 1.5 million/year = 15 million scoring sheets. Divide by 20,000 errors -> one in 7,500ish will have a problem. ROUGHLY, this approaches 0.01% error. </p>
<p>Of course, that's a huge error if you are talking about things like planes or something in such high volume that should be very reliable. Just pointing that out.</p>
<p>IMO, the single best improvement that the CB can make is not one that you can implement with a machine. Simply put, if there are enough requests for hand-scoring in a batch of tests, they would admit to a potential problem and rescore the entire set.</p>
<p>
[quote]
Wow. This reminds me of the story (which might even be true) of a man whose Social Security checks stopped arriving. He called the Social Security Administration to ask why. "Well, it's because you're deceased." Unable to convince the SSA over the phone that he wasn't dead, he was granted an appeal. Following a 45 minute meeting in which he displayed his birth certificate, driver's license, Social Security card, etc. he was told to go home and wait for the appeal board's decision. It arrived two weeks later. "The Appeals Board has concluded that you are indeed dead, and therefore not entitled to further Social Security benefits."
[/quote]
</p>
<p>That story may be taken from a similar episode in Heller's Catch-22.</p>
<p>Let me try this from the other side. </p>
<p>"If the scoring system is reliable then there is no problem."<br>
This may be a tough sell to those students were affected by the scoring systems' "reliability."</p>
<p>Well, don't forget that Booz Allen Hamilton was paid by College Board to conduct this study.</p>
<p>
[quote]
Booz Allen Hamilton was asked to conduct a review of the scanning process to identify any further specific improvement opportunities to minimize any risk that incorrect scores will be reported in the future due to scanning-related problems. The project scope included the new safeguards already instituted by the College Board in March.</p>
<p>The Booz Allen Hamilton report concluded " the current process is reliable and has prudent controls in place to safeguard scoring accuracy for those marks made in accordance with test directions. The operational changes made by College Board in response to the October administration further improved process reliability by introducing scanning redundancy, more frequent scoring checks, an environmental acclimation period to eliminate the effects of humidity, and other safeguards."
[/quote]
</p>
<p>So Booz Allen Hamilton found the process with changes that TCB instituted in March after the October scoring issues became public to be reliable.</p>
<p>I should have titled the thread: "College Board releases report about current scoring system put in place after the mess-up of October scores."</p>
<p>so if 495,000 students took teh Oct SAT and 27,000 were scored incorrectly, what are the odds that once these people move on from determining results of college placement and high school graduation exams, they will be looking at pysch testing to determine who will be hired to work at Walmart or the Post Office?</p>
<p>According to this NYT follow-up article, the College Board is not big on disclosure - I think we already figured that one out.</p>
<p>Well, their entire product is based on the idea that it is consistent and reliable.</p>
<p>Yes, many products are like that, but 'tis the nature of standardised testing that it ought to measure the same thing in Massachusetts in May to Kansas in October.</p>