<p>
</p>
<p>Hmmm, interesting. Yes, you’re right about how the questions are checked for that kind of flaw. A major source of info for the quality of a question comes from the plot of the percent of students getting the question correct versus their SAT scores on the same test. The best questions will have low percentages for low SAT scores, high percentages for high SAT scores, and a sharp rise in-between.</p>
<p>What’s funny is that stats below are suggestive of fewer A and E easier questions than one might expect normally. Perhaps more than suggestive for E answers.</p>
<p>Here is the distribution for level 1-3 questions:
Total 5-choice questions: 723
Expected number per letter: 144.6
Expected range per letter: 133.8 - 155.4 (one std deviation)</p>
<p>Results:</p>
<pre><code> observed expec one-var z chi-sq
-------- ----- --------- ------
A 134 (18.5%) 144.6 z = -0.99 0.777
B 154 (21.3%) 144.6 z = 0.87 0.611
C 155 (21.4%) 144.6 z = 0.97 0.748
D 158 (21.9%) 144.6 z = 1.25 1.242
E 122 (16.9%) 144.6 z = -2.10 3.532 <--
</code></pre>
<p>For one variable z, we need |z| > 1.96 for
95% confidence of a non-uniform distribution.</p>
<p>Total chi-squared = 6.91 (4 degrees of freedom)
</p>