<p>^That kind of accidental question would be caught in the pre-testing, I imagine.</p>
<p>Anyway, here are the stats for all the math questions I have:
Total 5-choice questions: 968
Expected number per letter: 193.6
Expected range per letter: 181.2 - 206.0 (one std deviation)</p>
<p>Results:</p>
<pre><code> observed expec one-var z chi-sq
-------- ----- --------- ------
A 187 (19.3%) 193.6 z = -0.53 0.225
B 201 (20.8%) 193.6 z = 0.59 0.283
C 200 (20.7%) 193.6 z = 0.51 0.212
D 201 (20.8%) 193.6 z = 0.59 0.283
E 179 (18.5%) 193.6 z = -1.17 1.101
</code></pre>
<p>For one variable z, we need |z| > 1.96 for
95% confidence of a non-uniform distribution.</p>
<p>Total chi-squared = 2.10 (4 degrees of freedom).
Here are the stats for the level 4 and level 5 questions only:
Total 5-choice questions: 245
Expected number per letter: 49.0
Expected range per letter: 42.7 - 55.3 (one std deviation)</p>
<p>Results:</p>
<pre><code> observed expec one-var z chi-sq
-------- ----- --------- ------
A 53 (21.6%) 49.0 z = 0.64 0.327
B 47 (19.2%) 49.0 z = -0.32 0.082
C 45 (18.4%) 49.0 z = -0.64 0.327
D 43 (17.6%) 49.0 z = -0.96 0.735
E 57 (23.3%) 49.0 z = 1.28 1.306
</code></pre>
<p>Total chi-squared = 2.78 (4 degrees of freedom).
</p>
<p>Looks like the harder questions are consistent with a uniform distribution as well.</p>