<p>"New question surveys," or pretests, are used to determine the approximate dificulty of the items before they are used. By pretesting items, the goal is to assure that every year's test is of the same difficulty. Unfortunately, life steps in, so the equating proces, using equator items from a previous form, is used to help understand any difference in difficulty or strength of test populations on this year's test compared to a previous year.</p>
<p>Comparability studies are performed about every five years in each subject. This consists of administering a previously administered test to a groups of college students. The colleges are not chosen from random, but are representative of the types of colleges that AP students typically apply to and receive credit for their AP performance.</p>
<p>The comp study tests are graded by the readers, using the same rubrics used for the AP students, and the grades are compared to those given by the college professors. This data is compared to the performance of the AP students who took the same test. All this information is then used to adjust the score distributions on the current test form.</p>
<p>The equating set is very important in determining cut points, as equating has been shown to be a reliable way of maintaining "continuity." However, the comp study performance, essay performance and overall raw score performance of the current testing population are all factors involved in determining valid and reliable cut points.</p>
<p>Unless a table leader has been to the score setting session, information would be second hand.</p>
<p>BigIs, the details of the grade setting process are very interesting to me. Do you perchance have a link to any more comprehensive information like this?</p>
<p>Different AP tests have different curves for different years. I remember my AP Psych teacher said that in one year the cutoff for a 5 was like a 70%, but in another year the cutoff was an 85%. Also, just because you think getting a 5 on an AP test is easy doesn't mean it's easy for everyone. (eg. I got a 4 on Physics C: Mechanics and a 5 on Physics C: E/M, and lots of other people I know got the other way around.)</p>
"New question surveys," or pretests, are used to determine the approximate dificulty of the items before they are used. By pretesting items, the goal is to assure that every year's test is of the same difficulty.
</p>
<p>I simply cannot believe that this is true, and for a number of reasons. Due to it being the only AP course that I am familiar with, I will restrict my remarks to those in AP Calculus.</p>
<p>(1) The aim of an AP Calculus course and a college level calculus course are inherently different. By necessity, the goal of an AP Calculus course is to teach calculus sufficiently to do well on a test, and oftentimes to prepare students for the AP specific test itself. This is frequently for a number of reasons, but primarily so that students can achieve placement credit at universities, and also so that schools can have their names in periodicals such as Newsweek magazine to make themselves look impressive.</p>
<p>Conversely, the goals of calculus at a collegiate level are to teach calculus to the extent necessary to meet the goals of the particular mathematics department. If a lot of the mathematics in a particular college focused on teaching delta-epsilon proofs for limits, those particular studenst would find themselves overly prepared for what appears on the exam. If the Trapezoidal Rule for approximating the area under a curve were not deemed a worthy task (especially since Simpson's Rule does a better job of approximating said area), these students would find themselves at a disadvantage on an AP test.</p>
<p>The goals of the two groups are (necessarily) different. Although both have calculus in common, the emphases of these two courses may be different. That is very much unlike any high-caliber AP course, where the teacher of said course knows that Trapezoidal Rule is a must, Simpson's Rule and delta-epsilon proofs for limits are not, and are therefore covered only at the teacher's discretion and time permitting.</p>
<p>(2) If the exams were desired to be equally different, why would AP exams see such drastically different curves in a relatively short time interval? Do the mock tests really get this so far wrong? If they do, how can they be considered reliable?</p>
<p>For instance, it was announced at the AP Calculus reading in Kansas City this year that the 2007 AP Calculus AB test was the hardest test ever given to AP Calculus students. Ever. And it wasn't even close.</p>
<p>But 21.0% received 5's and 25.7% received 1's, with a mean grade of 2.94. The 2005 exam had a nearly identical distribution, with 20.7% receiving 5's and 25.2% receiving 1's, also with a mean grade of 2.94. If the exams were designed to be equally difficult, why are the distributions so similar? Why are the means identical? And why need to adjust the raw cut score at all?</p>
<p>We recognize that no two tests can be the same in difficulty, and my conjecture is that the folks at the College Board do not even try to do this. Rather, my understanding is that the College Board tries to preserve equity in results: that a student who earned a solid 5 on one AP exam would almost ceratinly earn the same solid 5 by taking a different version of the same AP exam.</p>
<p>And that's where the consistency of the results within the multiple choice comes in.</p>
<p>
Comparability studies are performed about every five years in each subject. This consists of administering a previously administered test to a groups of college students. The colleges are not chosen from random, but are representative of the types of colleges that AP students typically apply to and receive credit for their AP performance.</p>
<p>The comp study tests are graded by the readers, using the same rubrics used for the AP students, and the grades are compared to those given by the college professors. This data is compared to the performance of the AP students who took the same test. All this information is then used to adjust the score distributions on the current test form.
<p>(1) If the current form is not the ones used in the comparability studies, then all you have truly confirmed is that the standards were not aligned in the year of the AP study. Those standards may have already been adjusted, or they may be further out of whack than indicated.</p>
<p>(2) It takes time in order to make the kinds of adjustments needed. Every change that occurs to an AP exam is given two years notice to AP teachers so that these changes can be incorporated into the curriculum.</p>
<p>As a for instance, sign charts used to be acceptable as a justification for relative minima and maxima on the AP Test. I imagine that these compatibility studies determined that college professors were not sufficiently impressed by these explanations. (It also could be that the test development committee was not sufficiently impressed, and just created the expectation, pushing the raw score downwards slightly to earn the same achievement.) But the change did not occur overnight. Rather, the change occurred as a deliberate course of action by the College Board pushing for a statement such as "f has a relative maximum at x = c because the sign of f ' changes from positive to negative at that point."</p>
<p>It is my claim that these comparability studies -- while they may provide information about additional points of emphases and information of additional topics to study or remove that are needed in the future -- are often not providing actual information that goes directly into the scoring for that particular year, but rather that it guides the long-term direction of the program.</p>
<p>If you have contradictory source information, I would love to see it. I admit that almost all of my information is second-hand knowledge from attending workshops for each of the last five years in order to maximize the success of the AP Calculus program at my particular school.</p>
<p>^The OP merely responded to bartleby's remark: "Sounds like somebody didn't find an exam as easy as it is supposed to be." Also, the OP actually has pretty compelling arguments. I agree one test is not a reflection of one's abilities. His post was not condescending in tone either.</p>
<p>I think the AP tests are a joke in the sense that people think the real world is equivalent to them. So you can answer multiples choice and write a coherent essay. Does that make you "smart"? Anyone that thinks that needs to get a grip on reality (I'm not accusing anyone here of believing that).</p>
<p>I also believe the AP Tests are a scam that a lot of kids are duped into doing because they feel they will get college credit. 140 dollars for a test is a joke. Yeah some people pass and get a reward at the end, but most (at least at the school I went to) did not. </p>
<p>Also college courses don't usually come down to one test. There is class work that combines with a final test score. </p>
<p>I obviously do not believe in AP Tests or the SAT for that matter. They are ridiculous and hopefully people can soon see beyond them.</p>
<p>College credits usually cost significantly more than the AP test fee (which was closer to $80-$85 at my school). And for schools that accept a 3 or better, that means that approximately 60% of the nation's test-takers are earning some kind of credit. That's not a bad deal.</p>
<p>While some college classes do have "class work", I believe that high school classes have significantly more that "counts" towards a grade than college courses do. My first English course had two papers and three tests. My first history class had one paper (25 pages!) and three tests. My first math class had only four tests (the last double weighted). There were only a handful of classes in my entire collegiate career that had more than five assignments/papers/tests/finals that counted in the course grade.</p>
<p>The truth is, the final frequently was make or break for your grade at the college that I went to, which admittedly was competitive, but certainly not Ivy-caliber. Maybe colleges have changed in the last ten to fifteen years, but if not, there could be a rude awakening waiting for folks who don't know how to prepare for one test that really counts.</p>
<p>Just caught up reading this thread, and what Math Prof said earlier is what I understand to be the case for Language too. In a general sense the bell curve with the numbers (1-5) delineating the percentages approximately. The Lang test 2 years ago was MUCH harder than the one from 2007 (The representative numbers are different in Lang-2006 about 21% 4s and 5s, (51% pass rate), and in 2007 nearer to 28% (58% overall pass). 2008 looked more like 2007, (we shall see with grade distributions).</p>
<p>Speaking to college correlation-I remember college classes about the same as Math Prof and tokenadult too. 2-4 tests, 1 was a final, and the only real daily work was science labs in most cases. If you weren't good on test day, you were toast, especially on the first one-hence a LOT of drops.</p>
<p>"So you can answer multiples choice and write a coherent essay. Does that make you "smart"?"</p>
<p>Considering that APs are probably very highly correlated with g - the general intelligence factor - scoring highly on APs probably does indicate you are smart.</p>
<p>"I also believe the AP Tests are a scam that a lot of kids are duped into doing because they feel they will get college credit...Yeah some people pass and get a reward at the end, but most (at least at the school I went to) did not."</p>
<p>We are all equal and know equally as much and nobody should fail. That's basically what you're saying. The College Board makes no guarantee that you will pass, or even if you do, that you will get college credit. If you're ignorant enough to think that is automatic, that's your own issue.</p>
<p>"Also college courses don't usually come down to one test. There is class work that combines with a final test score."</p>
<p>Some people do master the course that pass that is certain, but many others that pass do not. </p>
<p>Also there is a lot of good in freshman courses such as meeting new people as well as picking up on new information your high school teacher did not teach (there is a lot they missed). </p>
<p>I know someone is going to say,</p>
<p>"but the cost or you save money."</p>
<p>Yes this is true if you pass, but saving money isn't everything.</p>
<p>"Considering that APs are probably very highly correlated with g - the general intelligence factor - scoring highly on APs probably does indicate you are smart."</p>
<p>It's sad that kids have been tricked into thinking this.</p>
<p>AP tests don't measure how smart you are. </p>
<p>AP Tests measure how well you studied prior and how well you remember what you studied. After the test, you can forget everything (another good reason to take the freshman course), but whoever is grading the test won't ever know. They'll look at the number you got right and from that they'll judge how smart you are. You can become stupid within the next week, and it won't matter if you got a good score on the test.</p>
<p>It's like vocab tests. </p>
<p>How many of those words can you honestly say you remember now even though you got 20/20 on the test. </p>
<p>I went through AP Spanish and a year later I couldn't recall more than a handful of words. </p>
<p>Is everyone like this? No, but I'm sure more are than not.</p>
<p>"AP tests don't measure how smart you are."</p>
<p>Not directly. The results on them are almost certainly highly correlated with how smart you are. Even strong correlations have outliers, so it is quite possible that we know someone mediocre who got a 5, or someone brilliant who got a 1. But in the aggregate...</p>
<p>"AP Tests measure how well you studied prior and how well you remember what you studied."</p>
<p>...which is significantly easier when you are intelligent. However, AP tests (other than maybe the really easy ones like Human Geography or Psychology) are designed to test more than simple recall of information. They demand you be able to analyze it using skills learned through the curriculum, as well. </p>
<p>"After the test, you can forget everything (another good reason to take the freshman course)"</p>
<p>Just like you can forget everything after taking a college final. Although its unlikely you 'forget everything' - it is simply stored in your long-term memory that can be recalled much more easily when you review the material.</p>
<p>"You can become stupid within the next week"</p>
<p>No, you can't, unless you suffer brain trauma. Intelligence is fairly stable throughout life. </p>
<p>"They'll look at the number you got right and from that they'll judge how smart you are. "</p>
<p>No, they won't. They'll judge how well you know the skills intended to be learned from the curriculum. </p>
<p>"It's like vocab tests.</p>
<p>How many of those words can you honestly say you remember now even though you got 20/20 on the test.</p>
<p>I went through AP Spanish and a year later I couldn't recall more than a handful of words. </p>
<p>Is everyone like this? No, but I'm sure more are than not."</p>
<p>Of course, because intelligence is distributed on a bell curve-like distribution. The vast majority of people lie in the middle. The smartest people compromise a small tail of the distribution (it's not a tautology - the alternative is everyone being either really smart or really dumb, and few in between.)</p>
<p>There are not too many standardized tests left for you after college, but your ability to do well on those and essays during college is how you are measured.</p>
<p>It is all about the hoops that you are currently jumping through, like it or not.</p>