<p>It was never suggested that tea leaves are read or microscopes used.<br>
But what would (e.g.) Caltech do if you saw three successive [sittings of the]
SAT consisting of: all 500's, then all 800's, then all 500's?
I believe that at any of the top schools this would invite a further look into file,
if nothing else to get a sense of what if anything should be understood as the
"real" set of scores.</p>
<p>Joe (Caltech '04) wrote:</p>
<p>Frankly I can't really understand getting (significantly) less than 800 on math at the high school level if you'd done it in middle school either. Yeah, I took advanced math classes in high school, too, but it's not as if I forgot how to multiply... I've done 4 years of college and 2.5 years of grad school now and I still think I'd do fine if you sat me down and had me take an SAT math section right now.</p>
<hr>
<p>Perhaps it's due to my school or something, but I did some research on SAT I math results compared to results of the AP Calculus BC test and found some surprising results.</p>
<p>Five juniors took the Calculus BC test at my school last year and I talked to all of them. We had two 5s, two 4s, and a 3. The people in the latter two categories scored 800 on the SAT I Math, and the people who got 5s (myself and a close friend) scored rather low... 670 and 710. We two also received 5s on AP Statistics and AP Physics C, so our 5s weren't flukes.</p>
<p>When you compare the scores, they hardly make sense. Surely, the people who scored higher on the APs should naturally score higher on the SAT I Math as well? There must be a missing factor that we haven't considered...</p>
<p>Consider this: Perhaps it's the SETUP of the two examinations. As we progress through high school into the upper-level maths, we no longer have multiple choice tests. Everything is free response.</p>
<p>In these free response questions, it is often required that we personally get very close to these questions. It's not just looking at a question and knowing instinctively -- just to get started, one must think heavily. And as we get started upon these problems, there are more factors. We have to figure out such things as a convenient method to use and what coordinate systems would be most helpful. We are fully absorbed into this problem and it takes much concentration to figure out.</p>
<p>Moreover, the style we use to solve this problem is completely arbitrary. Some might solve a double-integral using polar coordinates while others might solve the same problem using rectangular coordinates. Both could come up with the correct answer, but the method is different. This is the same in physics: is g equal to 9.8 m/s^2 or -9.8 m/s^2. It really doesn't matter as long as you set up the problem well and have everything else match your sign preference.</p>
<p>The SAT doesn't allow for such individuality. As you get into upper-level math and physics, we all develop such mathematical personalities, and I believe because of this, some of us feel restrained and uncomfortable during timed multiple-choice tests. I personally think that the SAT is rather easy... but the problem is that with a few sign errors that you made because you were uncomfortable with the coordinate system, a great impact could be made on your score.</p>
<p>Am I the only one who really feels so uncomfortable with such tests around here? Like I said, it may be just be my school, but I often see that an inversely proportional relationship exists between doing well on free response and doing well on multiple choice.</p>
<p>(Sorry for long post.)</p>
<p>Personally, as a former member of the admissions committee, I would consider a 670 on the low side whether it was accompanied by a Calc BC 5 or not. Obviously if there was a long list of math-related accomplishments the 670 would be considered a fluke, but that doesn't change the fact that 670 is definitely below average for the student body at Caltech and many of the incoming freshmen also have or could have managed 5's on AP math tests.</p>
<p>Before explaining myself, I don't quite understand your response...</p>
<p>My point WAS that a 670 and 710 are quite low for SATs. And 3s and 4s are quite low for APs. But the low-scorers of the AP Exams scored higher on the SATs than the high-scorers of the AP Exams.</p>
<p>But moreover, what's a 670? It's missing 5 or 6 problems (out of 40 or 50 or so), which could definitely be attributed to uncomfortability with the examination itself, right? And this uncomfortability, for me at least, is with the style that the SAT exists as -- it's a quick-answer and go kind of exam. Someone who's acquainted with doing free response all the time will definitely be uncomfortable with this.</p>
<p>I believe my school is representative of this. It's a boarding school and we live here from junior to senior year. We haven't had a multiple-choice exam ONCE in the entire time I've been here. What's odd is that our school has 50% of the number of 5s for AP Physics C and about 20% of the 5s in Calculus BC IN THE STATE OF INDIANA-- and there are only 300 people here -- but the majority of the SAT scores are just so low that I'm suspicious that perhaps we just don't do well on the SAT because it's multiple-choice.</p>
<p>You have to admit that there SHOULD be a strong positive correlation between SAT scores and AP scores if they're both valid tests... and I haven't seen that here at all.</p>
<p>"But the low-scorers of the AP Exams scored higher on the SATs than the high-scorers of the AP Exams."</p>
<p>...in your five-person sample. I am 100% sure scores correlate over the whole pool.</p>
<p>Are you honestly suggesting that the average SAT score of those with an AP 5 is lower than the average SAT score of those with an AP 4? If you are, your small sample size has misled you. If you are not, your comment and subsequent analysis are much less relevant.</p>
<p>You can be sure that AP-to-SAT (and SAT II vs SAT I) correlation has been studied in detail and that if any discrepancy had been found, there would have been instant, loud and national calls for the elimination or further weakening of the SAT. That is, the correlation is positive and robust as one would expect.</p>
<p>I wasn't implying that I thought that scores don't correlate over the whole pool, because they DO.</p>
<p>What I am saying here is that in certain groups of people, one cannot take the SAT to be a definite factor for admission. You just can't take it seriously when there are groups of people who are going to be disadvantaged because of its layout! Especially when there are a number of things to better judge by, such as AP scores, course rigor, and grades in those courses.</p>
<p>Frankly, my argument is this. Results from free response tests are IN GENERAL more accurate than results from multiple-choice tests, so colleges should use free-response tests to determine admission rather than SATs.</p>
<p>I'm sure most people don't feel the same way I do with the uncomfortability I mentioned earlier, but it IS existent and I don't believe it should hurt the groups of people who feel this way. I don't believe that such uncomfortability exists for anyone for free response because there's much less restraint.</p>
<p>(I'm trying to avoid making it look like there's contradiction in my argument. There isn't, but I expressed some things in some weird ways... so yeah.)</p>
<p>And siserune, where do you see any studies? I googled and found nothing comparing the two.</p>
<p>I don't claim to have personally looked up the studies. I could do so given an hour or two at the nearby education school library, but there is no point, because:</p>
<p>ETS and College Board have statisticians working full time to analyze most of the obvious questions, and many more subtle ones as well, about the validity of their tests. You better believe that a negative correlation between any of SAT I, SAT II, and AP would be the biggest news in standardized tests in a long, long time (note that it would likely imply a reverse correlation with IQ, which would be even more of a bombshell). There are huge political incentives to locate such surprises if they exist, and to my knowledge nobody has found any.</p>
<p>Edit added: the free-response vs multiple choice can be argued both ways. There is much more flakiness and luck in AP results than in SAT results, and that may explain the entirety of your observations.</p>
<p>You really think CollegeBoard would disclose such a thing if it were to truly occur? It seems to me like CollegeBoard is all about money (IT COSTS MONEY TO APPLY FOR FINANCIAL AID, FOR CHRIST'S SAKE), and since that wouldn't exactly increase their profits, they'd probably keep it secret.</p>
<p>And you'd think you'd be able to find something about it online no matter what the results were if they really did carry out such a study...</p>
<p>Response to edit: You're right there. I have a friend who walked into the BC Exam and got a 4 and all he knew was what a derivative and integral was and how to use his TI-89.</p>
<p>I would, however, say that a 4 on the Calculus BC is a much clearer sign of one's inability in mathematics than a 700 on the SAT, since you just have to get 70% correct to get a 5... (MIT still awards credit for a 4 in Calc BC, oddly enough.)</p>
<p>What we need is a free-response test that's MEASURED like the SAT... instead of the 1-5 scale, use the 1-20 scale or something. It'd be much more apparent how one's mathematical ability were from seeing that score.</p>
<p>siserune is entirely right.</p>
<p>"Frankly, my argument is this. Results from free response tests are IN GENERAL more accurate than results from multiple-choice tests, so colleges should use free-response tests to determine admission rather than SATs."</p>
<p>This statement just isn't borne out. In truth, people who do well on AP math tests tend IN GENERAL (as you put it) to also do well on SAT math tests.</p>
<p>Of course individual results will vary, and you could select five students at random and possibly get results all over the map--but the overall correlation IS known, which is a strong argument against discrediting one test or another.</p>
<p>I would think a 3 and an 800 looked a bit strange, and I would also think a 5 and a 670 looked a bit strange, but I wouldn't be inclined to mentally discount the high or low result in either case.</p>
<p>Evidence, please? Unless you can give me a link to the study that tells me that there is a strong positive correlation between SATs and APs, I don't believe you. I think there's a positive correlation, but I don't think it's very strong at all. (R^2 = .65 or .7, perhaps -- not .9)</p>
<p>You're still missing one of Joe's arguments, which is that even though you can certainly find people with low SAT and high AP scores, you can also find plenty of people with high SAT and AP scores. All else being equal, why in the world would admissions staff not prefer the latter group?</p>
<p>Also, the College Board "conspiracy theory" is unimpressive. In reality the College Board directly releases reams of statistics on each test they give, and gives researchers access to even more data.</p>
<p>The odds of a "hiding-a-dirty-secret-about-the-SAT-Math" scenario being correct are close to nil, particularly given that the SAT gets so much publicity and is the subject of so much study.</p>
<p>The same reason why the admissions staff shouldn't pick an applicant who got straight As throughout high school over an applicant who got straight Bs in 9th grade and straight As from 10th-11th grade.</p>
<p>And Joe: Yeah, I know that the CollegeBoard studies the SAT a lot. I've seen it all over their website how they're all confirming that the SAT is a proper determination statistic for success in college and the like, even though it's being criticized harshly by many, many groups of people. And once again, if CollegeBoard releases so much data, simply give it to me... surely, if they've done so much research on this issue, then it's released online.</p>
<p>When I first started disliking the College Board, I saw a statement it made in court about an error in grading the SAT tests. The official statement started with a 30-minute speech about how CollegeBoard has spent this many years helping students succeed in high school and college, etc., etc. Their arrogance is unbelievable to me, and I don't doubt for a second that they'd hide the result of a study if attention beforehand wasn't given to it.</p>
<p>Ah, the "unless you can give me a link..." fallacy! Rarely seen on the Caltech boards, but a gem all the same.</p>
<p>It is common sense that the scores on two math tests given to high school students would be quite correlated, and quite frankly the SAT remains very important even in elite college admissions where many students submit AP scores. It is your role as debunker to do the research and provide data to disprove the common knowledge. If you cannot do this, you are an ineffective debunker.</p>
<p>If you really wish to delve into this, I would get in contact with the College Board.</p>
<p>Your description of the elaborate publicity that surrounded misgrading (if I recall correctly) a couple hundred or thousand SAT exams out of millions is only a good example of the intense media scrutiny that makes the "conspiracy to hide how badly the SAT-Math correlates with other College Board math exams" you all but assume, all but impossible.</p>
<p>The other point that's being lost here is that the SAT and AP exams are designed to test different things--the SAT a sort of general math, verbal, and writing "aptitude" and the AP exam mastery of specific well-defined subject matter. That is why we can discuss how correlations indicate general relevance for both tests, but can't discredit individual scores based on a "mis-match."</p>
<p>It is only logical that a student with high mathematical aptitude/potential would have an advantage in mastering mathematical subject material (which is why the correlation exists)--but given a choice, colleges prefer students who show both high aptitude AND mastery of whatever subjects they've previously studied.</p>
<p>"In reality the College Board directly releases reams of statistics on each test they give, and gives researchers access to even more data."</p>
<p>Oh, but I believe it was you who said this. Thus a data set would be quite easy to find if the test had been done and such results were found.</p>
<p>Googling gives some people asking for such data sets but no responses are given... this leads me to believe that no such studies have been done.</p>
<p>"and quite frankly the SAT remains quit important even in elite college admissions where many students submit AP scores."</p>
<p>Correct me if I'm wrong, but I believe Ben Golub said the opposite a while back.</p>
<p>"The other point that's being lost here is that the SAT and AP exams are designed to test different things--the SAT a sort of general math, verbal, and writing "aptitude" and the AP exam mastery of specific well-defined subject matter."</p>
<p>BUT the AP exams DEPEND on mastery of the material presented on the SAT. If you don't know algebra and geometry, you don't know calculus.</p>
<p>"Correct me if I'm wrong, but I believe Ben Golub said the opposite a while back."</p>
<p>Ben Golub--who is or was on the Caltech admissions committee, as I was when I was a Caltech undergraduate--said that there were more important factors than the SAT and that other accomplishments could outweigh a poor SAT score. Of course those are both correct. Admissions decisions are made based on a whole application, not just one number, whether than number is the result from the SAT <em>or</em> an AP exam. They do not change what's being said on this thread.</p>
<p>In the end, there are many applicants that have 5's and 800's. A student with a 5 and a 670 isn't going to look "just as good", and neither is a student with a 3 and an 800. It's like that admissions cliche: "Is it better to get a B in an honors class or an A in a regular class?" (Answer: "Most admitted students are able to get A's in honors classes.")</p>
<p>I'm done on this--if you'd like to debunk, get some data to back up your College Board conspiracy theory.</p>
<p>I wasn't trying to be belligerent, so hopefully you didn't take me wrongly...</p>
<p>What appears to be "obvious" or "common sense" isn't always correct, first of all. If there is evidence of a particular theory, that is when the debunker's job begins... but you cannot base a theory on common sense. Many things in physics agrees with this statement.</p>
<p>I believe it applies here as well. You can't make a theory upon common sense because common sense is often wrong. If CollegeBoard statisticians have found that there is a strong correlation, then let them post their findings. Otherwise, I'm not convinced, and I don't think you should be either... especially when there are many anomalies in the correlation between the SAT and the AP.</p>
<p>And as for your comparison of "Is it better to get a B in an honors class or an A in a regular class?" to my argument... I think it's a bad one. It's not chronologically intact. We're comparing an easy, basic exam with a more difficult exam founded upon the knowledge you find in easy, basic exam. It's like my analogy with 1st and 4th graders: If you get a bad score as a 1st grader but a good score as a 4th grader, you know that this 4th grader has mastered the 1st grade material. If you get a good score as a 1st grader and a bad score as a 4th grader, nothing is known but that the 1st grade content has been mastered.</p>
<p>I'm done here as well.</p>
<p>Speaking of Ben, let me clarify one of my remarks that I think may have been misunderstood as of his reply (the fault being my writing, not his reading):</p>
<p>I wrote about signals and noise that "this is the essence" of admissions work and that unusual SAT variance is an example of noise. The intended meaning of "essence" was not that a fine sifting of SAT scores is the essence of admissions work, but that the essence of admissions work is extracting signal from noisy data. If your SAT data are noisy enough, it may invite some further attention beyond the usual glance to check that the numbers are sensible in the context of the application (that is my understanding, at any rate).</p>
<p>re: College Board, it is very much their incentive to reveal and correct any SAT-to-AP discrepancies, as it would totally discredit these tests for a mismatch to exist. There is also a huge incentive for the many enemies of the SAT to find and exploit any such flaw, so you can be sure people outside the College Board have looked into this as well.</p>