SAT pointlessly long, Study finds- Could be partially replaced by 15 minute test

Parent Cafe
1

<p>A study finds that the SAT-V portion may be replaced with a 15 minute, 24 question test without any significant loss in predictive validity, meaning it's just as good as the current SAT-V in predicting freshman grades. The abstract follows:</p>

<p>Does the SAT Have to Be So Time Consuming?
Joseph F. Fagan and Cynthia Holland
Case Western Reserve University and Cuyahoga Community College, respectively.</p>

<p>The present study found that scores derived from a 24 item, 15-minute version of the type of questions typically asked on the SAT-V are as predictive of academic performance in college as are standard SAT-V scores. The purpose of the Scholastic Assessment Test (SAT) is to predict how well a student will do in college courses. Currently, questions have been raised in a series of articles in the New York Times as to the amount of time necessary to complete the SAT which is estimated at 3 hours and 45 minutes (longer for disabled students). Earlier reports in the New York Times also revealed a sizable error rate in the current scoring process for the SAT. In the present study we asked if a very brief, easily scored version of the SAT-V would be as predictive of college grades as the standard SAT. The sample included 502 students at three private universities attending introductory classes in Psychology. Questions of the sort traditionally tested on the SAT-V were taken from practice tests for the SAT-V to create a Brief Verbal Test which included 24 items, 8 of which tested knowledge of the meanings of words, 8 the knowledge of opposites, and 8 the knowledge of analogies. Measures of specific academic achievement (exam performance on objective tests) were obtained from the instructors of the Psychology courses. Cumulative grade point averages representing at least two semesters of attendance were also obtained from the registrar at each institution. Both the SAT-V and the Brief Verbal Test were successful in predicting academic performance. The predictions from each test to academic performance were virtually identical. No significant difference was found between the SAT-V prediction of class grade or the Brief Verbal Test’s prediction of class grade. The same was true for the predictions of each measure of aptitude to cumulative GPA. The present results tell us that scores derived from a brief version of the type of questions typically asked on the SAT-V are as predictive of academic performance in college as are scores derived from the SAT-V. Of course the SAT consists of more than the verbal portion of the test. There is a mathematical section to the SAT, the SAT-M. Perhaps a brief version of the SAT-M would also prove to be as predictive of academic performance as the lengthy standard version. That remains a question for further study. What we do know now is that a predictive, less time consuming, and more easily scored version of the SAT-V is possible.</p>

<p>Supported, in part, by a Leffingwell Professorship (Joseph F. Fagan), by an NIH grant under the Initiatives for Minority Students: Bridges to the Baccalaureate Program, 2R25 GM49010 (Cynthia R. Holland), and by a Contract from the Army Research Institute, W74V8H05K006 (Joseph F. Fagan).</p>

<p>By the way, the SAT has a freshman grades correlation of about .6-.8 ( <a href="http://www.collegeboard.com/repository/rdreport200_3919.pdf%5B/url%5D"&gt;http://www.collegeboard.com/repository/rdreport200_3919.pdf&lt;/a> p.32).</p>

2

<p>Though that is interesting, I'm not sure how accurate the study actually is. For one, I do not believe that SAT stands for Scholastic Assessment Test. That error really casts anything else they write in doubt, since it is such a simple matter. Second, this abstract is speaking as if the goal of the SAT is to predict college success, and I'm not sure that is the exact goal. The collegeboard website states: "The SAT assesses how well you analyze and solve problems—skills you learned in school that you'll need in college." Further, just because people with high grades score highly on both tests (and the other way around) doesn't mean that the tests are equal: I assume that there are numerous types of testing that would show the same correlation between score and GPA. </p>

<p>I think most of us recognize, though, that the SAT could be shorter.</p>

3

<p>Do they have this mini-test? I would like to take it.</p>

4

<p>I'm sorry but a 24? test?!? Ridiculous... I'd take it every chance I'd get then.</p>

5

<p>
[quote]
For one, I do not believe that SAT stands for Scholastic Assessment Test.

[/quote]
</p>

<p>SAT stood for "Scholastic Aptitude Test" for a LONG time until the College Board unilaterally declared that henceforth "SAT" would stand for nothing. Kind of appropriate, actually. </p>

<p>
[quote]
Second, this abstract is speaking as if the goal of the SAT is to predict college success, and I'm not sure that is the exact goal.

[/quote]
</p>

<p>If the SAT doesn't allegedly predict success in college, why would admissions officers care about it? Might as well have all candidates record their scores on -- say -- the 10 minute run and give preference to the fastest. I think you made a fairly bizarre assertion here.</p>

<p>(I actually like the 10-minute run idea. Think of all the new affirmative action possibilities, and the rush to track camps in well-to-do suburban areas. Might be the best thing possible for the health of our utes, too. No more staying up all night eating junk food to eke out the last 10 points on a test that doesn't actually claim to measure success in college. Students will spend their evenings playing computer games and their afternoons running. Works for me.)</p>

6

<p>
[quote]
For one, I do not believe that SAT stands for Scholastic Assessment Test.

[/quote]
</p>

<p>It did at one time. They keep changing it! It also used to stand for Scholastic Aptitude Test. Now it stands for nothing.</p>

<p>Edit: I see I'm repeating info. Oh well!</p>

7

<p>Yes. I was thinking that it was an old acronym, but I wasn't positive. It just worries me that a current report would use an improper name for the test about which it studies. </p>

<p>WashDad, I think that the SAT does work as a predictor of college success; however, I do not believe that is the precise goal of the College Board when devising the SAT, at least I've never heard them say as much. I believe the first goal of the SAT is to assess reasoning skills by testing the application of high school topics (that's paraphrased from the website). Colleges use this measure to predict collegiate success; however, the SAT's primary goal is the measurement of reasoning skills.</p>

8

<p>You guys are kidding me!!! "SAT" doesn't stand for anything? What about GMAT? LSAT? That is hilarious!! And PSAT? I can't deal with this. My world is turning upside down. Next your gonna tell me that ROFLMAO doesn't stand for anything either! This is surely the end of civilization as I know it.</p>

9

<p>I have not read this study but I work in this field. One of the first things that comes to mind is that tests typically include some number of field test items that automatically increase the length of the test. As an example, a 40 item test might include 7 field test items that are not used in computing the operational score. Field testing is used to determine the psychometric properties of proposed items including the discriminant validity of the item (typically using point biserial correlations) and that the item is relatively free from bias (differential item functioning). It is very difficult for item developers to anticipate how examinees will respond to items so field testing is a necessity. It is also conceivable that SAT protocols will drop items that differ by more than .3 logits from the field test difficulties or some other reason may be used to drop items from operational scoring.</p>

<p>Another observation is that if I were constructing a practice test, I would select items that have not only been used on tests but have good predictive validity and stability across administrations. I wonder if it is possible to repeat the results of these researchers on several test administrations per year.</p>

<p>Finally (and I could go on for hours) I wonder if the researchers have considered the domain that the SAT is intended to assess? On the tests that I work with, we can only sample a little more than half the domain we would like to sample. It is concievable that one might construct a short test of items that "hang together" resulting in high reliability and small standard errors of measurement, but they are not sampling across the domain. How did the researchers deal with that problem when selecting the items ex post?</p>

10

<p>URMs would take the test 19 times until they got the right result and ETS would then pretend the first 18 attempts didn't occur</p>

<p>Is this not the purpose of nearly all such studies?</p>

11

<p>When my DD was in eighth grade she took a computerized version of the SAT for talent search purposes. The computer automatically adjusted the difficulty of the questions it asked based on how well the test taker was doing on the questions. She completed the entire SAT (math and verbal then, no writing component) in one hour and did very well. (She also got her score as soon as she completed the test.) I always thought that the SAT of the future would be administered on a computer. Perhaps the logistical problem of giving so many tests on the computer are preventing that from happening, but it sure would make life easier for the students. I'd let them type their essays while we are at it!</p>

12

<p>The horror of actually having to sit down and take a 3.5 hour test - what a terrible fate for those seeking to go to college</p>

13

<p>
[quote]
For one, I do not believe that SAT stands for Scholastic Assessment Test. That error really casts anything else they write in doubt, since it is such a simple matter.

[/quote]

They probably picked that up from an old research paper. </p>

<p>
[quote]
Further, just because people with high grades score highly on both tests (and the other way around)

[/quote]

Their point was that the SAT makes a prediction, and a shorter version makes an identical prediction, so one could substitute the latter for the former while maintaining the same level of predictive power. If a $5 calculator can do what you want just as well as a TI-83, why spend $100 on the latter when the former will do just fine?</p>

<p>As for 'measuring skills,' as the goal of the SAT: Why would the college care what your skills/scores are, independent of their ability to predict your performance. Put differently, the number divorced from the prediction is meaningless and worthless. Its entire value stems from its predictive power.</p>

<p>I don't quite understand your questions, due to your use of jargon with which this layman is unfamiliar. After I posted it, I wondered, does a shorter test necessarily lead to a loss in the ability to discriminate/distinguish the upper and lower ends of the testers, i.e., the 1550 becomes identical to the 1600? </p>

<p>
[quote]
URMs would take the test 19 times until they got the right result and ETS would then pretend the first 18 attempts didn't occur</p>

<p>Is this not the purpose of nearly all such studies?

[/quote]

This applies to anyone who scores low. (Ironically, it might improve the predictive power of the test to only allow people to take it once, but then College Board would lose revenue.) The general reason is probably to save everyone's time and effort. If you could replace the MCATs, GMATs and LSATs with rather less consuming tests, why not? I sent you a message.</p>

<p>mother<em>of</em>perl,
The military for its Armed Forces Qualification Test (AFQT) has shifted to such a system. I believe it's a bit more accurate, as those who are overwhelmed are less likely to give up. I've also taken one such test for placement at a community college.</p>

14

<p>There is a test called the Miller Analogies Test, takes an hour, has a raw score that is the number of right answers. EVERY person I know who's taken it (an entire class of teachers--about 60 people, all with college degrees) got the same percentile on the MAT that they did on their SAT average. <a href="http://harcourtassessment.com/haiweb/Cultures/en-US/Harcourt/Community/PostSecondary/Products/MAT/mathome.htm%5B/url%5D"&gt;http://harcourtassessment.com/haiweb/Cultures/en-US/Harcourt/Community/PostSecondary/Products/MAT/mathome.htm&lt;/a&gt;&lt;/p>

<p>Of course, the SAT no longer has any analogies.</p>

<p>My hunch, however, is that a similar test could easily be developed.... but then people would say "how can college admission depend on a simple test that only takes an hour? Shouldn't it be more extensive?"</p>

15

<p>The unstated goal with such studies w/o any doubt is to destroy the SAT test. ETS has already had to spend years to revise many questions in the past to absolutely make sure there was no bias in the test, when in some cases certain legitimate bias issues were presented. Imagine a test which is now 1/12 the length of the original test - and someone or some group claims real, subconscious, or hidden "racism" or "bias" or "insensitivity" on a single question - then the SAT test would certainly be gone</p>

<p>A student that plans to spend 4 years in college should actually have the capability to sit still for 3.5 hours to take an exam - or (as I suspect) have we now reached the point where the the majority of prospective college students will now claim to have ADD/HD or some similar learning disability not permitting such lengthy test taking?</p>

<p>Should the bar exam now be only an hour? Two days is too much anyways</p>

<p>How about the CPA exam in 20 minutes?</p>

<p>MCAT tests for medical school - 10 minutes s/b sufficient -doctors are always on the move anyways</p>

16

<p>"The initials SAT have been used since the test was first introduced in 1901, when it was known as the Scholastic Achievement Test and was meant to measure the level achieved by students seeking college admission. The test was used mainly by colleges and universities in the northeastern United States. In 1941, after considerable development, the non-profit College Board changed the name to the Scholastic Aptitude Test, still the most popular name. The test became much more widely used in the 1950s and 1960s and once was almost universal.</p>

<p>The success of SAT coaching schools, such as Kaplan, Inc. and the Princeton Review, forced the College Board to change the name again. In 1990, the name was changed to Scholastic Assessment Test, since a test that can be coached clearly did not measure inherent "scholastic aptitude", but was influenced largely by what the test subject had learned in school. This was a major theoretical retreat by the College Board, which had previously maintained that the test measured inherent aptitude and was free of bias.</p>

<p>In 1994, however, the redundancy of the term assessment test was recognized and the name was changed to the neutral, and non-descriptive, SAT. At the time, the College Board announced, "Please note that SAT is not an initialism. It does not stand for anything."</p>

<p><a href="http://en.wikipedia.org/wiki/SAT%5B/url%5D"&gt;http://en.wikipedia.org/wiki/SAT&lt;/a&gt;&lt;/p>

17

<p>I certainly think it's worthwhile to look for better, shorter assessment tools, if we must have them. The sample used for the study cited by the OP is small, only about 500 and composed of private University students. This means they've all done well enough to be admitted. The kids taking the SAT are a huge diverse group whose scores are all over the map. See what I mean? This was a very narrow sample, maybe an ok place to start, but it doesn't prove anything.</p>

18

<p>
[quote]
A student that plans to spend 4 years in college should actually have the capability to sit still for 3.5 hours to take an exam.

[/quote]
</p>

<p>It actually takes five hours or more to administer the current version of the test, when you add in time for instructions, administrative tasks, and breaks. </p>

<p>Many high school students have little or no prior experience with tests of this length. Indeed, even if you wanted to give a five-hour exam on a regular school day, the law wouldn't allow it since students who are in school for four hours or more are entitled to eat lunch!</p>

<p>My daughter, a high school senior who has taken four AP exams and two IB exams, says that the only other tests that match the SAT in intensity are AP foreign language tests (which are especially intense because the material you are concentrating on for such a long period of time is not in your native language). For her, even the occasion on which she had to take the AP BC Calculus exam and the first half of the IB SL Math exam both on the same day was not as bad as the SAT.</p>

19

<p>I suspect the conclusion is correct. If you were able to look at current aggregate test results question by question, you could throw out the questions that nearly everybody answers correctly and just go to the ones that help you do what the test is supposed to do...................discriminate. </p>

<p>Of course, if you reduce the number of questions, then each question would be very important, and advocacy groups wishing to "job" the test have a much clearer target.</p>

20

<p>
[quote]
Imagine a test which is now 1/12 the length of the original test - and someone or some group claims real, subconscious, or hidden "racism" or "bias" or "insensitivity" on a single question - then the SAT test would certainly be gone

[/quote]

ETS is already quite scrupulous about it; I haven't heard of any recent 'scandals' over possibly biased questions. Also, you have a rather distorted view of the field of psychometrics, a field constantly wracked by controversy. I suggest you do a bit of research into it. </p>

<p>The purpose of such tests generally is to predict outcomes as best as possible, whether it's college freshman grades or workplace productivity ( <a href="http://www.owlnet.rice.edu/%7Epsyc231/Readings/schmidt.htm%5B/url%5D"&gt;http://www.owlnet.rice.edu/~psyc231/Readings/schmidt.htm&lt;/a> ). ETS due to public pressure, has secondary considerations, but has still retained the general form of the test, despite pressure to change it so as to eliminate gaps between groups.</p>

<p>I'm not so sure that the ability to concentrate for 3.5 hours is all that important if the outcome, the prediction, is the same (though in and of itself, it's very important). Also, a 15 minute test might remove the need for time extensions for those supposedly with ADD.</p>

<p>There's a difference between the Bar and the LSATs, in that the former is a test of the knowledge required to practice law, while the latter is a test of ability. A short test is inappropiate for tests of knowledge, but not necessarily for ability. The MCAT is an entrance examination, a test of ability. Were you to find a shorter but equally predictive one, why not? I suppose it all hinges on whether you deem the ability to concentrate for long hours one to be tested?</p>

<p>I'm for the shorter test, if it shows to be as good as the SAT-V, to reduce the frenzy that afflicts its takers.</p>

<p>
[quote]
advocacy groups wishing to "job" the test have a much clearer target.

[/quote]

Can you explain? What sort of advocacy groups?</p>

<p>I also wonder whether because the reduced number of questions, there would be more people with say, 750/800s because there is a higher probability of blindly answering them all right. The study suggests that this had no effect though.</p>