SAT: Part hoax and part fraud?

<p>

I’m guessing you calculated the 0.88 correlation with math SAT yourself, so there is not an external reference? In any case, I disagree with the conclusion. A correlation between grad rate and math SAT score means that selective colleges tend to have both higher math SAT scores and higher graduation rates, not that one causes the other, just as selective colleges also tend to have students with higher HS GPA and more rigorous classes. How do you know that the SAT score was the primary cause, rather than HS GPA and rigorous classes, or a combination of other factors? What if a college started ignoring SAT by becoming test optional and allowing lower SAT students to be admitted who were qualified based on GPA, but not test score? Your conclusion implies that the grad rate for that college would drop significantly. A few weeks ago a study was published that looks at exactly this question by analyzing 123,000 students over 33 colleges, looking at grad rate over large groups of students instead of just individuals – <a href=“http://www.nacacnet.org/research/research-data/nacac-research/Documents/DefiningPromise.pdf”>http://www.nacacnet.org/research/research-data/nacac-research/Documents/DefiningPromise.pdf&lt;/a&gt; . Their overall conclusion is quoted below:</p>

<p>“Few significant differences between submitters and non-submitters of testing were observed in cumulative GPAs and graduation rates, despite significant differences in SAT/ACT .”</p>

<p>“The kids are doing something actively different at those better schools, which results in higher scores and, eventually, higher income.” More likely, the kids are doing something actively different at home. We know that kids can attend the exact same school with different outcomes for different income levels. Better students in, better test scores out, gives you a “better” school.</p>

<p>Our school system defines “economically disadvantaged” as one of several special groups for which they need to “close the achievement gap”. By the end of 3rd grade, nearly twice as many “economically disadvantaged” students fail the reading and the math tests as compared with the overall failure rates. The kids are attending the same schools. </p>

<p>The nacanet.org study is not a serious statistical analysis. The authors are not trained in statistics, and It is not peer-reviewed. But put that aside for a moment and just ask yourself how you would judge applicants when one group submits a potentially useful piece of information (SAT test results) while one chooses not to. The first thing you would do is to infer that those who did not submit the SAT test results scored lower. (This is shown in the Dickert-Conlin et al paper cited above.) You might then hold the latter group to higher standards in other dimensions, be it grades, letters or ECs. In fact, some of the test-optional schools do that explicitly, requiring a minimum GPA or additional material. Another point is that applicants also self-select. A final point is that there are different grading standards across majors, so we cannot say whether grades are the same adjusting for difficulty of curriculum. There is so much scope for selection bias and unobserved heterogeneity that the results they claim are of little value. </p>

<p>Can I just mention that people should be looking to the parents, not the money. The money is one residue of what the parents are doing to produce the certain kinds of structure where the kids are more likely to succeed.</p>

<p>

The study also looked at schools that offer an auto admit based on class rank or GPA policy, so the test scores, ECs, and other factors had no impact on the decision. Comparisons between these effective submitter/non-submitter groups followed a similar pattern in that the grad rate correlated much more closely with HS GPA than test score.</p>

<p>

They do look at major distribution between different groups. The most shocking part of the distribution to me was how few engineering, math, and hard science majors there were in either group – submitter or non-submitter.</p>

<p>If you don’t like this study, many others are available, including some that have been referred to in this thread. Every study I’ve ever seen that had controls for both HS GPA and a measure of HS quality / course rigor came to a similar conclusion – SAT score added little beyond these factors in predicting grad rate, college GPA, chance of dropping out of STEM majors, etc., whether it be at an individual level or a group level (test scores generally had a benefit, but was small compared to the other available information). However, if a study looks at SAT alone without controls for other criteria that are available to colleges, such as HS GPA, it finds a more notable correlation and often comes to the opposite conclusion.</p>

<p>Data10-
I did the .88 correlation calculation myself based on data from the IPEDS website. I wouldn’t say that SAT “causes” a higher graduation rate nor would I say that HS grades or family income “cause” higher graduation rates. I am saying that SAT scores are an excellent predictor of aggregate student outcomes. But, are we really looking for the “cause” here of better student outcomes? If so, we are all barking up the wrong tree.</p>

<p>If you are looking for the real cause of better student outcomes, you should look first at genetics and then at the quality of parental care during the first 5 years of life. You are more likely to find the real cause there.</p>

<p>In your earlier post, you wrote that, “What this means is that colleges can raise their graduation rates by raising SAT scores.” Maybe raising SAT scores leads to higher graduation rates is a more accurate summary of the statement than causes.</p>

<p>If a selective college wanted to increase SAT scores, I’d expect they’d do so by admitting high scoring students over applicants who had other factors that led admissions to overlook the mediocre test scores under the old policies; such as a student who has a history of top grades in AP/IB-level courses, but had an off day when he took the SATs or just is not good at this type of multi-hour test. In this example, the increased SAT score comes at the expense of a reduced GPA and course rigor, which the referenced studies in this thread suggest it is likely to have the opposite of the intended effect and decrease graduation rate. This was my main point, SAT score isn’t the underlying factor that leads to the higher graduation rate, so modifying admissions policies to increase SAT at the expense of other areas may lead to a different result.</p>

<p>@Data10, well alternatively they could hold the line on the grades, and admit fewer kids who can juggle while riding a unicycle.</p>

<p>@mathyone - You stated, “More likely, the kids are doing something actively different at home.”</p>

<p>I completely agree with your point too. One of my earlier posts specifically looks at activities in the home that I believe are instrumental to high scores. </p>

<p>But to clarify what I meant in doing something actively different at school, what I was thinking on are the simple, important things - the kids behave, actually listen to the teacher, and make good use of the on-the-ground school resources. And these, in my view, are the more likely stuff learned and brought from home that allows the kids to take full advantage of school environment. </p>

<p>I will be a broken record here, but good behavior and the culture of education the kids bring to school and constructively use at home matter the most. Get those correct and high scores here we come. I do not think it is that hard or magical what is going on to do well on tests.</p>

<p>A home environment that values reading in particular and education in general is more of a predictor of success on the SAT that household income.</p>

<p>On one of these SAT threads someone commented that the average SAT score of 1715 for kids in the $200K+ income bracket isn’t all that high. That score isn’t going to get a kid into highly selective colleges. And if I’d paid thousands of dollars for pricey SAT prep tutors believing the often-repeated hype about how money for expensive preps gets you a high score, then gotten a result like that, I’d be pretty unhappy. I don’t know what fraction of those kids got pricey SAT prep, but if these families care anything about the SAT and have so much money, perhaps quite a few of them did. Another interesting point is that 1715 is actually lower than the average score at many high schools. There are around 50 high schools in California which have higher averages than that, going up to 1999. <a href=“Top Average SAT Scores - California Schools Guide - Los Angeles Times”>http://schools.latimes.com/sat-scores/ranking/page/1/&lt;/a&gt; Do all the kids from those high schools have incomes in a range well over $200K, and all have private SAT tutors? I don’t know.</p>

<p>I posted that number, mathyone (actually, 1714, I think). It came from the article in the New York Times magazine this past Sunday. It’s available online currently.</p>

<p>It seems to me that the collection of data with $200,000+ as the highest bracket obscures an issue of significant interest for admission to universities where the scores really matter. How long does the income correlation persist, as household incomes rise? And how high does the average get?</p>

<p>Many cheesy commercial sites that ask about our household income have a top category that starts higher than $200,000+.</p>

<p>@EarlyAction “1) All rich people aren’t smart. All those who are poor aren’t uneducated.”</p>

<p>So you are saying that there are no smart rich people. You are also saying that there are no poor, uneducated people.</p>

<p>Maybe you meant “Not all rich people are smart.” Well, no kidding. These studies are talking about averages. People with higher incomes are, ON AVERAGE, smarter than people with lower incomes. This should come as no surprise.</p>

<p>@QuantMech, 1715 was from a few years ago. Perhaps they don’t have enough respondents to look at higher income brackets. It’s a very good question though. There are places in this country where two married kindergarten teachers are in the $200k+ income bracket. The scores are still going up very linearly with income at $200k; they are not leveling off. School averages in very wealthy communities seem to top out at roughly 2000.</p>

<p>I’m wondering, if the college board suddenly thinks free SAT prep is so very important, why haven’t they posted their own SAT prep book on the web for free downloading? The students who took those tests already paid for their development with their test fees. Why should the college board be profiting from selling them?</p>

<p>

</p>

<p>If a college automatically admits students who are in the top 4% of their high school class, it is hardly surprising that the SAT does not explain too much of the variation in outcomes. These are students who have shown they can play the game. Peyton Manning was slow when he came out of college, but that doesn’t mean scouts should ignore how fast Johnny Manziel is. </p>

<p>

</p>

<p>Indeed, I quoted the Rothstein study, which found a smaller SAT effect than others did. I am not disputing that serious studies find a small effect, especially after controlling for SES variables; rather, I take issue with the nihilist position that the SAT provides no information. If it provides some information for some students, it is worth using. Suppose that unhooked students at your college have a 95% graduation rate and average GPA of 3.42. If I have a tool that helps me find a group of otherwise equivalent unhooked students who will graduate at a 96% rate and have a GPA of 3.44, I will use it. In fact, even if the tool found a group with a 95% graduation rate and a 3.42 it would be useful, because the applicants who would be crowded out would be weaker than the average. </p>

<p>The studies that are more convincing are the ones that look at a single college as they do not need to control for differences in grading standards across colleges or differences in admissions policies. One such study is by Arcidiacono et al, which uses data from Duke. The authors find, for example, that SAT scores are negatively related to the change in GPA and the change in class rank. The reason, though, is that low-SAT scorers are more likely to switch majors and go into less-demanding fields. (The authors argue that differences in academic background can account for differences in switching behavior.) <a href=“What happens after enrollment? An analysis of the time path of racial differences in GPA and major choice | IZA Journal of Labor Economics”>What happens after enrollment? An analysis of the time path of racial differences in GPA and major choice | IZA Journal of Labor Economics; </p>

<p>Regarding the idea that SATs “added little… to predicting graduation rates”, in stepwise multiple regression the decision about which variables to enter first should be based on a theory but is often arbitrary. So, one could say with equal justification that HS GPA adds little to the predictive power of SAT.</p>

<p>If you want a causal analysis, you could probably start with factors that affect the neurological and cognitive development of children before birth and during the few years after birth. By the time a child finishes first grade, you probably know enough to predict with some accuracy who will succeed in college.</p>

<p>

</p>

<p>Too logical. :)</p>

<p>I wanted to correct something in my last post. Above $60K, the scores are going up from 10.5 (I just took half of the 21 for the 160-200K range) to 38 for each additional $20K income. The 10.5 is at the low end of improvements, so perhaps the income effect is leveling out. But the $200K+ group shows an additional 89 point increase–that step is substantially more than any of the other groups over $60k are showing over the next lower group. It’s possible the advantage has begun to level out at $200K, but it seems that the advantage is still increasing at incomes quite a bit higher than $200K. </p>

<p>I doubt anything magic happens right at $200K. If you take the score increase of 21 for the $160-200K income range, and go up linearly from there, using 21 for each additional $40K income, you reach a score of 1709, close to what is stated for $200K+, at $320-$360K income. With this pattern of score increase, you could obtain an average score close to 1714 if you had incomes in the reported $200K+ group evenly distributed over the range $200K to about $460K (score would be about 1783 at $460K). These are just simple estimates of course. It would be interesting to see what the figures actually are.</p>

<p>@glido - I agree. I still have no idea why people think it is income, as if giving people more money would make them value reading or education more. Sounds more like a blame society argument rather than focusing on the behavior of the people who are not scoring as well as others. I met a good bit of not wealthy and poor kids in school with fantastic scores. One thing they had in common was parents on their butts about quality of their work in school. Gees, even in college the parents were on them. </p>

<p>It’s lack of critical thinking skills. People see that scores increase with income and they jump to the conclusion that this is because higher income people are buying expensive test prep, and then draw the further unfounded conclusion that you need to spend a lot of money to be successful on the SAT.</p>