Selectivity Ranking: National Us & LACs combined, USNEWS ~method

College Search & Selection
61

<p>Papa Chicken,</p>

<p>I had noticed the apparent SAT-ACT discrepancy at some schools, including Brown, and initially had thoughts similar to yours. But now I’m not so sure. The pattern is more widespread than Brown, and there a couple of other explanations. </p>

<p>First, many colleges superscore the SAT, but very few superscore the ACT, instead using only the highest single-sitting ACT composite. So you’re immediately comparing apples and oranges. Superscoring is going to give you higher 25th and 75th percentile medians than single-sitting scores, i.e., higher SATs than ACTs.</p>

<p>Second, I think the way US News reports “SAT CR+M” is, statistically speaking, just wrong. Colleges don’t report 25th and 75th percentiles for “SAT CR + M” on the Common Data Set. They report 25th and 75th percentiles for the SAT CR, and 25th and 75th percentiles for the SAT M, respectively. US News simply adds each school’s reported 75th percentile CR score to its 75th percentile M score to get a figure it then reports as “75th percentile CR + M.” But you can’t assume that everyone who’s above the 75th percentile in CR at a given school is also above the 75th percentile in M at that same school, and vice versa. For example, my D is well above the 75th percentile in CR for every school in the country; but for some of the most selective schools she’s in the second quartile in M. Many others are in the same boat, and just as many are in the same situation in reverse, with significantly higher M than CR scores, above the 75th percentile in M but below the 75th percentile in CR at their respective schools. Thus it’s likely that at every school, if you added everyone’s actual CR and M scores and then took the 75th percentile of those, you’d end up with a lower score than the figure US News reports as “75th percentile CR+M,” based on simply adding together the two reported 75th percentile scores from two separate and independent rankings of student scores. The other way to look at it is: at every school in the country, fewer than 25% of students (perhaps 20%? who knows?) will have higher actual combined CR + M scores than the “75th percentile CR + M” score as reported in US News. This is so because some of those with above-75th percentile CR scores have lower-than-75th percentile M scores, and vice-versa; and in some cases, they’ll be below one 75th percentile median figure by more than they’re above the other 75th percentile median, so their actual CR + M will be lower than the school’s US News-calculated “CR + M.” </p>

<p>If you don’t beleive me, run a few simple hypotheticals, e.g., a class of four students scoring:

  1. 800CR-600M = 1400 CR+M
  2. 700CR-640M = 1340 CR+M
  3. 625CR-700M = 1325 CR+M
  4. 600CR-750M = 1350 CR+M
    75th percentile CR = 700
    75th percentile M = 700
    US News-calculated 75th percentile CR+M = 1400
    actual 75th percentile CR + M = 1350 </p>

<p>[By the same token, I don’t believe the US News-reported “25th percentile SAT CR+M” scores are accurate, either; I think the US News-reported figures are lower than the actuals; see hypothetical above where Us News would calculate the 25th percentile CR+M at 1265 while the actual 25th CR + M is 1350. In general, because many students are stronger in one area than in the other, you’ll tend to get more kids bunched around the middle when you combine the two scores, and fewer in the high and low extremes].</p>

<p>Of course, you don’t have this problem with the ACT where all that’s reported is a single 75th percentile median score for highest single-sitting ACT composite; in that case, the actual scores and the US News-reported scores should be identical.</p>

<p>Combining the effects of superscoring with the US News misreporting phenomenon outlined above—both of which would tend to skew reported 75th percentile SAT CR + M scores higher than reported 75th percentile ACT scores— I don’t think a US News-reported “SAT CR+M 75th percentile” 50 points above the reported 75th percentile ACT equivalent is at all anomalous.</p>

62

<p>

</p>

<p>Do we need it for the CDS or IPEDS or for the USNews peer survey? That is where the real action is! After all, what does all that arguing about the “misreporting” of 25-75 percentiles amounts to? An incredibly small fraction of a percent. </p>

<p>By the way, would it surprise anyone that reporting a LOWER SAT score actually might help to obtain a higher ranking? If you need a hint, take a look at the impact of that wonderful graduation expectation index!</p>

<p>PS The only way USNews should deal with SAT optional schools is to park ALL of them in a separate category that is no longer ranked.</p>

63

<p>

</p>

<p>“An incredibly small fraction of a percent” of what? Of the reported SAT scores? I don’t think so, but I’d be happy to entertain counterarguments. </p>

<p>I think US News has created a completely false and potentiially highly misleading statistic here in the way it simply adds the reported 75th percent CR score to the reported 75th percentile M score to derive a phony “75th percentile CR+M” score that may bear no relation to reality. School 1 could have 100% overlap between its top 25% CR and top 25% M scores, in which case US News’ idiot method of simply adding the 75th percentile CR score to the 75th percentile M score would give you an accurate representation of the school’s 75th percentile CR+M. On the other hand, school 2 could have an entering class in which the CR scores of its top 25% M scorers were all 100 points below its 75th percentile CR score, and the M scores of its top 25% CR scorers were 100 points below its 75th percentile M score. Yet US News with its flawed “objective” methodology would show identical “75th percentile SAT CR+M scores” for the two schools. In the latter case, however, the school’s true 75th percentile CR+M score would be approximately 100 points below the true 75th percentile CR+M score for school 1, and approximately 100 points below the “75th percentile CR + M” score reported by US News—surely enough to drop the school a dozen or more places in the US News rankings, given that SAT/ACT scores count for 50% of the US News selectivity score, which in turn counts for a full 15% of the overall Us News score—one of the largest factors, and one that looms especially large in the minds of certain commentators (e.g., xiggi) who continue to hype “selectivity” as the quintessential measure of school quality. Turns out that for a lot of reasons, the US News-reported selectivity scores are pretty bogus, far from “objective,” and not measured in a statistically defensible fashion by US News.</p>

<p>Note also that schools that are into “gaming” the US News rankings—which I contend is a common practice—could easily “game” their reported CR+M numbers by choosing half the class primarily on the basis of high CR scores, and half the class on the basis of high M scores. The actual CR+M scores of their class might be lower than a school that sought more balance, but US News and its readers would be none the wiser.</p>

64

<p>

</p>

<p>Nice catch. I had the noticed that Colgate (the school I’m most familiar with) had a slightly higher avg ACT than Brown even though its SAT median was 70 points below Brown but just figured it was an anomoly, never have understood the ACT to SAT conversion and didn’t realize it was a consistent thing.</p>

<p>

</p>

<p>I have noticed this phenomenom as well but think the discrepancy is not as great as you think. I know for Colgate in the last year or two that the median SAT for accepted students was 1415 and the avg SAT was 1403, so the difference was only 12 points but will grant you it’s makes students think their chances are lower than what they may actually be. </p>

<p>

</p>

<p>That’s an interesting observation, although I’ve never looked closely at the expected graduation rates they calculate to understand how USNWR is doing it or what weight it ascribes to it vs other components.</p>

<p>

</p>

<p>I don’t agree with this as it’s going too far to me. USNWR has already gotten into too high of position of power with regard to college education than they should be; I don’t think they should be in a position of mandating that schools require the SAT. If a college wants to not require the SAT, that should be its perogative. Said college just shouldn’t be rewarded for doing so by appearing to have a stronger student profile than it actually does or receive a higher ranking in USNWR (or any other poll) as a result of doing so. Maybe, it should do something along the lines of what it appears Middlebury is doing and force the schools to tell them what their actual SAT of the entire enrolled class is and use this number in their calculations to compile their rankings.</p>

65

<p>Gellino:</p>

<p>since 'Gate now releases it’s CDS, it’s hard to conclude that the mean gpa is 1400+ ( the number formerly published as “admissions” data).</p>

<p>Colgate’s 75th % for ACT is a 32, or ~1420/2130. It’s SAT 25%-75% are: 630-730 (CR) and 640-730 (m). Thus, (improperly adding), we’d get a 75th % SAT score of 1440, and a arithmetical mean of ~1355. Even accounting for the fact that the SAT is skewed left (for highly selective schools), 50 points is a lot to make up.</p>

<p><a href=“http://portaldata.colgate.edu/imagegallerywww/4156/ImageGallery/ColgateU_Common%20Data%20Set%202008-09.pdf[/url]”>http://portaldata.colgate.edu/imagegallerywww/4156/ImageGallery/ColgateU_Common%20Data%20Set%202008-09.pdf&lt;/a&gt;&lt;/p&gt;

66

<p>^Right, Enrolled student profile is usually ~ 40-50 points lower than Accepted student profile as it was last year: median Accepted SAT: 1415, median Enrolled SAT: 1365 (not 1355 as your 75% SAT should be 1460 not 1440). Considering that princetonreview, collegedata, collegeboard, USNWR all have had Enrolled stats for Colgate for years, I wouldn’t think there should be the confusion, although reporting Accepted, Enrolled, Applied stats the way Bucknell does is probably the best approach.</p>

67

<p>^ bluebayou,
Don’t you mean a 75th % SAT score of 1460 (=730 + 730)?</p>

<p>

</p>

<p>Well, I’d like to see some more analysis. I think particularly at the 75th percentile level these figures can be highly misleading. It’s relatively easy to find applicants with high scores in CR or M; it’s much rarer to find both. According to the College Board, 9,648 college-bound 2008 seniors scored 790+ on CR, and 10,583 scored 790+ on M. But only 2146 scored a combined CR+M 1580+. In other words, about 80% of those scoring 790 or 800 on either CR or M didn’t match that performance on the other section.</p>

<p>US News shows both Harvard and Yale with 75th percentile CR+M of 1590. By that logic, a full 25% of the entering class at each school must have scored at least one 800 and one 790 (or another 800) on the CR and M section. But 25% of Harvard’s entering class is 415 students; 25% of Yale’s entering class is 330, for a combined total of 745 CR+M 1590+ scorers at the two schools. But that’s fully 56% of the global total of 1325 1590+ CR+M scorers reported by the College Board. I find this highly improbable as it leaves precious few 1590+ scorers for Princeton, Stanford, MIT, Caltech, Penn, Columbia, Duke, Dartmouth, etc., not to mention all the other fine schools out there that tend to have at least a few stellar SAT-takers. No doubt Harvard and Yale have more than their share of double-high scorers in CR and M; and I don’t doubt for a minute that their 75th percentile CR and 75th percentile M scores are what they say they are. But 25% of their freshman classes 1590+? I really, really doubt it. I’d take all these 75th percentile CR+M scores as reported by US News with a heavy grain of salt. I think they’re all skewed high by the erroneous methodology that publication uses, and to that extent deeply misleading.</p>

<p>Unfortunately, I think the US News-reported figures are also deeply misleading at the 25th percentile level, for similar reasons. This gives false hope to thousands of applicants who in reality don’t stand a snowball’s chance in Hades at many top schools. Stanford’s 25th percentile CR score is 660 and its 25th percentile M score is 680. US News translates this to a CR+M 25th percentile of 1380. Sounds pretty good, but not not so terribly high; and a full quarter of the class BELOW that level? Wow! Makes you think you might actually have a shot if you have a 660 CR and 680 M. But my guess is Stanford’s actual 25th percentile CR+M is considerably higher, because its adcom will overlook a certain number of low CR scores if they come attached to high M scores and vice-versa. The number of Stanford admits who have BOTH a CR score below 660 AND a M score below 680 could be vanishingly small; but if 25% of its admits are sub-660 CR scorers with M scores well into the 700s and another 25% are sub-680 M scorers with CR scores well into the 700s, it’s going to show up on US News with a (misleadingly labeled) 25th percentile “CR+M” score of 1340—even if nowhere near a quarter of its entering class actually has a CR+M score that low.</p>

<p>This may not affect the rankings all that much. But to judge by what appears on CC, a lot of people seem to rely pretty heavily on US News data to gauge their chances of admission. And US News is, to put it bluntly, providing faulty data that may be quite misleading at both the high and low ends.</p>

<p>Oh, and by the way: you can get exactly the same 50% median with a 75th percentile score that’s too high and a 25th percentile figure that’s too low by a like amount.</p>

68

<p>

</p>

<p>US News is in no position to tell any school what to do regarding the SAT or ACT, but they ARE in the position to refuse to rank any school that does not fit the criteria established for their rankings. Schools that do not fit the criteria should be EXCLUDED and listed accordingly. SAT optional schools should be delisted and placed in a separate alphabetized section where they can share the honors with the others schools who play games or are too different. </p>

<p>By the way, should all the schools do what Middlebury is doing? Does that mean included all applicants in the numbers submitted to USNews but only use the Fall enrollment to define the admission ratio, and “forget” the winter enrolled students?</p>

69

<p>

</p>

<p>Beg your pardon, Clinton! </p>

<p>While I can excuse you for NOT knowing what “looms large in my mind,” I think you should be a lot more careful in ascribing such conclusions to “certain commentators” without making the effort to either read what they wrote, or try to understand their argument. </p>

<p>Have I really “continued to hype “selectivity” as the quintessential measure of school quality?” Am I even saying that “selectivity” is the quintessential measure of school quality? </p>

<p>I believe you to be extremely mistaken about my position on selectivity, and surely about what I believe selectivity means.</p>

70

<p>

</p>

<p>I meant that if you’re not going to require the SAT to at least include the the scores of the entire enrolled class in your averages. I don’t know what they’re doing with the Feb-frosh data. Midd supporters claim that they’re the same quality level; theoretically at least, I’ve always begged to differ.</p>

71

<p>^^ Apologies, xiggi. Perhaps I misunderstood or am misremembering some of your earlier posts. I stand by my comment as regards some posters on CC, but I should not have attributed to you a view that you apparently do not hold. It was a cheap shot, and uncalled for.</p>

<p>As to delisting SAT-optional schools, I think there’s something to be said for it, but that would go only so far to cure the defects in the US News data. As long as some schools superscore the SAT and others don’t, you won’t get truly comparable data. As long as some schools rely on one test and some another, the data won’t be comparable, either—the ACT-SAT concordance is imperfect at best because these are quite different tests that measure different things. For that matter, as long as colleges allow students to submit either test, you’ll have SAT and/or ACT percentiles based on less than 100% of the entering class, and you won’t truly know what it means to compare the SAT data at a school where only 74% of enrolled freshman reported an SAT score (e.g., Amherst) to the figures at a school where 100% of the enrolled freshmen reported an SAT score (UC Berkeley). It makes all this 25th-75th percentile stuff little better than gibberish. And US News’ erroneous method of calculating 25th and 75th percentiles might also create additional opportunities for creative gamesmanship on the part of enterprising schools looking to boost their nominal selectivity rankings.</p>

72

<p>supporting bc’s point about the statistical problem with USN’s practice of adding the SAT M quartile & SAT CR quartile, to represent the 25-75 range for combined scores, I recall one college’s factbook that reports both ways…Colby.</p>

<p>Here’s their 2008 factbook:
<a href=“http://www.colby.edu/administration_cs/ir/factbook2008/upload/Admissions.pdf[/url]”>http://www.colby.edu/administration_cs/ir/factbook2008/upload/Admissions.pdf&lt;/a&gt;&lt;/p&gt;

<p>Here are the data for 2007:</p>

<p>2007 SAT CR 25%-50%-75%: 640-690-720
2007 SAT M 25%-50%-75%: 640-680-720
so, by the USN method, simply adding the separate totals together for 25% and 75%, they’d get: 1280-1440 (which I believe they did, don’t have that USN edition in front of me)
but Colby reports this combined total:
2007 SAT M+CR 25%-50%-75%: 1310-1370-1430
…30 point closer (higher) to median on the low end, and 10 points closer (lower) on the high end.</p>

<p>Same story, different variances, for each other year reported.</p>

<p>I’d expect that schools that tend to admit applicants less balanced in SAT scores (higher score variation between one student’s M & CR score) would have larger variances between ‘real’ SAT M+CR quartiles compared to the USN additive method.</p>

73

<p>No problem at all, Clinton. And, while I appreciate the apology, there was no need for it. I believe that the misunderstandings arise because most anyone who spends time debating the USNews ends up with some subtle contradictions. </p>

<p>For instance, I believe that USNews contains a number of misleading or counter-productive elements. Of course, that is MY opinion. I understand that for others the same “ingredients” I view with dismay are perfectly acceptable. To give an example, the expected graduation rates that are a direct result of the selectivity indexes assigned values of 99 to Harvey Mudd and (if I remember correctly) 96 to Pomona and Swarthmore. In turn, their lower real graduation rates yielded a substantial penalty. A few years ago, this caused Harvey Mudd to be DEAD LAST in this section since they had a negative 18 (99 versus 81.) I doubt that anyone would find very logical that the most selective LAC is supposed to graduate 99% of its students and maintain one of the the highest difficult program in the engineering in the nation. By the way, Caltech suffered similarly but found its graduation expectation number lowered. </p>

<p>All in all, we end up with this love-hate relationship with USNews. I know that I see plenty of problems, but do not see anything better and more cost-effective that the USNews rankings. I deplore their reluctance to make the ranking more user friendly through self-selected data that would allow the user to change the weights of the columns. I also deplore their lack of interest to weed out the gamers and condone the manipulation of the data by some. </p>

<p>But, in the end, it beats trying to get the data from the school directly and printing hundreds of CDS reports.</p>

74

<p>

I believe the new NRC rankings are supposed to incorporate this feature.
Perhaps USNews could enhance their web subscription to the rankings.</p>

75

<p>^^^ Thanks for the confirmation, Papa Chicken. The US News 2009 edition uses 2007 SAT data, and just as you predicted/remembered, they report the 25-75th percentile SAT CR+M for Colby as 1280-1440. Not far from the actual on the high end, but at the low end that 30 point difference could be huge for some poor fellow trying to decide whether it’s worth taking a shot. Unhooked and 1310 or below, I’d say not, especially with that 31% acceptance rate; but relying on US News the kid with a 1300 or 1310 would erroneously think his SAT scores fell somewhere in the middle 50% (well, lower-middle 50%) of enrolled freshmen. </p>

<p>

</p>

<p>UCB, any word as to when the NRC rankings are finally coming out? Will this happen in our lifetimes?</p>

<p>xiggi, I think your suggestion about user-friendly data selection tools is excellent. I should think this would make the online version of US News even more popular. Any idea why they’re reluctant go this route?</p>

76

<p>There’s something seriously wrong with you people</p>

77

<p>

</p>

<p>Actually, not only are they seemingly reluctant to add such feature, but have gone the opposite route. One added benefit of subscribing to the online edition was to be able to download the tables and (with a bit of effort) make them useful in an Excel format. They killed that ability by introducing a more graphic format and preset “weights.” The result is an idiot-proof presentation that is cumbersome and annoying. </p>

<p>Further, a few years ago, I discussed with Morse about offering “professional” versions of the rankings that would be sold via CDs (or downloadable) and contain the historical tables. I am pretty certain that many would not spending $25 for such product, or even more if ot would give access to additional background or raw data. </p>

<p>So far, I think comments from users are falling on deaf ears. We will soon know if Morse paid attention to the hate mail he must received about this “upgrade.” The soon part can be quantified on their website with that obnoxious Times Square look-alike clock. :)</p>

78

<p>Fascinating and informative thread.</p>

79

<p>I agree. Thanks for reviving it Marsden.</p>