Peer Assesment Rank

College Search & Selection
101

<p>USNEWS is nonsense. For many, many reasons. The notion that admissions deans are experts on academic quality of lots of other universities about which they have little knowledge is just one of the problems. Admissions deans are usually not even faculty members, so their knowledge about academic quality even at their own institutions is suspect at best. However, they do probably know more about this than do people with no association with higher education who likely have never set foot on campus. So if the choice is between the opinions of admissions deans vs man on the street for academic quality of colleges, admissions deans probably win. However, neither ranking is likely to be particularly informative.</p>

<p>Hawkette, Of course everyone is entitled to an opinion. This is America. </p>

<p>But if you know that the people offering the opinion don't know anything about, say, Classics, and for that matter cannot even read Greek or Latin, then why would you be interested in their opinions about which college has the better Classics department? The only thing of which you can be certain is that they have no idea what they are talking about. </p>

<p>Ask a thousand people who know nothing about Classics for their list of top Classics departments. You will get a list. The rankings may be internally consistent. That still does not mean they have anything to do with how good the departments really are.</p>

<p>The NRC asks English faculty about English departments, mathematicians about math departments, etc. So it is at least reasonable to expect the respondents to have useful knowledge about the fields they are rating.</p>

<p>I doubt "bias" is a significant problem, faculty members view other faculty members, or perhaps other departments, as colleagues and competitors. The notion that they define themselves by their university is more a CC fiction than reality. </p>

<p>Since the USNEWS rankings are garbage anyway, hardly worth much time worrying about imagined bias in this one factor.</p>

102

<p>afan,
Not to beat a dead horse, but....</p>

<p>Ask 100 employers about how well prepared the students are coming out of XYZ national university or ABC LAC. I believe that employers know a lot more about this and frankly, they have to because they are in business to make money and the people that they hire are commonly the most important decisions that they make. They won't know all colleges (just as the academics now being polled don't know all colleges), but they can see first-hand in an environment that has accountability whether the graduates from XYZ are as good as those from ABC or whether they are as good as the reputation of XYZ school. If I am a high school student looking for a college and I plan to work after college, knowing how employers view various colleges and how well prepared those students are would be of great, great value.</p>

103

<p>
[quote]
1) Inherent bias
2) Impossibility of "knowing" / having an expert opinion on dozens and dozens of colleges
3) Even if they did, it's not even their job to know, so why should we care about the Chancellor of XYZ's opinion on the relative merits of a Dartmouth vs. University of Wisconsin... opinions are just opinions they are like a-holes, everyone has one.
4) The lack of transparency (who actually is voting, how did they vote?)
5) No way to keep USNWR "honest", no way of double checking their numbers

[/quote]
</p>

<p>Okay, I'm going to take a crack at this. Points 1 through 3 simply demonstrate that any poll of opinions will be imperfect. Not useless, not grossly inaccurate - just imperfect. For example, I don't doubt that USC and Notre Dame suffer in PA because they are known for their sports teams, which some people feel is antithetical to the academic mission of a "serious" university. Still, the PA's published by USNWR are probably a generally pretty accurate reflection of the overall judgment among people in a position to have a somewhat informed opinion regarding the overall academic quality at the various schools. Nothing more, nothing less. It's a datum with some informative value.</p>

<p>4 and 5 are just silly conspiracy theory rants. As to 4: I've read the analyses of why voter X would want to raise or lower his or vote for school B. Folks: no one really cares that much. People responding to a survey like this will, in the overwhelming majority, simply try to give their honest assessment. Any attempt to job the system would just get lost in the survey "noise" even if anyone tried - and in reality they actually won't even try to do so. As to 5: What conceivable motive would USN have for altering these numbers? This is "aluminum foil hat" thinking.
[quote]
In sum, just stick to verifiable facts and there will be less scrutiny, less criticism.

[/quote]
As if! I, for one, find many of the "verifiable facts" incorporated into the USNWR ratings to have questionable relevance to the ultimate issue of academic quality. Alumni giving?!? So you get to choose between "hard data" which is of little actual informative value, and "imperfect" polls which at least directly address the question in people's minds. </p>

<p>Bottom line: look at the data you like, ignore the data you don't feel answers your questions, and take it all with a grain of sand. Don't expect perfection; you won't find it.</p>

104

<p>
[quote]
Okay, I'm going to take a crack at this. Points 1 through 3 simply demonstrate that any poll of opinions will be imperfect.

[/quote]
</p>

<p>Which is my main gripe, exactly. Glad you agree.</p>

105

<p>
[quote]
As to 5: What conceivable motive would USN have for altering these numbers? This is "aluminum foil hat" thinking.

[/quote]
</p>

<p>Simple. The motive is money. USNWR is a business. How many copies do you think they are going to sell if the rankings are the same old, same old, every year, year in and year out?</p>

<p>If they can create some "buzz" (and there is certainly buzz every year - last year was UChicago -- this year? Who knows? Gotta get the issue, that's the point). Why wouldn't they? They are in the business of selling as many copies as they can. God bless them. This is capitalism. Now if we, as consumers, don't call them on some BS in their attempts to make some coin when we smell it, then, frankly, shame on us. </p>

<p>Don't believe that there is major variance in the rankings in a single year? Take a quick gander at these numbers:</p>

<p>
[quote]
- Brown makes a ten notch jump from No. 18 to No. 8 over the course of four years ('93-'97)
- UPenn also makes the "ten notch" jump from No. 16 to No. 6 over the course of five years ('94-'99)
- Northwestern does the "ten notch" in one single year! (Go Purple!) - from No. 24 ('91) up No. 14 ('92) - then proceeds another five notches in five years to No. 9 in ('97)
- Cornell jumps nine spots from No. 15 ('95) to No. 6 ('99) in a span of four years
- Cal (Berkeley) does a nose dive of fourteen spots from No. 13 ('91) to a lowly outside "Top 25" - No. 27 in ('97)
- University of Michigan: freefalls 17 spots in a 5 year span, from no. 7 ('88) to no. 24 in ('93)
- Washington University: leapfrogs 12 spots in a 5 year span, from no. 20 ('98) to no. 9 in ('03)
- CalTech goes from No. 9 (for three straight years '97-'99) then inexplicably jumps to the No. 1 spot on '00! Man I bet the geeks at MIT (always the bridesmaid never the No. 1 bride) were so livid their taped specs steamed up at the mere thought that it might be the ultimate ingenious prank from Pasadena...
- But, the "Bob Beaman-Olympian-Like-Jump" Award goes to JHU. Johns Hopkins jumps a whopping twelve spots in one year from No. 22 ('95) to No. 10 ('96) then proceeds to break into the Top 10 to go to No. 7 ('00)</p>

<p>Of course the "usual suspects" HYPSM all occupy the 1-5/6/7 spots interchangeably year in, year out (I mean you HAVE to keep SOME semblance of credibility, right?) - the real "fun" / "controversy" (or should we call it "manufactured controversy for the sake of controversy" = 'did you see the latest ranking???' = sell copies") is to "spice it up" a la Vegas Style - "roll the dice, baby! who do we got this year!! - whatever happens at USNWR stays at USNWR!</p>

<p>Let's get real folks!!! You've been took. You've been hoodwinked, bamboozled, led astray, run amok...</p>

<p>It is absolutely ludicrous for any reasonable, educated (heck, uneducated) person to believe that a university (college) "jumps" (or falls) 10 notches over the short span of a year (let alone 12 notches). How does a university "do so well" over the course of a year to merit a leapfrog over 10 other institutions which were deemed "superior" a year (or two/three) ago? Conversely, how does a university "do so poorly" that it drops a similar amount in such a short time frame? Answer: it doesn't. they don't.</p>

<p>Viewing the USNWR decade compilation has been an eye-opener - an instant "top down" view of the insanity of trying to come up with a "legitimate college ranking".</p>

<p>Complete and utter nonsense.

[/quote]
</p>

<p>Now you are telling the consumer that an educational institution leaps in rankings over 10+ spots in a single year and others have the ability to drop a full 10+ spots over the course of a single year or a couple of years? Please.</p>

<p>If you believe for a second that USNWR wouldn't try and maximize their ability to sell the most copies possible (under the cover of providing some common good and appealing to the fears and greeds of prospective students and parents), I think its time to put down that "aluminum hat".</p>

106

<p>the schools that are hurt by the PA rank are non-east coat schools. If lets say Northwestern was located in New Jersey, it certainly would have higher than a 4.4 It shares the same rank as Brown and Dartmouth. Meanwhile, it produces a lot more research, which is directly correlated with the PA score.</p>

107

<p>This is big business folks. Let's stop being naive. In the world of academic rankings, the USNWR annual ranking is THE 800 lb gorilla - no stopping it.</p>

<p>So institutions will whore themselves on one end (WashU), others will start playing the game (UChicago), others will simply so "no" (LACs in the news). But don't for a second forget just how much money is at stake here. Reputations. Pride. Alumni. etc.</p>

<p>People say, stop being so paranoid with your aluminum hat wearing conspiracy theories... but if its so cut and dried, then why doesn't USNWR just release all of the data to the public? Why the secrecy? Especially in the face of such heated and growing criticism? It begs the simple question: "What do they have to hide?"</p>

108

<p>the_prestige,
I agree with many of your comments, but hoedown made a good point a few pages back about response rate and disclosure indicating that the combination of an annual survey and a public disclosure of votes could well end the institutions' willingness to cooperate and would reduce the response even further. </p>

<p>As for the changing of the schools over time, I think that there is probably more change than is admitted here. The students change and evolve. There is some faculty shifting. There are significant changes in endowment and state funding and what this means for the quality of the undergraduate product that a school can deliver. I mean, why should academia be the only place where nothing ever changes? I understand stability and it exists at the top with HYPSM, but otherwise so much else of what I see is an entrenched status quo that may be living off its reputation where other institutions are investing in their product, attracting higher quality students and faculty and presenting a more attractive undergraduate experience. Peer Assessment scoring protects that status quo and the result is rankings stability. But is that a true reflection TODAY of what the schools offer and what the undergraduate student will experience?</p>

109

<p>
[quote]
the schools that are hurt by the PA rank are non-east coat schools. If lets say Northwestern was located in New Jersey, it certainly would have higher than a 4.4 It shares the same rank as Brown and Dartmouth. Meanwhile, it produces a lot more research, which is directly correlated with the PA score.

[/quote]
</p>

<p>Which is exactly why those who criticize the PA score for simply being a crude proxy for graduate school rankings vs. what it is supposed to rank, which is undergrad rankings. Who cares if XYZ produces reams of research annually. Does that matter to an undergrad looking at the quality of the undergraduate education? One could argue, and I do, that the more important, the more powerful the grad school emphasis is vs. undergrad emphasis, this is clearly a net negative for undergrads.</p>

<p>That is why I've always argued that those exceptional institutions that emphasize undergraduate education over graduate students are hands down the best places to receive an undergraduate education: schools like Princeton, Brown and Dartmouth. None of those school have all three "big boy" professional schools (law, business, med -- some have a combination of the three but not all three, and Princeton doesn't have any), but would anyone argue that these schools suffer from the lack of research produced at these schools? Those schools certainly aren't lacking any facilities, faculty, resources, money, etc. to compete head to head with the best in the nation, and yet, the lion's share of those resources go to the undergrads. You can't say that at a large research oriented university, where you are effectively competing with fellow grad students for time, resources, etc.</p>

<p>So who cares about research? Maybe if you were a prospective grad student. But that's not what we are talking about. And that's not what the USNWR should be measuring.</p>

110

<p>It's pretty clear that those institutions with large and prominent graduate programs frequently will have a stronger PA than you might expect (public universities being the prime example). However, in judging the institutional use of resources, does anyone have a very clear idea of how these are divvied up between undergraduate students and graduate students? For example, if a college has a budget of $1bn and 10,000 undergrads and 5000 graduate students (med, law, business, other), can anyone give a rough, semi-informed guess on how that $1bn is spent?</p>

111

<p>hawkette, it goes beyond the dollars and cents (though this is, of course, an extremely important component in this discussion).</p>

<p>take for example, Brown, where each and every professor hired is required to teach an undergraduate class. You just can't say that at a large research oriented university. Many of the prominent professors at a research U are simply there to do their research (hence the name research U) and couldn't give a toss about the undergrads (some actually loathe them), and other professors simply go through the motions and hire TAs to do the lion's share of the work.... and they are the ones with the power at a research U not the undergrad students. its a case of the tail wagging the dog.</p>

<p>This contrasts sharply against a school where the basic raison detre is to provide a first class undergraduate education, first and foremost.</p>

112

<p>
[quote]
Simple. The motive is money. USNWR is a business. How many copies do you think they are going to sell if the rankings are the same old, same old, every year, year in and year out?
If they can create some "buzz" (and there is certainly buzz every year - last year was UChicago -- this year? Who knows? Gotta get the issue, that's the point). Why wouldn't they? They are in the business of selling as many copies as they can. God bless them. This is capitalism. Now if we, as consumers, don't call them on some BS in their attempts to make some coin when we smell it, then, frankly, shame on us.

[/quote]
the_prestige - do you honesly think that the sales of the USNWR college renkings publication are affected by "buzz" over which colleges have risen or fallen by a few slots? If so, you really need to get out more. The vast majority of purchases of this product are to the transient market of parents of high school students trying to get some information about the various schools little Johnnie or Suzy will be applying to. That market is huge (been there, done that.) The "market" of people affected in any way by the "buzz" is minute. I'm helping my third kid through this process, I buy the publication every other year, and I have no idea who has gone up or down a few slots, and don't care. Even if I did know it would have no effect on my purchasing decision. I've never bought one before, and after the next issue, I'll never buy one again - "buzz" or not.</p>

<p>Artificial "buzz" to boost sales: It's a great theory - it just has no relevance to the real world.</p>

113

<p>kluge,</p>

<p>let's peer through this looking glass the other way around.</p>

<p>if the USNWR annual rankings ended looking the same pretty much year in and year out, interest in this ranking would gradually diminish. who would pay for a copy of the rankings that basically re-hashes the same stuff over and over again?</p>

<p>do you think that USNWR sells copies by advertising: "gee, check out our new annual issue, same as the old issue." no, it does not. this is big business. we are talking about millions in ad money. </p>

<p>USNWR absolutely markets these issues very carefully. they get premium shelf placement. it is consistently one of their top selling issues every year. if you don't believe this cold hard fact, perhaps you need to get out more.</p>

114

<p>I see where Prestige is coming from. He is saying that students and their parents are being influenced by a magazine that has every incentive to have schools rise and fall yearly to keep up sales. He is right that no one would be interested in the rankings if they stayed consistent every year. With that said, one would be hard pressed to argue that the rankings do not capture the top schools well. The problem is that people cannot accept the fact that school x is 5 spots behind school y. The large picture is clear, and has always been clear. The top 15 or 20 schools for that matter have consistently maintained their ranking spots. There are thousands of schools out there. To consistently be ranked in the top 15 I think is more important than being ranked 10 in one year and 15 the next. That is trivial with respect to the larger picture. With all due respect, if Brown were ranked every year in the top 10, I highly doubt that the Prestige would have such a problem with the ranking.</p>

115

<p>
[quote]
With all due respect, if Brown were ranked every year in the top 10, I highly doubt that the Prestige would have such a problem with the ranking.

[/quote]
</p>

<p>I'm a Princeton grad and Wharton MBA alum.</p>

<p>That said, I absolutely do believe that Brown gets shafted unnecessarily by the USNWR rankings (for the many reasons I've argued). Its probably underranked by a minimum of 5 spots and should be considered a bonafide Top 10 school (by any reasonable undergraduate measure).</p>

<p>What's a better measure without the PA? I've made my own rankings in the past:</p>

<p><a href="http://talk.collegeconfidential.com/showthread.php?t=228347%5B/url%5D"&gt;http://talk.collegeconfidential.com/showthread.php?t=228347&lt;/a&gt;&lt;/p>

116

<p>And I do think Brown gets hurt by the ranking. It is clearly worthy of being ranked in the top 10. The problem is--are the schools that are in the top 10 now not worthy of being there?</p>

<p>T</p>

117

<p>What????????????????????</p>

<p>I am almost positive you have said before you went to Brown. Then why on earth are you so against the rankings. Without them, I argue that Princeton would not be where it is today. For years, it was lagging behind Harvard and Yale. Now, it is considered the main competitor for both.</p>

118

<p>
[quote]
I am almost positive you have said before you went to Brown. Then why on earth are you so against the rankings. Without them, I argue that Princeton would not be where it is today. For years, it was lagging behind Harvard and Yale. Now, it is considered the main competitor for both.

[/quote]
</p>

<p>The problem with your way of thinking / argument is that I am for or against the USNWR ranking on behalf of or against any particular school. (btw, Princeton does just fine on its own).</p>

<p>This is about an objection with regards to methodology / transparency and, frankly, downright corruption IMO. The USNWR rankings are totally bogus but they are somehow regarded as the gold standard in ranking colleges.</p>

119

<p>
[quote]
if the USNWR annual rankings ended looking the same pretty much year in and year out, interest in this ranking would gradually diminish. who would pay for a copy of the rankings that basically re-hashes the same stuff over and over again?

[/quote]
The rankings are available online for free. If the only thing in there was the "horse race", why would anyone ever buy it? Again, the market for this publication is the family with a high school junior or senior, looking for information about a lot of different colleges (plus a variety of articles on the application process, test prep, financing college, etc.) People aren't buying it to find out whether Brown has climbed or dropped a few slots in the "rankings" - no one cares enough to buy a magazine over that. In fact, most of the rankings don't change from year to year. That's okay - even if the average consumer cared (he doesn't) he's not going to buy a copy once little Johhny goes off to college anyway, and he is going to buy a copy when little sister Suzy's turn comes in a couple of years - even if the rankings haven't changed in the meantime - which will be the case for 99% of the colleges that family will be interested in.</p>

<p>"corruption" = tinfoil hat conspiracy theory</p>

120

<p>
[quote]
I'm not claiming that higher PA scores make for a bias. In fact, I content that public disclosure of the voting would remove any bias that might now exist

[/quote]
</p>

<p>Well, maybe it's because I'm one of those academics, but I tend to believe that if Joe Somebody, admissions director of WellKnown U, knew that untold millions of people could see his vote, as well as his peers at other institutions, he would choose being nice over being honest. It's a stretch for me to believe that every college's rating would be more "accurate" if it were higher. </p>

<p>
[quote]
Further down the ranks, I suspect there is some manipulation going on and we'd find out how some highly regarded schools are getting some surprisingly low scores.

[/quote]
</p>

<p>Such manipulaton would have be done by Synovate or USNews. It is beyond credibility to suggest that large groups of provosts, presidents, and admissions directors are all contacting one another to colluding in their votes. </p>

<p>
[quote]
A public disclosure of the votes would reveal if there is bias going on

[/quote]
</p>

<p>Can you explain how this would work? How can you tell bias from honest opinion in this way? I am having a hard time seeing the mechanism of discovery for bias. </p>

<p>
[quote]
is this just inherent favor being shown those schools that have been around longer and whose names are better known? Do the rankers even know about the teaching quality at ANY of these schools? They may, they may not, but how do we know?

[/quote]
</p>

<p>Will knowing the voting records of all respondents reveal this to you?</p>

<p>
[quote]
Why should academia be exempt from having outsiders give opinions

[/quote]
They are not exempt. For one glaring example, look at USNews itself. It is "opinion" that decides what is counted and what gets certain weights. Every single element they've included represents a subjective decision (not just the individual votes cast for PA). </p>

<p>Furthermore, colleges and universities are subject to outside opinion and evaluation from employers, students, parents, foundations, etc. Students make choices about applying, enrolling, and persisting. Employers make decisions about recruiting and hiring. Foundations decide where to give money. Bond raters evaluate their debt. You make it sound as if higher ed is devoid of outsiders making judgments on the work they do, but that simply doesn't ring true to me. </p>

<p>It seems to me that most of your complaints about PA should be directed at USNews, not the individuals who elected to return that rating survey. I think we may agree on many of the criticisms, but I can't understand why you keep laying it at the feet of the "status quo" and the "educational elite" and so on.</p>