Is Peer Assessment in USNWR Rankings based on Undergrad or Grad Reputation?

College Search & Selection
241

<p>


</p>

<p>Subtle trolling? I really have no idea how much more panegyrizing you'd need to be satisfied. Fwiw, here's the "origin" of my top 12 comment for Michigan:</p>

<p>My post on 02-18-2008 :<a href="http://talk.collegeconfidential.com/1059809645-post117.html%5B/url%5D"&gt;http://talk.collegeconfidential.com/1059809645-post117.html&lt;/a&gt;&lt;/p>

<p>
[quote]
1. I think a long dozen is good
2. I'll flip Michigan with San Diego</p>

<p>1 Harvard
2 Cambridge
3 Stanford
4 California - Berkeley
5 Massachusetts Inst Tech (MIT)
6 Caltech
7 Columbia
8 Princeton
9 Chicago
10 Oxford
11 Yale
12 Cornell
13 Univ Michigan - Ann Arbor</p>

<p>Now, the Quasi Emir of Dubai can smile too.

[/quote]
</p>

<p>A post that followed my earlier comment: <a href="http://talk.collegeconfidential.com/1059807140-post101.html%5B/url%5D"&gt;http://talk.collegeconfidential.com/1059807140-post101.html&lt;/a&gt;&lt;/p>

<p>


</p>

<p>And, YES, my comment "and its undergraduate should be easily among the best 50 in the country" meant to place Michigan's undergraduate among the best 50 in the country. Obviously, such list should include LACs and specialty schools. Forgive me for stooping this low and include unkwnown entities such as Williams, Pomona, or the military academies! And, fwiw, I DID write **EASILY **as well.</p>

<p>And, last but not least, I concluded with "And, one needs to understand that the "ranked" order is simply ... meaningless." That means that there is no meaningful difference between a school in the 10-20 range versus the 30-40 range. And, that also assumed that the "rank" is valid in the first place.<br>
/sigh</p>

242

<p>


</p>

<p>Agreed. But then so are all the rest of these rankings.</p>

<p>On the other hand, with a few outliers, there's a fair degree of consistency between the THE ranking of American universities and the US News PA score:</p>

<p>School USN PA rank* THE**
Princeton 1 3
Harvard 1 1
Stanford 1 11
MIT 1 6
Yale 5 2
Berkeley 5 14
Caltech 7 4
Columbia 8 7
Chicago 8 5
Cornell 8 13
JHU 8 10
Penn 12 9
Michigan 12 17
Duke 14 8
Brown 14 16
Dartmouth 16 25
Northwestern 16 15
UVA 16 39
CMU 19 12
UCLA 19 18
UNC 19 48
Wisconsin 22 23
Texas 22 21
WUSTL 22 52</p>

<ul>
<li>US News PA score, rank ordering of all national universities with PA scores > 4.0.
** THE rank ordering of U.S. universities making THE list of top 200 globally.</li>
</ul>

<p>I don't know what it means exactly, but the opinion of research universities held by the presidents and provosts surveyed in the US News PA rating generally seems to hold up in an international survey of academics and employers. All garbage beliefs, widely held? Or is something more going on here?</p>

243

<p>It's not surprising that the PA rankings agree with the THES rankings. US and international academics/employers alike are concerned with the university's contributions to academia, to the body of knowledge, and to the advancement of science. Harvard is Harvard, Cornell is Cornell, and Berkeley is Berkeley not because they can teach a mean Bio 101 course but because it is at these places that amazing discoveries are taking place and reputations are made.</p>

<p>Of course, you can always argue that the quality of graduate/academic programs has little to do with the quality of undergraduate education in those same departments (I would disagree). In that case, feel free to exclude the PA score and come up with your rankings and choose your college that way. However, don't expect US News to do it. Reputation (deserved or not) does matter in the real world. PA score is essentially a ranking of reputation and prestige and a fairly accurate one at that.</p>

244

<p>
[quote]
not because they can teach a mean Bio 101 course

[/quote]

If I were a top student, I wouldn't so concern about Bio 101; chances are I'd be placed out of that with my AP Bio. I would be more concern about who's teaching Bio 4xx and Bio 5xx ... or if these courses are even available.</p>

245

<p>THE is not as well respected as some of other rankings for its bias towards British universities.</p>

246

<p>^ Agreed, THE has a clear bias toward British universities and more broadly toward European universities. But it's interesting to pull out the U.S. universities and see how consistently academic reputation stands up across international lines. Or at least across the Atlantic. I think there's some merit to what norcalguy is saying ^^^. Like it or not, US News PA rating is a fair proxy for how these schools are seen by the academic community. You can choose to dismiss that if you like. To me it means something, certainly more than alumni giving rates and faculty salaries and some of the other goofy "objective" metrics U.S. News relies on which have nothing to do with academic quality or the quality of classroom instruction.</p>

247

<p>bclintonk, I like your posts. What do you teach?</p>

248

<p>^ Law, but my first and true love is philosophy.</p>

249

<p>My guess is you are one hell of a professor. I read your posts, and I see intelligence, logic...common sense. </p>

<p>I bet when students are finished with your class...their thinking skills are vastly improved.</p>

<p>Good luck with your daughter's college search.</p>

250

<p>^ Thanks for the vote of confidence. But hey, I don't teach at an LAC and I take my scholarship pretty seriously, so I can't be that great, right? ; )</p>

251

<p>I always appreciate your comments as they are genuine, constructive and very academic in nature. I wish more of professors like you would participate in and help contribute to cc forums. It's more than a vote of confidence but also one of great appreciation.</p>

252

<p>Hmmm. Bclintonk, you don't teach at a LAC. That is going to hurt your ranking. Objective data... your class sizes are 2.73625 too large. </p>

<p>You have three choices. </p>

<p>You can give your students a standardized test. Make them take the same test over and over again until their scores rise 80 points. At least now you will have stronger students in your class. </p>

<p>Ask students who have taken your class for donations so you can get your alumni giving rate higher. If they don't give, supply them with the money.</p>

<p>On the first day of each class, you can execute 2.73625 students. ;)</p>

253

<p>Re: my post #242. I went back and compared the US News PA rating for the top U.S. research universities with the "peer review" component of the THE rankings of U.S. universities. There's even a tighter correlation between these two figures than between US News PA rating and THE's overall ranking---which makes sense, when you think about it, because both the US News PA rating and THE's peer review rating attempt to measure the same thing, reputational standing within the academic community.</p>

<p>(Research universities with US News PA rating > 4.0)</p>

<p>School PA/rank THE-PR/rank</p>

<p>Princeton 4.9 1 100 1
Harvard 4.9 1 100 1
Stanford 4.9 1 100 1
MIT 4.9 1 100 1
Yale 4.8 5 100 1
Berkeley 4.8 5 100 1
Caltech 4.7 7 100 1
Columbia 4.6 8 100 1
Chicago 4.6 8 100 1
Cornell 4.6 8 100 1
JHU 4.6 8 99 12
Penn 4.5 12 97 15
Michigan 4.5 12 99 12
Duke 4.4 14 98 14
Brown 4.4 14 90 19
Dartmouth 4.3 16 60 *
N'western 4.3 16 88 20
UVA 4.3 16 63 *
CMU 4.2 19 96 16
UCLA 4.2 19 100 1
UNC 4.2 19 72 *
Wisconsin 4.1 22 94 18
Texas 4.1 22 95 17
WUSTL 4.1 22 72 *</p>

<ul>
<li>= too far down the THE peer reviews charts for me to calculate this late at night.</li>
</ul>

<p>The only outliers here are Dartmouth, UVA, UNC-Chapel Hill, and WUSTL, whose relatively strong US News PA scores are not matched by strong international peer review scores in the THE survey. The top 15 or so on each list track almost perfectly. Toward the bottom end of this group, CMU, Wisconsin, and Texas all score a little better in the THE international survey than in the US News PA ratings.</p>

<p>This does not prove the PA ratings are "correct," of course. But it does suggest that, at least at the top of the heap, a research university's reputation for excellence among its peers in the U.S.carries over to its reputation for excellence across international boundaries---or at least to Europe.</p>

254

<p>Since there are plenty of recognition of logic and intelligence of the mutual admiration society in the last posts, may I draw your attention to this exchange:</p>

<p>
[quote]
Quote: Originally Posted by barrons
The Times rankings are a joke. </p>

<p>Agreed. But then so are all the rest of these rankings.</p>

<p>On the other hand, with a few outliers, there's a fair degree of consistency between the THE ranking of American universities and the US News PA score

[/quote]
</p>

<p>If the THE rankings are a joke (a point the posters agree on) and there's a fair degree of consistency between the THE ranking of American universities and the US News PA score, logic and intelligence should force one to conclude that the PA scores are ... a joke.</p>

<p>Joke or not, both the PA and the THE are heavily influenced by the reputation of the graduate schools, And, accordingly, as far as the OP query goes, we may state ...</p>

<p>Quod erat demonstrandum! :D</p>

255

<p>just a small note. Times world rankings is known as THES world rankings, or Times Higher Education Supplement. THES world rankings for short. the other well known, probably more highly respected, is the Academic Ranking of World Universities by Shanghai Jiao Tong University, SJTU world rankings. these two regularly do the world ranking surveys and their survey results are very much cited for reference.</p>

<p>the other one, Newsweek Top 100 Global Universities, simply uses a combination of both THES and SJTU world rankings, with 50% of the score coming from equal parts of three measures used by Shanghai Jiaotong (number of highly cited researchers in various academic fields, number of articles published in Nature and Science, and the number of articles listed in the ISI Social Sciences and Arts & Humanities indices), 40% of the score coming from equal parts of four measures used by THES i.e. the percentage of int'l faculty, the percentage of int'l students, citations per faculty member and the ratio of faculty to students, with final 10% coming from library holdings (number of volumes).</p>

256

<p>bclintonk,
Very interesting insight. So PA scores do correspond with research productivities of universities.</p>

257

<p>The Times rankings are a joke because schools make wild moves in the rankings. That is a sure sign of a faulty methodology as schools change very little year to year.</p>

258

<p>


</p>

<p>I'd put it a little differently. Both the PA ratings and the THE rankings are heavily influenced by the scholarly reputation of the faculty. The reputation of the graduate programs is also almost entirely a function of the scholarly reputation of the faculty. It's not graduate schools per se that drive this; graduate school rankings are derivative of the faculty's reputation for scholarly excellence. It's all driven by academics' perceptions of other academics' scholarship. And the question is whether you think the scholarly reputation of the faculty should matter for purposes of evaluating undergraduate programs. Some say no. I say most definitely yes.</p>

<p>These rankings are a "joke" insofar as the surveys are very unscientific in their methodology and based on broad-brush impressions of overall faculty strength, rather than a more careful and fine-grained assessment of strengths faculty-by-faculty, as in the NRC rankings due out in the fall. What's an even bigger joke, however, is the other factors US News throws into its rankings---things like faculty salaries and alumni giving rate, which have little or no demonstrated correlation with either scholarship or teaching.</p>

<p>As for the wild swings in the THE rankings to which barrons refers ^, I haven't followed this closely, but I did note in reading THE's description of its 2007 survey that it had made radical changes in its methodology. Very likely these methodological changes account for much, possibly most of the year-to-year changes. My hunch is that the "peer review" component doesn't change as abruptly or as dramatically from year to year. Indeed, with respect to the US News PA ratings which seem to correlate quite closely with THE's "peer review" ratings, a frequent complaint by some posters here is that the ratings hardly change at all over the course of many years. Such is the stubborn nature of academic reputations.</p>

<p>For my money, a far better metric of faculty quality than either PA or THE peer review---though not without its faults--- is the NRC rating of faculty quality, field by field, which can be aggregated into an overall profile of a research university's academic strengths and weaknesses. Concededly, this is pitched as "faculty quality of graduate programs." But I don't know of a single research university that has separate graduate-level and undergraduate-level faculties in any field. The faculty members of the philosophy department teach both the undergrads and the graduate students; same in physics, and on down the line. Class sizes do matter; number of courses taught by TAs does matter. But separate metrics can capture those factors. Other things equal, however, I would insist that a school with a faculty consisting of the strongest scholars, the top people in their field, is a stronger faculty than one without as high a concentration of top scholars.</p>

259

<p>The problem in my mind is the very loose relationship between the PA and the actual experience of undergraduate students.
To the extent that numbers of distinguished faculty has a role in PA, this favors educational behemoths that can count large numbers of faculty and also have large numbers of graduate and undergraduates alums voting. Only one of the biases of the PA. And coincidentally, the big schools are where undergraduates may have the least involvement with the "distinguished". (Just an anecdote: in my PhD program at the University of Chicago, none of the professors wanted to teach undergraduates, and were not shy about telling us graduate students that. By and large they succeeded in avoiding undergraduates entirely, except for the most junior people.) Other biases include regional prejudice, skepticism and under-ranking of schools with a religious base, and a liberal-conservative bias that IMO is consistent and is most apparent in the LAC PAs.
Just junk the darn thing. It does have a vanity value for the faculty involved, I will concede. Let the vain check themselves out in the mirror.
Leave poor high school students, figuring out where to spend four years, alone.</p>

260

<p>^ It's not just the largest faculties, it's the faculties with the most distinguished scholarly reputations. Most very large faculties get middling, low, or very low PA ratings. And some small faculties get very high PA ratings. Caltech, for example, is a relatively small school with a terrific PA (4.7). Among LACs, Williams (4.7), Amherst (4.7), Swarthmore (4.6) and Wellesley (4.5) all have very high PAs.</p>

<p>Besides, I think the PA, imperfect as it is, does tell us something. It explains, for example, why most people in and outside academia would say Stanford is a cut above Duke. Apart from PA they're almost indistinguishable statistically. But as good as Duke's faculty is--and a 4.4 PA rating is nothing to sneeze at---what distinguishes Stanford is its extraordinarfy faculty, reflected in a 4.9 PA rating. </p>

<p>We can do far better than the current Us News PA methodology, but any "ranking" of colleges and universities that ignored differences in faculty strength would truly be a bad joke, and truly would do a disservice to high school students trying to sort out their college options.</p>