USNWR Peer Assessment Scores look biased to number of grad students

College Search & Selection
1

<p>I recently completed an analysis of the USNWR Top 50 ranked colleges. I sorted the schools by the size of their graduate enrollments and then looked at the PA scores of the various colleges. </p>

<p>Of the 25 colleges that had the highest number of graduate students, only 3 failed to receive a PA score of 4.0 or higher. </p>

<p>Of the 25 colleges that had the fewest number of graduate students, 15 failed to receive a PA score of 4.0 or higher.</p>

<p>Thus, Top 50 USNWR universities that had above the median number of graduate students were five times more likely to score at the 4.0 level or higher. </p>

<p>Here is the full data set (graduate school numbers taken from collegeboard.com):</p>

<p>Rank in absolute number of grad students, # of grad students, PA score, college</p>

<p>1 , 17740 , 4.6 Columbia
2 , 16477 , 3.8 NYU
3 , 14526 , 4.5 U Michigan
4 , 13950 , 4 USC
5 , 11439 , 3.6 U Florida
6 , 11101 , 4.1 U Texas
7 , 10901 , 4.2 UCLA
8 , 10285 , 4.9 Stanford
9 , 10211 , 4 U Illinois
10 , 10010 , 4.9 Harvard
11 , 9886 , 3.9 U Washington
12 , 9007 , 4.8 UC Berkeley
13 , 8832 , 4.1 U Wisconsin
14 , 8254 , 4.2 U North Carolina
15 , 7824 , 4.3 Northwestern
16 , 7693 , 4.3 U Virginia
17 , 6712 , 4.5 U Penn
18 , 6180 , 3.8 Penn State
19 , 6126 , 4.9 MIT
20 , 5834 , 4.6 U Chicago
21 , 5575 , 4 Georgia Tech
22 , 5350 , 4.4 Duke
23 , 5181 , 4.6 Cornell
24 , 4852 , 4.8 Yale
25 , 4715 , 4.1 Wash U
26 , 4589 , 3.8 UCSD
27 , 4490 , 4 Georgetown
28 , 4451 , 4.2 Carnegie Mellon
29 , 4088 , 3.6 UC Irvine
30 , 4080 , 4 Emory
31 , 4072 , 3.8 UC Davis
32 , 3928 , 3.5 Case Western
33 , 3925 , 4 Vanderbilt
34 , 3842 , 3.6 Boston Coll
35 , 3539 , 3.4 U Rochester
36 , 2977 , 3.6 Tufts
37 , 2870 , 3.5 UC Santa Barbara
38 , 2646 , 3.9 Notre Dame
39 , 2305 , 3.3 Tulane
40 , 2240 , 3.5 Rensselaer
41 , 2115 , 3.2 Lehigh
42 , 2070 , 4 Rice
43 , 2010 , 4.9 Princeton
44 , 2009 , 3.6 Brandeis
45 , 1756 , 4.4 Brown
46 , 1646 , 4.6 Johns Hopkins
47 , 1393 , 3.5 Wake Forest
48 , 1360 , 3.7 W&M
49 , 1352 , 4.3 Dartmouth
50 , 1222 , 4.7 Caltech</p>

2

<p>Hawkette: So? What am I missing, here?
Isn't it interesting that Princeton, with a PA of 4.9, has 15,000 fewer grad students than has Columbia, with a PA of 4.6? And Columbia and Hopkins share the same PA (4.6), yet there are at least 15,000 more grad students at Columbia? And Harvard, with a PA of 4.9 has 6,000 fewer grad students than does NYU, which has a PA of 3.8. My conclusion? PA has not much to do with the number of grad students at a given university.</p>

3

<p>A ratio of a 5 to 1 is pretty striking and pretty hard to explain away. It is pretty clear that PA scores are biased in favor of the schools with large graduate student populations. </p>

<p>There are exceptions (like Princeton and Hopkins), but there is definitely a relationship that favors the larger graduate program schools over the smaller graduate program schools. Even the schools in the Ivy League that get the lowest PA scores (Brown and Dartmouth) have the smallest graduate programs. </p>

<p>Beyond the stark 5 to 1 ratio, there is another comparison that raises eyebrows. If you were to compare the PA scores of the top 25 with the bottom 25, the difference in PA score is 4.32 to 3.86.</p>

4

<p>Remember the common fallacy: correlation does not mean causation. It could be simply that those schools who have large graduate programs have large PA scores because they're great institutions. While of course a 5:1 ratio is high, there are sufficient counter examples ("anomalies") to dispel your conclusion, I think.</p>

5

<p>You, of course, are correct, Kyle. Drawing inferences from these numbers presupposes a causation that does not necessarily exist. Otherwise, the schools with the larger grad population would have the higher PA scores...Which they do not, necessarily.
Hawkette...What are you getting at? That there is something wrong with PA because it favors schools with large numbers of graduate students? Is that what you are attempting to show? Perhaps PA is affected by graduate programs...So? And if this were always the case, then how do you explain the relatively high PAs of many of the LACs???? Think Williams, Amherst, Swarthmore, Vassar, Middlebury, Weslyan, etc. Nah...I fear that your reasoning has an abyss in it.</p>

6

<p>While kyledavid and gabriellaah both make good points, the other thing to note here is that the schools in the bottom 25 in number of grad students that have low PAs generally do worse in other areas of the US News rankings (as do the three schools with low PAs in the top 25 in number of grad students). If you compared number of grad students to overall US News scores (or even overall scores factoring out PA), would the results really be that different?</p>

7

<p>Gabriellaah, </p>

<p>The reason some LACs have high PAs is because they are voting for only each other. So of course some will have high PAs...dare I say, duh?</p>

8

<p>slipper is correct. Williams isnt being judged against Cal on a 1-5 scale. They are being judges against their peer Amherst and Wes. Hawkette is dead on with this analysis. The schools that dont do well in PA but have stronger student bodies vs their ranking are the small-medium National U. BC, Georgetown, Rice, Tufts, William & Mary are all grossly underrated. One prime reason is that their PA score is lower. Comparing Rice to Michigan is ridiculous but that is what the salesmen at US News are trying to do.</p>

<p>PA
W&M 3.7
Rice 4.0
Georgetown 4.0
Tufts 3.6
BC 3.6
Wake Forest 3.5</p>

<p>UTexas 4.1
Wisconsin 4.1
UNC 4.2
Michigan 4.5
UCLA 4.2
Cal 4.8</p>

<p>Compare this against SAT scores and you see the the grad school bias come out brightly.</p>

9

<p>The peer assessment rating is highly correlated with freshman retention, graduation rate, SAT scores, the proportion of students in the top 10% of their class, acceptance rate for BOTH LACs (which don't have grad students)and universities. Perhaps universities with large grad programs ALSO attract great freshman and have high grad rates.</p>

10

<p>While the PA scores may indeed be heavily influenced by graduate programs, it is still incorrect to assume they directly reflect graduate program strength. It is more likely that they are <em>based</em> on graduate strength, but with some sort of qualitative adjustment for undergrad quality. This is readily apparent comparing scores. Of the top publics, for example, Texas and Wisconsin have much stronger faculty/grad programs overall than UNC and Virginia (looking at faculty awards, NAS/NAE membership, PhD rankings, etc.), yet have lower PA scores. This is obviously a correction for undergrad. The same holds true versus privates. In Texas, UT-Austin is #1 in nearly every academic discipline per the NRC and USNWR graduate program rankings, yet its PA score of 4.1 is barely above Rice's 4.0. Again, if this was clearly based on graduate programs the spread should be higher. Since it is not, we can only assume their is some level of "correction" built in.</p>

11

<p>In using the same list of the USNWR Top 50 colleges and ranking by absolute size of graduate students, there is another measure that further clarifies the great PA benefit that the larger graduate programs deliver to a school’s PA score.</p>

<p>I looked at each of the colleges and compared their PA rank to their overall USNWR rank. The differences in the two groups were quite dramatic.</p>

<p>Top group of 25 colleges as ranked by absolute number of graduate students
19 Number of times the PA rank was higher than the overall rank
5 Number of times the PA rank was lower than the overall rank
1 Number of times the PA rank was equal to the overall rank</p>

<p>Lower group of 25 colleges as ranked by absolute number of graduate students
5 Number of times the PA rank was higher than the overall rank
18 Number of times the PA rank was lower than the overall rank
2 Number of times the PA rank was equal to the overall rank</p>

<p>Conclusion: If a college is one of the 25 colleges with the largest number of graduate students, there is more than a 75% chance that the PA rank will be higher than the overall rank. Conversely, if a school has a smaller number of graduate students, it will have an 80% chance that its PA score will not be as high as its overall rank. </p>

<p>Collegehelp,
I tested your hypothesis about student quality and graduation rates, but there is no pattern that would support your thought. When measuring the 25 schools with the largest graduate student populations, for Student Selectivity rank, 15 were in the Top 25 graduate programs and 10 were not. For Graduation & Retention ranks, 14 were in the Top 25 graduate programs and 11 were not. I don’t consider either of these results as particularly dramatic and indicative of anything like the 5 to 1 ratio as described above. </p>

<p>JWT,
You may be right although I suspect there is more than a bit of anti-Texas sentiment among academics. IMO, Rice at 4.0 is a travesty.</p>

12

<p>Since you keep pointing out the 5:1 stat, I thought I should mention that you miscounted originally. 4 of the schools in the top 25 in grad students have PAs below 4.0 (NYU, Florida, Washington, Penn State). So it's actually 3.75:1.
I've never claimed that PA isn't influenced by strength of grad programs (though I would say strength, rather than size, look at Princeton for instance, very strong grad programs as rated by the NRC, SJT, etc, but the 8th smallest grad programs, produces the highest PA. Caltech has a similar result). On the other hand, it's not the only thing affecting PA. If it was, there is no way Dartmouth would have a PA of 4.3. It's a wonderful undergraduate college, but it doesn't even crack the top-100 in a research ranking like the SJT. If faculty research and graduate strength were all that affected the rankings, it would have a PA of 3.0, if not below. Similarly, in a ranking based on research, Berkeley would have the highest PA (0.1 may not seem like a large difference, but no professor, ranking only on research, would put Princeton ahead of Berkeley). These, and many similar examples (like JWT's example of Rice and Texas) demonstrate that PA is reporting something more than just research quality. It hardly means it's a wonderful rating system, but size of grad programs is a very imperfect proxy.</p>

13

<p>US News has clearly noticed that the PA scores favor larger public universities with bigger departments that have more graduates and graduate students and often more visibility (I don't think that the actual graduate student population per se is the cause of this ranking disparity). In fact, one of the reasons that all of the objective data was added was because PA alone led to large institutions often dominating. There is a very interesting article in this link written by a US News official about how and why changes to the methodology have occurred.</p>

<p><a href="http://www.ihep.org/Pubs/PDF/College_and_University_Ranking_Systems_Final_Report.pdf%5B/url%5D"&gt;http://www.ihep.org/Pubs/PDF/College_and_University_Ranking_Systems_Final_Report.pdf&lt;/a&gt;&lt;/p>

<p>Hawkette, once again, I don't know why you treat these issues as a sort of "gotcha" moment, in your constant attempt to prove that US News is guilty of bias and PA is bad. PA is interesting but imperfect and probably accurately reflects academic views of the strength of an institution, US News has made constant attempts to change its methodology to respond to criticisms, and rankings are not perfect and can be harmful, particularly when they lead students and parents to assume that one school is "better" than another based upon higher placement in the rankings.</p>

<p>I have stated before, however, that most of your fixes seem naive (student surveys, a la Princeton Review, don't seem to me to reflect much of anything and employer surveys would be way too limited and would also reflect a strong alumni and regional bias that wouldn't prove anything about the educational quality of a school). I also don't agree with you that PA is somehow inherently wrong--it is entirely possible that the schools assessed by peers as the best also have excellent teaching (that is clearly one of the factors being surveyed). If there is a measurement that would accurately assess teaching quality alone I would love to see it, but so far, I haven't seen anything that's particularly useful.</p>

14

<p>a lot of the most famous professors only teach grad students and/or spend most of their time doing research. i would think that this would skew the numbers a bit, since they actually aren't involved in the undergraduate experience.</p>

15

<p>Slipper: Right, of course. But the argument thins when LACs can have such high PAs without grad schools. For a valid outcome, we cannot treat LACs as though they do not exist. There is no way to measure how the universities would fare if they did not have grad schools. There are too many problems with these assumptions, that cannot be accounted for.
In essence, what is being implied here, is that the LAC peer assessment is perhaps more reliable than the university PA because it reflects opinion only about undergrad education. No? I do believe, however, that LACs have some limited grad programs. Does this mean, that in this regard, Princeton's PA is more reliable than Columbia's, because there are so many fewer grad students/programs skewing the results? Or that Hopkins PA is a more reliable number than Columbia's (both 4.6)...and we might as well include Chicago and Cornell (4.6, too), with approximately 4 times more grad students... because of it has fewer grad students? I don't think that we can reason along these lines and come up with anything meaningful. There are too many huge holes to plug up.</p>

16

<p>Hawkette: ditto midatlmom.
Also, it has got to be exhausting to be constantly searching for ways to prove your point about PA. It is okay for people to disagree. It is really not necessary to continuously work on winning your point. Some will agree, and some will disagree...which is fine.</p>

17

<p>If you want to prove bias, you need to do this differently. It's pretty iffy drawing conclusions based on a correlation calculated on extreme cases. </p>

<p>There are over 600 schools which enroll graduate students, about 1.5 million of them in total But your analysis includes quite a small sample and it's about as non-random as it can get--the top 25-50 by ranking? </p>

<p>Consider the fact that public schools enroll almost twice as many graduate students as private schools in the U.S. But do public institutions dominate the PA assessment? That right there should be a heads-up that PA may not be closely tied to grad enrollment.</p>

18

<p>Based on the following data, I found a correlation of about .79 between peer assessment and SAT and admit percent but a much lower correlation of .4 between graduate (including professional) enrollment and peer assessment. The relationship between peer assessment and freshman selectivity is much stronger than between peer assessment and graduate enrollment.</p>

<p>The peer assessment scores are from 2007 (last year). The enrollment data is from last year (IPEDS). The SATs and accept rate are from IPEDS.</p>

<p>school, SAT, accept rate, graduate enrollment, first professional enrollment, total of grad plus professional school enrollment </p>

<p>Harvard University 1590 9 15370 4.9
Massachusetts Institute of Technology 1560 13 6126 4.9
Princeton University 1590 10 2275 4.9
Stanford University 1540 11 12466 4.9
Yale University 1580 10 6065 4.9
California Institute of Technology 1570 17 1222 4.7
University of California-Berkeley 1450 27 9980 4.7
University of Chicago 1560 40 9486 4.7
Cornell University 1490 25 6079 4.6
Columbia University in the City of New York 1540 12 15525 4.6
Johns Hopkins University 1490 28 13970 4.6
Duke University 1560 24 6890 4.5
University of Michigan-Ann Arbor 1420 47 12290 4.5
University of Pennsylvania 1490 18 11811 4.5
Brown University 1530 14 2134 4.4
Dartmouth College 1550 16 1689 4.4
Northwestern University 1500 30 9325 4.4
University of California-Los Angeles 1410 27 11015 4.3
University of Virginia-Main Campus 1430 37 9140 4.3
Carnegie Mellon University 1490 34 4451 4.2
University of North Carolina at Chapel Hill 1390 37 10428 4.2
University of Wisconsin-Madison 1370 73 13920 4.2
Georgetown University 1490 22 7295 4.1
Rice University 1530 24 2013 4.1
The University of Texas at Austin 1350 49 12660 4.1
Vanderbilt University 1460 35 5229 4.1
Washington University in St Louis 1530 21 6027 4.1
Emory University 1470 37 5692 4
Georgia Institute of Technology-Main Campus 1400 67 5576 4
University of Illinois at Urbana-Champaign 1400 75 11234 4
University of Notre Dame 1500 27 3251 3.9
University of Southern California 1460 25 16660 3.9
University of Washington-Seattle Campus 1320 67 11688 3.9
College of William and Mary 1440 32 1967 3.8
Indiana University-Bloomington 1240 78 8419 3.8
New York University 1410 36 19299 3.8
Pennsylvania State University-Penn State Main Campus 1300 62 6101 3.8
Purdue University-Main Campus 1250 85 7939 3.8
University of California-San Diego 1370 43 4436 3.8
University of Minnesota-Twin Cities 1360 57 18289 3.8
Ohio State University-Main Campus 1310 65 13339 3.7
Tufts University 1480 27 4643 3.7
University of California-Davis 1300 61 6760 3.7
University of Maryland-College Park 1390 44 9948 3.7
Boston College 1420 29 4632 3.6
Brandeis University 1440 38 2005 3.6
Rensselaer Polytechnic Institute 1420 67 1417 3.6
Texas A & M University 1290 73 8800 3.6
University of California-Irvine 1300 60 4085 3.6
University of Florida 1360 48 15802 3.6
University of Iowa 1320 83 8078 3.6
Case Western Reserve University 1430 67 5512 3.5
George Washington University 1390 37 13631 3.5
Michigan State University 1290 73 9699 3.5
University of California-Santa Barbara 1320 53 3000 3.5
University of Colorado at Boulder 1280 88 5236 3.5
University of Pittsburgh-Main Campus 1330 53 9614 3.5
Wake Forest University 1400 43 2418 3.5
Boston University 1370 57 12855 3.4
Miami University-Oxford 1300 78 1341 3.4
Rutgers University-New Brunswick/Piscataway 1310 58 7656 3.4
Syracuse University 1330 51 5964 3.4
Tulane University of Louisiana 1425 38 4073 3.4
University of Georgia 1330 65 8513 3.4
University of Rochester 1420 44 3811 3.4
Virginia Polytechnic Institute and State University 1290 68 6473 3.4
Iowa State University 1350 90 5022 3.3
University of Missouri-Columbia 1300 78 6700 3.3
Baylor University 1310 43 2209 3.2
Lehigh University 1390 39 2116 3.2
Southern Methodist University 1320 54 4645 3.2
The University of Tennessee 1270 74 8136 3.2
University of California-Santa Cruz 1270 74 1540 3.2
University of Connecticut 1290 51 7210 3.2
University of Delaware 1290 47 3446 3.2
University of Miami 1360 40 5161 3.2
Auburn University Main Campus 1230 72 4180 3.1
Brigham Young University 1350 70 3573 3.1
Clemson University 1310 55 3167 3.1
Fordham University 1290 47 7031 3.1
Pepperdine University 1350 30 4254 3.1
University of California-Riverside 1200 80 2112 3.1
Saint Louis University-Main Campus 1300 67 4514 3
SUNY at Binghamton 1350 43 2919 3
Yeshiva University 1340 79 3161 3
American University 1370 53 5223 2.9
Clark University 1310 60 809 2.9
Marquette University 1300 70 3500 2.9
Worcester Polytechnic Institute 1390 67 1027 2.8
Stevens Institute of Technology 1360 54 2976 2.7
University of Denver 1290 73 5701 2.7</p>

19

<p>collegehelp, out of curiosity, what's the correlation between SAT score/admit rate and size of grad program?</p>

20

<p>correlation peer assessment with:
SAT verbal .73
SAT math .79
SAT total .78
accept rate -.65
graduate plus professional enrollment .38
graduate enrollment .40</p>

<p>All are statistically significant but the correlations with graduate enrollment are substantially lower.</p>