Is Peer Assessment in USNWR Rankings based on Undergrad or Grad Reputation?

College Search & Selection
1

<p>There have been number of threads/posts on cc regarding Peer Assessment used for USNWR rankings. Some say it's pretty much based on undergrad program reputation while some say it's grad program that is heavily factored in. Or is it somewhere in between? What is the true answer to this? </p>

<p>Based on knowledgeable cc posters' comments, national public schools have been enjoying the advantage of their grad school reputation as a cover for their relatively weaker undergrad program. Were it not for their PA scores being raised due to grad program reputation and thus their overall rankings, these national public schools (UCBerkeley, UMichigan etc.) would have been ranked lower than they might have been.</p>

<p>Does USNWR publish or make public their questionnaire sent to schools when conducting the survey? </p>

<p>I hope to use this forum to seek the true answer to all this.</p>

2

<p>Those doing PA (the peers) don't mentally separate grad and undergrad, usually; they rate the school on their perception of its academics and such. Many don't realize that those with strong undergrad and comparatively weak grad do well in PA, too.</p>

3

<p>It's really somewhere in between. Looking at the actual numbers makes it clear. If it were purely based on graduate programs, schools like UVA and Brown wouldn't have higher PAs than schools like UT and Wisconsin. Similarly, even though schools like UT and Wisconsin have higher scores than schools like Rice, Emory, and Vanderbilt, they would be much higher than they currently are based on grad program strength. So the PA score is some sort of mental grad+undergrad composite by academics when asked to judge schools.</p>

4

<p>I agree with JWT.</p>

5

<p>How different is the quality of the undergraduate education that faculty offer at the folllowing groups of colleges?</p>

<p>PA Score , PA Rank , School (USN Rank)</p>

<p>4.4 , 14 , Duke (8)
4.2 , 19 , U North Carolina (28)
3.5 , 52 , Wake Forest (30)
3.3 , 67 , Tulane (50)</p>

<p>4.3 , 16 , U Virginia (23)
3.7 , 39 , W&M (33)</p>

<p>4.6 , 8 , Columbia (9)
4.3 , 16 , Dartmouth (11)
3.6 , 43 , Tufts (28)
3.6 , 43 , Boston Coll (35)
3.2 , 73 , Lehigh (31)</p>

<p>4.6 , 8 , U Chicago (9)
4.5 , 12 , U Michigan (25)
4.3 , 16 , Northwestern (14)
4.1 , 22 , Wash U (12)
3.9 , 32 , Notre Dame (19)</p>

<p>4.1 , 22 , U Texas (44)
4 , 25 , Rice (17)</p>

<p>4.8 , 5 , UC Berkeley (21)
4.2 , 19 , UCLA (25)
4 , 25 , USC (27) </p>

<p>If you think the faculty and the actual undergraduate education that is delivered to students are linked, please help explain these (sometimes signficant) gaps in colleges that many regard as regional peers.</p>

6

<p>since when is tufts a regional peer of columbia?</p>

7

<p>I don't know if they make public the questionnaire or not. It's at least 40 pages long. Yeah.</p>

<p>As for the question they ask for the peer ranking, U.S News says that respondents are asked "To rate peer schools' academic programs on a scale from 1 (marginal) to 5 (distinguished)." </p>

<p>It's hard to know exactly what respondents take into consideration in their ranking. However, it's certainly likely that excellent graduate programs have a halo effect on the regard for undergraduate programs. I think JWT86 sums it up nicely.</p>

8

<p>I agree with JWT. Here's a comparison of USNWR's undergrad PA with its grad departmental PA/ranking.</p>

<p><grad dept="" pa="" rank="">...Wisc...UIUC...WUSTL...Brown...Dartmouth
UG PA... 4.1 ... 4.0 ... 4.1 ... 4.4 ... 4.5
<business>... 3.6 ... 3.5 ... 3.6 ... NA ... 4.3
<engineering>... 4.1 ... 4.5 ... 3.2 ... 3.2 ... 2.9
<medicine>... 27 ... NA ... 3 ... 23 ... 31
<math>... 14 ... 18 ... 40 ... 14 ... 48
<computer>... 11 ... 5 ... 39 ... 20 ... 48
<biological sci="">... 15 ... 29 ... 7 ... 34 ... 34
<chemistry>... 7 ... 7 ... 43 ... NR ... NR
<physics>... 16 ... 8 ... 48 ... 29 ... 56
<earth sci="">... 15 ... 34 ... 25 ... 19 ... 49
<economics>... 11 ... 28 ... 36 ... 21 ... NR
<political sci="">... 16 ... 22 ... 16 ... 46 ... NR
<psychology>... 9 ... 5 ... NR ... 36 ... 55
<sociology>... 1 ... 34 ... NR ... 27 ... NR
<english>... 16 ... 19 ... 39 ... 15 ... NR
<history>... 11 ... 22 ... 39 ... 15 ... NR
<education>... 12 ... 25 ... 38 ... NR ... NA</education></history></english></sociology></psychology></political></economics></earth></physics></chemistry></biological></computer></math></medicine></engineering></business></grad></p>

9

<p>I would say that graduate program reputation contributes about 6% to Peer Assessment ratings. The other 94% can be explained by data associated with the undergraduate program. This is based on statistical analysis.</p>

10

<p>As is obvious by Hawkette's numbers, PA is more biased toward grad programs. Wake Forest, William & Mary, Rice, Tufts, Georgetown, Notre Dame, Vanderbilt, Emory, are all ridiculously underrated, esp compared to their peer groups. </p>

<p>Its a shame b/c high schoolers and families are the ones buying the mag and the info is set up for prospective grad students.</p>

11

<p>
[quote]
I would say that graduate program reputation contributes about 6% to Peer Assessment ratings. The other 94% can be explained by data associated with the undergraduate program. This is based on statistical analysis.

[/quote]
I'm pretty sure you're misunderstanding regression if that's your conclusion...</p>

12

<p>Igellar-
There is a 6% increase in the explanatory power of the regression when you add NRC ratings to the US News data. That is how I came to my conclusion. 94% of PA is explained by the US News data alone. When you add NRC ratings, almost 100% is explained.</p>

13

<p>^^How is Rice underrated with a 4? How is Notre Dame underrated with a 3.9? How are Wake Forest and WIlliam & Mary underrated either? Your post makes no sense these Pa scores are pretty accurate with what people think. I do agree Tufts is low though. Just like I think Berkley's is far overrated (5th in the nation--I think not)</p>

14

<p>
[quote]
There is a 6% increase in the explanatory power of the regression when you add NRC ratings to the US News data. That is how I came to my conclusion. 94% of PA is explained by the US News data alone. When you add NRC ratings, almost 100% is explained.

[/quote]
If you take the U.S. News scores and take out 25% of the variation caused by the PA (because PA counts for 25%), you have the U.S. News scores without PA. Then, if you check the R-squared for the PA scores and the U.S. News scores without PA, you get about 71%. </p>

<p>What are you doing?</p>

15

<p>
[quote]
What are you doing?

[/quote]
</p>

<p>Demonstrating (conclusively) that the Peer Assessment has the largest individual impact on the final rankings, and that all other components somewhat correlate to a final score that is dominated by the PA.</p>

16

<p>bescraze, here's the explanation:</p>

<p>Ranked in USNWR Top 25 Teaching Colleges? , % of students who felt that faculty involvement was better than most colleges, College</p>

<p>YES , 79% , Wake Forest
YES , 73% , Notre Dame
YES , 68% , Princeton
YES , 68% , Vanderbilt
YES , 64% , Dartmouth
YES , 64% , Emory
YES , 58% , U Chicago
YES , 58% , Georgetown
YES , 54% , Stanford
YES , 53% ,Tufts
no , 50% , U Penn
YES , 48% , Yale
YES , 44% , Wash U
no , 43% , Carnegie Mellon
YES , 41% , Duke
YES , 40% , Brown
YES , 38% , Caltech
YES , 38% , U Virginia
no , 36% , MIT
YES , 33% , Rice
YES , 32% , U North Carolina
no , 31% , Cornell
no , 28% ,Johns Hopkins
YES , 27% , Northwestern
no , 27% ,USC
no , 25% , Columbia
YES , 22% , Harvard
no , 21% , U Michigan
no , na , UC Berkeley
no , na , UCLA</p>

17

<p>and here</p>

<p>The website, Sparknotes, provides a lot of data about colleges, including comments that they have garnered from students at colleges all across the USA.</p>

<p>One of the questions that they ask is: “How involved is the faculty in your academic life?”</p>

<p>A series of choices were provided for students to respond to, eg,:
1) More than most. They know when I’m lying if I try to use the “my grandmother died” excuse for the third semester in a row.
2) Purely on school hours. We have a casual, in-class relationship. They see other students; I see other professors.
3) Not much. A secret service-like wall of TAs surrounds all of my professors.
4) Not at all. I once saw one of my professors through a high powered telescope.</p>

<p>Most of the answers were for #s 1 and 2, but there were a few outliers.</p>

<p>Following are the responses for each college ranked in the USNWR Top 30. Keep in mind that this is not a scientifically done survey, but I suspect that the conclusions they suggest are broadly accurate.</p>

<p>“How involved is the faculty in your academic life?”</p>

<p>More than most , Purely on school hours , Not much/Not at all , College , # of Students responding</p>

<p>79% , 18% , na , Wake Forest , 56
73% , 16% , na , Notre Dame , 105
68% , 15% , na , Princeton , 40
68% , 23% , na , Vanderbilt , 47
64% , 23% , na , Dartmouth , 47
64% , 33% , na , Emory , 36
58% , 23% , na , U Chicago , 48
58% , 38% , na , Georgetown , 66
54% , 20% , na , Stanford , 50
53% , 27% , na , Tufts , 30
50% , 35% , na , U Penn , 54
48% , 26% , na , Yale , 82
44% , 53% , na , Wash U , 34
43% , 35% , na , Carnegie Mellon , 23
41% , 41% , na , Duke , 51
40% , 45% , na , Brown , 47
38% , 31% , na , Caltech , 13
38% , 54% , na , U Virginia , 52
36% , na , 28% , MIT , 39
33% , 61% , na , Rice , 18
32% , 50% , na , U North Carolina , 68
31% , 50% , na , Cornell , 52
28% , 59% , na , Johns Hopkins , 39
27% , 51% , na , Northwestern , 45
27% , 47% , na , USC , 64
25% , 67% , na , Columbia , 69
22% , na , 34% , Harvard , 217
21% , 59% , na , U Michigan , 95
na , 64% , 17% , UC Berkeley , 58
na , 49% , 20% , UCLA , 79</p>

18

<p>here is more usnews</p>

<p>School Percent dissatisfied (categories #3 and #4)</p>

<ol>
<li>Wake Forest 3%</li>
<li>Emory 3%</li>
<li>Wash U 3%</li>
<li>Georgetown 4%</li>
<li>Rice 6%</li>
<li>UVA 8%</li>
<li>Columbia 8%</li>
<li>Vanderbilt 9%</li>
<li>Notre Dame 11%</li>
<li>Dartmouth 13%</li>
<li>Johns Hopkins 13%</li>
<li>Brown 15%</li>
<li>Penn 15%</li>
<li>Princeton 17%</li>
<li>Berkeley 17%</li>
<li>Duke 18%</li>
<li>UNC 18%</li>
<li>U Chicago 19%</li>
<li>Cornell 19%</li>
<li>Tufts 20%</li>
<li>Michigan 20%</li>
<li>UCLA 20%</li>
<li>Northwestern 22%</li>
<li>Carnegie Mellon 22%</li>
<li>Yale 26%</li>
<li>Stanford 26%</li>
<li>USC 26%</li>
<li>MIT 28%</li>
<li>Caltech 31%</li>
<li>Harvard 35%</li>
</ol>

19

<p>here is more usnews</p>

<p>School Percent dissatisfied with faculty (categories #3 and #4)</p>

<ol>
<li>Wake Forest 3%</li>
<li>Emory 3%</li>
<li>Wash U 3%</li>
<li>Georgetown 4%</li>
<li>Rice 6%</li>
<li>UVA 8%</li>
<li>Columbia 8%</li>
<li>Vanderbilt 9%</li>
<li>Notre Dame 11%</li>
<li>Dartmouth 13%</li>
<li>Johns Hopkins 13%</li>
<li>Brown 15%</li>
<li>Penn 15%</li>
<li>Princeton 17%</li>
<li>Berkeley 17%</li>
<li>Duke 18%</li>
<li>UNC 18%</li>
<li>U Chicago 19%</li>
<li>Cornell 19%</li>
<li>Tufts 20%</li>
<li>Michigan 20%</li>
<li>UCLA 20%</li>
<li>Northwestern 22%</li>
<li>Carnegie Mellon 22%</li>
<li>Yale 26%</li>
<li>Stanford 26%</li>
<li>USC 26%</li>
<li>MIT 28%</li>
<li>Caltech 31%</li>
<li>Harvard 35%</li>
</ol>

20

<p>What I am doing is multiple regression to determine whether PA can be explained by hard data. The 25% weight given to the PA in calculating the overall score by US News is something different.</p>

<p>People who make the PA ratings are asked to base their ratings on the quality of undergraduate programs. The statistical analysis shows that 94% of the PA can be explained by the hard data listed in US News Best Colleges such as graduation rate, student-faculty ratio, SAT scores, acceptance rate, alumni giving rate, and so on. When I add ratings of graduate programs from the NRC database, the combination of US News hard data and NRC ratings of graduate programs adds about 6% to the explanatory power of the US News hard data alone.</p>

<p>So, I conclude that 6% of the PA can be attributed to the quality of the graduate programs. I've answered the OP's original question.</p>