The Wisdom of US News Peer Assessment Rating

<p>Collegehelp, would you please clarify the following,</p>

<ol>
<li><p>Are the “weights” you listed in message # 17, just the raw weights to be multiplied with the input data, which aren’t normalized? Then it would be more revealing to list the products of the “weights” with the input data and present as percentages of the total, to show the relative contribution of each category.</p></li>
<li><p>How do you numerically input private vs. public? +1 vs. 0? And does Cornell count as private or semi-private?</p></li>
<li><p>How much do Berkeley’s and Michigan’s actual PA scores deviate from this modeling?</p></li>
</ol>

<p>Collegehelp, I often use multiple regression in my field and I don’t normally consider such a mix of variable transformations (mix of linear, squared and natural log) unless there is reason to believe that it makes sense (like velocity squared being acceleration). What was the relationship with those variables only in linear relationships?</p>

<p>Yes, the weights stay the same for all colleges.</p>

<p>The weights are the raw weights to be multiplied by the input data to calculate an estimated PA score.</p>

<p>Public and private are coded as 10 and 11 by IPEDS, as I recall. I think Cornell is private, I will have to check.</p>

<p>I think I was able to find a model with an R-squared of about .9 (90%) without using transformed data. So the transformed versions led to a small gain. Squaring the data helps capture curvilinear relationships with one inflection. I think logs help linearize data that is curvilinear. Transformations like this are done to better fit a linear model.</p>

<p>Regarding, the deviations from the predicted values for Michigan and Berkeley and calculating the actual contribution of each factor after weighting, that will have to wait until tomorrow when I am back at school which is where my data is.</p>

<p>The data is all objective, correct?</p>

<p>Then the conclusions based on the data must be accurate.</p>

<p>TOP RESEARCH UNIVERSITIES IN THE 2005 FACULTY SCHOLARLY PRODUCTIVITY INDEX</p>

<p>The 2005 index compiles overall institutional rankings on 166 large research universities, which include 15 or more Ph.D. programs, as well as 61 smaller research universities, which contain between one and 14 Ph.D. programs. Here are the top 50 large research universities and the top 20 smaller ones.
Large Research Universities
Rank Institution Faculty Scholarly Productivity Index Number of programs
1 Harvard U. +1.68 38
2 California Institute of Technology +1.59 19
2 U. of California at San Francisco +1.59 15
4 Massachusetts Institute of Technology +1.44 26
5 Yale U. +1.35 55
6 Carnegie Mellon U. +1.18 27
7 Washington U. in St. Louis +1.16 33
8 Vanderbilt U. +1.09 48
9 Johns Hopkins U. +1.08 49
10 Duke U. +1.07 52
11 U. of Pennsylvania +1.06 55
12 Princeton U. +1.03 43
12 U. of California at Berkeley +1.03 70
14 U. of Wisconsin at Madison +0.90 83
15 New York U. +0.89 56
15 Stanford U. +0.89 52
17 U. of Washington +0.82 79
18 U. of Virginia +0.81 48
19 State U. of New York at Stony Brook +0.80 41
20 Cornell U. endowed colleges +0.73 68
20 Dartmouth College +0.73 21
22 Emory U. +0.71 41
22 Rice U. +0.71 27
24 Georgia Institute of Technology +0.69 29
25 U. of North Carolina at Chapel Hill +0.67 56
26 Columbia U. +0.66 59
27 U. of Michigan at Ann Arbor +0.65 74
28 Northwestern U. +0.64 46
28 Pennsylvania State U. +0.64 85
28 U. of California at San Diego +0.64 33
31 U. of Maryland at College Park +0.60 68
32 U. of Southern California +0.57 58
33 U. of Chicago +0.56 40
34 U. of Illinois at Urbana-Champaign +0.55 71
35 Case Western Reserve U. +0.54 34
36 City U. of New York Graduate Center +0.52 31
37 U. of Iowa +0.46 66
38 Michigan State U. +0.43 76
38 U. of California at Los Angeles +0.43 64
38 U. of California at Santa Barbara +0.43 46
41 U. of California at Davis +0.41 60
41 U. of Kentucky +0.41 50
43 U. of California at Irvine +0.40 36
44 U. of Illinois at Chicago +0.34 43
45 Indiana U. at Bloomington +0.31 62
46 Boston U. +0.30 32
46 Purdue U. +0.30 57
46 Rensselaer Polytechnic Institute +0.30 23
46 U. of California at Riverside +0.30 36
50 U. of Texas at Austin +0.28 68</p>

<p>COHE and Academic Analytics. Wisconsin and Penn State have the most ranked programs. Darn impressive.</p>

<p>barrons-
Isn’t the faculty scholarly productivity index done at SUNY Stonybrook? They place pretty high. It makes me wonder whether they know how to sqeeze the most out of the method.</p>

<p>Don’t get me wrong; I like the FSPI. </p>

<p>dstark-
Yes, all the data is objective and quantitative but there is always interpretation involved.</p>

<p>

</p>

<p>

</p>

<p>Haha, could we skip the next 100 posts and jump to the conclusion most of us has read a few times? By now, we know that the correlation of a large pool will bury the outliers. And, we know that the outliers are exactly what constitute the bone of contention for most who refute the “validity” of the PA. It is the Smith College versus Harvey Mudd “discrepancy” as well as the stratospherically high score from that little school by the bay that stick out, not the averages of the averages. </p>

<p>Until the day when USNews makes the entire survey public and allows outsiders to measure the true degree of cronyism, gamesmanship, and manipulation, the PA will remain what it is … a mere tool, a prop, a device to level the playing field between public and private universities and a tool to protect the grande dames of yesterday’s liberal arts education. The objective AND use of the PA are clearly established by USNews, and Morse will never make the adjustments needed nor incorporate a modicum of integrity in the process. </p>

<p>Fwiw, I really believe that Morse and his staff must laugh uncontrollably at all the CC “intelligent” discussions on this issue.</p>

<p>"Yes, all the data is objective and quantitative but there is always interpretation involved. "</p>

<p>No can’t be any interpretation involved. It’s objective. ;)</p>

<p>xiggi, I knew you’d weigh in on one of our favorite talking points. :)</p>

<p>

I’m glad you seem to agree that Stanford’s PA score is too high… :D</p>

<p>HAHAHAH. I love Stanford vs. UC Berkeley rivalry :D</p>

<p>Fascinating analysis, Collegehelp.</p>

<p>

</p>

<p>xiggi: yeah, that Junior University on The Farm really does stick out. :D</p>

<p>Here are the predicted Peer Assessment ratings compared with the actual Peer Assessment ratings. The correlation between actual and predicted is +.96 (very high). Yet there are some anomalies (see U Chicago). Michigan’s prediction is almost perfect. Berkeley’s prediction is very close. </p>

<p>When I see the discrepancy for U Chicago, I am inclined to think that the actual PA is correct and my prediction model doesn’t work well for Chicago. Hence the title of this thread: “The Wisdom of US News Peer Assessment”. Sometimes there is wisdom in the collective judgement of experts that can’t be captured in the numbers.</p>

<p>rank, actual PA, predicted PA, school</p>

<p>1 4.9 4.8 Harvard University
2 4.9 4.9 Massachusetts Institute of Technology
3 4.9 4.9 Stanford University
4 4.8 4.9 Princeton University
5 4.8 4.8 Yale University
6 4.7 4.4 University of California-Berkeley
7 4.6 4.7 California Institute of Technology
8 4.6 3.9 University of Chicago
9 4.5 4.5 Columbia University in the City of New York
10 4.5 4.3 Cornell University
11 4.5 4.2 Johns Hopkins University
12 4.5 4.4 University of Pennsylvania
13 4.4 4.5 Duke University
14 4.4 4.3 University of Michigan-Ann Arbor
15 4.3 4.5 Brown University
16 4.3 4.4 Dartmouth College
17 4.3 4.2 Northwestern University
18 4.3 4.1 University of Virginia-Main Campus
19 4.2 4.3 University of California-Los Angeles
20 4.1 4.2 Carnegie Mellon University
21 4.1 3.7 University of North Carolina at Chapel Hill
22 4.1 3.9 University of Wisconsin-Madison
23 4.1 4.3 Washington University in St Louis
24 4.0 3.9 Georgetown University
25 4.0 3.6 Georgia Institute of Technology-Main Campus
26 4.0 4.1 Rice University
27 4.0 3.9 The University of Texas at Austin
28 4.0 4.2 University of Illinois at Urbana-Champaign
29 4.0 4.1 Vanderbilt University
30 3.9 3.9 Emory University
31 3.9 4.0 University of Notre Dame
32 3.9 4.0 University of Southern California
33 3.9 3.5 University of Washington-Seattle Campus
34 3.8 3.8 New York University
35 3.8 3.4 University of California-Davis
36 3.8 3.9 University of California-San Diego
37 3.7 3.6 College of William and Mary
38 3.7 3.5 Indiana University-Bloomington
39 3.7 3.9 Pennsylvania State University-Main Campus
40 3.7 3.5 Purdue University-Main Campus
41 3.6 3.5 Ohio State University-Main Campus
42 3.6 3.9 Tufts University
43 3.6 3.8 University of Florida
44 3.6 3.7 University of Maryland-College Park
45 3.6 3.5 University of Minnesota-Twin Cities
46 3.5 3.8 Boston College
47 3.5 3.6 Brandeis University
48 3.5 3.4 Rensselaer Polytechnic Institute
49 3.5 3.7 Texas A & M University
50 3.5 3.2 University of Arizona
51 3.5 3.5 University of California-Irvine
52 3.5 3.5 University of California-Santa Barbara
53 3.5 3.2 University of Iowa
54 3.5 3.4 Wake Forest University
55 3.4 3.4 Boston University
56 3.4 3.5 Case Western Reserve University
57 3.4 3.2 George Washington University
58 3.4 3.6 Michigan State University
59 3.4 . University of Colorado at Boulder
60 3.4 3.3 University of Georgia
61 3.4 3.5 University of Pittsburgh-Pittsburgh Campus
62 3.4 3.5 University of Rochester
63 3.3 3.4 Rutgers University-New Brunswick
64 3.3 3.1 Syracuse University
65 3.3 3.4 Tulane University of Louisiana
66 3.3 2.9 University of Oregon
67 3.3 3.5 Virginia Polytechnic Institute and State University
68 3.2 3.1 Arizona State University
69 3.2 3.4 Lehigh University
70 3.2 3.2 Miami University-Oxford
71 3.2 2.9 Stony Brook University
72 3.2 3.1 University of Massachusetts Amherst
73 3.2 3.2 University of Miami
74 3.2 3.1 University of Missouri-Columbia
75 3.1 2.9 Baylor University
76 3.1 3.2 Clemson University</p>

<p>The FSPI is better for some fields than others. Last time we examined their methodology, for example, they were counting publications the same across fields, even though in academe books carry more scholarly “weight” in some fields, less than others.</p>

<p>I also don’t like the way they “weight” awards longitudinally, instead of directly. In any given year a Fullbright is as important as a Nobel.</p>

<p>Perhaps they’ve changed it since then.</p>

<p>Here are the individual correlations between the input variables and the actual PA ratings:</p>

<p>.72 graduation percent (squared it was .76)
.71 SAT math 75th
.68 SAT math 25th
.66 SAT CR 75th
.65 SAT CR 25th
.56 retention after one year
.49 admissions percent
.32 endowment per FTE
.23 research expenditures percent
.09 public or private
.04 size of freshman class
.04 number of bachelors degrees awarded
.02 public service expenditures percent
.01 yield
0 academic support expenditures percent
0 highest degree offered
0 instructional expenditures percent</p>

<p>This line of thinking that is rampant on CC cracks me up. This is like trying to come up with a mathmatical formula to decide who is a better artist Warhol v Wyeth v Picasso v Bernini. Its not surprising that the big research schools do well and the schools more focused on teaching do poorly. Sometimes you miss the forest for the trees. Penn St and Ohio St over Tufts?? Illinois over Emory Vandy and Notre Dame?? UNC over Wash U and Georgetown? </p>

<p>How about some common sense here. Answer me this. You need emergency heart surgery. What diploma do you want to see on the wall before the surgeon cuts into your chest? </p>

<p>Use this criterion to rank schools and I guarantee you wouldnt see Indiana over Tufts, Brandeis, Boston College or Wake Forest.</p>

<p>Here are the best models with various numbers of input variables:
The R-square is a measure of how much of the variability in PA is accounted for by that combination of input variables. The maximum is 1.</p>

<p>Perhaps the most parsimonius model is the 4-factor version.</p>

<p>.76 graduation percent squared</p>

<p>.83 SAT math 75th squared, freshman class size natural log</p>

<p>.87 SAT math 75th squared, graduation percent squared, freshman class size natural log</p>

<p>.90 SAT math 75th, freshman class size, SAT math 75th squared, graduation percent squared</p>

<p>.91 SAT math 75th, freshman class size, research $ percent, SAT math 75th sq, graduation percent squared</p>

<p>.92 SAT math 75th, admissions percent, yield, freshman class size, research $ percent, SAT math 75th squared, graduation percent squared, admissions percent natural log</p>

<p>doctorb-
Wouldn’t it matter more where the surgeon went to medical school?</p>

<p>You know, Collegehelp’s analysis is similar to finding that there’s a very high correlation between baseball players’ offensive statistics and their likelihood of being voted onto their league’s All-Star team. I have no doubt we’d find that that’s indeed the case. But even then, I doubt we’d generate a correlation of .96.</p>

<p>doctor:</p>

<p>Depends on the training. A grad from Hopkins med who was chief resident at U-Dub (known for primary care) wouldn’t give me a great comfort level…</p>