USNWR Rankings - The Metrics

College Search & Selection
81

<p>tk, unfortunately, I have seen no evidence that statistical data can accurately determine institutional quality because there isn’t a standardized system that is properly observed and audited. Universities have different ways of listing SAT/ACT ranges, of classifying classes and class size, of calculating student to faculty ratios, of calculating financial resources, even of classifying alumni giving rates. Many universities have been playing the rankings game by manipulating statistical data. </p>

<p>At any rate, even if the data were reported in a uniform and standardized manner across the entire range, it would still not give a complete picture. But as it stands, statistical data is completely worthless.</p>

82

<br>

<br>

<p>Maybe that is the whole point. The PA has a flywheel effect. Transient movements in admit rate, etc., have to overcome the inertia of collective judgements based on previous rankings.</p>

83

<p>You only took part of my quote, tk. “You may be right that this “tightness” means that the PA have little impact, but one really cannot say that just because the scale is only a 5 point scale and therefore the numbers tend to vary by a few tenths in many cases.” My point is that just saying things look tightly bunched may not mean a thing, if small movements have a significant impact on the final ranking. Maybe that is what you meant too, I am not sure.</p>

<p>In any case, I see some of these peer assesments and have to wonder how much is due to proximity, sheer size, sports fame leading to more general awareness, etc. Or for that matter if having one famous area (Iowa for writing, for example) leads to an inflated judgement for the entire school. Not that there is anything wrong with Iowa, but I have to question whether or not it really deserves a PA above GW, Pitt, Tulane, Lehigh, Fordham, etc. Seems off to me, but that is just an opinion, which may be the whole point. People can say these are informed opinions, but based on conversations I have had with profs from many schools, they say they know very little about other schools outside of their area of expertise, and feel this kind of judging is total BS.</p>

84

<p>

Perhaps USNWR should throw out the PA score and use faculty academic merit valuations (ie. academy membership, Nobel prize, etc.) instead. USNWR current methodology does not include any measure for faculty quality…however, the PA score is a crude method for this.</p>

<p>And if you don’t think current PA scores are for undergraduate programs, I point to Berkeley’s score, which should be 4.9 to 5.0 if looking only at depth and breadth of its faculty and academic program offerings. However, it is not as high as it could be due to Berkeley’s larger undergraduate population using up the resources offered…I think PA scores take these “intangible” factors into account…Just as USNWR says they do.</p>

85

<p>After the 2009 USNWR rankings were released, I did a reranking of the USNWR Top 50 national universities WITHOUT the PA scores. It’s not exact because all of the USNWR data is not publicized, but it’s pretty close. </p>

<p>Below are the 2009 Rankings Ex-PA Scores and their change from the regular USNWR rankings. I don’t think that there were big changes in 2010.</p>

<p>Ex-PA Ranking , Change from 2009 USNWR Ranking , College</p>

<p>1 , 0 , Harvard
2 , 0 , Princeton
3 , 0 , Yale
4 , 2 , U Penn
5 , 1 , Caltech
6 , 2 , Duke
7 , 1 , Columbia
8 , -4 , Stanford
9 , 3 , Wash U
10 , -6 , MIT
11 , 0 , Dartmouth
12 , 0 , Northwestern
13 , 3 , Brown
14 , -6 , U Chicago
15 , -1 , Cornell
16 , 2 , Notre Dame
17 , 0 , Rice
18 , -3 , Johns Hopkins
19 , -1 , Emory
20 , -2 , Vanderbilt
21 , 2 , Georgetown
21 , 7 , Tufts
23 , 4 , USC
24 , -2 , Carnegie Mellon
25 , 3 , Wake Forest
26 , 5 , Brandeis
27 , -4 , U Virginia
28 , 7 , Lehigh
29 , -8 , UC Berkeley
30 , -5 , UCLA
31 , 4 , U Rochester
32 , -2 , U North Carolina
33 , 17 , Yeshiva
34 , 0 , Boston Coll
35 , -2 , NYU
36 , 5 , Rensselaer
37 , -11 , U Michigan
38 , 3 , Case Western
39 , -7 , W&M
40 , 4 , UC Santa Barbara
41 , 10 , Tulane
42 , 2 , UC Irvine
43 , -8 , Georgia Tech
44 , -9 , U Wisconsin
45 , -10 , UCSD
46 , -6 , U Illinois
47 , -3 , UC Davis
48 , 1 , U Florida
49 , -8 , U Washington
50 , -3 , U Texas
50 , -3 , Penn State</p>

86

<p>

</p>

<p>That is exactly the point of PA scores. PA scores are measuring distinguished academic programs. Iowa’s creative writing program is a top-notch, well regarded, distinguished academic program - so its PA score is enhanced versus others.</p>

87

<p>

Proof that PA scores should remain. There is much more to a college than being small and selective.</p>

88

<p>In the opinions of the students who are actually partaking of these “distinguished academic programs,” you find the following grades for academics:</p>

<p>A- Tulane
A- Lehigh
B+ George Washington
B+ U Pittsburgh
B Fordham
B U Iowa</p>

<p>A high ACADEMICS grade generally indicates that professors are knowledgable, accessible and geniunely interested in their students’ welfare. Other determining facors include class size, how well professors communicate, and whether or not classes are engaging.</p>

89

<p>^ Hawkette, I think those grades say more about the size of the college than anything else. Do you think U Iowa’s creative writing program has huge class sizes and out-of-touch profs?</p>

90

<p>I don’t know, but U Iowa has about 21,000 undergrads. I think that several hundred students gave their opinions on what they received in the classroom. My guess is that their views represented a much broader cross section of classes than just creative writing. And IMO rightly so. The entire school should not be graded based solely or even mostly on how its Creative Writing program is perceived.</p>

91

<p>Do you know what percentage of the undergrad population responded to that survey?</p>

92

<p>

Wow, that seems wildly off base to me. I don’t at all agree with your other statement about Nobel Prizes and the like either. That’s great for grad school, but for undergrad not so much. Anyway, having one great program like writing (and not saying that is all Iowa has, but…) does not represent the holistic program, but it might have a severe yet undeserved halo effect. People naturally think “well if that program is good then probably others are too” but that might not be so at all. I suppose it is better than being ranked higher than deserved because of sports, but still highly flawed.</p>

93

<p>^ I’m not saying I agree with it, I’m saying that those are the factors that go into peer assessment. You’re asking academics to rate schools…Academics will look for things important and distinctive to them - Nobel prizes and the like are the epitome of academic distinction - to ACADEMICS (they’re the ones asked to fill out the PA survey). </p>

<p>U Iowa has a well known academic program (creative writing) that could be classified as a “distinguished” program. I cannot name an academic program from Tulane or Fordham that is tops in its respective field (ie distinguished).</p>

94

<p>Hawkette, I do not believe those “teaching” grades. It think some universities instruct their students to give A+ratings while other universities do not interfere. I know that some Business schools do that while filling the BW teaching survey. Any university that gets an A+ or even an A for teaching is manipulating data, I guarantee it. While at Michigan, even award winning professors (teaching awards) who really were excellent instructors, such as Brian Coppola and Sidney Fine, seldom got ratings better than 4.2 or 4.3 out of 5. That is not surprising. At top universities, you are bound to have (in great numbers I might add):</p>

<p>1) Students with huge egos who are never pleased and always want more
2) Students who get lower grades than they expected/anticipated</p>

<p>Those students will give poor reviews, no matter what. I find it very suspecious when a universities has a faculty that averages A+ or A for teaching. Are those ratings audited by an independent agencies? Are the students chosen to fill those forms picked at random or are did they fit a specific profile? Were they instructed on how to fill the forms? etc… I am always very skeptical of student-based data.</p>

95

<p>Alex,
I understand your concerns about student opinions. We all want accuracy and no one wants to be misled by a bad grade from a few disgruntled students. </p>

<p>For that reason, I look at multiple sources and see how much the student opinions change from one publication to the next. In most cases, they don’t. Heck, the most surprising discovery to me was how consistent different sources (eg, CP, SN, BW) were in their assessments, including for your U Michigan. </p>

<p>I’ve also looked at several sites where students reviewed individual profs. However, I think that these are least useful because you will have the disgruntled student effect here.</p>

<p>For an individual institution, the NSSE reports can be very informative. It may be hard to compare directly to another school, but you will find different levels of satisfaction and enthusiasm at many schools and this can be instructive. </p>

<p>No one student opinion student opinion source is definitive, but collectively they provide important and usually accurate signals about how students at an institution view the quality of the product that they receive in the classroom. Students usually know good teaching when they see it. </p>

<p>For colleges ranked in the USNWR Top 75 national universities and the Top 40 LACs that graded at B or better, here is how one source grades the Academics offered:</p>

<p>GRADE OF A+ </p>

<p>Caltech
Dartmouth<br>
MIT
Princeton<br>
Stanford<br>
U Chicago<br>
Bowdoin
Harvey Mudd
Williams</p>

<p>GRADE OF A </p>

<p>Brown<br>
Carnegie Mellon
Columbia<br>
Duke<br>
Emory<br>
Harvard
Northwestern<br>
Rice<br>
Tufts<br>
U Penn<br>
U Rochester
Vanderbilt<br>
Yale<br>
Amherst
Bard
Barnard
Bryn Mawr
Carleton
Claremont McKenna
Davidson
Macalester
Middlebury
New College
Oberlin
Pomona
Reed
Scripps
Smith
St.Olaf
Swarthmore
Ursinus
Vassar
W&L
Wellesley</p>

<p>GRADE OF A- </p>

<p>Brandeis<br>
Cornell
Georgetown<br>
Georgia Tech<br>
Johns Hopkins<br>
Lehigh<br>
McGill<br>
Notre Dame<br>
Tulane<br>
U Virginia<br>
UC Berkeley
UCLA<br>
USC
W&M
Wash U<br>
Bates
Belmont
Clark
Colby
Colgate
Colorado College
Connecticut College
Fashion Institute
Grinnell
Hamilton
Hanover
Hastings
Haverford
Kenyon
Lawrence
RI School of Design
Trinity
Wesleyan
Williamette</p>

<p>GRADE OF B+</p>

<p>Boston College
Case Western
George Washington
NYU
Purdue
Rensselaer
U Illinois
U Maryland
U Miami
U Michigan
U North Carolina
U Pittsburgh
U Texas
UC San Diego<br>
UC Santa Barbara<br>
Virginia Tech<br>
Wake Forest
Albion
Alverno
American
Babson
Birmingham Southern
Bucknell
Catholic
Centre
Holy Cross
Wooster
Depauw
Dickinson
Earlham
Emerson
Franklin & Marshall
Grove City
Hampshire
Howard
Illinois Wesleyan
Lafayette
Mount Holyoke
Muhlenberg
Occidental
Pitzer
Rhodes
Rochester Inst. Technology
St. Joseph’s
Sarah Lawrence
Susquehanna
U Puget Sound
U Richmond
Valparaiso
Villanova
Wheaton (IL)
Wheaton (MA)
Whitman</p>

<p>GRADE OF B </p>

<p>Boston University<br>
BYU
Fordham
Indiana U<br>
Ohio State
Penn State
Rutgers
SMU
Syracuse
U Connecticut
U Delaware
U Florida
U Georgia
U Iowa
U Minnesota
U Washington
U Wisconsin
UC Davis
UC Irvine
UC Riverside
UC Santa Cruz<br>
Alfred
Allegheny
Ball State
Beloit
Bentley
Bradley
Cal Poly State U
Centenary
College of Charleston
Creighton
Denison
Duquesne
Elon
Furman
Geneva
Gettysburg
Gonzaga
Goucher
Guilford
Gustavus Adolphus
Hollins
Idaho State
Illinois State
Ithaca
Juniata
La Roche
Lewis & Clark
Loyola (MD)
Loyola Marymount
Luther
Marlboro
Marquette
Millsaps
NC State
Northeastern
Ohio Wesleyan
Pepperdine
Providence
Rollins
Santa Clara Univ
Seattle
Seton Hall
Simmons
Skidmore
Southwestern
Spelman
Stetson
Texas Tech
Trinity (TX)
Truman State
Union
U Buffalo
U Arizona
U Cincinnati
U Denver
U Kansas
U Mass
U Nebraska
U Oklahoma
U Rhode Island
U San Diego
U South Carolina
U South Dakota
U Vermont
Virginia Commonwealth
Warren Wilson
Washington & Jefferson
US Military Academy
Xavier</p>

<p>GRADE OF B- </p>

<p>Clemson
Michigan State<br>
Texas A&M<br>
Arizona State
Auburn
Baruch
Baylor
Binghampton
Cal Poly Pomona
Cal State Northridge
Clark Atlanta
Drexel
East Carolina
Florida State
Hampton
Hofstra
Hunter
IUPUI
Iowa State
James Madison
Kansas State
Kent State
LSU
Loyola (IL)
Miami U
Mid Tenn State
Montana State
Northern Arizona
Northern Illinois
Ohio
Old Dominion
Sacramento State
San Diego State
Slippery Rock
St. John’s
St. Louis
Stony Brook
Temple
Tennessee State
TCU
Towson
U Albany
U Alabama
U Central Florida
U Colorado
U Kentucky
U Maine
U Mississippi
U Missouri
UNLV
U New Hampshire
U Oregon
U San Franscisco
U South Florida
U Tennessee
U Utah
U West Virginia
Wilkes</p>

96

<p>

</p>

<p>UCB, fully knowing that we will never agree about the validity of the PA, allow me to point to the multiple threads of the past where a few members of this august forum TRIED to explain what the PA actually was, and justify its existence. And this with little success! Except for one notable effort in statistical science, it is a given that most of the supporters like it for one particular and obvious reason: it boosts the ranking of their favorite school. There is no need to cite names … you know who you are! </p>

<p>However, it really does not matter that the fans cannot agree on what the PA truly measures (as the past threads demonstrate the amusing contradictions in successive posts by the same fans, especially when discussing quality of education versus “perception of the perception of the reputation of the reputation of a faculty” or something even more nebulous. </p>

<p>Perhaps we should simply look at the content of the questionaire sent to the good people who are asked to share their wisdom and knowledge in an impartial manner. You do have the list of the question asked, don’t you? Since you and your Michigan friends always speak with such authority about what the PA “really measure” and how the opinion of professional in education bests our amateurish speculation, may I assume that you have read the PA survey and know the intimate details of the very precise questions that are posed to the Presidents, Provosts, and other academic luminaries who haunt our ivory towers of education. </p>

<p>To make my life easier and educate this forum, would you mind sharing the first 10-12 questions of the Peer Assessment Survey? And perhaps, let us know how many questions adorn that amazing piece of research. Because it must be amazing with rich details and precision, isn’it?</p>

97

<p>There is a FAR more acceptable survey that many schools use for such purposes as student satisfaction. It’s NSSE. You still have the problem of lack of comparability–is a happy student at Harvard the same as a happy student at Cal? But at least the methodology is more statistically accurate as students are selected at random with follow-up etc.</p>

98

<p>Tulane’s Latin American Studies Program as well as its Finance department are among the best. </p>

<p>I think the problem most have with the PA, in theory, is that it’s 25% of the overall score. I personally think the PA score is good idea. At 25% though, it could lead admins and academics to introduce their bias to influence the overall rankings. It also introduces a great deal of regional bias. The main bias that would be introduced though, is that if you’re an academic in say economics for example, you will be extremely familiar with schools that have strong econ programs. You might not know anything of schools with english as their focus. </p>

<p>I have a lot less of a problem with public or large universities getting a benefit from the PA. The nature of their size,funding, ect can too often hurt them in other stats.</p>

99

<p>

Xiggi, you’ve been around the block enough here to have seen the PA form.</p>

<p>It lists all national universities (all LACs for those asked to evaluate LACs) and asks the participant to circle a score of 5, 4, 3, 2, 1, or “don’t know” about the school’s academic program offerings - with 5 being “distinguished” to 1 being “marginal”. </p>

<p>It’s a simple form.</p>

<p>You need to keep it simple to encourage participation.</p>

<p>I understand what you are getting at…I’m just explaining what I think goes into academics minds when they rate a college’s academic programs as 5 = distinguished or a 4 = strong.</p>

100

<p>Hawkette, Michigan’s rating of B+ for instruction is fair. I have trouble believing any institution that grants itself a rating of A or A+. Any rating that gives a university a rating of A+ or A for instruction is flawed. </p>

<p>My larger issue here is the claim that quality of instruction can be measured. It cannot. It is impossible to come close to an accurate rating for instruction.</p>