<p>While it’s true that the intention of the R-based method may be to “systematize” what evaluators FEEL (not think, the S-based is more to what the evaluators think are the right criterions for a strong program) are the “better” programs, my previous post has amply demonstrated the “systematization” procedure of the R-based methodology is nothing short of ridiculous. I’m sure no evaluator is willing to come out and say “I think having private work space for graduate students is the most important criteria of a successful program”.</p>
<p>That said, I’m having immense trouble why you would find the survey-based (S) method (i.e. asking evaluators to list what they think are the more important criterions of a good program) weird. To me, the R & S ranking is a reflection of “what I feel is right” and “what I know is right”.</p>
<p>Does anyone other than me, admittedly biased, find themselves amazed at how well Princeton did? Given its small size? And location in suburban New Jersey?</p>
<p>I looked at the English Department ranking of Berkeley. Now, for all of my adult life (and before it), Berkeley has been considered the #1 or #2 English program. Maybe #3. In the R method, it is essentially tied with Yale (another former #1 or #2) for 5th behind a four-way tie for first (among other recognizably strong programs), and has a range of 2-17, or something like that. In the S rankings, its range is 27-56, which works out to 42nd place. Well, guess what? There aren’t 41 better places to get an English PhD than Berkeley. There aren’t 26 better places, either. I’m not positive there are 4 better places, but I could certainly understand how a strong case could be made for that. Maybe even 7 or 8 better places. Not 20+</p>
<p>The S rankings suggest that the English PhD programs at Purdue and Berkeley are comparable, but Purdue is probably a little better. A student would have to have his head examined to pursue an English PhD at Purdue rather than Berkeley, without some really idiosyncratic reason, and without being certain he didn’t want an academic job teaching English. That’s why I characterize the S results as “weird”.</p>
<p>Now, I’m sure that the S results reflect something rational and important. If I were the Provost or Dean at Berkeley, or chair of the English department, I would be looking very hard at why it comes up so short in so many areas people say are important quality measures for an English department. Long-term, those S results auger badly for Berkeley English, and probably explain why its R range doesn’t include #1 and extends below #10. Something is rotten there. But 40 places rotten? That’s three or four tiers as a practical matter. Hasn’t happened yet.</p>
<p>I have also been looking closely at the rankings in a social science field in which one of my children is planning to apply to PhD programs. The R rankings pretty much correspond to what my child’s faculty mentors advise, once one takes into account that different universities focus more or less on different aspects of the field. The S rankings don’t.</p>
<p>Thanks for your detailed explanation. I agree that on paper, the S ranking may have done Berkeley a huge disservice, while the R ranking has saved Berkeley from a total fall from grace. I’m not well-informed on the reputational status of the English Department @ Berkeley, but if what you say is true (that Berkeley has traditionally been considered to be #1 or #2), then the better ranking on the R-measure makes total sense (since the R methodology is based on correlating weights of variable based on reputational surveys). I suspect your child’s faculty mentors were also advising from their impressionistic view on which department are more reputable, thus the stronger correspondence with R ranking.</p>
<p>However, as “weird” as the S-ranking for Berkeley’s English Department is, it is as you say a “rational and important” methodology that doesn’t “reek of molding data to fit the desired results” (to quote another poster on this trend). From this perspective, the R ranking is a whole lot more unreliable as a reflection of real strength. As to why Berkeley has done so poorly on the more sensible S methodology, I suggest a closer look at the basics of how the ranking is done, namely what the weights are for the various variables under the S methodology for English Departments (as I have done for Linguistics). Perhaps an answer could be found there.</p>
<p>limni – I haven’t undertaken that because (a) I don’t have unlimited time to waste, and (b) I don’t care ALL that much. I strongly suspect that the essential problem, at least in something like English, is that things with numerical measures tend miss the point of what makes a faculty and a program great. You can count publications and citation, you can count how long it takes people to finish and whether they finish and whether they have jobs, and student GRE averages, and all of that matters somewhat, but it doesn’t tell you how thrilling anything is. Or the fact that if as many people from Purdue have jobs as from Berkeley, the Purdue people are all teaching in community colleges, and some significant number of the Berkeley people are teaching at Harvard or Yale, etc.</p>
<p>Is this the place where I should gloat about how well my program did in the S-Rankings? Because that’s totally what I feel like I should be doing.</p>
<p>I have taken a quick look at the S rankings. They are not specific enough to accomplish what the NRC authors intend them to do. For example, # of citations does not qualify WHICH publications… important ones? Well regarded ones? There are probably ten or so “metrics” used in the S Rankings that are too gross in what they measure, that flatten real, qualitative differences into nothingness.</p>
<p>While the S rankings are a good enough idea, they are not implemented properly.</p>
<p>The R rankings are composed of reputational subcategories with dozens of variables stored in the brains of the voters over 10,20, 30 years, For example, where is the S ranking category for “% of alumni securing Assistant or Associate Professorships at Top 10 Programs within x period of time?” Those observations are stored in the brains of the voters and compose a portion of the R rankings. The R rankings measure very fine differences betwen programs that the S rankings are not designed to measure. The S Instrument needs refining. It is too blunt.</p>
<p>DunninLA – I agree with you in general, but both sets of rankings suffer from the same problem: No one has measured % of graduates (and I would argue for number of graduates, not percent) in top-10 faculties (a circular problem, anyway). So while people are basing their reputational judgments on something like that, the computers are deciding that the measured statistical factor most closely correlated with the preference for well-respected programs is the inverse of the number of bathrooms per student, and so every program with a bathroom shortage gets a little boost in the R rankings.</p>
<p>I agree with you that the variables used by NRC maybe too blunt to measure the quality you want at that specific level.</p>
<p>However, also note that both S and R ranking uses the SAME variables and the SAME data, when calculating the rankings. Therefore your criticisms of gross-ness in the variables will apply to both S and R rankings.</p>
<p>Furthermore, while the evaluators may or may not have the “reputational subcategories with dozens of variables stored in the brains” you suggested of them (but honestly, I don’t think the evaluators did their reputational survey by hastily calculating the “dozen of variables” stored in their brains), it remains that the reputational surveys of the R-based method is not used to rank the departments directly, but as a means to CALCULATE THE WEIGHTINGS to be applied to the variables. This regression process has resulted in a illogical and untenable weighting of the variables used to do R rankings, as I have demonstrated (I think this is also what JHS was getting at in post #70).</p>
<p>As such, perhaps an improvement in the specificity of the variables is in order for the next round of NRC exercise, but at this moment, I think the S ranking would still be more reliable and tenable than the R ranking.</p>
<p>P.S. (at JHS’s post #67): Indeed, in pure humanities subject like English and English literature, immeasurable qualities like how thrilling a topic is to the individual can hardly be validated to determine how great the department is. Unfortunately, I think no objectively based ranking can rectify that, bar a total regression to a ranking based on hearsay (i.e. reputation).</p>
<p>I have university assessment from 1925-1982, The columns are for the rankings in order from 1925(Hughes), 1934(Hughes), 1959(Keniston), 1965(Cartter) ,1970(Roose), 1979(Ladd/Lipset) and the 1982 assessment, 1993 and 2010. I thought that it would be interesting to average out those years. I was surprised to see Michigan, Wisconsin, Harvard, Berkeley, and Yale as the only schools to stay in the top 10 during that period. I used the summary from this blog as the top 10 although I realize that it is by no means the final rankings, but it is probably very close. the last column is the average rank, sorry I wasn’t able to make the grid fit on this page.</p>
<p>I found an even better example of disparity between R- and S- rankings: UCLA’s sociology department. In the R- rankings, its range is 8-25, and it basically comes 14th (but in a virtual tie with ## 12 and 13). In the S- rankings, its range is 49-86, and it is in 65th place.</p>
<p>Now, based on external information I “know” UCLA’s sociology department is well-regarded, considered top-10 (maybe) and certainly top-20. The R- rankings reflect that. (Especially given that some universities show up more than once in the rankings for different programs, so that the top 10 includes Harvard twice and two other programs that are not mainstream sociology.) It’s hard to imagine how it could fall so short on everything everybody ostensibly considers important and still retain its reputation as a top program.</p>
<p>So . . . procrastinating wildly this afternoon, I looked at some of the weightings, and some stuff becomes much more clear. The R ratings basically supervalue student GRE scores, awards per faculty member, and citations per faculty member (but not publications per faculty member). Other things that have some impact include stuff like percentage of female faculty (in a non-humanities field). It’s not terribly surprising that those kinds of measurable factors are what is consistent with reputation. The S ratings supervalue publications per faculty member, citations per faculty member, percentage of students completing in 6 years, percent of faculty with grants, and give some meaningful impact to a bunch of student quality-of-life factors that are certainly morally desirable but that don’t have anything to do with academics. And with those things, it’s all binary – a program has the item or not, the quality of the item isn’t addressed – so that all that distinguishes one program from another in this regard is the number of bells and whistles it has, not at all how they sound together.</p>
<p>So, in the end, I have to say that both sets of rankings are complete BS. The S rankings sort on the basis of qualities everyone purports to admire, but doesn’t really. There is certainly something to be said for them, and as between two programs with similar reputations the one with the better range of S rankings is probably meaningfully more attractive. The R rankings approximate reputation only imprecisely, because only a few of the data categories used have any meaningful correlation with reputation, and among these average GRE scores loom very, very large. (Which is a little interesting in and of itself, because I have yet to meet anyone who will admit that GRE scores are important in distinguishing among students.)</p>
<p>That’s true. But faculty seem to discount GRE scores almost completely when they talk about selecting students for PhD programs. They talk as if there’s a very wide qualifying band, and no one cares about variations within that band. But with the exception of some small outlier programs (that probably ARE selecting students based wholly on GRE), the factor that probably most closely resembles reputation is average relevant GRE score for students.</p>
<p>I don’t understand why, with today’s technology, that they couldn’t come up with the rankings in a timely manner. Even if we toss aside questionable metrics and assume, for the sake of argument, that they are the best anyone can do, we still end up with old and possible irrelevant data.</p>