Inside Higher Ed: 'Manipulating,' Er, Influencing 'U.S. News'

<p>'Manipulating,</a>' Er, Influencing 'U.S. News':</p>

<p>
[quote]
June 3, 2009
ATLANTA -- Higher education's love-hate relationship with college rankings was on full display here this week at the annual forum of the Association for Institutional Research, where -- despite the continuing campaign by some campus presidents to marginalize rankings -- campus number crunchers were treated/subjected to at least a half-dozen sessions on the subject.</p>

<p>As is common at this gathering of self-described wonks, some of the sessions (like the one on using Microsoft Excel to adjust raw rankings data) were practical to the point of being arcane, but most aimed to defend, dissect or debunk them. Officials from U.S. News & World Report and the Times Higher Education Supplement were on hand to tell the men and women who in many cases are responsible for submitting information about their campuses how their rankings were evolving and what might be coming down the road, for instance. In another session, researchers from Michigan State and Pennsylvania State Universities examined a decade's worth of U.S. News results to show how little change there was in colleges' performance in the rankings' controversial "reputational" score, and that what movement did occur was tied mostly to changes in the selectivity of their student bodies.</p>

<p>One session, however, revealed more than any other the extent to which the rankings, for all the protestations to the contrary, influence colleges' behavior. A presentation by Catherine Watt, the former institutional researcher and now a professor at Clemson University, laid bare in a way that is usually left to the imagination the steps that Clemson has (rather brazenly) taken since 2001 to move</a> from 38th to 22nd in U.S. News's ranking of public research universities...

[/quote]
</p>

<p>I hate to hear it but a university official came forward from my school, Clemson saying that the university crossed some “unethical” lines when reporting US News data. The article can be found here: [News:</a> ‘Manipulating,’ Er, Influencing ‘U.S. News’ - Inside Higher Ed](<a href=“http://www.insidehighered.com/news/2009/06/03/rankings#Comments]News:”>http://www.insidehighered.com/news/2009/06/03/rankings#Comments)</p>

<p>Some of the stuff Clemson has done to rise in the rankings include:
-Ranking many schools above them on the peer assessment as “below average”
-Only admitting students in the top third of the HS class to improve selectivity
-adjusting class sizes so that a section of 21-25 students goes “under 20”
etc…</p>

<p>To me, this just proves how stupid the US News rankings are. The rankings can clearly be manipulated and god knows how many other schools are acting the same way as Clemson is to move themselves up. The problem is that people put so much weight in the US News Rankings to determine a college’s reputation when selecting colleges that colleges feel that they have to move up in the rankings which may mean doing unethical stuff and manipulating the ranking criteria.</p>

<p>Some parts are very troubling - playing with salary numbers, ratings of other colleges, etc. Others not really - more small classes. Overall I wonder how the state legislature feels about this. Will Clemson become the premier U in the state (ala the UC’s in CA) with USC coming along behind (ala the CSU’s)?</p>

<p>Pierre, This was already posted by StitchInTime.</p>

<p>I really doubt if Clemson is the only school that is doing this. I remember last year that Baylor payed its freshmen accepted students to “retake” the SAT so as to improve their score.</p>

<p>I imagine all the schools are doing this, so won’t it just even out in the end?</p>

<p>If people stopped relying so much on US News to pick a college (I’m going to this school because it’s ranked higher than this school etc…) colleges wouldn’t be doing this and having campaigns to be a “top 20 public school” or “a top 50 school”. </p>

<p>Clemson clearly has exposed some flaws in the US News rankings such as the peer assessment score which can clearly be manipulated.</p>

<p>Other examples of colleges potentially “gaming” the rankings:</p>

<p>[Magazine</a> ranking key item in ASU chief’s bonus | Tempe News | eastvalleytribune.com](<a href=“http://www.eastvalleytribune.com/story/86052]Magazine”>http://www.eastvalleytribune.com/story/86052)</p>

<p>from: <a href=“http://chronicle.com/free/v53/i38/38a01101.htm:[/url]”>http://chronicle.com/free/v53/i38/38a01101.htm:&lt;/a&gt;&lt;/p&gt;

<p>"Take, for example, Chapman University.</p>

<p>Chapman, in the heart of Orange County, Calif., has long been known as a college that gave a second chance to underachieving high-school students who showed promise. When James L. Doti became president, in 1991, he says, Chapman essentially had no admissions criteria, other than the best judgment of the staff.</p>

<p>Students were “using Chapman like a community college,” he says. Only 42 percent of students graduated within five years. The university had one endowed chair. There was almost no money for merit-based financial aid.</p>

<p>So Mr. Doti dropped the athletics program from Division II to Division III, thereby eliminating all athletics scholarships.</p>

<p>“We took that $2-million a year in athletic aid and added it to the financial-aid budget,” he says. The institution increased its tuition one year by 25 percent, so parents and students would perceive that the college had as good a program as “the colleges we wanted to compete with.”</p>

<p>Mr. Doti decided to set a minimum SAT score required for admission. “It was 740, which is nothing great, but for Chapman, at least it was something,” he says. “The next year, it was 760. That lops off a lot of people at the bottom. Every year we went up another 10 or 20 points.” The university began a scholars program with grants for high-achieving students.</p>

<p>Almost all the changes were designed expressly to help the college rise in the U.S. News rankings. “I can quibble with the methodology, but what else is out there?” says Mr. Doti. “We probably use it more than anything else to give us objective data to see if we are making progress on our strategic goals.”</p>

<p>In less than 20 years, Chapman has come to top the “selectivity rank” among master’s-level institution in the West, according to U.S. News. The minimum SAT score is now 1050. It has 45 endowed chairs. The endowment has grown from $20-million to $250-million. When U.S. News expanded the universe of colleges it ranks in 1993 by adding regional institutions, Chapman was in the second quartile of all such institutions in the West, and its academic reputation was ranked 90th among its 112 peers. It now ranks 11th over all among master’s-level institutions in the West, and its academic reputation is tied for 14th highest in that group."</p>

<p>Lies, damned lies and statistics.</p>

<p>Even so, the criticisms of rankings should be forced on every person who joins CC.</p>

<p>There are a number of academic criticisms of USNWR’s rankings - all of which conclude they’re crap or worse than crap - but of course they refuse to release their data and they not only change their methodology slightly year to year but the critics have found evidence they don’t actually follow the methods they list. </p>

<p>(I don’t know if anyone remembers, but they hired a person to run the rankings who tried to be more statistically valid and the rankings turned up, I think, CalTech as the top school. That person was quickly gone and the ratings reverted to normal.)</p>

<p>I’ve wasted far too much time trying to explain basic statistics to kids and parents who cite 5 or 10 or 20 notch differences as being important, but anyway.</p>

<p>A few days ago I spoke with an admissions director at a very good school. He thinks ratings are sad and related his experience filling out the survey - and he’s spent decades in college admissions. We agreed the survey is built into the method as a way to maintain a basic prestige structure. Because they don’t share the data, we can’t know how many schools are actually ranked on each survey, how many are ranked by competitors (up or down) or other important measures that would help us figure out the surveys actually say.</p>

<p>I haven’t seen this posted so thank you. </p>

<p>I can’t imagine why anyone would think all universities didn’t lie, um, I mean, uh, fudge the numbers. It’s disgusting but they’re going to do it. Perhaps USNWR isn’t the be all to end all list but that’s the one every parent has easy access to and will be pouring over it with their checkbook. It comes down to the almighty dollar. Every point a school moves up is that much more they can raise their tuition. Every extra point gives them bragging rights and a larger paycheck.</p>

<p>pierre:</p>

<p>what that article fails to point out is the Chapman was nearly broke when Doti took over, and it was bordering on bankruptcy. But, like ASU how throws money at NMSF’s, I have no problem with a college giving more money to students so it can increase its USNews ranking. Isn’t the point of the rankings to identify areas where a college is lagging?</p>

<p>“And to actual gasps from some members of the audience, Watt said that Clemson officials, in filling out the reputational survey form for presidents, “rates all programs other than Clemson below average,” to make the university look better. “And I’m confident my president is not the only one who does that,” Watt said.”</p>

<p>^ I think the point of the article is that the schools don’t need to lie or even “fudge” numbers. The US News rankings are driving many of them to alter their actual behaviors in ways that may get them higher rankings but do not materially improve educational quality. It’s relatively easy to set new 19-person enrollment caps on classes that previously served 21 or 22 students. It doesn’t serve the students because the qualitative difference between a 19-person and a 22-person class is insignificant; in fact, that change arguably hurts students by making it somewhat more difficult to get the classes they want or need. But it does allow the school to show a higher percentage of its classes with fewer than 20 students—the arbitrary cut-off US News uses to measure the percentage of “small” classes. At the other end of the scale, a 50-person class counts the same to US News as a 100-person class or a 1,000-person class, so the logical strategy for a school set on improving its US News ranking would be to consolidate all multiple-section classes of 50 or more students into a smaller number of single-section mega-classes; or, for example, to offer the 50- or 100-person introductory class only in the Fall rather than both semesters, forcing students into a single larger class and thereby cutting in half the number of “large” classes. No one could argue with a straight face that this improves educational quality, but it would allow the school to show a lower percentage of its total course offerings as “large” (50 or more) classes, and therefore marginally improve its US News rating. These moves in themselves would have only a tiny impact on the school’s overall ranking, but they can easily be combined with a lot of other moves—like jacking up tuition to pay higher faculty salaries, or jacking up tuition and recycling the additional revenue into offsetting financial aid thereby showing greater (nominal) spending per student. Or reducing the size of the freshman class to boost selectivity, offsetting the lost tuition revenue by admitting an increased number of (possibly less well qualified) transfer students whose stats won’t show up in the school’s freshman “selectivity” rating. Or luring large numbers of applicants by cynically dangling a vanishingly small number of extremely generous merit scholarships in front of them so as to boost selectivity. Or going SAT-optional on the theory that only the higher scorers will then submit SAT scores, thereby boosting the school’s 25th-75th percentile reported SAT scores. And so on.</p>

<p>It doesn’t surprise me in the least that the kinds of practices described in the article are going on. I fault uninformed applicants who rely too heavily on the US News rankings, as much as I do college and university administrators. If applicants didn’t take US News so seriously, the colleges could safely ignore it. But many feel they’re trapped in a destructive “arms race” with their competitors; if they don’t engage in these kinds of statistical-manipulation behaviors, their competitors will, and they’ll slip further behind in the rankings.</p>

<p>Oddly, though, the one bit of statistical manipulation most often complained about on CC is, in my opinion, probably the least harmful. The article reports that Clemson administrators regularly downgrade their competitors in the Peer Assessment survey, on the theory that this will make their own school look better by comparison. This is intellectually dishonest and morally bankrupt, but probably utterly ineffective. You’ve got to assume that other colleges do the same. Across the thousands of colleges surveyed, these distortions should pretty much completely cancel each other out.</p>

<p>Fascinating article! Thanks for the link!</p>

<p>The President of my Alma Mater is among those leading the charge to de-emphasize the USNews rankings. He wrote the letter that he and many other colleges have signed refusing to use any USNWR ranking numbers in their marketing material. Of course, Alma Mater has slipped in the rankings every year since he joined this campaign.</p>

<p>My dad is a Clemson alum, and he has told me several times that Clemson’s goal is to be ranked in the top 20 Univs in the country. The same thing was repeated by the tour guide when we toured campus 3 years ago. A few weeks ago my dad told me that Clemson had “passed” Texas A&M in the rankings. I decided not to enlighten him on my opinion on the rankings…</p>

<p>haha yeah the US News rankings are a whole bunch of BS, I’d like to tell James Barker that I didn’t pick his school because of some magazine ranking table</p>

<p>I can almost hear this future father-son discussion:</p>

<p>Dad: So how’s the application list looking Son?</p>

<p>Son: Well, I’ve got Stanford, MIT and Vandy as my reaches with Clemson, UVA and Cal-Berkeley as matches. For my safety I’m trying to decide between UMichigan, William and Mary, and UNH-CH.</p>

<p>Yeah, right.</p>

<p>
[quote=]
Only admitting students in the top third of the HS class to improve selectivity

[/quote]
</p>

<p>What is so bad about that? Don’t a lot of top schools do that? and if you’re not in the top 1/3, you need to have some crazy good hook.</p>

<p>Go back to the article, scroll to the bottom and read the comments at the bottom.</p>

<p>It seems like there is a divide from the comments between the faculty at Clemson over the top 20 initiative. Some are 100% behind the plan, others are skeptical and angry at President Barker’s actions.</p>

<p>Some snipets from 4 different commenters:</p>

<p>"# Posted by Martine LaBerge, Ph.D., F.B.S.E. , Professor and Chair of Bioengineering at Clemson University on June 3, 2009 at 4:00pm EDT
The information presented in this article was extracted from an Institutional Research Conference. The facts stated should have been verified before publication. Additionally, what has been printed and what Catherine Watt said are irrelevant to facts and have little to do with research.
Catherine Watt said that Clemson is only concerned with attaining a high score from US News. Yet, everyday she has to walk beside the construction site of our new Bioengineering Building. It is a $12M investment. Yet, it has nothing to do with US News rankings.
Clemson’s administration has worked diligently in the past 10 years to achieve the recognition Clemson University deserves. It provides an utmost environment for integrating knowledge and preparing our students to fulfill outstanding careers. It provides the highest level of creative thinking, learning, and innovation. And, we are very proud of that. Attaining a high magazine ranking is something that all universities in the United States seek. However, facts are checked by US News
Since President Barker has taken office, we are more focused and as a result we have better students, our faculty salary and benefits are more competitive, our classes are smaller, our graduation and retention rates are better and our research expenditures are higher. These accomplishments clearly have not been mentioned by Catherine, or perhaps, omitted in this article.
As a department chair, I take part in the US News survey and rank other bioengineering departments. I rank other programs based entirely on my knowledge of them. I know my colleagues do the same. We are professionals and would never do anything else. "</p>

<p>“Ms. Watt, the faculty at Clemson owe you a big thanks for having the courage to present the facts about Clemson and its obsession with gaming the US News rankings. Millions of dollars have been squandered in this meaningless effort. During the past 10 years faculty and staff have watched the quality of education decrease. Good faculty left in disgust, science labs are woefully outdated, and “Smart classrooms” are anything but. But millions were directed to flashy brochures about Clemson, new branding techniques, and to academic success activities for students with high SAT scores to increase retention and graduation rates only to improve rankings. Many of us are also aware that inaccurate data was submitted to US News. Most of the faculty are sick of hearing President Barker’s litany that we are # 22 in US News and # 8 in this and #10th in that. We know that the “hard” data like the amount of federal research dollars, number scholarly articles cited, and major academic awards for faculty and students have not changed. Now we are in the midst of an economic crisis and major budget cuts to the university and as another person commented the University is in a free fall. It is so sad that so much time, money and effort was wasted on the equivalent of winning the swim suit competition in a beauty contest.”</p>

<p>“My name is John Ballato and I had the great honor and privilege to serve the Clemson University faculty for 3 years as the Faculty Representative to the Board of Trustees. During my term, which ended this past December, I participated in many very frank and open discussions with faculty, staff, and administration, including closed Trustee retreats, about strategies and tactics.
My first comment in response to this article is to make abundantly clear that Ms. Watt is not the voice of the faculty, staff, or students at Clemson. At best, she (and the un-named “Truth At Last” author) represent a vocal minority, but a minority none-the-less.
Secondly, and unlike the other Clemson employees contributing to this dialog, being a first-hand participant as a representative of the faculty in institutional strategic planning I can say without reservation that the focus was always on quality - quality of faculty, staff, students, and facilities - that has driven Clemson’s efforts. Clemson’s growth in US News rankings is a result of the hard decisions that have been made on where resources should be invested and not a result of chasing a US News number. Anyone who thinks Clemson’s focus is not on the students and faculty are out of touch. It needs to be kept in mind that we live in a State that is not committed to higher education and the state institutions have seen a disproportional cut in funding. Furloughs, deferred hiring, reduced travel are unfortunate but necessary evils when many of our sister schools around the country are laying off faculty. Times are tough but there is no better job than a tenured faculty member where one has the luxury to spend time putting down the very institution that is working hard to keep them in a job.
Thirdly, during this Top-20 period, Clemson’s SAT scores have grown greatly, sponsored research awards have tripled, we have more spin-off companies creating higher wage jobs than ever before and we are more focused on our mission rather than deviating from it to pursue a number.”
Along this lines, it is worth noting that, as a Professor, I also have been the largest generator of research funds at Clemson over the past 6 years at least, among the top in scholarly output (publications and citations), have spun out a successful company, all while teaching my classes and graduating students. I see considerably better students now than we did 10 years ago and Clemson’s investment in our programs are growing, not declining; another clear indication that resources are being targeted towards where the greatest productivity lies. For those more disgruntled, I suspect that it is because productivity does not warrant investment.
Lastly, I am amazed that Inside Higher Ed and the Chronicle of Higher Education would write an article using purely anecdotal accusations without permitting the administration time (more than a late Tuesday night email for a Wednesday AM article) to respond in kind or contacting faculty, staff, or student representatives for their comments. This level of journalism is more expected from the likes of People Magazine or the National Inquirer. </p>

<p>“To the fierce defenders of CU, recruited through an email message that went out to faculty earlier in the day, your attempts to shame the reporters of legitimate concerns show your true colors, or lack thereof. You and your ilk are fostering the insidious, retaliatory culture that presently exists at the university. Your attempts to squelch debate are quintessential CU.”</p>