U of C's 6 place jump by manipulation shows how small the dif is among top schools

College Search & Selection
1

<p>The fact that the University of Chicago was able to jump 6 places from #15 to #9 in the USnews rankings by recalculating some of its data illustrates how miniscule the difference is between being top 15 and top 10. It is unfortunate that so many people think that changes in the order of the top 15 schools means so much and that now U of C is where it rightfully belongs and now it must be that much better than say its neighbor Northwestern at #14.</p>

<p>I fully expect all schools to rush out and make sure all its data is correct. It is incredible that U of C's vice president for university relations and dean of college enrollment, Michael Behnke, went to Washington, D.C to meet with magazine researchers and editors to find out if there was anything they could do to manipulate the data. (note they say "way toimprove" as in "Officials also found a way to improve the alumni giving rank") If it were to just see if they were doing everything correctly why such a secret comment by Behnke:</p>

<p>Behnke said the university also changed other calculations, but he declined to say in which areas.
"Frankly, I don't want to help my competitors," he said. "Let them figure it out. The problem is that they probably already figured it out. We're late to the game."</p>

<p>I'm not so sure everyone has figured it out or is paying so close attention, but maybe they should. Does Northwestern, for example, include their Freshman seminars and exclude alumni they can't find from their financial giving percentage? I'm not so sure that USnews gives schools a hundred page pamphlet detailing every situation and what they think is ok to do. I think many schools are already booking fights to DC.</p>

<p>From today's Chicago Tribune:</p>

<p>U. of C. jumps to 9th place in ranking of universities
New look at numbers boosts its standing</p>

<p>By Jodi S. Cohen
Tribune higher education reporter</p>

<p>August 18, 2006</p>

<p>Size counts, money counts, and maybe most important, how you count counts, the University of Chicago discovered as it engineered a dramatic jump in a national college ranking.</p>

<p>When the highly publicized U.S. News & World Report rankings are released Friday, the University of Chicago will come in ninth place after finishing last year at No. 15.</p>

<p>It's rare for a college to move more than a spot or two on the list, so U. of C.'s jump six places may raise some eyebrows.</p>

<p>But university officials said they realized they had been miscalculating several categories, including the number of small classes and educational spending, errors that led to a steady drop in position since it was last in the top 10 in 2002.</p>

<p>In the issue that hits newsstands Monday, Princeton University in New Jersey is ranked No. 1 among national universities. Northwestern University is No. 14, and the University of Illinois at Urbana-Champaign is No. 41.</p>

<p>The annual list is a double-edged sword for colleges. Admissions officials frequently criticize the report, challenging everything from the magazine's methodology to the presumption that one college is best for everyone. But administrators sometimes concede that they ignore the rankings at their peril.</p>

<p>Concerned that a continued slide in rankings might affect the University of Chicago's reputation, Michael Behnke, the vice president for university relations and dean of college enrollment, went to Washington, D.C., with other top officials to meet with magazine researchers and editors.</p>

<p>The magazine evaluates about 20 factors when ranking the universities, including class sizes, student retention, graduation rate, alumni giving rate and SAT scores.</p>

<p>"They concluded that we were misinterpreting some of their definitions," Behnke said.</p>

<p>In calculating the number of classes with fewer than 20 students, for example, university officials did not count the freshmen writing courses that have an average of eight students.</p>

<p>By including the writing classes, the percentage of classes under 20 increased to about 67 percent, from 60 percent, Behnke said.</p>

<p>"That was a `duh' moment. Why aren't we including these all along?" Behnke said.</p>

<p>Officials also found a way to improve the alumni giving rank--the percentage of alumni who donate to the university--by excluding graduates who couldn't be located.</p>

<p>The university also improved its per-student spending calculation by relabeling $15 million in annual library expenditures that had been incorrectly filed under a category other than educational expenditures--information that also is submitted to the federal government. The additional per-student spending improved the university's position in the "financial resources" category.</p>

<p>Bob Morse, director of data research at U.S. News & World Report, said he was surprised that university officials had done such a poor job checking their data in the past.</p>

<p>"It is rare that a school like the University of Chicago admitted that they were not doing their federal financial data correctly," he said. "They came across as an institution ... that in some cases wasn't doing as serious a job reporting some of their data as they could have."</p>

<p>Behnke said the university also changed other calculations, but he declined to say in which areas.</p>

<p>"Frankly, I don't want to help my competitors," he said. "Let them figure it out. The problem is that they probably already figured it out. We're late to the game."</p>

<p><a href="http://www.chicagotribune.com/news/local/northwest/chi-0608180238aug18,1,4041502,print.story?coll=chi-newslocalnorthwest-hed%5B/url%5D"&gt;http://www.chicagotribune.com/news/local/northwest/chi-0608180238aug18,1,4041502,print.story?coll=chi-newslocalnorthwest-hed&lt;/a&gt;&lt;/p>

2

<p>Here's the manipulation: In peer assessment (academic reputation), Chicago ties for 6th with Cal Tech and Berkeley, one notch down from HYPSM. Compared to that, most of the other stuff in the USNWR rankings is b.s. Percentage of classes with under 20 students is a very tenuous "quality" metric. Chicago, Berkeley, and Michigan have always stuck out like sore thumbs in USNWR, because their overall rankings were so out of line with their peer assessment ranking, and still are for Cal (19 I think vs. 6) and Mich (24 vs. 12). Michigan and Penn have the same peer assessment and a difference of 17 places in USNWR. Sure, Penn deserves to be ranked higher because of the other factors measured, but you would expect them to be closer.</p>

3

<p>If you play the game well, you reap the rewards. Conversely, if you don't play the game at all, you shall be punished: just ask Reed and St. John's.</p>

4

<p>I don't know why the OP's first paragraph was so negative about Chicago's effort... USNWR and the Tribune have stated that Chicago went ahead and did things the right way... that for a long time it had been misrepresenting its numbers... why is that a bad thing? Also, I agree with JHS... peer assessment is indeed the most important category of the rankings, and the fact that Chicago has moved closer to the PA ranking in the overall category just means that their new calculations have simply brought things closer to reality.</p>

5

<p>A couple of things:</p>

<p>The peer assessment IS already the most important item in the USNews, but that does not change that it should be. Simply stated, the PA is an exercise in futility and ... abject cronyism, especially on a geographical basis. The process of completing the PA is NOT what it is purported to be., and is marred by favoritism and true lack of knowledge of the change made at other schools. Pretending that the Dean of Lane knows what happens at Washington and Lee is nothing but a fraud. The only way for this most despicable creation to gain some credibility is to PUBLISH the entire questionaire and let the world see how the schools vote for one another. </p>

<p>People familiar with the methodology know that USNews does NOT punish Reed. Actually, it has made efforts to showcase the school in its magazine, including one of the best article written on the school. The reality is that Reed is milking its iconocalstic position by playing the role of a victim, and downplaying the negative statistics that plague the school. Considering the overall numbers, Reed is more or less where it belongs. At least, until they discover the value of using the creativity they are known for in filling the Common Data Set forms, and make a mockery of a system relying on self-reporting. </p>

<p>It is obvious that this year rankings clearly demonstrate how the USNews model can--and is-- being manipulated. Exactly what I predicted last year after witnessing aberrant movements.</p>

6

<p>Reed is no way the 40-somethig best LAC in the nation, where it is placed by USNWR. It is usually consider one of the very top, as highly regarded as Pomona on the West.</p>

7

<p>I don't see how Reed is "punished" ... the US News uses objective data, if someone could prove this it would be interesting</p>

<p>Also, PA is already the biggest factor. I'm not sure how you could expect Penn and Michigan to be closer. Penn's student body is more academically gifted based on scores, its alumni giving is much higher, other advantages such as less students per faculty etc. Student selectivity is the most important criteria and Penn dominates Michigan in that area.</p>

8

<p>thethoughtprocess: "I don't see how Reed is "punished" ... the US News uses objective data, if someone could prove this it would be interesting"</p>

<p>There is no one around to see if the "objective" data can be "improved" like the U of C did.</p>

9

<p>Yeah, but I mean, that implies Chicago lied about their objective data, and also that Reed is only "punished" by US News because it refuses to republish its objective data.</p>

<p>There was an article in it in recent years about how cool Reed is...I doubt thats punishing them.</p>

10

<p>Bob Morse, director of data research at U.S. News & World Report, said he was surprised that U of C officials had done such a poor job checking their data in the past. Who's there to check Reed's data if they are not willing participants? If the data is so "objective" and straightforward how was U of C self-admittedly able to change so many calculations and then not want to admit which ones because they don't want to help the competion "figure it out"? One certainly can help "game" the system because obviously data can be presented in the best possible light by rules that aren't all detailed by US News. Obviously it is not straightforward if it requires face to face meetings and questions like, "can we do this?, can we exclude this?, exactly what U of C did. Reed is being "punished" because they refuse to play along and no one is checking and calculating data in an "improved" fashion for them that can help their standing.</p>

11

<p>Reed shouldn't even be ranked at all if they are unwilling to participate. I find it disrespectful to soil the reputation of such a fine school by putting them so low.</p>

12

<p>exactly... Chicago certainly didnt lie about their stats. Consider the fact that they didnt include their freshmen writing seminars in the list, with an average of 8 students per class. That's akin to Harvard not including the hundreds of freshmen seminars they have which are capped at 15... I'm sure they included those, and Chicago was remiss in not including its writing seminars all these years.</p>

13

<p>So US News isn't punishing them, Reed punishes itself by not clarifying its data, and US News reports everthing as accurately as possible under the circumstances...right?</p>

14

<p>dwincho, I agree... but I think that withdrawing altogether might further ruin their reputation. If I were at Reed I'd just give up and succumb to the magazine's influence... much like the way Chicago did when it decided to work a little bit harder to push itself further up on the list.</p>

15

<p>It's a free country, why does Reed have to subvert itself to the whims of a second-rate magazine?</p>

16

<p>felipecocco: "exactly... Chicago certainly didnt lie about their stats. Consider the fact that they didnt include their freshmen writing seminars in the list, with an average of 8 students per class. That's akin to Harvard not including the hundreds of freshmen seminars they have which are capped at 15... I'm sure they included those, and Chicago was remiss in not including its writing seminars all these years."</p>

<p>Are you so sure? What about small night seminar classes available to degree and non-degree students? I certainly don't think Chicago "lies" about their stats. But obviously which data to put in can push the envelope and may not be obvious if it involves direct questioning of US News officials. Especially when given the fact that U of C got so many calculations "wrong" and the U of C official doesn't want to alert their competitors because then those competitors can do the same manipulation. Again if this is so "above board" why doesn't the U of C official just tell all the calculations they did wrong and what other schools should do to correct theirs?</p>

17

<p>Reed appears to have no practical problem with its treatment by US News; its selectivity has been steadily increasing as more and more top students apply. Reed has repeatedly asked to be dropped from the listings, to no avail.</p>

18

<p>Again, I don't understand why Reed's reputation is soiled if US News continues to use the best information it has available to them to rank it...I could see a legitimate argument for that if US News makes up facts, or gives them a 0 rating for a category or something, but it doesn't...</p>

19

<p>Werner,</p>

<p>I certainly agree with you that stats can be manipulated every which way, but I'd like to point out that Chicago wasn't questioned about their reporting... in fact, much of the opposite took place. From the article: "Bob Morse, director of data research at U.S. News & World Report, said he was surprised that university officials had done such a poor job checking their data in the past." i.e. this year's stats probably better reflect reality.
And as for not telling what the other changes in the calculations were, I think Chicago is well within its own right to do that... and I would be surprised if their so-called "new" ways to report information haven't been discovered and used by other schools for a long time. </p>

<p>After all this is said and done, this is just another push by admissions departments to get more talented students to apply to their schools. If they all agressively call you, send you mail, come to your cities... why not agressively push to get themselves ahead in a ranking -- despite horrible -- which will be read by hundreds of thousands of graduating American seniors?</p>

<p>Further, in the case of Chicago, I think you'd be somewhat surprised by how some people have taken the news... some have considered it to be a negative thing. Chicago enjoyed a very high acceptance rate while still maintaining excellent stats for its incoming class due to its self-selective applicant pool. Many people feel that this new place will get rid of that, and one thing which many people liked about the application process at Chicago, the very personal touch, will be lost...</p>

20

<p>felipecocco: "I would be surprised if their so-called "new" ways to report information haven't been discovered and used by other schools for a long time."</p>

<p>felipecocco,</p>

<p>This is where we primarily disagree. I'm not so sure the other schools have figured out the so-called "new" ways to report information. There shouldn't be any figuring out to do. It should be straightforward which evidently it isn't if U of C couldn't figure it out without a face to face meeting and hides their changes because it doesn't want to help competitors to figure it out.</p>

<p>Unfortunately, the changes I see in Chicago and most schools is the realization that to be highly ranked in US News IS important, VERY important. Wash U is a perfect example. Their rise would have been impossible without US News. Until 1996 Wash U was ranked around 20. But through heavy advertising, they got more applicants to apply (thus increasing selectivity). They also provided merit aid (which most of the top rank schools don't preferring to give all financial aid) which caused the quality to increase. This lead to a higher ranking in US News. When HS students saw the higher ranking in US News, more and better students applied the following cycle which again caused a bump in the rankings, and on went this cycle. Wash U rose steadily from #20 in 1996 to #9 in 2004. None of this would have been possible without the buzz from US News because no one would have known without US News. Eventually, the buzz reaches the "peer reviewers" and their assessment/reputation scores increase as well.</p>

<p>Because US News is so important, if you want "objective" rankings you need "objective" stats. And currently these stats can be too easily "gamed" because you need face to face meetings with US News editors to figure out what you can and can not do and clearly these manipulations are not obvious.</p>

<p>U of C's behavior illustrates how important US News is (it was obviously incredibly important to U of C) and we may have entered a new era in how important the ability to "improve" the stats by the universities has become.</p>