<p>
[quote]
Obviously I knew that the Presidents or Provosts of mega-universities such as Michigan State, Florida State, or South Dakota State University are supposedly the best assessors of Caltech.
[/quote]
</p>
<p>For all we know, the Presidents et al of such Unis could have attended CMC or USC or Cal State LA and took classes at Cal Tech. Your assumption that they know NOTHING about Cal Tech is as valid as mine -- I assume that they know something of its value vis a vis other Unis.</p>
<p>I wasn't commenting on the test scores to PA correlation, rather the correlation between overall ranking to test scores, hence the reasoning behind the quote.</p>
<p>
[quote]
They must be calculating the median differently than when I was a kid. I thought they did it like this.
[/quote]
</p>
<p>Nope, you're right, that's how they do it when you know everyone's score or response or value or whatever. You line 'em up and find the one where half the scores are below it. What I'm talking about is finding the median (or any percentile) for data that's grouped up and you don't actually know the individual scores. So let's say that instead of knowing everyone's SAT, you knew that 12% scored between 1500-1600, 27% scored between 1400-1490, 28% scored between 1300-1390... you get the picture. You can tell that the 75% percentile must fall somewhere in that 1400-1490 range, but where? This formula gives you an estimate. </p>
<p>It seems far-fetched to me that this is the situation that the college found itself in when calculating the percentile, but I was just pointing out that one shouldn't assume foul play when you see a percentile score that doesn't look like an actual score any individual might get. </p>
<p>A purely academic point with rapidly decreasing relevance to the topic at hand: quintessential hoedown.</p>
<p>hoedown-
There is some interpretation involved with Principal Components Analysis, as you may know. After studying the results further, I would exclude grad rate over- and under-performance from the Alumni Giving factor. I mis-read the results before. I am not really an expert at this. Others have been helping me. </p>
<p>So, I would say that graduation rate over- and under-performance stands alone in its own component. Its "weight" is far higher than any other data in that component (.89).</p>
<p>I would also like to add that "faculty resources" seems to be pretty important in all four components. It doesn't seem to "load" neatly into just one of the four component, with "weights" of -.51, .47, ..46, and .31.</p>
<p>I had some difficulty interpreting the second component but I would now say that the second component of the US News data has to do with size.</p>
<p>Here are the "weights" that the defining pieces of US News data have in each component. The names I gave each component are the result of my interpretation of the weights. Zero means the data had no relationship with that component. The closer the weight is to either 1 or -1 the stronger the relationship to that component. Do you agree with my choice of names?</p>
<ol>
<li><p>Academic Quality (by far the most important component)
overall score .91
peer assessment .84
graduation rate rank -.93 (minus because low number means better rank)
freshmen retention .94
predicted graduation rate .88
actual graduation rate .94
faculty resources rank -.51 (low number is better)
selectivity rank -.88 (low number is better)
SAT 25th percentile .85
SAT 75th percentile .87
top 10% of class .88
acceptance rate -.82 (low number is better)
financial resources -.71 (low number is better)</p></li>
<li><p>Size
classes under 20 -.63
classes over 50 .89
student-faculty ratio .66
percent faculty full-time .71
faculty resources .47</p></li>
<li><p>graduation rate over- and under-performance
graduation rate over- and under-performance .89
faculty resources .46</p></li>
<li><p>Alumni Giving
alumni giving rank .77
alumni giving percent -.63</p></li>
</ol>
<p>The analysis I did is saying that these four components account for almost all the information used by US News. US News uses a conceptual approachfor assigning weights to do its rankings. This method uses a mathematical approach. It says that the individual peices of data that fall within the same component have a lot of overlap. That is, they "mean" about the same thing in statistical terms (although they may seem different conceptually). Height and weight are different conceptually but they both are related to size. They have a relationship (overlap) with each other. Another way to say it is that this technique examines the redundancy in the data and has reduced the information to four broad areas. The four broad areas are NOT related to each other in a statistical sense. They are independent. They are not redundant.</p>
<p>Another thing you might say is that if you know the following peices of information, you know most of what US News data has to say:
graduation rate
graduation rate over- and under-performance
percent of classes over 50
alumni giving rank</p>
<p>A2Wolves6-
When I said that 85% of the US News total score is SAT 75th percentile I was naming the single piece of data that was most closely related to the total score. Other pieces of data are closely related too:
peer assessment 83%
SAT 25th percentile 82%
actual grad rate 80%
predicted grad rate .79
freshman retention .78
grad rate rank .73
top 10 percent .71
acceptance rate .71
selectivity rank .68
financial resources rank .58
faculty resources .46
student-faculty ratio .41
and so on...</p>
<p>But, many of the above are closely related to each other and don't really tell you anything new. For example, a low acceptance rate implies a high graduation rate.</p>
<p>Gotcha, that's what I got from the data, I don't know if I said it correctly though. I think it's pretty ridiculous how the overall score is essentially what your SAT range is (referring to the 85% 75th percentile and the 82% 25th percentile) and not other things. I always wonder why GPA isn't in these rankings.</p>
<p>It seems to me everyone is discussing different assessments and their weights, whether certain assessments are valuable and whether they carry the right weight. It appears that everyone agrees that there can be no real ranking of schools, we are looking for what is right for each student.</p>
<p>As such, what I would like to see US News redo their online version of the rankings and enable everyone to weight the categories as they see fit. They could provide a screen that allows you to define what weight you would apply to each category making sure it adds up to 100%. The web application could then present a personalized ranking for the student.</p>
<p>Of course I would like for them to go a bit deeper. To me, Peer assessment of USC vs Williams College does not make sense unless it is tied to a particular program. For example, the film school at USC would have a higher peer assessment score than a program at Williams. This is not to diminish the fact that Williams is a great school, just that for particular programs certain schools may be a bit better.</p>
<p>Anyway, in the age of the internet and personalization of information and feeds this should be a trivial thing for US News to implement. For those who are looking for a definitive list, this would not be good for them.</p>
<p>The bottom line is that none of the categories are being correctly measured by USNWR. This has been explained before in countless posts (by me and others, on other threads) and articles in various national publications, so I won't bother to get into it here.</p>
<p>The only categories that USNWR currently measures accurately are: graduation rate, retention rate, admissions rate, standardized test scores, and, arguably to some extent, the peer assessment score. The data in the other categories, most notably within faculty resources, are NOTORIOUSLY incorrect.</p>
<p>The statistical problems in USNWR's analysis makes the rankings almost completely useless, except as to distinguish among "tiers" of schools. Look at the numerical number, not the rank order, to distinguish. HYP are basically tied at the top with scores of 98-100. After that, there is a big gap down to a lower tier, and then a whole bunch of schools with scores of 88, 89, 90, etc. Given that much of USNWR's underlying data is invalid or imprecise, you can basically say that if a school is within a few points of any other, they are statistically tied. There is obviously a huge gap between Dartmouth and Tufts, or between HYP and Northwestern, but there's no significant difference among Dartmouth, Duke, Brown, Penn, Columbia, Northwestern and Cornell, at least if you factor in that USNWR's numbers are misleading and poorly measured.</p>
<p>Agree. That would be a good one -- not just because of the library itself, but because it would correlate heavily, and accurately, with the amount of resources that actually are available in areas OTHER THAN the library.</p>
<p>Anyways, as an example, see the measures below -- and what they mean to a student:</p>
<p>Source for all is COHE 2005.</p>
<p>Library books/student -- Do you have to wait for recalls? Are obscure books in every topic area easily and readily available?</p>
<p>Yale University 1100
Harvard University 820
Princeton University 810
Chicago 550
Dartmouth College 520
Brown University 470
Columbia University 440
Cornell University 330
University of Pennsylvania 290
University of Texas 180</p>
<p>Library expenditure/student -- Are there enough quality resources in
general? are the lights in the library burned out? are there enough computers? enough knowledgeable staff to help with any conceivable topic of research? </p>
<p>Yale University $6,300
Harvard University $5,400
Princeton University $4,500
Dartmouth College $3,500
Brown University $2,500
Columbia University $2,400
Chicago $2,100
Cornell University $1,800
University of Pennsylvania $1,600
University of Texas $780</p>
<p>Serial subscriptions/student -- Are there a lot of magazines and rare
scholarly journals to read? Is cutting-edge information easily and rapidly accessible?</p>
<p>Dartmouth College 10
Princeton University 8
Yale University 7
Harvard University 5
Brown University 4
Columbia University 4
Cornell University 4
Chicago 3
University of Pennsylvania 2
University of Texas 1</p>
<p>Library staff/100students -- Are staff friendly, available, helpful, and
willing to help you find that obscure item for your paper? note this
is permanent staff only; colleges also hire student workers but those
are not tracked here.</p>
<p>Harvard University 5.7
Yale University 5.7
Princeton University 4.4
Dartmouth College 3.6
Brown University 2.4
Columbia University 2.4
Cornell University 1.9
Chicago 1.8
University of Pennsylvania 1.5
University of Texas 0.9</p>
<p>I like to see the alumni giving rate. Alums tend to give if they are loyal to their schools and enjoyed their time at the schools. If there are high giving rates it means more of the students enjoyed their time in the school and wish to remain loyal to the institution. However, I don't think this works well in public vs private schools because publics just have a ton more students and alums.</p>
<p>
[quote]
I like to see the alumni giving rate. Alums tend to give if they are loyal to their schools
[/quote]
The problem with the way USNWR accounts for alumni giving rate is that they only measure the number of alumni donating but not the total amount donated. That is, an alumni giving $10 is counted the same as one giving $10,000,000. One would think that an alumni giving $10M is a million time more loyal to the school ... or more successful.</p>
<p>
[quote]
If there are high giving rates it means more of the students enjoyed their time in the school and wish to remain loyal to the institution.
[/quote]
</p>
<p>However, it does not mean the reverse it true, i.e., low alumni giving rates mean that students did not "enjoy" their time at school. But, alumni giving also reflects the income of the students who matriculate into those schools...</p>
<p>I also believe that, for public universities, many have the mindset that they are already making a contribution to the school via their state income taxes which go to fund part of the college. This factor and the wealth of the alumni factor are the two major reasons why alumni giving is NOT a good metric for USNWR ranking purposes.</p>
<p>
[quote]
I also believe that, for public universities, many have the mindset that they are already making a contribution to the school via their state income taxes which go to fund part of the college. This factor and the wealth of the alumni factor are the two major reasons why alumni giving is NOT a good metric for USNWR ranking purposes.
[/quote]
</p>
<p>By that rationale, I've been a huge supporter of both the SUNY and UC systems even though I've never attended a single school in those systems... there is a disconnect there somewhere.</p>
<p>I just want to add one more comment about my analysis discussed in post #86.</p>
<p>That analysis boiled down the information in US News for National Universities into four "components":
Academic Quality
Size
Graduation Rate Over- and Under-performance
Alumni Giving.</p>
<p>Academic Quality data represents about 60-70% of the total information. The other three components are smaller.</p>
<p>I want to point out that these four components are statistically unrelated to each other. They are independent factors.</p>
<p>Therefore, it implies that academic quality measures are not correlated with size. Both small and large universities can have excellent academic quality.</p>
<p>It also suggests that grad rate over/under performance is not related to academic quality. This piece of the US News data purports to measure how well a college graduates its students relative to the quality (selectivity) of its incoming students. The analysis supports that idea. Over/under performance indicates graduation performance controlling for the quality of the students.</p>
<p>Alumni giving participation seems to result from something other than academic quality/selectivity.</p>
<p>It would be interesting to repeat this analysis for LACs...would they present a similar pattern?</p>
<p>That was a fascinating analysis. Please repeat for LAC's.</p>
<p>A note. When people do principal components analysis they often avoid giving the components value-loaded names. For example, many people would disagree about whether any of the items in the "academic quality" component actually are indicators of this value. Instead, they are perhaps proxies for what the academic quality might be. Having a talented student body is nice, but the academic quality may depend on what happens at the college. How well are the courses run, how much faculty guidance do the students receive, how much do they learn, are they engaged in independent work, or simply parroting back information on multiple choice tests? Since the USNews data do not address these issues, they may not measure academic quality at all.</p>
<p>So, for example, your first component says that enrolling talented students and graduating them at a high rate correlates with a high overall score. These features are common to colleges that end up at the top of USNews rankings. However, they do not, by themselves, tell us that these students are getting a better education. </p>
<p>The peer assessment may be an attempt to measure what informed observers believe academic quality to be, but it is impossible to verify the reliability of these results.</p>
<p>Ever thought of giving USNews a hand and letting you publish an ongoing analysis of their work? Your analysis is far more informative than the arbitrary weights the magazine assigns. I am amazed that people take the rankings so seriously when USNews refuses to provide any rationale for the weights it employs, or the omission of any outcome data other than grad rate.</p>
<p>I repeated the same analysis for the top 125 LACs in US News. LACs are more homogeneous than universities in terms of size, type of student (all undergraduates), and curriculum (generally all Arts and Sciences: science, math, humanities, social sciences, art and music).</p>
<p>For LACs, the 20 or so pieces of data in US News could be boiled down to the same four broad components although the individual pieces of data were weighted a little differently in each component.</p>
<p>I named the four components as follows</p>
<p>Academic Quality:
overall score .77
peer assessment .77
graduation rank -.82
freshman retention .85
predicted graduation rate .81
actual graduation rate .79
percent of classes over 50 .59
selectivity rank -.85
SAT 25th percentile .88
SAT 75th percentile .81
students in top 10% of class .86
acceptance rate -.74</p>
<p>Size (has more to do with class size):
faculty resources .76
percent of classes with under 20 students -.84
student/faculty ratio .82
financial resources rank .74</p>
<p>Graduation Rate Over- and Under-performance (value added):
grad rate over- and under-performance .94</p>
<p>Together, these 4 components accounted for 83% of the information (variability) in the US News data for LACs.
academic quality accounts for 57%
size (class size?) accounts for 14%
alumni giving 7%
graduation rate over/under performance 6%</p>
<p>Each of these 4 broad areas are uncorrelated with each other. </p>
<p>If you include peer assessment as one of the pieces of information, 98% of the overall score could be explained by three things:
Peer Assessment
Graduation Rate Rank
Faculty Rresources rank</p>
<p>If you exclude Peer Assessment from the formula and just use objective data, then 92% of the overall score can be explained by a combination of
actual graduation rate,
SAT 25th percentile,
and financial resources rank.</p>
<p>Peer Assessment itself can be explained 78% by:
classes under 20
SAT 25th percentile
financial resources rank</p>
<p>Much of the information used by US News is redundant in the statistical sense. If you know one piece of information, then you can guess the others pretty accurately. Here are some correlations (1 or -1 means very highly correlated, 0 means not correlated):
peer assessment and SAT 25th percentile .83
freshman retention and graduation rate rank .95
graduation rate and SAT 25th percentile .75
faculty resources rank with financial resources rank .72
student/faculty ratio with financial resources rank .72
acceptance rate with the students in top 10% of HS class -.75
alumni giving rate with alumni giving rank -.96</p>
<p>So, in terms of value added, you could say that, once you know SAT scores or top 10%, the only things that add much in the way of new information are financial resources (or one of its related items such as student/faculty ratio), alumni giving, and graduation rate over/under performance.</p>
<p>Agree. That would be a good one -- not just because of the library itself, but because it would correlate heavily, and accurately, with the amount of resources that actually are available in areas OTHER THAN the library.</p>
<p>Anyways, as an example, see the measures below -- and what they mean to a student:</p>
<p>Source for all is COHE 2005.</p>
<p>Library books/student -- Do you have to wait for recalls? Are obscure books in every topic area easily and readily available?</p>
<p>Yale University 1100
Harvard University 820
Princeton University 810
Chicago 550
Dartmouth College 520
Brown University 470
Columbia University 440
Cornell University 330
University of Pennsylvania 290
University of Texas 180</p>
<p>Library expenditure/student -- Are there enough quality resources in
general? are the lights in the library burned out? are there enough computers? enough knowledgeable staff to help with any conceivable topic of research? </p>
<p>Yale University $6,300
Harvard University $5,400
Princeton University $4,500
Dartmouth College $3,500
Brown University $2,500
Columbia University $2,400
Chicago $2,100
Cornell University $1,800
University of Pennsylvania $1,600
University of Texas $780</p>
<p>Serial subscriptions/student -- Are there a lot of magazines and rare
scholarly journals to read? Is cutting-edge information easily and rapidly accessible?</p>
<p>Dartmouth College 10
Princeton University 8
Yale University 7
Harvard University 5
Brown University 4
Columbia University 4
Cornell University 4
Chicago 3
University of Pennsylvania 2
University of Texas 1</p>
<p>Library staff/100students -- Are staff friendly, available, helpful, and
willing to help you find that obscure item for your paper? note this
is permanent staff only; colleges also hire student workers but those
are not tracked here.</p>
<p>Harvard University 5.7
Yale University 5.7
Princeton University 4.4
Dartmouth College 3.6
Brown University 2.4
Columbia University 2.4
Cornell University 1.9
Chicago 1.8
University of Pennsylvania 1.5
University of Texas 0.9
</p>
<p>Like most measures, per capita doesn't provide as accurate a measure as total resources. In the case of libraries, it's like supercomputing resources. The fact that a university has the resource in the first place - at all - is more important than a meanginless ratio derived from student population. It doesn't matter that 500, 5000, or 50000 students have access to a priceless manuscript, or rare book. The university needs to have the resource in the first place. It's not like it can be circulated or that everyone would need access at the same time.</p>
<p>Using UT-Austin from your example, the per capita numbers distort the fact that UT has superior library resources to most, if not all, of the other universities listed. This doesn't mean it has more staff or "spending per student" - it clearly doesn't. It does have a world-class library that is the envy of much of the Western world. What does it really matter that 5 library workers are there for you at one school's library? What does that actually translate to in practice? If anything, it speaks of inefficiency. Economies of scale are more important.</p>
<p>Here are some comments regarding UT's library collections that demonstrate why per capita numbers don't really mean much:</p>
<p>"The Ransom Center (library at UT-Austin) easily outmaneuvers rivals such as Yale, Harvard, and the British Library."</p>
<p>"There's a good deal of awe at the speed at which the Ransom has been able to build such extensive collections by the dint of a lot of energy and a good deal of money," said Jean Ashton, director of the rare books and manuscript library at Columbia University. "We admire it and are more than a little bit jealous."</p>
<p>"Unquestionably, it's the second best collection of English literature after the British Library," Ferdinand Mount, editor of London's Times Literary Supplement, said during a recent visit to the Ransom Center.</p>
<p>Increasingly, Princeton curators have found themselves standing glumly on the sidelines. “Because we don’t have a lot of money, dealers don’t even bother to come,” says Primer. Horowitz confirms that. He did not call Princeton about the Mailer papers. Why bother? He knew that Princeton almost certainly would not match the kind of money he could expect to get from the Ransom Center. </p>
<p>"They're in the very top tier in the United States, which means they're top-tier internationally as well," said Barbara Shailor, director of the Beneicke Library at Yale. "They don't specialize the way the Morgan Library or the Getty Museum do. They're strong overall. They excel in so many ways." </p>
<p>"Lifting the Lid on a Treasure Chest", STEPHEN KINZER, The New York Times (2/4/2003)</p>
<p>"How important is the Carlton Lake Collection [at UT-Austin's Ransom Humanities Research Center]? Florence de Lussy, conservateur en chef de manuscrits at the Bibliotheque Nationale in Paris, had a straightforward answer... "Remarkable," she said... "In certain areas, for example Paul Valery, the most important writer in 20th century France, you must go to Texas if you want to study the man thoroughly... Consequently, the Carlton Lake Collection is essential, and very well known here in France. I wish it were here and not there."</p>
<ul>
<li>from "A Gentle Madness", Nicholas Basbanes</li>
</ul>