USNWR Peer Assessment Scores look biased to number of grad students

College Search & Selection
21

<p>I believe most of you are neglecting in your ridiculous analysis of these statistics, the fact that the PA is 100% subjective. The presidents, provosts, deans, etc.... do not sit to analyze the information to the degree that you are trying to do here. They give a 5 or a 1 on the basis of what they "feel" the institution represents to them. That is what throws the PA 's credibility out the window. Personal likes or dislikes, personal experiences, long standing reputations as well as second hand information, they all play a role at the time they have to write down their little number. The PA is nothing more than a "glorified survey" and just like any survey it is very subjective. These people are giving their own opinions and an "objective opinion" is nothing more than an oxymoron.</p>

<p>The fact that a 25% weight is given to a glorified survey that only 51% of those asked chose to return, should be more than enough to know what those rankings really stand for. Please, stop dissecting data to support your positions. It is getting beyond ridiculous now,</p>

22

<p>
[quote]
They give a 5 or a 1 on the basis of what they "feel" the institution represents to them. That is what throws the PA 's credibility out the window. Personal likes or dislikes, personal experiences, long standing reputations as well as second hand information, they all play a role at the time they have to write down their little number.

[/quote]
</p>

<p>You honestly think that the presidents of these universities are that clumsy in their response? I highly doubt you've ever filled one of the surveys out, so I don't think you really know what goes through their heads when they fill it out. But I'll say one thing: it's highly unlikely that the presidents are going just on general "feelings" or "perceptions" of the universities -- rather, on what they know of them, facts, etc.</p>

23

<p>^^^ Do you honestly think that the presidents of these universities can not be "that clumsy"? Do you honestly think that the presidents of these universities are infallible ? How many surveys have you filled out? Have you ever seen one? </p>

<p>Why is it highly unlikely that the presidents are not going on general 'feelings" or "perceptions" ? What "facts" do they know about the other schools? Even better, how do you know what they know or what they don't know? </p>

<p>Do you know what a survey is? Do you know what SUBJECTIVE is? Why don't you say anything about the fact that only 51% of those asked bothered to answer the survey? What if the people most biased towards any kind of opinion are those who were motivated to answer?</p>

<p>Do you get it??</p>

24

<p>
[quote]
Do you honestly think that the presidents of these universities can not be "that clumsy"?

[/quote]
</p>

<p>Yes, I do.</p>

<p>
[quote]
Do you honestly think that the presidents of these universities are infallible ?

[/quote]
</p>

<p>I never asserted that they're infallible -- not clumsy =/= infallible.</p>

<p>
[quote]
Why is it highly unlikely that the presidents are not going on general 'feelings" or "perceptions" ?

[/quote]
</p>

<p>Because they're presidents of universities. I think it's safe to assume they're not idiots.</p>

<p>
[quote]
Do you know what a survey is? Do you know what SUBJECTIVE is? Why don't you say anything about the fact that only 51% of those asked bothered to answer the survey? What if the people most biased towards any kind of opinion are those who were motivated to answer?

[/quote]
</p>

<p>You don't need to attempt to insult my intelligence. A 51% turnout is great and most likely follows normal distribution of responses (i.e. there are enough unbiased opinions given so that the supposedly biased ones don't have much of an effect).</p>

25

<p>
[quote]
A 51% turnout is great and most likely follows normal distribution of responses (i.e. there are enough unbiased opinions given so that the supposedly biased ones don't have much of an effect).

[/quote]
</p>

<p>What?? Absurd! You have just attempted to defend the weakest point of the PA. There is plenty of data out there that demonstrates that you are way wrong, just with basic statistic analysis. No need to take this argument further.</p>

26

<p>
[quote]
There is plenty of data out there that demonstrates that you are way wrong, just with basic statistic analysis.

[/quote]
</p>

<p>Why don't you point out that data? I haven't seen any such data.</p>

27

<p>"Why is it highly unlikely that the presidents are not going on general 'feelings" or "perceptions" ? What "facts" do they know about the other schools? Even better, how do you know what they know or what they don't know? </p>

<p>Do you know what a survey is? Do you know what SUBJECTIVE is? Why don't you say anything about the fact that only 51% of those asked bothered to answer the survey? What if the people most biased towards any kind of opinion are those who were motivated to answer?</p>

<p>Do you get it??"</p>

<p>The PA score is subjective, yes, but to a certain extent, the way one perceives a school IS reality. Besides, even if you were to factor out PA and only look at SAT scores, GPA, admit rates, etc. the top 25-30 schools would basically stay the same save for some shuffling around here and there and random schools dropping in/out.</p>

28

<p>
[quote]
I believe most of you are neglecting in your ridiculous analysis of these statistics, the fact that the PA is 100% subjective.... Please, stop dissecting data to support your positions. It is getting beyond ridiculous now

[/quote]
</p>

<p>It is sound to use objective analysis to try to discern what institutional characteristics might influence perceptions. These ephemeral "feelings" of which you are so dismissive came from somewhere--they don't pull impressions of institutions out of a hat. </p>

<p>What you call "ridiculous" I see as an attempt to refute some far-fetched claims. And it wasn't exactly invented in this thread--social science research has long been trying to determine the correlates of attitudes and beliefs. Your sweeping dismissal of applying the same kind of thinking to these threads seems sorely misplaced.</p>

<p>I'd also point out that some of us aren't "defending our positions" but are rather trying to eradicate misconceptions and ignorance. I don't have a "position" on what influences attitudes--but I am certainly open to finding out what they might be. Correlations, regressions, and factor analysis are one of the first things this social scientist would consider in that search. Granted, we're not doing high-level statistical analysis in this thread, but the basic application is the same here.</p>

<p>
[quote]
There is plenty of data out there that demonstrates that you are way wrong, just with basic statistic analysis. No need to take this argument further.

[/quote]

No, I don't think that is the end of it. If you're going to slam the use of bi- and multi-variate analysis to find the correlates of perception & opinions, then I'm eager to know about the statistical analysis you're speaking of.</p>

29

<p>As we all know, the vast majority of ranking controversies involve schools ranked in the USNWR Top 50 and thus my initial analysis of these schools and their graduate programs was limited to this universe of colleges. These are the most visible schools in the country and no one has yet provided a fair explanation of why a college listed among the top half of large graduate programs has a 75% likelihood of scoring higher than its overall USNWR rank and one in the lower half has an 80% likelihood of scoring lower than its overall USNWR rank.</p>

<p>I know all of the arguments and comments about correlation and causation and perhaps this is just a statistical anomaly, but maybe it’s not. Maybe it is a reflection of a blind spot in the way that PA scores are assigned and reflects what I think is the reality, ie, that graders have great difficulty distinguishing between overall university reputation (including size and strength of graduate programs) and undergraduate quality and/or don’t have the breadth of familiarity to make fine distinctions between what is going on at the UNDERGRADUATE level at U Texas vs what is going on at Wash U vs what is going on at Wake Forest, etc. </p>

<p>To midatlmom,
I appreciate your comments but we have several differences on what PA is and what it does, including:</p>

<ol>
<li><p>I see the subjectivity and Rorschach Patch nature of PA as rendering this number as ill-defined at best and heavily misguided at worst. I have a greater appreciation and understanding today than a year ago for those viewpoints opposite my own, but this has actually led to greater conviction than ever that this number subverts the rankings more than helps them. An appropriate compromise, as suggested repeatedly by xiggi and others including me, is to separate out the PA and rank the schools based on the objective data and then do another ranking based on PA. This would hopefully satisfy those PA backers while allowing the balance of the rankings to be accepted by others (like me) that a more objective result has been created (and then we can fight over whether the other factors are fairly weighted :)). </p></li>
<li><p>You believe that teaching is one of the factors being surveyed in the PA. I don’t reach that conclusion (otherwise a school like W&M or Wake would be a lot higher) and really am disappointed that this feature was discontinued with the 1995 survey that left many traditional PA powers out of the Top 25. Certainly it is possible you are right that teaching is considered by some voters, but we will never know given the way that PA is currently constructed and reported.</p></li>
<li><p>The criticisms that I and many others have directed at PA is not some sort of “gotcha” game as you suggest. But it is useful to see how some defenders of PA respond to challenges to these numbers (eg, attack the motives of those challenging the academic status quo and believe me, it would be easy to respond that nearly all PA defenders have close associations with colleges that have high PAs). For me, part of the frustration is that so much of the academic world (and many defenders of PA) act as if nothing can ever change and that institutions don’t evolve. Bull. </p></li>
</ol>

<p>Among other things, I believe that PA perpetuates a pecking order in the world of academia that is not representative of what I see in the student quality going in or coming out of these schools and the ways that employers across the country view many of these institutions. American society has changed so much over the last 30-40 years and yet there has been remarkably little evolution in the academic world, including college rankings, and God save those schools who have worked the rankings to upset that order (see Wash U). </p>

<p>I believe in competition and I think that the PA stifles competition as I see this score as a loaded game with in-bred biases, such as the inclination toward colleges that have larger graduate student bodies. So I suggest some less traditional (but perhaps more relevant) faculty assessments that you consider na</p>

30

<p>And critics of PA seem to have pet schools that are lower on PA--Wake Forest, W&M.
Here's a little real world result we can both enjoy.</p>

<p>"The top ten schools on the (2006) CPA exam are: (1) Wake Forrest University, (2) University of Wisconsin-Madison, (3) University of Washington, (4) University of NC-Chapel Hill, (5) San Diego State University, (6) University of Texas at Dallas, (7) Brigham Young University Main, (8) Cleveland State University, (9) University of IL-Urbana-Champaign and (10) Bently College, DePaul University and University of Georgia (tie).</p>

<p><a href="http://www-rohan.sdsu.edu/%7Ecba/news/story/acct-ranking.html%5B/url%5D"&gt;http://www-rohan.sdsu.edu/~cba/news/story/acct-ranking.html&lt;/a&gt;&lt;/p>

31

<p>
[quote]
it would be easy to respond that nearly all PA defenders have close associations with colleges that have high PAs

[/quote]
</p>

<p>Ah, I'm skewered. Before getting my degree, my committee had some final instructions for me. They said "Never mind what you learned here, there is one lesson more important than all others. PROTECT THE STATUS QUO!" Then they slaughtered a goat, smeared a red "PA" on my forehead in its blood, and threw up the Phi Beta Kappa gang sign. </p>

<p>It's a touching ceremony, dating back from the guild days at the University of Padua in the 13th century.</p>

32

<p>Finally, the code of silence has been broken and the truth has been revealed.....Btw, please one more secret-what is the PBK gang sign? :)</p>

<p>On a more serious note, would you care to comment on the idea of providing rankings for PA separately from the objective data?</p>

33

<p>
[quote]
I believe that PA perpetuates a pecking order in the world of academia..

[/quote]
</p>

<p>hawkette: so does nearly every other [so-called] objective measure. :)</p>

<p>If you recall, the orginal USNews "ranking" had several publics in the top 20. But, the cynic in me says that demonstrating that a public was a good as [fill in your NE private name] was not good for selling magazines. Voila, change the objective data and all is now right with the world. :D</p>

<p>In answer to your query and xiggi's recommendation: If USNews wants to allow data manipulation on thier website, they surely can. But, if they don't wanna, they don't wanna. [hey, xiggi, would you be so anti PA if UT-Austin had a 4.9?]</p>

34

<p>Wow, hawkette, I'm a little surprised at your post, which seemed to get personal, but so be it. To answer some of your points:</p>

<ol>
<li><p>You continue to cite the 1995 survey about teaching as if it were some kind of Holy Grail. Yet those figures came from a survey of the same academics who you assume are biased perpetuators of the status quo, determined to misguidedly place the same old schools on top regardless of changing facts. You really can't have it both ways--either PA is useless or PA has some validity. It can't be valid for some questions, but not for others.</p></li>
<li><p>You say you can't reach the conclusion that quality of teaching is one of the aspects being surveyed in the PA questionnaire. Yet in the intro to the Peer Assessment discussion in USNWR, it states that "the peer assessment survey allows the top academics we consult . . . to account for intangibles such as faculty dedication to teaching." Well, this clearly indicates to me that USNWR is asking survey participants to include quality of teaching in their responses. While I think that assessing quality of teaching is quite difficult (and highly subjective) and I haven't seen a great way of measuring it, it is clear that it is one of the factors that survey participants are being asked to comment on. They may do a poor job, they may not all have the background, but I can't imagine that they are ignoring it.</p></li>
<li><p>As I have mentioned in several posts, I have a relative who is a college administrator. I have enormous respect for his opinions and that probably colors my opinion of PA as being valuable. Rather than being the sort of smug perpetuator of the status quo you seem to envision, he is a genuinely curious, intellectual, very modest guy who really cares about the quality of teaching. When we speak about colleges, he seems to know a lot about the ins and outs of various schools, particularly faculty/administration relations, strength of departments and whether a department is fractured due to conflicts etc. I think his insight is quite valuable and I believe that there are many similar administrators across the country, who probably have a better knowledge of colleges than you or I.</p></li>
<li><p>I actually have no horse in this race (I went to Princeton, but I don't think that its status would change much if PA were not a factor), nor am I the one who is constantly starting threads in an attempt to prove that PA is biased, that Wake Forest has great teaching and that schools with great sports programs are better than schools without them. I agree with you that there are lots of excellent schools and that the process of precisely trying to rank them is ridiculous. I agree with you that parents and students often treat the rankings as if they are sacrosanct, and that they might miss out on some excellent schools because of over-reliance on rankings. I disagree with you that PA is inherently awful and that student surveys and employer surveys are inherently better. I think that if PA were taken out of the mix, most of the state schools would drop a notch and some of the private schools might rise a little, but I do not think it would give us a better understanding of quality of teaching. </p></li>
</ol>

<p>Personally, I have always thought that if people want rankings, a tiered approach to schools rather than precise rankings would be the most helpful. However, I doubt that USNWR would consider it because it would probably not sell as many magazines (and it would of course still be controversial). I also think that on an individual level, we should all tell our children that labels and rankings do not really add that much and that an understanding of the student's preferences about location, size, academic programs available, percentage of greek life etc., and a college visit if possible, are the best ways to determine if a school is right.</p>

35

<p>Collegehelp has shown that PA is more closely associated with many other factors than with number of graduate students. So why continue to harp on this? </p>

<p>It was an interesting theory that number of grad students is a major predictor of PA. </p>

<p>It turns out that it was not true. </p>

<p>So move on.</p>

36

<p>
[quote]
Collegehelp has shown that PA is more closely associated with many other factors than with number of graduate students. So why continue to harp on this?

[/quote]
</p>

<p>Could it be because CollegeHelp smorgasbords of numbers are far from being as conclusive as he and some might think. </p>

<p>Doesn't College also believe that say that: </p>

<p>"The peer assessment rating is highly correlated with freshman retention, graduation rate, SAT scores, the proportion of students in the top 10% of their class, acceptance rate for BOTH LACs (which don't have grad students)and universities. Perhaps universities with large grad programs ALSO attract great freshman and have high grad rates."</p>

<p>This assessment is only valid for a generic evaluation of the data, and this generic evaluation does not amount to much at all because it fails to uncover the outliers and the statisticals anomalies. </p>

<p>And guess where the problems with USNEws and its PA start and end: the blatant anomalies that cause schools to be underrated or overrated in the PA category, and skew the objective data of the remaining 75% of the ranking. </p>

<p>To put il mildly, I dont see what the --supposedly-- high correlation factor of SAT scores, the proportion of students in the top 10% of their class, acceptance rate (could be shortened to selectivity index) and the PA for the entire LAC realm might mean, if I am ONLY interested in comparing the PA and selectivity index of say, Smith College and Harvey Mudd.</p>

<p>So, how does that mythical correlation look here?</p>

<p>


Schools ..... HMC Smith</p>

<p>25% SAT         1420  1140
75% SAT          1550   1370
Average SAT 1485    1255</p>

<p>Top 10% HS  94% 61%
Admit Rate  30% 53%
Selectivity     1   46</p>

<p>USN Rank    15  17</p>

<p>PA Raw #    **4.1  4.3**
PA Ranking  18  6

</p>

<p>I'll wager that the correlation would be far from a 1.00 here. Still care to pretend that "peer assessment rating is highly correlated with freshman retention, graduation rate, SAT scores, the proportion of students in the top 10% of their class, acceptance rate for LACs ?"</p>

37

<p>
[quote]
[hey, xiggi, would you be so anti PA if UT-Austin had a 4.9?]

[/quote]
</p>

<p>What, are you telling me UT-Austin does not have a 4.9? The horror, the horror!</p>

<p>Oh, fwiw, it would make no difference whatsoever in what I think about the PA. The bull***** takes place about 25 spots higher than UT's, and in the next section. I'll stop being anti-PA the day USNews stops using it.</p>

38

<p>Most people use more than 2 data points for any stat. Do the top 20 LACs and come back. Smith is an old money school with a long history. That never hurts the image. My car mechanic went to HMC. He's a great guy.</p>

<p>"Chris is one of the best Porsche mechanics in the Pacific Northwest. What does a degree in Chemistry from Harvey Mudd College do for him in his Bellevue, Washington Porsche Shop?"</p>

39

<p>Sure, Barrons! You make as much sense as ever and continue to show yor inabiliy to understand the simplest of points.</p>

40

<p>
[quote]
The horror, the horror!

[/quote]
</p>

<p>Sometimes we agree to disagree, but you are still one of my favorite posters. :)</p>