Peer Assessment-Is this a useful tool in the College Selection process?

College Search & Selection
61

<p>I pulled out my old copy (2005) of the USNWR rankings and scanned the peer assessment scores. For the most part as I scan down the 'best national universities' list, the PA scores decline. But there are some interesting anomolies that stick out. WUSTL is one. It is ranked #11, but with a PA score of 4.1, yet it is surrounded by schools with scores of 4.4-4.7. Could it be that WUSTL's PA doesn't fully reflect its rise as a premier institution? If so, this seems to indicate that PA is a lagging indicator. The same is true for Notre Dame, ranked #18 with a 3.9 PA. OTOH, Cal Berkeley, #21, has a PA of 4.8 amidst schools with PA's averageing about 4.1. Is Berkeley's PA also a lagging indicator? Only 5 other schools have a higher PA and all are in the top 5 (or tied).</p>

<p>The LAC PA ratings seem to be a bit more in line.</p>

62

<p>Tarhunt, hoedown or anyone,
If you have the PA numbers from the early 80s or even before (when were they first published??), could you post them? Maybe you’re right that I’m way off and there have been some shake-ups since that time.</p>

<p>I made the statement about my perception of changes in Amercan education, so I guess I better try to explain my thinking. I will concede that much of this is my opinion formed over years of work, travel, and living in various areas of the country over the last twenty-five years. So much has changed in American life and American business over that time and I think American business people are recognizing the improved talent in all of the regions of the country. Perhaps, tarhunt, you will have different experiences and conclusions from your OB consulting. I’ll be curious to read more of your thoughts. </p>

<p>Several factors have led me to the conclusion that the quality of American education is today much more broadly distributed than ever before and that several important, inexorable trends are in place that will continue this. First, we all know that several demographic factors have led to a large increase in the number of students applying and attending all colleges. Combined with a greater acceptance of students of color, of women, of greater religious tolerance, of non-Americans, the quality of the pool pursuing a college education has greatly expanded and improved. At the same time, the great wealth creation of the 80s and the 90s has fueled a greater than ever competition for developing high school students able to achieve at the highest levels. It is true that the absolute number of college students is expected to level off and decline in a few years as the baby boomer children age. But for the moment, the student demographics are a powerful force and we see evidence of it everywhere in the lower college acceptance rates, especially at the top 20 schools (30% or lower at all, 15% or less at many). </p>

<p>I think we all know plenty of people (perhaps including you) who attended an Ivy school in the 1960-80 period who laugh now about how much easier it was to get in back in those days and thank God that they aren’t applying in 2006-07. Today, many of those students would likely matriculate at schools outside of the Ivy League and perhaps very far away geographically (WashU, Rice, Emory, Vanderbilt, some of the top state universities in other parts of the country, etc.). This larger and better prepared applicant pool has been a sort of supply-side academics and led outstanding students to consider and go to new schools and new geographies. The result is that American student excellence has become more broadly distributed (and I think that this is a great thing for the country both economically and socially). </p>

<p>Second, the role of technology has been huge. In the early 80s, Microsoft was a baby and to see how PCs have changed our world is to understand how these devices have empowered so many people with tools and computing power previously available to only a few (many of whom were located in the academic world). With the arrival of email a decade later, the increased ease and quantity and quality of communication reduced barriers and allowed those in more remote areas with fewer of the traditional resources (eg, libraries) to learn and grow and prosper. As a result, faculty in far-flung colleges have quick access to studies being performed in Princeton or Ithaca or New Haven and an ability to communicate individually and collectively with others as if they were in the next room. The value of this change cannot be overstated. </p>

<p>Third, and still on technology, the internet continued and daily accelerates the devolutionary process that was begun with the increased use of personal computers and then email. The competitive information edge that Ivies and other top NE located privates enjoyed was narrowing and would be exploded by the arrival of the internet. Twenty-five years ago, the average American might have looked to the Ivy League as the bastion and perhaps single source of excellence in American college education. While still recognized today as exceptional, the Ivies’ information advantage has been completely exploded with the internet. Now a student in West Texas (or Jaipur or Guangzhou or Rabat) has nearly the same access to information as a student in Cambridge or New Haven. As Tom Friedman has made clear, electronic distribution of resources have completely changed the map in the business world and I believe that this more broadly distributed knowledge is driving the same changes in American colleges same (albeit at a slower rate and in less noticeable ways). </p>

<p>IMO, the net result is that the Ivies and some other top NE and CA privates may have maintained a prestige edge, but in practical terms, there are more great students than ever before and an Ivy-level academic education is available at many, many places around the country and in areas that have been traditionally considered academic backwaters. Such places may not get the regular mentions in the NYT, but that does not diminish their quality which is understood by corporate America. Furthermore, IMO, if America is able to sustain its history of attracting many of the best and brightest students from around the world (and that is somewhat doubtful today), then this process of educational devolution will continue and spread. </p>

<p>If you are from outside the educational status quo, it’s a pretty exciting time as the people in the less recognized schools and geographies are playing on an increasingly level field. They know the quality of their students and the quality of their educational product and increasingly employers are embracing it with the growth of their regional economies. </p>

<p>Meanwhile, the Ivies, while still pre-eminent in prestige (particularly among academics), are likely working harder than ever to maintain their position. Money is a big part of their defense and positions them well to remain atop of the heap, but after a certain point, the amount of endowment reaches of a point of diminishing returns. Furthermore, there is now money in all parts of the country and with that has come greater resources and a fuller ability to deliver undergraduate excellence. </p>

<p>IMO, the Peer Assessment is one more way that the educational establishment (as represented by the Ivies) is working to perpetuate the traditional ranking of American colleges. But I think business people are ahead of the curve on this and increasingly understand that while the Ivies are very strong, so too are many other places and certainly more so now than at any other time in our history. </p>

<p>Boy, sorry about the length. I guess I can get pretty worked up about this topic. Anyway, thanks for reading thru this mostly pretty obvious stuff.</p>

63

<p>While one can communicate fine from anywhere, you better be at a place that has the physical plant to do your research, other quality people to help, and a reputation that helps get research funding. Otherwise you will be a nobody as far as academe is concerned. More work today involves multiple disciplines and more cash. The easy stuff has been done.</p>

64

<p>Hawkette:</p>

<p>No need to apologize for length. Sometimes, subjects take a bit of explanation, and I'm glad you took the time to do that. As for obvious, well, I don't think what you said is so very obvious. At least, it's not obvious to me.</p>

<p>Let's start first with student skills as opposed to "talent." I believe that skills are, roughly, talent x work. Talent is nice, but it doesn't equal skills. It simply defines how easily one acquires skills and/or how much one can develop one's skills, given infinite work.</p>

<p>I teach in a flagship public university. Like most such universities, the student body, as measured by standardized test scores, is a bit better than average. I must say that, instead of seeing "more great students than ever before," I'm seeing a precipitous and alarming plunge in skills. I'm one of those who is quite skeptical of attacks on whatever the newest generation happens to be, but I have to say that I've joined the chorus on this. The current students I'm getting (with exceptions, of course) are, by and large, shockingly awful. </p>

<p>Very few can write cogently. Their thoughts tend to be badly disconnected and scattered, when they actually have thoughts. Ask them to take some principle(s) they have learned in class and apply it to a real life example, and they are simply flummoxed. They cannot do it, and they complain bitterly that "that wasn't taught in class." Many of them call their parents, who call me, also complaining bitterly. I have had physics professors tell me it's the same with them. Ask students to apply principles to some problem they haven't been walked through in class, and they simply cannot do it. They're great at cooperating. They're so good, in fact, that I will venture a guess that this generation will give new meaning to the term "group think."</p>

<p>Are skills more distributed right at the moment than they were in 1982? Probably. Since we're reaching the crest of the post baby-boomlet, and since most of the elite schools haven't increased their size, some kids who would have gotten in to the very elite schools are now being pushed out to what would once have been their backup schools, and they, in turn, have pushed other students down the admissions ladder. But the distribution of skills on a relative basis has changed little, if at all. The elite schools still attract the top talent if for no other reason than simply being admitted to them indicates extraordinary high school success. Being admitted to that "brand" means something.</p>

<p>I think you have a good point about the availability of information on line. Once, it was important to be near a good library with 20,000 periodicals arriving constantly. Now, one can access most of those publications on line. Having said that, I can assure you that much of what one needs for research is still in books, and most books have not been committed to ones and zeros. The same is true of dissertations. If I want a dissertation on, say, Assyrian chariot parts, I will need to find that in a single place.</p>

<p>Are faculty better than ever before? Are today's Nobel Prize winners better than those from the past? I doubt it. I certainly see no evidence that those in the academies are any better these days than they were in 1982, though they may well be better, overall, than they were in 1942.</p>

<p>Back to my hypothesis, hawkette, which you must be sick of by now. Put skilled students together with skilled faculty in small classrooms and good things tend to happen. The elite, rich schools still tend to have the best students and the smallest class sizes. Whether the faculty is actually better at teaching than faculty elsewhere is very debatable, but I'd suggest they are probably not any worse. Yale still does this better than AOregon State, in my opinion (not picking on Oregon State; I could have used any large U).</p>

<p>To reinterate, I don't think talent distribution has changed much among human beings. What has changed, and much for the worse, are skills. Kids don't read much anymore, and it shows in their ability to absorb abstraction and think write about abstract concepts. My international students tend to be much better than American students, even when writing in English when English is their second language.</p>

<p>I wonder if you've ever seen "The Paper Chase"? There's a scene in there in which the main professor tells his students that their minds are mush, and that he will make them sharp. I've always loved that scene, because it defined what I thought was important about the part of my job that involves teaching. I found it inspirational. Unfortunately, it is possible to accomplish this only in a relatively small number of circumstances, now. Most of the students don't have mushy minds; they have liquified minds.</p>

65

<p>Take a look at the founders of the AAU in 1900. A few majors have been added (Mit etc) but most top research schools where tops then too.</p>

<p><a href="http://en.wikipedia.org/wiki/Association_of_American_Universities%5B/url%5D"&gt;http://en.wikipedia.org/wiki/Association_of_American_Universities&lt;/a&gt;&lt;/p>

66

<p>tarhunt,
Great stuff. You are clearly much closer to the students than I and that gives your words real power with me. As you point out, the quantitative data show that there are many more and better applicants to colleges generally in America today. Along with my own experiences and discussions with business contacts, I interpreted and extrapolated that to mean that the students coming out of colleges today are more talented and precocious than ever before. I still believe that, but your comments give me pause and I want to think about it some more. </p>

<p>With regard to skills, I completely agree, but in defense of these students, they have great skills in other areas that I (and perhaps you) can only dream of. For example, their ease with technology and applying that in everyday, new ways is evident with things like Facebook, MySpace, and applications involving technology devices like iPod, cellphones, video equipment, etc. And the enormous impact of these ideas/products on popular culture or on the business world cannot be denied, even if you don’t approve of them or particularly like them. Perhaps the kids today have more skills than you appreciate, but they are displayed in ways that are different from how you and I learned. As an optimist, I’d like to believe that. :)</p>

<p>Barrons,
While I appreciate that research work can have great value to certain areas of study and work, I question the relevance of much of it to most students. In your opinion, if a student wants to study landscape architecture or film or hotel management or some other non-technical field of study, should a college’s Peer Assessment score have any bearing on a high schooler’s college choice? If so, why?</p>

67

<p>use peer assesment somewhat like a GPA.</p>

<p>a 3.9 and a 4.1 are pretty similiar - much like a 2.9 gpa and a 3.1 gpa (they're both really just B's). what does it tell you though? the kid with the 2.9 got more c's than a's - and mostly b's, and the kid with the 3.1 got more a's than c's - with mostly b's.</p>

<p>a school with a peer assesment score of 3.9 got more "3s" than "5s", while a school with a 4.1 got more "5s" than "3s." </p>

<p>what does this mean? it means that while a 3.9 school and a 4.1 school are both "good" schools, the 4.1 school was seen as more distinguished by more people than a 3.9 school. does it matter? yes, i think it does, while both schools are "solid," more people think the 4.1 is a "5" school than the 3.9 school. </p>

<p>its obviously not going to tell you that the faculty are better at one place than another or which is better, but I belive PA (while a number arrived to subjectively) is the only OBJECTIVE data put into the USnews rankings. I know it sound strange, but while you can pick and choose any other piece of data to include in your rankings, (does graduation and retention over/underperformance really make 1 school better than another?) what professionals in the field think of other schools is a good barometer of the quality of an institution.</p>

68

<p>jags,</p>

<p>There is no statistical difference between 3.9 and 4.1. The difference is well within the margin of error for the sample size.</p>

69

<p>barrons,
A few more follow-up questions, if I may. Your point is well made about communication and information access being only one piece of the equation and there is also a need for quality facilities, quality faculty, and a good reputation to attract grants. The issue of facilities has been raised on a few occasions here and in other threads. What do you really mean by this? What I am trying to understand is whether this is a sustainable competitive advantage. If this is merely a bricks & mortar issue relating to lab facilities and such, there may be an advantage now or historically, but this can be overcome if the have-nots raise enough capital to build equal or better facilities. Right?</p>

<p>As for quality faculty and reputation, obviously these are essential to any "elite" university, but other than the previously mentioned criteria for faculty recognition (grants, papers published, awards, etc), do you believe that there are any other factors that are germane? In your opinion, should it matter how well (or how often) a "top" professor actually performs in the classroom? Should existing students or alumni or employers have any say in judging the quality of a faculty? </p>

<p>As for reputation, my impression is that much of this (at least among academics) is driven by factors that have little to do with the students in non-technical fields of study. I suspect that you would disagree with this, but I don't understand the reasoning why.</p>

70

<p>hawkette:</p>

<p>Perhaps you are right, but I will have to be convinced that knowing how to operate an iPod or being really hot with a joystick is as important to personal and societal success has being able to use the basic rules of logic.</p>

<p>A clarification. There are (simplified) two kinds of students I see coming out these days. A very few are extraordinarily well prepared. They have access to educational opportunities that were simply unavailable 25 years ago, and they READ. A LOT. Past them, we have a cliff to the people who can memorize information just fine, but can't connect the dots. I don't mean they don't connect the dots. I mean they can't. They don't know how.</p>

<p>As an example, read one of the threads on this board where kids are ranking schools. They don't use criteria. They don't even know that they need criteria. Opinion is all that counts, even if that opinion is based on nothing but some sort of "gut." Or follow along on a thread in which the debate hinges on the definition of a term or terms. They get into heated arguments without ever even knowing they need to define their terms.</p>

<p>I learned both those things in grade 8 in a dismal school system in the rural South but, the fact is, those things were also learned by osmosis as I read things.</p>

<p>So, hawkette, let me tell you what happened in the last undergrad course I taught two years ago. In one class, I ran them through the basics of semantic differential testing, explaining what it is, why it's useful, and the cognitive impact of language on behavior. I also took them through the way semantics of individual words have changed, historically, and both historical and modern attempts to deliberately move semantic differentials across broad populations.</p>

<p>OK. So, we get to the test, right? One of the items on the test was something like this: "Choose a loaded word in modern American society and explain how, if you were to perform a semantic differential survey, the results might look. Taking those results, explain how you might use this word to some advantage in influencing mass behavior. Be sure to support your conclusion with relevant cognitive research results."</p>

<p>So, I get the tests back. The kids have memorized pretty well. But the answers to this question were ... troubling. Deeply troubling. This was not a hard question. All it required was that one take what one has learned and apply it to a theoretical construct in a useful way. Frankly, it's obvious. All but one kid in that class of around 35 couldn't do it. Most of them couldn't even construct sentences or sequence information in some sort of logical order. </p>

<p>Predictably, I got a long string of distraught kids and even some phone calls from parents. The general line was that the test wasn't fair because I hadn't covered this specific example in class. </p>

<p>This is not just me. I have a good friend who teaches elementary, classical physics on occasion. He gave his class a simple problem of the energy necesary to turn a crank handle on a well. All it requires is that one treat the handle like the lever that it is and plug in an equation. The class imploded.</p>

<p>I have seen the pundits who opine that reading is kaput, and that we will get our information in different ways in the future. I have also seen some neurological studies that suggest that, short of an evolutionary leap, only reading will stimulate that part of the brain and form the neural connections needed to build symbolic constructs, and it is symbolic constructs that lead to human advances.</p>

<p>I guess I'm not as optimistic as you are ;-).</p>

71

<p>standrews,</p>

<p>there is no "margin of error" in a survey such as this. what are you talking about? Unless you actually believe that a multimillion dollar company can't keep track of 200 votes...</p>

72

<p>jags:</p>

<p>I think standrews has this mixed up. Obviously, if the response from 200+ schools is 100%, there can be no margin of error, since there is a 100% sampling. Of course there's a margin of error based on rater bias, but that's not measurable in this way.</p>

<p>Now, if we were talking about a purely random sampling of a very large population, he would be substantially correct.</p>

73

<p>Yes, bricks and concrete can be added but it tends to be a fact that the top schools have an advantage in this area too. Look at the lists of top fundraisers and endowments--not many lower tier level schools in the Top 20 on either. As to top profs, the fact is that a connecton with a top person in their field will carry more weight with grad schools and employers than one from a great teacher nobody has heard of. Also in my experience top profs tend to be interesting and engaging and at least decent teachers.</p>

74

<p>barrons,correctly points out that the top schools have impressive endowments ( at #25, Brown has an endowment of just over $2bn) as the rich have definitely gotten richer in America in the last decade and this pattern continues in academia. But the endowment numbers at many lower ranked schools have also grown significantly. As of 6/30/06, 62 colleges had endowments in excess of $1bn. I conclude from this that many lower ranked colleges, while still lagging in absolute endowment dollars, have the wherewithal to build competitive, world-class research facilities if this is an institution's strategic priority. Whatever facilities edge the "elite" schools might currently have, this is likely not a sustainable advantage over many similarly motivated lower ranked schools. </p>

<p>I understand the affection for high quality faculty, but I am trying to understand how to measure this and whether this measurement, which may be useful in the academic world, is relevant to current high schoolers as they make their college selections. While I'm sure that many top professors at elite schools are interesting and engaging, this is certainly not unique to colleges with high PA scores. Other than the grants/papers/awards criteria that was previously mentioned, is there any other way to measure the quality of a faculty, particularly one that might be in a lower profile part of the country? Also, should other stakeholders (students, alumni, corporate recruiters) have a role in making this faculty assessment?</p>

75

<p>The price of being really competitive is very high. Wisconsin for example has two major building programs underway for just the sciences--Wistar and Biostar. The total cost of these is over $1 Billion over 10 years. (and a Billion at UW might cost nearly twice that in Boston, NY or the west coast) That's a lot of cheese.</p>

<p>Now there are what I will call mid-majors (see Northeastern) that can do one nice $100 Million facility but that will not catch them up to the big boys.</p>

76

<p>Tarhunt, I read your post #70 and I see the problem as your students' education before they get to your class.</p>

<p>Kids from elemenatry school, middle school and high school are taught similar subjects in different ways.</p>

<p>Kids from lower and middle class schools are generally taught a lot of memorization. You learn concepts, but you don't have to think how to use the concepts. The leaders do that.</p>

<p>The kids from the upper middle class and upper class schools are taught how to present ideas. They are taught how to take a concept and how to use it in other situations. They are taught how to communicate their ideas (even if the ideas are nonsense). </p>

<p>So, the kids in your class may never have been taught how to take a concept and use it in other situations. They may never have been taught how to present their ideas in a well thought out way.</p>

<p>Your students may think the idea of education is to learn facts, not how to use facts.</p>

<p>What's the make-up of your class?</p>

77

<p>Those building costs at Wisconsin sound awfully high. Can I bid on some of that? :)</p>

<p>You're right about the relative cash levels and I am not contending that the "mid-majors" will ever catch up and I'm don't think they ever need to. It's just that the marginal utility of each dollar above a certain level ($2bn? $3bn, $5bn?) diminishes. These lower ranked schools, particularly as measured on a per capita basis, will still have pretty sizable resources to invest in their facilities, their faculty and their students. </p>

<p>Any thoughts on my faculty quality assessments questions?</p>

78

<p>The NSSE for schools that use it is pretty good. </p>

<p>I think UW is very cost efficient on their construction. They don't do the archictect controlled things that are expensive but of dubious utility. </p>

<p>Here's an example from two very similar buildings I have seen. $8.5 Million vs. $18 Million in the same year. I will say the expensive architect had better photos.</p>

<p><a href="http://www.uwbadgers.com/facilities/boat_house/index_75.html%5B/url%5D"&gt;http://www.uwbadgers.com/facilities/boat_house/index_75.html&lt;/a&gt;&lt;/p>

<p><a href="http://www.millerhull.com/html/index.htm?http%3A//www.millerhull.com/html/institutional/conibear.htm%5B/url%5D"&gt;http://www.millerhull.com/html/index.htm?http%3A//www.millerhull.com/html/institutional/conibear.htm&lt;/a&gt;&lt;/p>

79

<p>Nice photos. Even at the higher $18mm cost, you could build a heckuva lot of buildings for $1bn. Should be a pretty awesome place when it is done.</p>

<p>Any thoughts on student, alumni, corporate stakeholders having a say in the faculty assessment process?</p>

80

<p>The impact of Peer Assessment scoring is seen most strongly in the USNWR rankings of public universities. I surveyed the top 15 National Public Universities both with and without Peer Assessment scoring. </p>

<p>If you remove Peer Assessment scoring, the rankings for the Top 15 National public universities would be adjusted as follows:</p>

<ol>
<li>University of Virginia (USNWR ranking of 23rd without PA, 24th with PA)</li>
<li>University of California-Los Angeles (28th, 26th) </li>
<li>U of North Carolina-Chapel Hill (28th, 27th)</li>
<li>University of California-Berkeley (28th, 21st)</li>
<li>University of Michigan-Ann Arbor (34th, 24th)</li>
<li>College of William and Mary (34th, 31st) </li>
<li>Univ. of California-San Diego (38th, 38th)</li>
<li>Univ. of Wisconsin-Madison (42nd, 34th) </li>
<li>Univ. of California-Santa Barbara (42nd, 47th)</li>
<li>University of California-Irvine (46th, 44th)</li>
<li>U of Illinois-Urbana Champaign (47th, 41st)</li>
<li>University of Florida (48th, 47th)</li>
<li>University of Washington (49th, 42nd)</li>
<li>Pennsylvania State University (50th, 47th)</li>
<li>University of California-Davis (51st, 47th)</li>
</ol>

<p>Of the top 15 National public universities, the ranking of only two schools are helped by the removal of the Peer Assessment scores (U Virginia and UC Santa Barbara). </p>

<p>The other 13 schools benefit from USNWR’s Peer Assessment scoring. U Michigan (helped by 10 ranking spots), U Wisconsin (helped by 8 ranking spots), and U Washington (helped by 7 rankings spots) are the biggest beneficiaries of the inclusion of Peer Assessment scores. </p>

<p>It would appear from this data that PA scoring works to the great benefit of most public universities in the calculation of the USNWR rankings.</p>