Peer Assessment-Is this a useful tool in the College Selection process?

College Search & Selection
121

<p>hoedown,
As you probably guessed, I know nothing about NSSE and heard about it for the first time on this thread. I am glad to learn of its existence and wish that its results were more broadly publicized and available. For my comment on the sample size, your response makes absolute sense to me. I don't mean to appear critical of the schools as they have their own self interest to protect. If I were a university administrator, turning this information over to someone else to publish or perhaps interpret would be a little scary unless I had high confidence in the result (absolutely and relatively). On the 8.8% number, I am mostly lamenting the poor overall, single-year response rate. </p>

<p>Bottom-line for me, however, is that this is very useful information (for multiple stakeholders) and at least we have a clearer idea of the questions and the nature of the responses. This stands in pretty stark contrast to the PA. Does anything like this ever get created on the PA?</p>

122

<p>I suppose it is scary for some, but some schools have participated in NSSE multiple times, so they have an idea what the data will reveal even before they participate the second time. It's not really fear of the unknown that prevents them from turning it over to USNews. I suppose schools with les-than-flattering results would have concerns, that's true. But the bottom line is, even schools who do relatively well on NSSE measures don't elect to turn it over to USNews. </p>

<p>
[quote]
this is very useful information (for multiple stakeholders) and at least we have a clearer idea of the questions and the nature of the responses. This stands in pretty stark contrast to the PA. Does anything like this ever get created on the PA?

[/quote]
</p>

<p>I think, actually, your concern with PA lies elsewhere. It's not that we don't know the question, it's that the question may be too vague to know what the respondents were thinking about when they responded. On some levels, PA is as transparent as you are going to get when it comes to a survey and knowing the questions and responses. It's one question, with a simple scale, with the results printed in mean form. That doesn't mean it's a good measure, but I don't think its flaw is lack of transparency.</p>

123

<p>PA desperately needs more than one question (NSSE studies had many questions) and it needs much much greater definition on what is being asked and evaluated. Right now it is one question that is open to very wide and very different interpretations by different administrators with different backgrounds at different types of schools in different geographies. The current PA measures…what? </p>

<p>Beyond greater detail of what academics mean when they make their evaluations of other school, PA also needs explicit inclusion of the impressions of non-academic groups (students, parents (?), alumni, corporate recruiters). IMO, the single biggest difference in all of academia vs the corporate world is the lack of accountability. What consequences are there to a faculty member if his/her students consistently underperform (however you want to measure it-but please measure it!)? Expanding PA to include the views of more groups will increase accountability and add a little more competition to the college educational system. That can only be good for students who are the reason that the schools exist in the first place.</p>

124

<p>
[quote]
Expanding PA to include the views of more groups will increase accountability and add a little more competition to the college educational system. That can only be good for students who are the reason that the schools exist in the first place.

[/quote]
</p>

<p>I don't have a problem with improvements to the PA. However, it's almost as though you blame schools for its shortcomings. </p>

<p>It's USNews who developed the PA (which as you'll recall was its only ranking criteria back in 1987) and who has elected to continue using it the way that it has. Institutions are not a part of that decision-making. It's may be compelling to you or me to state that students are the raison d'etre for higher education, but is that an issue for USNews? I hate to sound cynical, but I don't think USNews wants to do what is good for students. They want to do something that is useful and interesting enough to sell magazines. It has to have face value, of course. But beyond that, I don't think USNews feels any moral imperative to do what's good for students.</p>

125

<p>hoedown, I don't blame the schools for the PA's shortcomings and I suspect that many schools share our disgruntlement. My objections are to those partisans who elsewhere have promulgated PA as dispassionate and tangible evidence that a certain faculty is superior to another. As for USNWR, their primary obligations are to their shareholders and their interests are served when magazine sales are maximized. </p>

<p>My impression, reinforced in this discussion, is that the PA and its amorphous quality of faculty link is strongly influenced by reputations forged through research work and other non-classroom activities. While this is legitimate and relevant for a portion of potential college applicants, ie, those who might seek a career in academia, it is far from an insightful or accurate depiction of the quality and contribution of a school's entire faculty. </p>

<p>The PA is so ill-defined in what it represents. This thought has been widely accepted here and IMO this consensus invalidates the use of the PA as part of any ranking system. Does anybody believe that it should stay, particularly in its current form?</p>

126

<p>Research work is a key element of any university worth the name. The advancement of knowledge is the building block of all higher education. Otherwise we could still be using textbooks written in 1900. People who do the research are generally the best and the brightest and they can bring that cutting edge outlook into any classroom.</p>

127

<p>barrons,
No one is claiming that research and the advancement of knowledge are not important. Obviously, they are. However, is this the sole (or even the main purpose) of a college? Also, how much do you think the average college applicant (especially with a non-scientific or non-technical interest) should care or be impacted by the research efforts of a school? Or why should a faculty that has expertise in non-research driven fields be considered less than a faculty that has research as its priority? Do you believe that they, as faculty members, are less worthy or talented or deserve less academic respect than their research-oriented colleagues? </p>

<p>As you know, there is also a tremendous amount of research that comes out of the corporate world, but we don't wholly judge those firms on the quality of their research. Research is just one function within a larger company and its impact fluctuates depending on the nature of the research, the business and the marketplace. Certainly, Pfizer's ability to regenerate its drug pipeline or Intel's ability to create new semiconductors is crucial, but so are many other aspects of those companies (finance, marketing, distribution, management, non-US issues, etc.). </p>

<p>What if your business is not technical, but rather something like real estate or trucking or insurance or apparel or any number of other businesses that don't have research as their raison d'etre. I would guess that far more college graduates go into these non-technical fields than not. Given this, why should the academic world have research as the primary driver when evaluations are done of faculty excellence?</p>

128

<p>In the academic world everyone does research. There are journals for finance, insurance, real estate, accounting and everything else from art history to zoology.<br>
Wharton gained much of its current fame due to the research and theories of their finance profs who basically changed the theory of pricing assets. My real estate dept gained a strong reputation because they invented a new approach to analyzing land and development based on urban land theory developed at UW.</p>

129

<p>I think the Peer Assessment has limitations, which have been well discussed here, but I don't think it's as bad as you make out. Your main complaints seem to be:</p>

<p>(1) That it's very mysterious
(2) That it doesn't measure what matters
(3) That it's weighted too heavily in USNews' calculations
(4) That (some) people buy into it too much </p>

<p>I agree with some of this, but I don't think PA (or, should I say, PA exclusively) should warrant this level of concern.</p>

<p>As for #1, several people have detailed the nature of the question, the respondents, the response scale, and the way USNews reports it. While we can't crack open the brains of people who answered, that's the nature of survey research. Even with a better-written question, we'd still have to live with not knowing what the respondents were thinking. </p>

<p>As for #2, it may not measure things we'd like to see measured, but it's pretty clear in what it does measure: peer reputation. Not student learning or satisfaction, not faculty quality, productivity, or pedagogy; not employer or corporate-world reputation, not research excellence, not parental opinions. It's what their "peers" (as defined by USNews) think in general about a college's academic programs. USNews isn't making any claims to the contrary. </p>

<p>As for #3). Maybe it is. One thing USNews does is provide this measure in its own column, divulging the value, enabling people to evaluate PA separately (or not at all) if they so wish. It's time-consuming, but one should only need to do this for the handful of schools one is seriously considering. </p>

<p>As for #4). True, some people put a lot of stock in this measure, and perhaps too much. From where I sit, there are lots of things to hate about how some people assess colleges. What can anyone do? I think it's important to remember that USNews is one tool among many. What's positive is that is isn't the last word on college choice. Many people (including institutions themselves, including those you've speculated are 'afraid' to share data with USNews, and those you claim are 'working to perpetuate the traditional ranking') devote considerable resources to make other college-choice information available and encourage students to use it. </p>

<p>USNews is what it is, it's useful to a point; some people will value it more than others. There's lots that you don't like about PA, but there exist people who have similar and valid concerns about some of the other measures USNews includes, or the weight USNews puts on them, or the way people fawn over them. I guess ultimately I don't wholly accept that the USNews ranking would be that much better if they got rid of PA (or changed it, or weighted it differently). There would remain a lot about USNews' ranking that troubles me. That probably fuels my antipathy in trying to come up with a weighting system or a measurement that would be "better." </p>

<p>And yes, in the interest of full disclosure, I do work at an institition which generally does pretty well in the rankings, and in fact benefits a good bit from the PA. When PA was the only measure, my institution ranked very highly. Yet I still don't care for the influence USNews seems to have on some people.</p>

130

<p>barrons
As a business person, I would much prefer my new employees receive excellent, personal instruction in the classroom from someone who can relate to them in a pragmatic fashion and help them apply their lessons in the real world. As a business person, what is written in an article in some obscure magazine or what awards a certain professor has won have relatively little value to me. I understand why it would be important to those in academia, but if I encounter an interviewee who is focused on such matters, that student rarely makes it to the next stage. A capitalistic economy requires this. And if I'm hiring a graduate from the history or zoology department to work in my hypothetical real estate firm, I'm not looking for what the student knows about history or zoology or how his/her professors won a research grant. I'm looking for a student who has outstanding critical thinking skills, an ability to initiate, create, monitor & complete a task, an ability to work effectively with and learn from others, etc. I don't believe that the traditional measures used to bolster one's reputation in academia do much to build these skills. Nor do I believe that most students, parents, alumni or other employers put a great deal of weight on these measures either. </p>

<p>hoedown,
I appreciate the effort(s) that you have made. You have written a fair description of my concerns about PA. I don't think we will ever get rid of the influence that USNWR has, so my approach is to work with it and control how it is interpreted and used. The PA, however, runs counter to this and severely obfuscates the final ranks with its subjective, mysterious score and 25% weighting. As a result, consumers will see the final ranks, do little further inquiry and accept the results without understanding them. This serves no one's interest save for perhaps the status quo institutions which historically have benefited most from the inclusion of PA.</p>

<p>Most importantly, hoedown, I believe your conclusion is not decisive enough. IMO, PAs do little to further illuminate the process of college search & selection. And if the intent of USNWR (and all of us) is to inform and assist in this process, then PA does more harm than good in bringing this about.</p>

131

<p>All those factors are essentially personal character issues and not really learned in any school except as part of doing the work. Most big RE firms don't hire many zoology grads. I have worked for three of the biggest in the commercial RE field. We hire people trained in the latest anaytical techniques with all the personal qualities you noted. Most students put considerable weight on the reputation of the school which is mostly built on the reputation of the faculty.</p>

132

<p>If we are to take USNWR at its word regarding its college rankings, let those words be,</p>

<p>"We say this is a great starting point, but we don’t pretend it’s anything more than that.” - Brian Kelly, U.S. News & World Report Executive Editor</p>

133

<p>Some of you question the validity of the PA citing lack of transparency or the credibility of the person responding to the survey. Before you dismiss PA altogether, it makes sense to look at the other USNWR rankings. The USNWR rankings for undergrad business/engineering and individual graduate programs are all based on PA. Here the credibility is less disputable as the surveys are sent to "department heads and directors of graduate studies at each program in each discipline", people who are supposed to be in-the-know of their own fields. However, if you compare the two sets of rankings, you will find that the undergrad rankings track very well with their respective grad rankings. Moreover, schools with more highly ranked departments generally score high in PA. That may explain why schools like WUSTL, Notre Dame and Tufts score relatively low in PA despite their high rankings.</p>

<p>You may disagree with the way USNWR weighs the PA factor, however, it is undeniable that PA reflects the opinions of people in academia regarding a school's quality in faculty, facilities and UG curriculum. And that ought to count for something.</p>

134

<p>Exactly. You might not like the methodology but the results track well with virtually all other similar studies. If you don't care about the faculty, fine--send the kids to the closest small school with those small classes and dedicated teachers. Then watch them try to get a job with a major company.</p>

135

<p>"Then watch them try to get a job with a major company."</p>

<p>Barrons,
Just wondering. If you were in a position with a major company, (assuming you aren't), and had the responsibility to hire people in your area, what weight would you give to graduates of schools ranked high on USNWR realizing as you do that some candidates come from "small school[s] with those small classes and dedicated teachers", that do not rank as high on USNRW's list?</p>

136

<p>barrons,
I disagree if you truly believe that critical thinking skills and project management and teamwork are personal character qualities that cannot be taught. These can (and IMO must) be taught (or further developed) at the collegiate level. Otherwise, you are saying that the intellectual die is cast before the student even gets to college and faculty has no role in their development. Have I misunderstood your comment?</p>

<p>As for RE and zoology, there aren't many zoology grads generally and probably few find their way to the interview rooms for real estate jobs. (An interesting question might be how many who decide to interview get the offer and why and is this in any way related to their school and the zoology faculty.) But that was not really the point. The point was that the major field of study of the large majority of college students is not the field that they have a career in. So, individualized knowledge in these fields of study (and the research experience, literary publications, etc of their professors) is not nearly as valuable as the practical, skill developmental training that they receive. </p>

<p>GoBlue,
I agree with your comment that the PA has value in academia and should count for something. It is this distinction (IN ACADEMIA) that is often lost in many discussions on CC and elsewhere. I also agree that it should count for something (although it REALLY needs to be defined better), just not 25% of a total score and particularly when no weight is given to the views of students, alumni and corporate recruiters. Given the current lack of other stakeholder input, I would prefer that the PA be a wholly separate list and rank. </p>

<p>As for PA in business school ranks, I'm afraid that it also suffers many of the same shortcomings that have already been noted here. Still, the tighter circle of commentators probably increases its accuracy as saying anything about reputation (again among academics) and I concur that this is a good thing. However, I don't fully understand your comments and point on the role of grad programs and their effect on PAs. Could you run that by me again?</p>

137

<p>
[quote]
While I appreciate that research work can have great value to certain areas of study and work, I question the relevance of much of it to most students... As for reputation, my impression is that much of this (at least among academics) is driven by factors that have little to do with the students in non-technical fields of study.

[/quote]

Perhaps you are not aware of how extensive undergrad research is nowadays. And why is it limited to technical fields? The University of Washington reports that:
"- Every year, 7000 undergraduates (25% of the student body) participate in research;
- More than 160 undergraduates currently participate in research with the Department of Physics (out of about 60 graduates each year);
- 100% of Materials Science & Engineering majors are involved in undergraduate research."</p>

<p>I am quoting UW as I have the data handy. I'd expect more at schools that encourage freshman research (e.g., MIT, Duke and Michigan).</p>

138

<p>
[quote]
However, I don't fully understand your comments and point on the role of grad programs and their effect on PAs. Could you run that by me again?

[/quote]

Someone questioned the credibility of the person responding to the PA survey. I mentioned the grad programs as there is no question (less anyway) that the department heads or director of graduate studies should be well informed of their peers in their own fields. And since the undergrad rankings tracks well with the grad's, it infers that the PA survey is credible.</p>

<p>And since the schools with more highly ranked grad departments generally score high in PA, one can infer that PA reflects the consensus opinion of people in academia about a school's quality of faculty, facilities and UG curriculum...the same standards the grad programs are evaluated.</p>

139

<p>GoBlue,
Maybe I'm missing something but the UW 2006-07 CDS data that I am looking at shows the following:</p>

<p>Engineering 9.11% (of students graduating with this degree)
Natural Resources/Envi Sci 1.43%
Biological/Life Sciences 8.84%
Physical Sciences 3.2%
Health professions & related sciences 3.7%</p>

<p>All together, these fields, which I would describe as oriented to technical skills and are most likely to engage in research activities, comprise 26.28% of the graduates which nearly matches the number that you gave. I don't think we are disagreeing here.</p>

140

<p>You are assuming ALL of the undergrad students in technical fields do research, which is certainly not the case. Research or independent study are generally not a requirement for graduation. It's more likely that less than half of the class will take the initiative to participate. The rest will have to come from the less technical majors. For example, there are plenty of research projects in business and social sciences, especially in psychology or related fields.</p>

<p>Here are some of the current undergrad research opportunities at UW:
<a href="http://webapps.ued.washington.edu/opportunities/Opportunities.aspx%5B/url%5D"&gt;http://webapps.ued.washington.edu/opportunities/Opportunities.aspx&lt;/a&gt;&lt;/p>