Peer Assessment-Is this a useful tool in the College Selection process?

<p>We had to do original research projects in our real estate program. We did real field surveys (in the sometimes cold Wisconsin winter) and market analysis using all sorts of primary and secondary sorces. We also did some advanced stats analysis on block level census data in order to study the impact of prisons on property values. Many of our projects were real world consulting assignments.</p>

<p>Research numbers vary greatly from school to school. Many, many schools (like Dartmouth) are not focused on research and I would not automatically conclude from that that their faculty are inferior (you may be aware that Dartmouth, with Brown, has the lowest PA among the Ivies). Furthermore, we do know that a large portion of the research work is happening in the fields that I referenced. The research work is most commonly found at the state universities and this group of schools benefits most from the ongoing inclusion of PA scores. And even accepting in your UW example that half of the research is done in non-technical fields, this still only involves 12.5% of the student body (and it looked like all of your examples came from the psychology department). </p>

<p>Your efforts are appreciated, but it does not alter the problems with PA-lack of definition on what it measures, that it gives no guidance or standard on which to make measurements, that there is no assurance of a knowledge of the school being judged, that it ignores the interests of important stakeholders, etc. It is just a very, very flawed number. Not completely useless as it likely has value to academics, but I cannot endorse its use by the average student, his/her parents, alumni, or recruiters and I certainly object to its gross 25% weighting in USNWR.</p>

<p>The state universities benefit from the PA because they hire and develop many of the top people in their fields. At least until some of the Ivy schools or Stanford come calling and lure them away with more money and less teaching load. You can prefer the no names all you wish but most of the population would prefer getting to work with the best people in academe. They seem to be clustered at the schools with high PA scores.</p>

<p>barrons,
I think you may be speaking from the academic perspective and I respect that. However, the next non-technical business person that I meet who tells me that the PA is relevant to his/her real world will be the first. As an employer, I am much, much more interested in the student and how they think and apply their thinking than I am in whether young Suzy took classes with Professor Smith at State U. Frankly, the PA of the student's school and his/her professors mean nothing once the interview begins (and very rarely even before that). A high PA does not get her the job and definitely does not pay off her student loans.</p>

<p>While employment is important it is not really the major goal of college education on the whole. Getting the interview is the biggest hurdle and the top firms generally recruit at the schools with the highest PA's--coincidence?? I think not.
And when you have a Prof that is well known in his field such as RE his word will get you that interview--then you are on your own. I have seen it happen many times.</p>

<p>
[quote]
lack of definition on what it measures, that it gives no guidance or standard on which to make measurements.... It is just a very, very flawed number.

[/quote]
</p>

<p>I feel like I'm repeating myself here, but some of your statements seem like we're talking about two different things. </p>

<p>It measures what provosts, presidents, and admissions directors think of the schools' academic programs. It is based on their personal knowledge of the school and/or its reputation.</p>

<p>Maybe I'd better understand your deeply-held concern over the harm of PA if you better explained the harm you see done. Maybe some examples? For which institutions do you think the peer measurement is incorrect? What's your assessment of those instituitions' reputation among their peers? How much is it off by, in your estimation? What is the impact on their ultimate rating? Who do you think is harmed the most by it? The enrolled students? The students who didn't enroll? Other schools who look better or worse by comparison?</p>

<p>
[quote]
As for PA in business school ranks, I'm afraid that it also suffers many of the same shortcomings that have already been noted here.

[/quote]

Well, for grad business/engineering rankings, the recruiter assessment scores are in general agreement with peer assessment. And since the undergrad rankings track very well the grad's, it confirms its validity as well. At least the academia and the recruiters are in agreement here.</p>

<p>Barrons - “While employment is important it is not really the major goal of college education on the whole.” </p>

<p>More than anything else, this comment makes clear how differently you and I see college education. If you think that getting a job is not, by far, the single biggest reason that kids go to college, I think we can stop right now. We are never going to agree.</p>

<p>Hoedown, we’re both repeating ourselves, but I’ll try again. Provosts, Presidents & Deans of Admissions, like all of us, have different values and priorities. They differ from one another and also differ from others in similar positions at other institutions. As my exchange with barrons so clearly portrays, what people value about a college education varies and sometimes quite considerably. Given this variety, the lack of more guidance from USNWR to PP & DoAs on what they are to measure only makes the interpretation of the result that much more difficult. Thus, in their comparisons, PP & DoAs aren’t comparing apples to apples or apples to oranges, they are comparing apples to oranges to grapefruits to tangelos to kiwi to…..These are all fruits, but does the Provost favor one type of fruit (research) while the President favors another type (classroom skills) to the Dean of Admission favors a third type of fruit (preparation for grad school) and they score highly those fruits that reflect the style that they favor (and that assumes that they have that depth of knowledge about more than a few schools, which I doubt). Maybe not the greatest analogy, but a strain of truth comes through. </p>

<p>I object not to the scores themselves nor even to the scores of certain institutions (although I think the obvious public school tip only further delegitimatizes the number). Beyond the lack of standards and the subjectivity, an additional complaint is that, if a subjective element must be included, it should include the views of all stakeholders and not just the irregular views of academics. If the input of other stakeholders cannot be included, then separate out the view of the academics, do a distinct listing for those who care about this and leave the objective data out of it. We would all be much better off without the mixing of the objective and the subjective.</p>

<p>GoBlue,
Still not sure I understand or follow the connection you are making. Grad school recruiting and undergrad recruiting are very different things with very different expectations on the part of the recruiters and very different skill levels of the students. I would be very hesitant to connect grad recruiters’ assessments to undergrad departments.</p>

<p>As I said, even if you believe college is all about the job prospects the best jobs go to the grads of high PA schools. I majored in a very career oriented track and it worked out fine with a good job waiting at graduation through the school placement office with the head of the hiring office a fellow alum. Not much interview stress that day. He just tried to sell me on the company.</p>

<p>First, what is a "best" job to you is quite likely not a "best" job to someone else. American business is replete with stories of students from low prestige schools who achieved great things. Frankly, my experience has been that the ones from the prestige schools make for good lawyers or good doctors or good investment bankers because those are safe fields, but give them a situation with risk, guts and a need for innovative thinking and you're asking for trouble. </p>

<p>Second, did you major in a technical or non-technical field? Glad it worked out for you in your interview. Maybe you're just a stud. :)</p>

<p>For business and engineering:
Grad recuiter assessments agree with grad PA -> validating grad PA
Undergrad PAs agree with grad PA -> validating academia consensus on school's quality in faculty, facilities and UG curriculum.</p>

<p>If you know anything about the engineering industry, you should know that the highly ranked engineering programs are also the favorites of national engineering recruiters. You can say the same about the top ranked undergrad business schools.</p>

<p>In your opinion, which of the top business/engineering programs do not deserve the high ranking?</p>

<p>Anyone that thinks IB or the professions are safe fields is just talking though their hat. Most require long hours and ultra competition to land and keep big deals/accts. Now my school which is not Top 25 but in the next group is #1 for producing CEOs ahead of Harvard and well ahead of the next public. It also does very well in Peace Corps volunteers and Teach for America as well as Pultizer prizes and even a good number of Nobels. Maybe it's the best of both worlds.</p>

<p>barrons (aka Interview Stud),
I didn't say that they don't work long hours. God knows they do, but they are not being trained as risk takers. In fact, just the opposite and I am extremely sure that I am not talking thru my or anybody's hat. </p>

<p>As for UW, I think it and others of its ilk might be the best of both worlds. Smart kids who are motivated and don't have a huge sense of entitlement. That has huge appeal for an employer. It has even greater appeal inside the workplace as these students typically have humility and know how to work with others (character traits and skills that are sorely lacking at many of the elite schools and certain professions like I-banking).</p>

<p>
[quote]
We would all be much better off without the mixing of the objective and the subjective.

[/quote]
</p>

<p>I think I'd better understand this argument if you suggested that USNews and World Report listed everything separately and didn't use any kind of formula for ranking. It would seem more consistent to me if you were arguing that U.S. News just put the data out there for people to look at, so they can weighing elements according to how important they are for them personally. </p>

<p>You herald these "objective" measures but that seems to ignore the fact that USNews is still putting a "subjective" element in there, by deciding what to include and exclude, by deciding how much weight to place on certain elements, to decide whether to go with a raw value or some recalculation of a variable. Simply by taking the same "subjective" measures, but calculating things differently, you can see an institution move almost 10 places up or down in the ranking. CalTech, anyone?</p>

<p>I am beginning to think the PA discussion looks a lot like stock market analysis. Would you rather have an expert's opinion on a stock, S&P or Goldman Sachs, in terms of a buy/hold/sell recommendation or would you rather look at the numbers for yourself? While I don't have the expertise of the so-called experts, I'm somewhat suspicious of their motives. While I don't have to act upon all of their recommendations, I do have the ability to check out the recommendations that I think may have merit. Ostensibly, that is all USNWR is asking its readers to do. "Here's our rankings; check it out for yourself." It's easy to use expert opinion when your money is not on the line. But my money is on the line. Barrons is bringing up the other side of the money equation: getting a good job. Big companies tend to be risk averse in their hiring and will stick with reputation. USNWR makes it more convenient for them, even though it is contrary to USNWR's stated purpose.</p>

<p>hoedown,
I read somewhere recently that Business Week is responding to your thought. Present the objective data and then provide a template that the user (high school student or parent, alumni, corporate recruiter, academic) could weight the variables as he/she saw fit and create personalized rankings based on what is important to the user. I think that this is a terrific idea and I hope that it comes to pass with the data that USNWR provides.</p>

<p>standrews,
Your stock analogy does not work for me. All of the stock observers have the same objective-they want to make money. It's not clear what the academics want or value. </p>

<p>Many stocks may be grossly overvalued or undervalued in the market and analysts (Wall Street or otherwise and frankly, given Wall Street's record over the last decade, I'd suggest going to some other advisor) will adjust their ranking based on the market's price. Analysts may have some similar opinions on the underlying quality of the corporate franchise that is being evaluated, but this is not necessarily (and usually is not) accurately reflected in the stock price. These different judgments are what makes a market. </p>

<p>Another difference is that stock prices reflect future expectations. By contrast, I would say that college PA rankings reflect the historical views of some academics about institutions, many of which they may have very limited knowledge of.</p>

<p>hawkette,</p>

<p>Yes the analogy breaks down quickly. My comparison is with respect to trusting the opinions of experts, whether as a basis for investment or finding the best fit for attending a college or university.</p>

<p>Interesting how some people focus on PA when comparing Michigan to "higher ranked" schools (by this, I mean the USNWR) and then downplay PA when UoM is compared to Cal.</p>

<p>As has been noted elsewhere, several schools are talking about a boycott of the USNWR survey and a draft letter has been circulated. One comment in particular caught my eye and has direct application to this thread:</p>

<p>"Several college presidents suggested that they personally could evaluate only five to 10 schools-a far cry from the hundreds on the list. "We know each other through reputation, but that's different than having the kind of intimate knowledge you should have when you are making a ranking," says Robert Weisbuch, president of Drew University in Madison, N.J., who plans to sign the letter."</p>

<p>Peer Assessment, at least as currently constructed, is a badly flawed number and even the participants in the survey don't trust it. Why should anyone else?</p>

<p>I wouldnt trust the PA # for my kids going to college. It is too subjective and is focused more on research and "opinion" that anything. I have a bias toward teaching during the undergrad years rather than the strngth of the grad school.</p>

<p>I think the following have a much stronger student body than their PA scores suggest:
Notre Dame 3.9
Tufts 3.7
Wake Forest 3.5
W&M 3.8
Georgetown 4.1
Vanderbilt 4.1
Rice 4.1
Wash U 4.1
Emory 4.0</p>

<p>Compared to overated schools:</p>

<p>Cal 4.7
Michigan 4.5
UCLA 4.3
UNC 4.2
Texas 4.1
Ga Tech 4</p>

<p>IMO this overwieght towards research schools is exactly what is wrong the US News rankings. I would like to see someone compare grad rates and SAT scores vs PA. That would probably be very illuminating.</p>