<p>Many posters have presented passionate arguments both for and against the Peer Assessment scoring now done by USNWR. Part of the ensuing discussion has been what an undergraduate student actually experiences in the classroom and whether this should be considered when a school's PA score is created. What are the responsibilities of the faculty to the undergraduate student and should their efforts in the classroom, both good and bad, be considered in the assignment of PA scores? Who should judge the quality of classroom instruction by faculty members? Finally, should classroom instructional quality have ranking consequences?</p>
<p>The problem (as I see it) is the "peer" in "peer assessment." I don't know how you could expect institutions to judge each other on specific behaviors that happen inside the classroom or on "quality" of instruction during the course meeting times.</p>
<p>You could find a way to measure some of what goes on in the classroom--perhaps using NSSE data--but this would not be a "peer" assessment. Which suggests this would be a new element, not an addition to PA.</p>
<p>If quality of instruction (both in terms of ground-breaking content and method of communicating it) could be measured, it would be of great importance and should definitely have ranking consequences.</p>
<p>But it is impossible to measure quality of instruction. Students have no basis for comparson nor a common frame of reference to adequately rate quality of instruction. A neutral source that is qualified to rate instruction would have to be responsible for any such rating, but I don't think such an organization can possibly sit in on thousands of classes at hundreds of universities and come up with an accurate rating of instruction.</p>
<p>The quality of instruction is the most important thing about a school. Everything else is ancillary. Quality of instruction is mainly the quality of lectures but not only that. Quality of textbook choice, evaluation methods, classroom discussion, availablity outside class, timeliness and quality of feedback on performance, choice of problems and papers, level of expectation...these all matter too. You would have to get inside student's heads to find out what is going on.</p>
<p>Course evaluations capture some of it but demanding professors get lower ratings to some extent. And, that information is not shared publicly.</p>
<p>I think the quality of the students conributes to quality of instruction. It enables and encourages better instruction. So maybe selectivity is indirectly an indication of good instruction.</p>
<p>Retention and graduation rates may be indirect indicators of good instruction, especially over- and under-performance.</p>
<p>But, my main point is that the quality of instruction is the most important feature of a school and parents and students and accrediting agencies and the US Department of Education should be more aggressive about demanding excellence in this area.</p>
<p>hoedown,
You make a good point as a "peer assessment" involving classroom work would be next to impossible. But I'm trying to understand if there is a consensus about the importance of classroom instruction and how this should filter through to the reputation of the faculty. Can a faculty be considered weak if it is exceptional in the classroom but poor in research work or can they be considered great if they are outstanding in research, but not necessarily in the classroom?</p>
<p>hawkette:</p>
<p>Who's buying? If you're an undergrad and you want the best possible learning facilitation, then research is just what gets in the way of having your teachers focus on instruction. </p>
<p>If you're government or NGO, you have a vested interest in funding productive research.</p>
<p>So, I think the answer to your question is, "Who's buying?"</p>
<p>Tarhunt,
Good point and distinction.</p>
<p>I am asking from the standpoint of the undergrad student. I am also asking if a school can be considered great (poor) if its research is excellent (weak) and its classroom instruction lacking (very good). </p>
<p>As I see it right now, the USNWR survey and its PA score is asking from the standpoint of the institution and its (likely highly graduate school influenced) research reputation, but packaging and selling it to the public as if it is asking from the standpoint of the student.</p>
<p>
[quote]
Can a faculty be considered weak if it is exceptional in the classroom but poor in research work or can they be considered great if they are outstanding in research, but not necessarily in the classroom?
[/quote]
</p>
<p>I think you can't generalise about the entire faculty body at a research university in this manner. Some faculty excel at both, some don't, and I think there will be a mix on campus. If for some reason a school has an unusual number of poor teachers, then I think students do really suffer, just as they benefit from a better-than-average number of stellar teachers. But the statements you are suggesting we might make about entire universities? They just don't seem realistic.</p>
<p>It's just doggedly hard to measure how hundreds of faculty on a campus collectively teach.</p>
<p>Maybe a better improvement would be to make it easier for prospective students to find out from current students what their classroom experiences are. Let students narrow down schools in their usual way, but then make it possible for them to find out this information for their top choices, from students who are taking classes (especially in their chosen field).</p>
<p>hoedown,
Your comments are perfectly reasonable but part of this harkens back to the PA discussions where opinions and scores are given there about whole universities. Undoubtedly, universities also have some professors who are very good at research and others who are not, yet PA scores are given for an entire university. If it is ok for PA, and if you think classroom teaching is important, then why not for instructional quality? </p>
<p>But, really, the fundamental question is should a school's faculty be considered great if its classroom instruction is not?</p>
<p>
[quote]
If it is ok for PA, and if you think classroom teaching is important, then why not for instructional quality?
[/quote]
</p>
<p>PA is a general measure of how an entire institution is regarded by its peers. It's about a general reputation. It's no secret that by capturing how university leaders think about an institution generally, PA is going to miss numerous nuances, including quality that varies from department to department, pedagogy that is worse with some faculty than others, research which is superior in some departments, etc. </p>
<p>Any student that is impressed by a high PA score should also know that it's measuring something general. It tells you something, but that something is not enough. They need to do additional research to get information about specific experiences they may have at the university, whether it be in the classroom, with the aid office, in the residence halls, or wherever.</p>
<p>There are lots of things I think are important about an institution. But I do not expect PA to capture them. I know what PA measures, and I know its limitations. The same could be said for the other measures USNews chooses to put into its formula. PA is a little more elusive because it's conveying opinion, but there is no reason to demand that it accurately capture every facet of the undergraduate experience--particularly when that would be a wholly impossible task.</p>
<p>
[quote]
As I see it right now, the USNWR survey and its PA score is asking from the standpoint of the institution and its (likely highly graduate school influenced) research reputation, but packaging and selling it to the public as if it is asking from the standpoint of the student.
[/quote]
</p>
<p>And I agree that the PA ranking runs suspiciously closely to research/publication reputation for the faculty. This is why the PAs for large, state schools tend to be higher than the other stats would suggest.</p>
<p>Here's another way to look at it, though. All things being equal, I believe that it's better to have a top researcher teaching your class than a non-researcher. This ASSUMES, of course, that learning facilitation skills are the same for both the researchers and non-researchers. Personally, I don't make that assumption because I believe that research and grant-writing respsonsibilities generally interfere with the ability to be a good teacher.</p>
<p>BUT, I can't prove it.</p>
<p>Adding to this, I think that small classrooms tend to mitigate, to the point of elimination, the differences between those who are primarily researchers and those who are primarily teachers. This does NOT mean that one can't find a terrible teacher regardless of classroom size. It just means that, in my opinion and experience, pedagogical weaknesses tend to work themselves out in a discussion atmosphere.</p>
<p>Look, it's not a secret that "National Universities" have other priorities besides undergraduate instruction. Their PA scores reflect that fact. </p>
<p>If you think that PA scores should primarily reflect the quality of undergraduate instruction, then you are looking in the wrong place. Try the US News "National Liberal Arts Colleges" rankings, instead of the "National Universities" rankings.</p>
<p>Notice that for 2008, both Cornell and Swarthmore have identically high (4.6) USN&WR PA scores, despite the fact that they are obviously very different institutions. Obviously the PA score is measuring different things here. In the case of Swarthmore, the PA score presumably does reflect the quality of undergraduate instruction (it's certainly not based on Swarthmore's research or professional schools). </p>
<p>Personally, I rather doubt that quality of undergraduate instruction is in fact the #1 student priority. If it was, then there would be overwhelming demand for LACs, where undergraduate instruction is the #1 institutional priority. Yet in fact, LACs only educate a minority of even the best qualified students. </p>
<p>My impression (after having spent too much time reading threads on College Confidential) is that most posters are more concerned about issues like name recognition and prestige, and less concerned about issues like class size and TAs.</p>
<p>^^^I agree. the PA score is a measure of prestige which is usually tied with research capabilities. A professor (and a university) becomes famous through advancing science not because he's willing to hold an extra office hour each week for his students. In no way should the PA score reflect the quality of undergraduate teaching. If you wish to incorporate that in some other manner, fine.</p>
<p>When someone comes up with a reliable way to measure quality of teaching, then it should not simply be included in endeavors like the US News ranking - it should make up a majority of it. The problem is that there is no method for reliably evaluating the quality of teaching. In fact, many people disagree on what makes a good teacher.
hawkette's two favorite answers - employer and student evaluations - simply don't work. Students at different schools don't judge teaching on the same scale or according to the same standards. The course evaluations that students fill out for university purposes certainly wouldn't work, as students evaluate classes in terms of how they compare to other classes in the university. Theoretically, the average class should be rated as "average" at any university (in current student evaluations), and any deviations from that reflect differing tendencies of students when evaluating. On the other hand, asking students to evaluate courses based on some grand, universal standard of excellence is ridiculous. No two university student bodies would do this in the same way and the results would be arbitrary.
As for employer evaluations, this is an even more ridiculous idea. Employers are interested in the quality of their future employees. No matter how bad the teaching at Harvard, its graduates will still be better prepared to work in most fields than will the grads of Southwestern Podunk State University. And even when the differences in student body quality aren't that extreme, they can mask a lot of difference in quality of teaching. Measuring teaching quality by solely looking at the results makes for a very poor metric.</p>
<p>So, yes, teaching quality is more important to the prospective undergrad than research quality, and a school could, supposedly, be a terrific research university and a terrible school for undergrads (though it would probably have to go about hiring faculty with the intent of getting bad teachers for this to actually occur). But there is no good way to determine this, not PA, not student surveys, not employer rankings, not US News. hawkette likes to treat the failure of US News to include teaching quality as an easily remedied omission that is only left out because some cabal of professors wants to ensure that it is never expected to devote any effort to educating undergrads (I know this is a bit of an unfair exaggeration, but hawkette comes pretty close to this position in some posts). This is simply not the case. If US News had remotely decent some way to consider the strength of teaching at the schools it evaluates, it would probably do so.</p>
<p>There's a theory that the best single indicator of undergraduate teaching quality in the US News rankings is: alumni giving rate. </p>
<p>According to this theory, the schools that "connect" most effectively to undergraduates in the classroom are more likely to be supported by their alumni after graduation. Schools that don't particularly care about undergraduate teaching are less likely to get alumni support.</p>
<p>In other words, if your school cares about you as a student, you will be more likely to care about your school later on. </p>
<p>It may sound simplistic, but the idea is not crazy. In practice, the schools with the highest giving rates (many top LACs, certain universities like Princeton and Dartmouth) do tend to be schools that are also known for their emphasis on undergraduate education. </p>
<p>Ironically, hawkette has completely [url=<a href="http://talk.collegeconfidential.com/showthread.php?t=383429%5Ddismissed%5B/url">http://talk.collegeconfidential.com/showthread.php?t=383429]dismissed[/url</a>] the use of alumni giving rate in the US News rankings:
[quote]
Here is my personal view: ... 7. Alumni Giving: This has got to go.
[/quote]
</p>
<p>Corbett,
You highlight one of the flaws with PA-its seeming derivation from a school’s graduate school activity. IMO that has no place in an undergraduate ranking and thus my frequent objections to its heavy weighting, if not its use at all in the USNWR rankings. Xiggi probably has the best solution of breaking out the PA and giving a separate rank based on PA so that all those who love this metric will still be able to reference it while those of us who prefer the objective, quantifiable data will then get to fight over the weighting of those factors. :)</p>
<p>But this topic is about classroom instruction which I think, along with the quality of your fellow students, is perhaps the most important aspect of the undergraduate academic experience. I understand the objections of svalbardlutefisk and others to student and employer surveys as part of a ranking system (I have elsewhere proposed Faculty Assessment be weighted at 15% of a total score with 6% from academics, and 3% each from students, alumni, and employers). My objective is to get rankings that are more informative to students for the things that IMO they most care about, eg, classroom instruction, postgraduate opportunities, etc. rather than the research accomplishments of some graduate school professors. So I am trying to understand if others value classroom instruction as much as I and if people think that a faculty can be great for undergraduates even if they don’t teach or do so only in large classes or aren’t very good or interested in teaching. Or if a school with low PA prestige but a strong teaching reputation (eg, Wake Forest) should be recognized for this good work. I’m also looking for anyone with any decent ideas on how to measure this and incorporate it into a rankings system.</p>
<p>I was surprised by your comment about Alumni Giving. Certainly a novel idea. Please know that I actually think that alumni giving is a good thing to measure and track and a pretty good indicator of the bonds that a school creates with its graduates, eg, Princeton, Dartmouth, Notre Dame, etc. However, I think its inclusion in the USNWR results unfairly penalizes public universities and that is why I think it should go.</p>
<p>Look hawkette, I think MOST people would agree that quality of teaching would be very important to most undergrads. The issue is finding a decent measurement, and there aren't any universal ones. Doubtless, some schools have mentoring programs, feedback, and the like for faculty. You'd have a riot if you tried that at my university, but I would support it.</p>
<p>But that's a sidetrack. There is no universally acceptable measure of this factor as yet. There is a survey of student engagement used by some schools, but those schools rarely share the data. And I believe the instrument is deeply flawed. Student evaluation? At my school, student evaluation on all factors is highly correlated to the grades given out. Employer evaluation? Please. GIGO/DIDO. </p>
<p>We're pretty much stuck with the PA as an imperfect measure.</p>
<p>
[quote]
But this topic is about classroom instruction which I think, along with the quality of your fellow students, is perhaps the most important aspect of the undergraduate academic experience...So I am trying to understand if others value classroom instruction as much as I
[/quote]
Here's your answer: No. For most people, classroom instruction is not the top priority. </p>
<p>As I indicated before, the proof is simple. If undergraduate instruction was the #1 applicant priority, then everyone would want to go to LACs, where undergraduate instruction is the #1 institutional priority. Some people do value classroom instruction (I do), and do attend LACs (I did), but they are in the minority. </p>
<p>For national universities, PA measures things like prestige and Nobel Prizes and sports championships and name recognition, as well as classroom experience. But most people care about these other things just as much (or more) as they do about the classroom. That's why they go to national universities in the first place. So for most people, a PA that measures all of these things makes sense.</p>
<p>
[quote]
I understand the objections of svalbardlutefisk and others to student and employer surveys as part of a ranking system
[/quote]
Note that the "alumni giving rate" is, in effect, the result of an extremely comprehensive survey conducted every year, by every college, of all its alumni. The survey basically asks the following question: "Do you care enough about the college's mission to support it financially?" </p>
<p>Notice that it costs $$ to answer this question affirmatively. So the results are also quite reliable: you can safely assume that people won't answer "yes", unless they really mean it.</p>
<p>corbett, alumni giving has a lot of flaws as a measure of teaching quality/undergrad focus. The most obvious is that it is highly dependent on things entirely unrelated to the quality of education - how strong the alumni network is, how good a given university is at getting money out of people (things like whether it put out 15-17 alumni magazine issues a year like Princeton, or only 6, like Stanford), how much money alumni earn (yes, it only measures total number of people who donate, but alumni who aren't doing well financially are probably still less likely to give), loyalty developed through sports or religious affiliation (why do you think Notre Dame does so well?), how many alums are ****ed off because their kids just got rejected, etc. It's hard to call it simply (or even mostly) a measure of undergrad focus/teaching quality (or even overall college experience) when so much goes into it.</p>
<p>The second major problem is that, to the extent that it does measure quality of education, it is measuring quality of education 50 years ago, just as much as quality of education today. That is, all alums count in the giving rate, whether they graduated in 2007 or 1937. To me, the opinions of alums from more than 15 or 20 years ago (making up a very solid majority of total alums at most institutions) is not really that relevant. </p>
<p>
[quote]
As I indicated before, the proof is simple. If undergraduate instruction was the #1 applicant priority, then everyone would want to go to LACs, where undergraduate instruction is the #1 institutional priority. Some people do value classroom instruction (I do), and do attend LACs (I did), but they are in the minority.
[/quote]
</p>
<p>I hardly think that failure to choose to attend a LAC shows that one's first priority is not undergraduate instruction. I chose to attend a research university rather than a LAC in the belief (which I still hold) that the quality of undergraduate education at my institution is as good as or better than that at any LAC. What it may lose by having faculty who are more involved in research, it makes up for by having a wide array of resources that few if any LACs could equal. Many prefer the environment at LACs, and I do not consider them objectively inferior to research Us (you can look through my post history if you want to see me defending the quality of education offered at LACs), and many who chose research Us did so for reasons other than educational quality (prestige, cost, sports, etc). But to imply that LACs are somehow universally better at undergraduate education because they don't do anything else, or that one who chooses a National University is not making classroom instruction his or her first priority is hardly a fair claim.</p>
<p>
[quote]
But to imply that LACs are somehow universally better at undergraduate education because they don't do anything else, or that one who chooses a National University is not making classroom instruction his or her first priority is hardly a fair claim.
[/quote]
Look, nothing is universal. My point is simply that LACs are only a niche market; the collective enrollments of National LACs are obviously dwarfed by the collective enrollments of National Universities. If college applicants are really looking for schools that value and emphasize undergraduate teaching, then why are LACs -- which provide, as you note, precisely that -- such a small niche?</p>