Benjamin Barton (University of Tennessee, Knoxville - College of Law) has posted Is There a Correlation Between Scholarly Productivity, Scholarly Influence and Teaching Effectiveness in American Law Schools? An Empirical Study on SSRN. Here is the abstract:
This empirical study attempts to answer an age-old debate in legal academia; whether scholarly productivity helps or hurts teaching. The study is of an unprecedented size and scope. It covers every tenured or tenure-track faculty member at 19 American law schools, a total of 623 professors. The study gathers four years of teaching evaluation data (calendar years 2000-03) and creates an index for teaching effectiveness. This index was then correlated against five different measures of research productivity. The first three measure each professor's productivity for the years 2000-03. These productivity measures include a raw count of publications and two weighted counts. The scholarly productivity measure weights scholarly books and top-20 or peer reviewed law review articles above casebooks, treatises or other publications. By comparison, the practice-oriented productivity measure weights casebooks, treatises and practitioner articles at the top of the scale. There are also two measures of scholarly influence. One is a lifetime citation count, and the other is a count of citations per year. These five measures of research productivity cover virtually any definition of research productivity. Combined with four years of teaching evaluation data the study provides a powerful measure of both sides of the teaching versus scholarship debate. The study correlates each of these five different research measures against the teaching evaluation index for all 623 professors, and each individual law school. The results are counter-intuitive: there is no correlation between teaching effectiveness and any of the five measures of research productivity. Given the breadth of the study, this finding is quite robust. The study should prove invaluable to anyone interested in the priorities of American law schools, and anyone interested in the interaction between scholarship and teaching in higher education.
And here is a bit more from the paper:
The teaching evaluation data came in different forms for different institutions, from access to a university website that gathered the data, to a single page amalgamation, to physical copies of every student evaluation during the period. From these data I chose the question on the evaluation sheet that most closely measured teaching effectiveness. For example, the University of Tennessee form actually asks the students to rank the professor from 1-5 (with 5 being the highest ranking) on the “Instructor's effectiveness in teaching material.” The results can be found on a publicly accessible website (University of Tennessee 2006). Of the 19 schools, 13 schools asked a somewhat similar question and ranked the professor from 1-5. Two of the other schools ranked from 5-1 (with 1 being the best ranking), one ranked from 4-1 (again with 1 as the best), and one each ranked from 1-4, 1-7, and 1-9, with 1 being the lowest.
One more point--the study examines the correlation between global teaching effectiveness (across courses) and global scholarly productivity (across fields) and did not attempt to study correlations between writing that is salient to the course for which teaching effectiveness is being measure.
And one more issue--what about peer versus student evaluations. Again, a bit more from the paper:
I also am aware that the use of teaching evaluations as a proxy for teaching effectiveness is somewhat controversial. There are studies, both within law schools and higher education in general, that show that teaching evaluations have biases, including biases based on race (Smith 1999), gender (Farley 1996), and even physical attractiveness (O’Reilly 1987). Other studies have shown that student teaching evaluations are positively correlated with other measures of teaching effectiveness, including peer reviews and output studies, suggesting at least that student measures track other alternative measures (Bok 2004). Many law faculty members have nevertheless argued to me that teaching evaluations are little more than a popularity contest. Some have even argued that teaching effectiveness is inversely correlated with teaching evaluations, since students tend to highly rank easy professors of little substance, while ranking professors who challenge them comparatively lower. For better or worse, I believe teacher evaluations are the only viable way to measure teaching effectiveness for a study of this breadth. My other choices were exceedingly unpalatable: 1) attempting to gather peer evaluation data, which is rarely if ever expressed numerically, and would also almost certainly not be provided by the host institutions; 2) some type of personal subjective measure of teaching effectiveness, potentially requiring me to personally visit classes and make my own call on teaching effectiveness.
At one level these results are completely unsurprising. What mechanism would result in a correlation between research productivity and teaching effectiveness? Here are some possibilities: --More research and more effective teaching might both be products of some underlying trait--such as diligence. --More research might result in more knowledge, which might result in more effective teaching. --More research might result in more knowledge, which might result in less effective teaching. --More research might divert effort from teaching, which might result in less effective teaching. And so forth. It is possible, however, that some of these effects might be observed with a different research design. If it were possible to do reliable assessments of the objective accuracy of information conveyed and to compare that to research productivity in the particular field, for example, there might be some correlation between productivity and teaching effectivenss (in the objective sense). But that would not necessarily correlate with student ratings of teaching effectiveness? Why not? Because generally law students are incapable of evaluating "knowledge of the subject matter." For one thing, they lack a good baseline for comparison, because the truth is that the general level of knowledge of subject-mater among legal academics is fairly shallow. And a student rarely learns enough about a subject to actually get ahead of the professor. Of course, we all know that occasionally newbie professors get caught in gaffs--but most experienced teachers learn how to avoid this--which is mostly a matter of not saying things you don't know, not mastering the subject so deeply that you can answer any question about any point accurately. But with that caveat aside, this is clearly valuable research! Highly recommended for all legal academics!
