Ted Seto, a tax law professor at Loyola University New Orleans, and a fellow contributor on MoneyLaw recently published Understanding the US News Law School Rankings ,60 SMU L. Rev. 493 (2007). Seto criticizes the USNews law school rankings, among other reasons, because they undervalue or do not take into account at all some attributes law students deem most important in evaluating law school performance. The November 2007 issue of National Jurist revealed the results of its survey of law students on what matters most to them. Over half the student respondents, 66%, said that quality teaching was the most important attribute in law school quality, outweighing all twelve of the factors the USNews rankings track. Indeed, USNews does not collect data on teaching quality and does not factor it in the rankings. Runners up according to students: bar passage rates, job placement rates nine months after graduation, and practical skills training (also not a factor in the USNews rankings).
It's no surprise that consumers of legal education value most highly the product they consume -- legal education. A quick look at the USNews graduate schools ranking website is all it takes to see that USNews produces the rankings to help students comparison shop for legal education. Why isn't USNews responding to market demand and accounting for what students value most?
USNews doesn't consider qualilty of law school teaching because there is no accepted metric for it. What makes a law teacher great? Teaching and learning are intensely personal and idiosyncratic activities. Students in the same class perceive the professor differently, sometimes radically so. Moreover, evaluating relative quality of teaching takes perspective and comparative experience that law students lack. My story, I suspect, is typical. The quality of my law teachers appeared most clearly in the rear view mirror after I graduated and had occasion to hear their words in my head: "Consider the impact of procedural posture." Or, "You'll just have to come up with something better than that." We could correct somewhat for the temporal problem of judging teaching quality by asking alumni for their retrospective assessments. But, we return to the fundamental and inescapable truth. Teaching quality is in the eye of the beholder and thus controversial.
We law professors know that student evaluations are fickle measures of teacher effectiveness. We review them for "warning signs" not so much of quality but of social deviance: drunkeness, distracting bigotry, sexual predation, and the like. We tend to discount glowing recommendations by attributing some of the glow to a teacher's decision to lower the performance bar for students in order to raise their short term satisfaction. We are uncomfortable passing on the quality of our colleagues' teaching. This is puzzling. We justify our collective abdication of a meaningful quality control role as a noble exercise of respect for academic freedom. While academic freedom surely protects external intrusion into the substance of what we profess in the classroom, I see a kind of double standard. We tend to be far less reticent in passing on the "quality" of a colleague's scholarship (although we are much happier to simply count the number of articles than actually read them). This is so, even though most of the time our evaluation of scholarship is more about presentation than substance, and presentation is a written manifestation of communication skills rendered live in the classroom.
Given the difficulty, expense and controversy inherent in evaluating quality in teaching, it is no surprise both that law schools don't invest much in evaluating it, and USNews doesn't try to account for it directly in the rankings. Ted Seto's observations about the rankings are powerful. I can't help but try to defend the market though. A law student buys a portfolio of law teachers. In any year, at any school, some teachers will pay off and others will not. A quality teaching faculty, like a quality mutual fund, is one that outperforms its competitors over time as a group. Some of the factors USNews does measure may be the best we can do under the circumstances to find quantifiable surrogates for the overall performance of a law school's teacher portfolio.