qualitative approaches to empirical legal research

• Advanced Search
• All new items
• Journal articles
• Manuscripts
• All Categories
• Metaphysics and Epistemology
• Epistemology
• Metaphilosophy
• Metaphysics
• Philosophy of Action
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Religion
• Value Theory
• Applied Ethics
• Meta-Ethics
• Normative Ethics
• Philosophy of Gender, Race, and Sexuality
• Philosophy of Law
• Social and Political Philosophy
• Value Theory, Miscellaneous
• Science, Logic, and Mathematics
• Logic and Philosophy of Logic
• Philosophy of Biology
• Philosophy of Cognitive Science
• Philosophy of Computing and Information
• Philosophy of Mathematics
• Philosophy of Physical Science
• Philosophy of Social Science
• Philosophy of Probability
• General Philosophy of Science
• Philosophy of Science, Misc
• History of Western Philosophy
• Ancient Greek and Roman Philosophy
• Medieval and Renaissance Philosophy
• 17th/18th Century Philosophy
• 19th Century Philosophy
• 20th Century Philosophy
• History of Western Philosophy, Misc
• Philosophical Traditions
• African/Africana Philosophy
• Asian Philosophy
• Continental Philosophy
• European Philosophy
• Philosophy of the Americas
• Philosophical Traditions, Miscellaneous
• Philosophy, Misc
• Philosophy, Introductions and Anthologies
• Philosophy, General Works
• Teaching Philosophy
• Philosophy, Miscellaneous
• Other Academic Areas
• Natural Sciences
• Social Sciences
• Cognitive Sciences
• Formal Sciences
• Arts and Humanities
• Professional Areas
• Other Academic Areas, Misc
• Submit a book or article
• Upload a bibliography
• Personal page tracking
• Archives we track
• Information for publishers
• Introduction
• Submitting to PhilPapers
• Frequently Asked Questions
• Subscriptions
• Editor's Guide
• The Categorization Project
• For Publishers
• For Archive Admins
• PhilPapers Surveys
• Bargain Finder
• About PhilPapers
• Create an account

Qualitative approaches to empirical legal research

Reprint years, philarchive, external links.

• From the Publisher via CrossRef (no proxy)
• dx.doi.org (no proxy)

Through your library

• Sign in / register and customize your OpenURL resolver
• Configure custom resolver

Similar books and articles

Citations of this work, references found in this work.

No references found.

• Help & FAQ

Qualitative approaches to empirical legal research

• Birmingham Law School

Research output : Chapter in Book/Report/Conference proceeding › Chapter

Publication series

T1 - Qualitative approaches to empirical legal research

AU - Webley, Lisa

PY - 2010/11/11

Y1 - 2010/11/11

UR - https://global.oup.com/academic/product/the-oxford-handbook-of-empirical-legal-research-9780199542475?cc=gb&lang=en&

M3 - Chapter

SN - 9780199542475

SN - 9780199659944

T3 - Oxford Handbooks

BT - The Oxford Handbook of Empirical Legal Research

A2 - Cane, Peter

A2 - Kritzer, Herbert

PB - Oxford University Press

• Book chapter

Qualitative approaches to empirical legal research

Webley, L. 2010. Qualitative approaches to empirical legal research. in: Cane, P. and Kritzer, H. (ed.) Oxford handbook of empirical legal research Oxford Oxford University Press. pp. 926-950

Related outputs

Gender and the legal academy in the UK: a product of proxies and hiring and promotion practices Duff, L. and Webley, L. 2021. Gender and the legal academy in the UK: a product of proxies and hiring and promotion practices. in: Auchmuty, R.S., Ulrike Schultz, U., Shaw, G. and Thornton, M. (ed.) Gender and Careers in the Legal Academy UK Hart Publishing.

Women in Law Literature Review Duff, L., Mason, M. and Webley, L. 2019. Women in Law Literature Review. London Law Society.

When is mediation mediatory and when is it really adjudicatory? Religion, Norms and decision-making Webley, L. 2017. When is mediation mediatory and when is it really adjudicatory? Religion, Norms and decision-making. in: Bano, S. (ed.) Gender and Justice in Family Law Disputes: Women, Mediation, and Religious Arbitration University Press of New England.

Stumbling Blocks in Empirical Legal Research: Case Study Research Webley, L. 2016. Stumbling Blocks in Empirical Legal Research: Case Study Research. Law and Method. https://doi.org/10.5553/REM/.000020

Interception of communications and legal professional privilege and the rule of law Webley, L. 2016. Interception of communications and legal professional privilege and the rule of law. Legal Ethics. 19 (1), pp. 173-176. https://doi.org/10.1080/1460728x.2016.1193954

Diversity and Inclusion As the Key to Innovating Talent Management in Law Firms? Webley, L. and Duff, L. 2016. Diversity and Inclusion As the Key to Innovating Talent Management in Law Firms? in: Mottershead, T. (ed.) Innovating Talent Management in Law Firms National Association for Law Placement. pp. 235-254

Legal Writing (4th edition) Webley, L. 2016. Legal Writing (4th edition). Abingdon Routledge.

Access to a Career in the Legal Profession in England and Wales: Race, Class, and the Role of Educational Background Webley, L., Tomlinson, J., Muzio, D., Sommerlad, H. and Duff, L. 2016. Access to a Career in the Legal Profession in England and Wales: Race, Class, and the Role of Educational Background. in: Headworth, S., Nelson, R.L., Dinovitzer, R. and Wilkins, D.B. (ed.) Diversity In Practice: Race, Gender, and Class in Legal and Professional Careers Cambridge Cambridge University Press. pp. 198-225

European Citizenship, Freedom, Security, Justice and Equality Webley, L. 2015. European Citizenship, Freedom, Security, Justice and Equality. in: European Citizenship Challenges and Opportunities Porto Portugal Universidade Catolica Editora Porto. pp. 143-162

Complete Public Law Text, Cases and Materials, 3rd edition Webley, L. and Samuels, H. 2015. Complete Public Law Text, Cases and Materials, 3rd edition. Oxford Oxford University Press.

When is a Family Lawyer, a Lawyer? Webley, L. 2015. When is a Family Lawyer, a Lawyer? in: Mavis Maclean, John Eekalaar and Bastard Benoit (ed.) Delivering Family Justice in the 21st Century Oxford, UK Hart. pp. 305-321

Legal Professional De(Re)Regulation, Equality, and Inclusion, and the Contested Space of Professionalism Within the Legal Market in England and Wales Webley, L. 2015. Legal Professional De(Re)Regulation, Equality, and Inclusion, and the Contested Space of Professionalism Within the Legal Market in England and Wales. Fordham Law Review. 83 (5), pp. 2349-2367.

The Former Legal Director of the London Times, Legal Professional Privilege and the Duty Not to Mislead the Court in England and Wales Webley, L. 2014. The Former Legal Director of the London Times, Legal Professional Privilege and the Duty Not to Mislead the Court in England and Wales. Legal Ethics. 17 (2), pp. 310-312.

Legal Writing (3rd Edition) Webley, L. 2013. Legal Writing (3rd Edition). London, UK Routledge.

United Kingdom: What Robinson v Solicitors Regulation Authority Tells Us about the Contested Terrain of Race and Disciplinary Processes Webley, L. 2013. United Kingdom: What Robinson v Solicitors Regulation Authority Tells Us about the Contested Terrain of Race and Disciplinary Processes. Legal Ethics. 16 (1), pp. 236-241. https://doi.org/10.5235/1460728X.1.1.236

Structure, agency and the career strategies of women and BME individuals in the legal profession Tomlinson, J., Muzio, D., Sommerlad, H., Webley, L. and Duff, L. 2013. Structure, agency and the career strategies of women and BME individuals in the legal profession. Human Relations. 66 (2), pp. 245-269. https://doi.org/10.1177/0018726712460556

Complete Public Law: Text, Cases and Materials (2nd Edition) Webley, L. and Samuels, H. 2012. Complete Public Law: Text, Cases and Materials (2nd Edition). Oxford, UK Oxford University Press.

The Legal Education Training Review: Correspondent’s Report from the United Kingdom Webley, L. 2012. The Legal Education Training Review: Correspondent’s Report from the United Kingdom. Legal Ethics. 15 (1), pp. 140-141. https://doi.org/10.5235/146072812801292773

Legal Ethics and the Legal Education Training Review: Correspondent’s Report from the United Kingdom Webley, L. 2012. Legal Ethics and the Legal Education Training Review: Correspondent’s Report from the United Kingdom. Legal Ethics. 15 (2), pp. 402-404.

Gender, Hierarchy, Power and Inequality: What Sociological Theory Adds to our Understanding of Sex-Discrimination Webley, L. 2012. Gender, Hierarchy, Power and Inequality: What Sociological Theory Adds to our Understanding of Sex-Discrimination. Westminster Law Review . 1 (1), pp. 37-42.

Legal ethics in the academic curriculum: correspondent’s report from the United Kingdom Webley, L. 2011. Legal ethics in the academic curriculum: correspondent’s report from the United Kingdom. Legal Ethics. 14 (1), pp. 132-134.

Law Students as Peer Mentors: Developing the Reflective Practitioner and/or Embedding Legal Learning: Correspondent’s Report from the United Kingdom Webley, L. 2011. Law Students as Peer Mentors: Developing the Reflective Practitioner and/or Embedding Legal Learning: Correspondent’s Report from the United Kingdom. Legal Ethics. 14 (2), pp. 250-251.

Solicitors as imagined masculine, family mediators as fictive feminine and the hybridization of divorce solicitors Webley, L. 2011. Solicitors as imagined masculine, family mediators as fictive feminine and the hybridization of divorce solicitors. in: Bartlett, F., Mortenson, R. and Tranter, K. (ed.) Alternative perspectives on lawyers and legal ethics Abingdon Routledge. pp. 132-150

Gate-keeper, supervisor or mentor? The role of professional bodies in the regulation and professional development of solicitors and family mediators undertaking divorce matters in England and Wales Webley, L. 2010. Gate-keeper, supervisor or mentor? The role of professional bodies in the regulation and professional development of solicitors and family mediators undertaking divorce matters in England and Wales. Journal of Social Welfare and Family Law. 32 (2), pp. 119-133. https://doi.org/10.1080/09649069.2010.506308

Diversity in the legal profession: a business or an ethical rationale?: correspondent's report from the United Kingdom Webley, L. 2010. Diversity in the legal profession: a business or an ethical rationale?: correspondent's report from the United Kingdom. Legal Ethics. 13 (2), pp. 223-224.

Adversarialism and consensus? The professions’ construction of solicitor and family mediator identity and role Webley, L. 2010. Adversarialism and consensus? The professions’ construction of solicitor and family mediator identity and role. New Orleans Quid Pro.

Diversity in the legal profession in England and Wales: A qualitative study of barriers and individual choices Sommerlad, H., Webley, L., Duff, L., Muzio, D. and Tomlinson, J. 2010. Diversity in the legal profession in England and Wales: A qualitative study of barriers and individual choices. London Legal Services Board.

Complete Public Law: Test Cases and Materials Webley, L. and Samuels, H. 2009. Complete Public Law: Test Cases and Materials. Oxford, UK Oxford University Press.

Legal writing. 2nd edition Webley, L. 2009. Legal writing. 2nd edition. London, UK Cavendish Publishing.

The next generation of legal aid solicitors: the LSC’s Training Grants Scheme Bacquet, S., Boon, A., Webley, L. and Whyte, A. 2009. The next generation of legal aid solicitors: the LSC’s Training Grants Scheme. in: Buck, A. (ed.) Reaching further: innovation, access and quality in legal services : papers from the Legal Services Research Centre's seventh international research conference, Reaching further, Royal Naval Academy in Greenwich, London, 18th to 20th June 2008 London Stationary Office. pp. 143-165

Evaluation of the Legal Service Commission's Training Contract Grant Scheme Bacquet, S., Boon, A., Webley, L. and Whyte, A. 2008. Evaluation of the Legal Service Commission's Training Contract Grant Scheme. Legal Services Commission.

The European criminal record in England and Wales Webley, L. 2008. The European criminal record in England and Wales. in: Stefanou, C. and Xanthaki, H. (ed.) Towards a European criminal record Cambridge Cambridge University Press. pp. 281-307

Paralegals Webley, L. 2008. Paralegals. in: Cane, P. and Conaghan, J. (ed.) The new Oxford companion to law Oxford Oxford University Press. pp. 862-863

Evaluation of the Legal Service Commission's Training Contract Grant Scheme Webley, L. 2008. Evaluation of the Legal Service Commission's Training Contract Grant Scheme. Society of Legal Scholars Annual Conference 2008: The Impact of Legal Scholarship. London School of Economics 15 - 18 Sep 2008

Encouraging the next generation of legal aid solicitors: the LSC’s Training Grants Scheme Webley, L. 2008. Encouraging the next generation of legal aid solicitors: the LSC’s Training Grants Scheme. Seventh Legal Services Research Centre (LSRC) International Research Conference, "Reaching Further: New Approaches to the Delivery of Legal Services". Naval College, Greenwich, London 18 - 20 Jun 2008

Costs of complaining and claiming Webley, L. 2008. Costs of complaining and claiming. in: Cane, P. and Conaghan, J. (ed.) The new Oxford companion to law Oxford Oxford University Press. pp. 243-245

Court-based mediation in England and Wales: are judges evading their responsibilities? Webley, L. 2007. Court-based mediation in England and Wales: are judges evading their responsibilities? Cleveland-Marshall College of Law, Cleveland State University Ohio, Seminar Series. Cleveland State University, Ohio 2007

Women solicitors as a barometer for problems within the legal profession: time to put values before profits? Webley, L. and Duff, L. 2007. Women solicitors as a barometer for problems within the legal profession: time to put values before profits? Journal of Law and Society. 34 (3), pp. 374-402. https://doi.org/10.1111/j.1467-6478.2007.00397.x

A review of how other countries provide information and advice to the vulnerable on consumer and social issues Webley, L. and Bacquet, S. 2006. A review of how other countries provide information and advice to the vulnerable on consumer and social issues. London, UK DTI. https://doi.org/URN06/1871

Mediation through the Birmingham court-based scheme Webley, L., Abrams, P. and Bacquet, S. 2006. Mediation through the Birmingham court-based scheme. Journal of Housing Law. 9 (1), pp. 9-12.

Evaluation of the Birmingham court-based Civil (non-family) mediation scheme Webley, L., Abrams, P. and Bacquet, S. 2006. Evaluation of the Birmingham court-based Civil (non-family) mediation scheme. London, UK Department for Constitutional Affairs.

Divorce solicitors and ethical approaches: the best interests of the client and/or the best interests of the family? Webley, L. 2005. Divorce solicitors and ethical approaches: the best interests of the client and/or the best interests of the family? Legal Ethics. 7 (2), pp. 231-250.

Criminal records and organised crime in England and Wales Webley, L. 2005. Criminal records and organised crime in England and Wales. in: Stefanou, C. and Xanthaki, H. (ed.) Financial crime in the EU: criminal records as effective tools or missed opportunities? The Hague, Netherlands Kluwer Law International. pp. 133-148

The loneliness of the pro bono coordinator Webley, L. 2004. The loneliness of the pro bono coordinator. Law Institute Journal. 78 (12), p. 92.

Legal aid and legal need Webley, L. 2004. Legal aid and legal need. Law Institute Journal. 78 (11), p. 91.

Access key in the UK Webley, L. 2004. Access key in the UK. Law Institute Journal. 78 (9), p. 83.

Equality and diversity: women solicitors research study 48 Duff, L. and Webley, L. 2004. Equality and diversity: women solicitors research study 48. London, UK Law Society.

Pro bono and young solicitors: views from the front line Webley, L. 2001. Pro bono and young solicitors: views from the front line. Legal Ethics. 3 (2), pp. 152-168.

Quality and cost: final report on the contracting of Civil, Non-Family Advice and Assistance Pilot Moorhead, R., Sherr, A., Webley, L., Rogers, S., Sherr, L., Patterson, A. and Domberger, S. 2001. Quality and cost: final report on the contracting of Civil, Non-Family Advice and Assistance Pilot. London, UK The Stationary Office.

Permalink - https://westminsterresearch.westminster.ac.uk/item/90872/qualitative-approaches-to-empirical-legal-research

Usage statistics

Harvard Empirical Legal Studies Series

5005 Wasserstein Hall (WCC) 1585 Massachusetts Avenue Cambridge, MA02138

Contact the Graduate Program

The  Harvard Empirical Legal Studies (HELS) Series  explores a range of empirical methods, both qualitative and quantitative, and their application in legal scholarship in different areas of the law. It is a platform for engaging with current empirical research, hearing from leading scholars working in a variety of fields, and developing ideas and empirical projects.

HELS is open to all students and scholars with an interest in empirical research. No prior background in empirical legal research is necessary. If you would like to join HELS and receive information about our sessions, please subscribe to our mailing list by completing the HELS mailing list form .

If you have any questions, do not hesitate to contact the current HELS coordinator,  Tiran Bajgiran.

All times are provided in U.S. Eastern Time (UTC/GMT-0400).

Spring 2024 Sessions

Empire and the shaping of american constitutional law.

Aziz Rana, Wellesley

Monday, Mar. 25, 12:15 PM Lewis 214

This talk will explore how US imperial practice has influenced the methods and boundaries of American constitutional study.

Historical Approaches to Neoliberal Legality

Quinn Slobodian, Boston University

Thursday, Mar. 28, 12:15 PM Lewis 202

Fall 2023 Sessions

On critical quantitative methods.

Hendrik Theine , WU, Vienna/Univ. of Pennsylvania Monday, Nov. 6, 12:30 PM Lewis 202

Economic inequality is a profound challenge in the United States. Both income and wealth inequality increased remarkably since the 1980s. This growing concentration of economic inequality creates real-world political and societal problems which are increasingly reflected by social science scholarship. Among those detriments is for instance the increasing economic and political power of the super-rich. The research at hand takes a new radical look at media discourses of economic inequality over four decades in various elite US newspapers by way of quantitative critical discourse analysis. It shows that up until recently, there was minimal media coverage of economic inequality, but interest has steadily increased since then. Initially, the focus was primarily on income inequality, but over time, it has expanded to encompass broader issues of inequality. Notably, the discourse on economic inequality is significantly influenced by party politics and elections. The study also highlights certain limitations in the discourse. Critiques of inequality tend to remain at a general level, discussing concepts like capitalist and racial inequality. There is relatively less focus on policy-related discussions, such as tax reform, or discussions centered around specific actors, like the wealthy and their charitable contributions.

Spring 2023 Sessions

How to conduct qualitative empirical legal scholarship.

Jessica Silbey , Professor of Law at Boston University Yanakakis Faculty Research Scholar

Friday, March 31, 12:30 PM WCC 3034

This session explores the benefits and some limitations of qualitative research methods to study intellectual property law. It compares quantitative research methods and the economic analysis of law in the same field as other kinds of empirical inquiry that are helpful in collaboration but limited in isolation. Creativity and innovation, the practices intellectual property law purports to regulate, are not amenable to quantification without identifying qualitative variables. The lessons from this session apply across fields of legal research.

Fall 2022 Sessions

How to read quantitative empirical legal scholarship.

Holger Spamann , Lawrence R. Grove Professor of Law

Friday, September 13, 12:30 PM WCC 3007

As legal scholars, what tools do we need to read critically and engage productively with quantitative empirical scholarship? In the first session of the 2022-2023 Harvard Empirical Legal Studies Series, Harvard Law School Professor Holger Spamann will compare and discuss different quantitative studies. This session will be a first approximation to be able to understand and eventually produce empirical legal scholarship. All students and scholars interested in empirical research are welcome and encouraged to attend.

How do People Learn from Not Being Caught? An Experimental Investigation of a “Non-Occurrence Bias”

Tom Zur , John M. Olin Fellow and SJD candidate, HLS

Friday, November 4, 2:00 PM WCC 3007

The law and economics literature on specific deterrence has long theorized that offenders rationally learn from being caught and sanctioned. This paper presents evidence from a randomized controlled trial showing that offenders learn differently when not being caught as compared to being caught, which we call a “non-occurrence bias.” This implies that the socially optimal level of investment in law enforcement should be lower than stipulated by rational choice theory, even on grounds of deterrence alone.

Empirical Legal Research: Using Data and Methodology to Craft a Research Agenda

Florencia Marotta-Wurgler , NYU Boxer Family Professor of Law Faculty Director, NYU Law in Buenos Aires

Monday, November 14, 12:30 PM Lewis 202

Using a series of examples, this discussion will focus on strategies to conduct empirical legal research and develop a robust research agenda. Topics will include creating a data set and leveraging to answer unexplored questions, developing meaningful methodologies to address legal questions, building on existing work to develop a robust research agenda, and engaging the process of automation and scaling up to develop large scale data sets using machine learning approaches. 

Resources for Empirical Research

• HLS Library Empirical Research Service
• Harvard Institute for Quantitative Social Research (IQSS)
• Harvard Committee on the Use of Human Subjects
• Qualtrics Harvard
• Harvard Kennedy School Behavioral Insights Group

Past HELS Sessions

Holger Spamann (Lawrence R. Grove Professor of Law) – How to Read Quantitative Empirical Legal Scholarship?

Katerina Linos (Professor of Law at UC Berkeley School of Law) – Qualitative Methods for Law Review Writing

Aziza Ahmed (Professor of Law at UC Irvine School of Law) – Risk and Rage: How Feminists Transformed the Law and Science of AIDS

Amy Kapczynski and Yochai Benkler –(Professor of Law at Yale; Professor of Law at Harvard) Law & Political Economy and the Question of Method

Jessica Silbey – (Boston University School of Law) Ethnography in Legal Scholarship

Roberto Tallarita – (Lecturer on Law, and Associate Director of the Program on Corporate Governance at Harvard) The Limits of Portfolio Primacy

Susan S. Silbey – (Leon and Anne Goldberg Professor of Humanities, Sociology and Anthropology at MIT) HELS with Susan Silbey: Analyzing Ethnographic Data and Producting New Theory

Cass R. Sunstein  (University Professor at Harvard) – Optimal Sludge? The Price of Program Integrity

Scott L. Cummings  (Professor of Legal Ethics and Professor of Law at UCLA School of Law) – The Making of Public Interest Lawyers

Elliot Ash  (Assistant Professor of Law, Economics, and Data Science at ETH Zürich) – Gender Attitudes in the Judiciary: Evidence from U.S. Circuit Courts

Kathleen Thelen  (Ford Professor of Political Science at MIT) – Employer Organization in the United States: Historical Legacies and the Long Shadow of the American Courts

Omer Kimhi  (Associate Professor at Haifa University Law School) – Caught In a Circle of Debt – Consumer Bankruptcy Discharge and Its Aftereffects

Suresh Naidu  (Professor in Economics and International and Public Affairs, Columbia School of International and Public Affairs) – Ideas Have Consequences: The Impact of Law and Economics on American Justice

Vardit Ravitsky  (Full Professor at the Bioethics Program, School of Public Health, University of Montreal) – Empirical Bioethics: The Example of Research on Prenatal Testing

Johnnie Lotesta  (Postdoctoral Democracy Fellow at the Ash Center for Democratic Governance and Innovation at the Harvard Kennedy School) – Opinion Crafting and the Making of U.S. Labor Law in the States

David Hagmann  (Harvard Kennedy School) – The Agent-Selection Dilemma in Distributive Bargaining

Cass R. Sunstein  (Harvard Law School) – Rear Visibility and Some Problems for Economic Analysis (with Particular Reference to Experience Goods)

Talia Gillis  (Ph.D. Candidate and S.J.D. Candidate, Harvard Business School and Graduate School of Arts and Sciences and Harvard Law School) – False Dreams of Algorithmic Fairness: The Case of Credit Pricing

Tzachi Raz (Ph.D. Candidate in Economics at Harvard University) – There’s No Such Thing as Free Land: The Homestead Act and Economic Development

Crystal Yang (Harvard Law School) – Fear and the Safety Net: Evidence from Secure Communities

Adaner Usmani (Harvard Sociology) – The Origins of Mass Incarceration

Jim Greiner (Harvard Law School) – Randomized Control Trials in the Legal Profession

Talia Shiff  (Postdoctoral Fellow, Weatherhead Center for International Affairs and Department of Sociology, Harvard University) – Legal Standards and Moral Worth in Frontline Decision-Making: Evaluations of Victimization in US Asylum Determinations

Francesca Gino (Harvard Business School) – Rebel Talent

Joscha Legewie (Department of Sociology, Harvard University) – The Effects of Policing on Educational Outcomes and Health of Minority Youth

Ryan D. Enos (Department of Government, Harvard University) – The Space Between Us: Social Geography and Politics

Katerina Linos (Berkeley Law, University of California) – How Technology Transforms Refugee Law

Roie Hauser (Visiting Researcher at the Program on Corporate Governance, Harvard Law School) – Term Length and the Role of Independent Directors in Acquisitions

Anina Schwarzenbach (Fellow, National Security Program, the Belfer Center for Science and International Affairs, Harvard Kennedy School) – A Challenge to Legitimacy: Effects of Stop-and-Search Police Contacts on Young People’s Relations with the Police

Cass R. Sunstein (Harvard Law School) – Willingness to Pay to Use Facebook, Twitter, Youtube, Instagram, Snapchat, and More: A National Survey

Netta Barak-Corren (Hebrew University of Jerusalem) – The War Within

James Greiner & Holger Spamann (Harvard Law School) – Panel: Why​ ​Does​ ​the​ ​Legal​ ​Profession​ ​Resist​ ​Rigorous​ ​Empiricism?

Mila Versteeg (University of Virginia School of Law) (with Adam Chilton) – Do Constitutional Rights Make a Difference?

Susan S. Silbey (MIT Department of Anthropology) (with Patricia Ewick) – The Common Place of Law

Holger Spamann (Harvard Law School) – Empirical Legal Studies: What They Are and How NOT to Do Them

Arevik Avedian (Harvard Law School) – How to Read an Empirical Paper in Law

James Greiner (Harvard Law School) – Randomized Experiments in the Law

Robert MacCoun (Stanford Law School) – Coping with Rapidly Changing Standards and Practices in the Empirical Sciences (including ELS)

Mario Small (Harvard Department of Sociology) – Qualitative Research in the Big Data Era

Adam Chilton (University of Chicago Law School) – Trade Openness and Antitrust Law

Jennifer Lerner (Harvard Kennedy School and Department of Psychology) – Anger in Legal Decision Making

Sarah Dryden-Peterson (Harvard Graduate School of Education) – Respect, Reciprocity, and Relationships in Interview-Based Research

Charles Wang (Harvard Business School) – Natural Experiments and Court Rulings

Guhan Subramanian (Harvard Law School) – Determining Fair Value

James Greiner (Harvard Law School) – Randomized Control Trials and the Impact of Legal Aid

Maya Sen (Harvard Kennedy School) – The Political Ideologies of Law Clerks and their Judges

Daria Roithmayr (University of Southern California Law School) – The Dynamics of Police Violence

Crystal Yang (Harvard Law School) – Empiricism in the Service of Criminal Law and Theory

Oren Bar-Gill (Harvard Law School) – Is Empirical Legal Studies Changing Law and Economics?

Elizabeth Linos (Harvard Kennedy School; VP, Head of Research and Evaluation, North America, Behavioral Insights Team) – Behavioral Law and Economics in Action: BIT, BIG, and the policymaking of choice architecture

Meira Levinson (Harvard School of Education) – Justice in Schools: Qualitative Sociological Research and Normative Ethics in Schools

Howell Jackson (HLS) – Cost-Benefit Analysis

Michael Heise (Cornell Law School) – Quantitative Research in Law: An Introductory Workshop

Susan Silbey (MIT) – Interviews: An Introductory Workshop

Kevin Quinn (UC Berkeley) – Quantifying Judicial Decisions

Holger Spamman (Harvard Law School) – Comparative Empirical Research

James Greiner (Harvard Law School) – Randomized Controlled Trials in the Research of Legal Problems

Michael Heise (Cornell Law School) – Quantitative Research in Law

James Greiner (Harvard Law School) – A Typology of Empirical Methods in Law

David Wilkins (Harvard Law School) – Mixed Methods Work and the Legal Profession

Tom Tyler (Yale Law School) – Fairness and Policing

Modal Gallery

Qualitative Methods for Law Review Writing

We are extremely grateful to Catherine Albiston, Lauren Edelman, Stavros Gadinis, David Lieberman, Aila Matanock, Alison Post, Kevin Quinn, Karen Tani, and participants at the Berkeley Law Faculty Workshop for their generous comments.

• Share The University of Chicago Law Review | Qualitative Methods for Law Review Writing on Facebook
• Share The University of Chicago Law Review | Qualitative Methods for Law Review Writing on Twitter
• Share The University of Chicago Law Review | Qualitative Methods for Law Review Writing on Email
• Share The University of Chicago Law Review | Qualitative Methods for Law Review Writing on LinkedIn

Typical law review articles not only clarify what the law is, but also examine the history of the current rules, assess the status quo, and present reform proposals. To make theoretical arguments more plausible, legal scholars frequently use examples: they draw on cases, statutes, political debates, and other sources. But legal scholars often pick their examples unsystematically and explore them armed with only the tools for doctrinal analysis. Unsystematically chosen examples can help develop plausible theories, but they rarely suffice to convince readers that these theories are true, especially when plausible alternative explanations exist. This project presents methodological insights from multiple social science disciplines and from history that could strengthen legal scholarship by improving research design, case selection, and case analysis. We describe qualitative techniques rarely found in law review writing, such as process tracing, theoretically informed sampling, and most similar case design, among others. We provide examples of best practice and illustrate how each technique can be adapted for legal sources and arguments.

I.  Imagining Alternatives and Identifying a Puzzle

“[A]ll you really need to have is an ‘explanandum’—a puzzle, paradox, or conundrum about the social world that in one way or another upsets our expectations, and for which there is no ready answer. But this is not at all a trivial accomplishment.” 16

For social scientific research, the starting point—and perhaps half the battle—is identifying a puzzle that cannot be easily solved. Legal advocacy training does not highlight this element of puzzlement. In fact, many masterful legal strategists downplay the novelty of their arguments so that courts can more easily accept them.

To identify a puzzle, one can begin by imagining alternative outcomes to the one that occurred. The sources legal scholars regularly use are superb starting points for this task. The adversarial process inherently offers (at least) two alternative ways of understanding a set of facts—the plaintiff’s and the defendant’s. Amicus briefs and other third-party interventions can also help sketch out alternative options. Additionally, separate opinions from judges, including powerful concurrences and dissents, provide a range of plausible alternative legal outcomes. Furthermore, trial and appellate court judges can offer different answers to the same question, creating legally plausible alternative conclusions. In short, the legal process itself offers a broad range of well-constructed alternatives.

Legal scholars often go beyond these first steps to construct plausible but nonobvious alternative worlds, and draw comparisons across historical periods, legal fields, and jurisdictions. For example, in Pigs and Positivism , Professor Hendrik Hartog constructs a nonobvious but plausible counterfactual by examining a case concerning pig owners’ right to let pigs roam in urban settings. 17 Predictably, the prosecution emphasized the risks and nuisances pigs create, while the defense minimized them. 18 Drawing on historical and comparative evidence, Hartog spells out a plausible, alternative understanding of the case. Defense lawyers could have argued that pig keepers possess a customary right to let their pigs roam freely because this was a commonly accepted practice historically. 19 Despite its plausibility, the defense did not make a claim about custom—why?

By identifying this third plausible alternative, Hartog demonstrates that, while prosecutors and defense attorneys predictably disagree, the terms of disagreement explain the bounds of what is legally acceptable in particular times and places. 20 Hartog shows that an argument about custom was just outside the bounds of acceptability in early nineteenth-century New York City, even though it might have been entirely acceptable at a slightly earlier moment, in a more rural American setting, or in contemporary Britain. 21

After imagining plausible alternatives, scholars select cases that allow them to effectively explore why a particular path was or should have been chosen rather than its alternative. In the Part that follows, we present useful techniques for scholars to systematically select cases.

II.  Sampling and Case Selection

Concerns about case selection and sampling are widespread among legal scholars, particularly the worry of cherry-picking cases that best fit an argument. What is less well-known is how to create representative samples and select cases to make credible, generalizable causal claims. We introduce some helpful sampling and case selection techniques in the paragraphs that follow.

A.    Sampling

Through sampling, researchers gather a subset of units from which they can make inferences about a broader population. Sampling techniques are useful for scholars pursuing doctrinal projects because the credibility of a generalization about doctrine depends on the representativeness of chosen examples. Sampling also holds important advantages for scholars pursuing causal arguments because it helps eliminate alternative explanations of the outcome. Below, we start with some general considerations about carefully sampling legal cases. We then present two particularly useful sampling techniques: random sampling and theoretically informed sampling. We discuss random sampling to dispel the assumption that it is too complicated to use in qualitative research. We present theoretically informed sampling because it allows scholars who work with few cases to make valid inferences.

Careful sampling requires scholars to clearly define the scope of their generalizations and the population to which their inferences apply. To see careful sampling in practice, we turn to Multiple Disadvantages: An Empirical Test of Intersectionality Theory in EEO Litigation . 22 Professors Rachel Best, Lauren Edelman, Linda Krieger, and Scott Eliason sample judicial opinions in equal employment opportunity cases in US federal courts to argue that antidiscrimination lawsuits provide the least protection for plaintiffs with multiple social disadvantages. 23 Plaintiffs who allege discrimination based on multiple traits, such as race and gender, are only half as likely to win their cases as other plaintiffs. 24

Careful sampling is critical in making this claim persuasive. First, the authors select the appropriate unit in which to test their theory: federal circuit and district court cases. 25 Circuit decisions establish precedent, while district courts handle a substantial number of discrimination cases and are thus “the primary federal locale for civil rights dispute resolution.” 26 If the authors had used Supreme Court cases as their unit of analysis, it would have been harder to assess whether plaintiff characteristics influence judicial rulings. Supreme Court cases are idiosyncratic; they often involve novel issues and particularly motivated parties. The authors could not draw valid general inferences from these cases.

Second, the authors clearly explain their sample’s limitations and define the scope of their inferences. The authors randomly sampled from relevant district and circuit court opinions available on Westlaw. 27 The authors emphasize that they could not include disputes that were resolved before reaching the courts or court opinions that were never published. 28 By defining the limits of their sample, the authors strengthen the plausibility of their inferences.

1.   Random sampling and systematic sampling.

Random sampling is widely used in the social sciences. Random sampling involves selecting subjects from a larger population by chance; each subject has equal probability of being selected. Random sampling has distinct advantages because it eliminates the possibility that the characteristics of selected units influence the outcome. This technique allows scholars with limited information about the universe of cases to draw generalizations efficiently.

Random sampling is critical to Best and her colleagues’ ability to make a general claim about plaintiffs’ success in antidiscrimination lawsuits. The authors collected all relevant district and circuit court opinions between 1965 and 1999 available on Westlaw, from which they randomly chose 2 percent. 29 Each district court opinion has unique characteristics that could influence its outcome; moreover, the authors do not possess anywhere near complete knowledge about every district court case. Random sampling allows the authors to make valid generalizations to all published district and circuit court cases despite these challenges.

A related technique—systematic sampling—can also produce credible generalizations. Systematic sampling involves randomly choosing a starting point and then selecting cases based on a fixed interval. 30 For example, for his book Habeas Corpus: From England to Empire , Professor Paul Halliday creates a systematic sample of all uses of the writ of habeas corpus issued by the courts of the King’s Bench from 1500 to 1800. 31 Starting in 1502, Halliday chooses petitions filed in every fourth year until 1798. 32 Creating this systematic sample allows Halliday to identify common case characteristics and make generalizations about how people approached law. 33 Systematic sampling also allows scholars to correlate outcomes to variables; this is important for Halliday, who “correlat[es] outcomes to . . . the wrongs for which prisoners were held and the jurisdictions that ordered confinement.” 34

Random sampling has an important limitation: it requires the researcher to select a relatively large number of cases. We turn next to theoretically informed sampling, which is more appropriate for studying smaller numbers of cases.

2.  Theoretically informed sampling.

Theoretically informed sampling holds distinct advantages for producing causal claims and credible generalizations with a small number of cases. First, the researcher identifies theoretically important characteristics that could influence the outcome. The researcher then sorts cases into categories defined by these characteristics and selects cases from each category. 35

For example, if a researcher was interested in treaty compliance, she would begin by identifying state characteristics that could delay compliance, such as limited bureaucratic capacity, poverty, or federalism. The researcher would then create categories defined by different combinations of these variables (for example, a wealthy federal state with high bureaucratic capacity) and sort states into each category. She would then select cases from each category, either randomly or based on practical and theoretical concerns. For example, because US treaty ratification behavior is very different from that of other wealthy federal states with high bureaucratic capacity, the researcher might want to include additional wealthy federal states. Ultimately, the researcher should “select[ ] a manageable number of cases that are diverse in terms of theoretically important traits.” 36

Theoretically informed sampling is more difficult to carry out than random sampling and more likely to lead the researcher to introduce bias into the selection process. Despite these drawbacks, theoretically informed sampling has distinct advantages over random sampling for scholars working with a small number of cases. Random sampling has poor small-sample properties: the chances that a researcher who randomly selects five countries will end up with five developing countries, or five agricultural economies, rather than five diverse states, are surprisingly high. Scholars cannot then make valid generalizations because the cases selected have particular, shared characteristics. 37

We could not locate exemplary uses of theoretically informed sampling in the legal literature. This makes our description more challenging, yet more likely to be useful. Below is an example that illustrates some of the steps outlined above, but that has important limitations. In Legalizing Gender Inequality: Courts, Markets, and Unequal Pay for Women in America , Professors Robert Nelson and William Bridges investigate “wage differences between jobs held primarily by women and those held primarily by men within the same organization. ” 38 Al­though relevant literature argues that market principles produce these differences, Nelson and Bridges argue that organizational processes cause pay differences between typically “male” and “female” jobs. 39 Undergirding this argument are four case studies of gender discrimination lawsuits. 40

The authors select these cases to capture theoretically important variation across lawsuits. 41 The authors define the universe of cases, which includes defendant organizations large enough to have sufficiently differentiated occupations, internal labor markets, and bureaucratic personnel systems. 42 Within these parameters, the authors identify firm characteristics that might influence their outcome of interest, development of gender inequality. The potentially influential characteristics include whether organizations are public or private and the proportion of the workforce with firm-specific skills. 43 After creating four categories (for example, public companies requiring firm-specific skills), the authors select cases from each category according to practical considerations, namely, whether evidence was accessible. 44 Essentially, the authors select cases based on the values of potentially influential variables because it allows the authors to effectively evaluate whether and how organization type and skill requirements influence the outcome. Because the authors demonstrate that these other variables do not fully account for the patterns they observe, it strengthens their argument that their independent variable of interest is driving the outcome. As such, by using theoretically informed sampling, researchers can use few cases to assess their independent variable’s effect on the outcome.

Despite their use of theoretically informed sampling, the authors’ selection process raises important questions. For example, they examine only organizations sued for gender discrimination; these organizations may have especially egregious practices, and thus may be unrepresentative. 45 The authors try to alleviate this concern by, among other things, comparing employment numbers to similarly sized firms and including statements from employers that the firms sued were not unusual. 46

B.    Case Selection Techniques

While sampling techniques strengthen generalizations about the prevalence of certain population characteristics, case selection techniques are used to make structured and focused comparisons across cases, strengthening causal claims. We describe several case selection techniques below.

1.   Most difficult case design.

Selecting cases in which one’s theory is least likely to hold true can offer strong theoretical leverage. These cases, called “least-likely” cases, 47 undergird most difficult case design. If a researcher demonstrates that her theory holds true in an unlikely case, the argument is likely to hold in a broader range of cases. 48 In The Hollow Hope: Can Courts Bring About Social Change? , Professor Gerald Rosenberg uses two prominent US Supreme Court cases, Roe and Brown , to argue that the US Supreme Court’s influence on public policy is limited. 49

Using a least-likely case selection strategy is particularly effective for increasing the causal strength and generalizability of Rosenberg’s argument. The Supreme Court is more visible and influential than any other court in the American political system. 50 Roe and Brown are considered prime examples of a court producing significant social reform. 51 If Rosenberg’s theory holds true in the cases in which it is most likely to fail, it is plausible that his hypothesis could hold true in other, “easier” cases. If Rosenberg had instead chosen a case from a lower court believed to have little impact on social reform, his claim would have been far less plausible, and would have generated far less interest.

2.   Most similar case design.

In most similar case selection, the researcher chooses cases that have similar values on theoretically important characteristics, but differ on the independent variable of interest. 52 This allows the researcher to “hold constant” the other characteristics’ effects. 53 In Judicial Comparativism and Judicial Diplomacy , Professor David Law uses a most similar case design to explore why some courts use foreign law more than others. 54 Law hypothesizes that a court’s institutional capacity to learn about foreign law, and the emphasis a legal education system places on foreign law, shapes a court’s use of foreign law. 55        

Law selects the Japanese Supreme Court, the Korean Constitutional Court, and the Taiwanese Constitutional Court because they share characteristics that potentially explain judicial engagement in comparativism. 56 These countries are geographically adjacent, are democratic, share security and economic alliances with the United States, train judges similarly, have German-influenced civil law systems, have comparable popular attitudes toward comparativism, and share welcoming attitudes toward foreign law. 57

Despite their similarities, these courts differ on the outcome and explanatory variables of interest, namely, the court’s use of foreign law, the court’s institutional capacity for comparativism, and the use of comparativism in legal education. The use of foreign law by Japan’s highest court is minimal relative to Korea’s Constitutional Court, which draws on foreign law in a majority of cases, 58 and to Taiwan’s Constitutional Court, which consults foreign constitutional materials almost automatically. 59 While neither the Japanese justices nor their clerks conduct foreign legal research routinely, 60 the Korean Court has extensive foreign law research mechanisms, including a research institute for comparative constitutional scholarship. 61 Moreover, each country’s legal education system emphasizes comparativism differently. In top South Korean and Taiwanese universities, all constitutional law professors studied law abroad, compared to 25 percent to 66 percent in top Japanese universities. 62 While law professors regularly work for the Korean Constitutional Court 63 and a majority of the Taiwanese Constitutional Court justices are former legal professors, Japanese professors rarely hold seats on Japan’s Supreme Court. 64 By using most similar case design, Law effectively isolates important differences between the countries at issue, demonstrating how the highlighted differences influence judicial usage of foreign law. 65

3.   Variants on most similar case design.

Variants on most similar case design have distinct advantages for assessing claims that are of particular interest to legal scholars, such as whether particular legal devices are necessary or sufficient to produce an outcome of interest. For example, many legal scholars want to know whether particular legal rules are essential for well-functioning markets, effective political participation, or robust environmental protection. Similarly, many legal scholars wonder whether adopting similar laws (for example, a model code) in different jurisdictions will result in largely similar outcomes.

In Private Enforcement of Corporate Law: An Empirical Comparison of the United Kingdom and the United States , Professors John Armour, Bernard Black, Brian Cheffins, and Richard Nolan use a variation of most similar case design to assess whether formal private enforcement of corporate law is necessary for strong securities markets. 66 The authors select the United States and the United Kingdom because they share similar values on important characteristics. 67 “Both are common-law jurisdictions with strong judiciaries, low levels of government corruption, [ ] highly developed stock markets,” liquid securities markets, and many publicly traded firms. 68

The authors argue that, “[i]f private enforcement is [indeed] essential for robust stock markets,” they should observe “vigorous private enforcement of corporate law in both” countries, as these countries are otherwise similar in relevant respects. 69 The rate of private enforcement, however, drastically differs. The United States possesses a relatively high frequency of suits brought against directors of public companies. These suits are almost nonexistent in the United Kingdom. 70 By selecting cases that share otherwise-similar characteristics and outcomes, Armour and his coauthors trace back from the outcome and determine if the development of strong stock markets depends crucially on the private enforcement of corporate law. By showing that, contrary to expectations, private enforcement is not present in both cases, the authors effectively eliminate this as an essential precondition for strong securities markets.

Variations of most similar case design are also useful for legal scholars evaluating whether similar legal frameworks are used in the same way, or produce similar effects, across contexts. In How Dispute Resolution System Design Matters , Professor Shauhin Talesh examines why California and Vermont consumers receive different protections despite the fact that these states have nearly identical automobile consumer protection laws, or “lemon laws.” 71

Starting with nearly identical lemon laws, Talesh identifies differences between the contexts that could influence the implementation of these laws. Talesh finds that California and Vermont vary in terms of public and private control of dispute resolution structures. 72 In California, disputes are resolved in forums funded by automobile manufacturers but operated by external third-party organizations. 73 In Vermont, consumer disputes are resolved in a state-operated dispute resolution structure. 74 These dispute resolution structures filter business and consumer preferences differently, giving similar lemon laws distinct meanings. California’s managerial-justice adjudicatory model stresses business values of efficiency and managerial discretion. Vermont, by contrast, uses a collaborative justice model that reflects consumer values. 75

It is not only similarly structured laws, but also identical words, that are interpreted in very different ways. For example, both Vermont and California emphasize impartiality and neutrality in the fact-finding process; however, these words’ meanings differ across states. In California, arbitrators who actively investigate facts “compromise” impartiality and neutrality, while Vermont arbitrators must actively investigate facts to establish impartiality and neutrality. 76 This distinction leads California arbitrators to provide advantages for businesses, while Vermont arbitrators favor consumers. 77 Ultimately, by selecting cases with similar laws yet different outcomes, Talesh effectively establishes the critical role of varied implementation. 78

4.   Most different case design.

In most different case design, researchers select cases that differ on all relevant characteristics except the explanatory variable and outcome. 79 As such, most different case designs can suggest that the same variable produces the same effect across extremely different contexts. In The Euro-Crisis and the Courts: Judicial Review and the Political Process in Comparative Perspective , Professor Federico Fabbrini argues that, in response to the European debt crisis (the Euro-crisis) and new legal architecture of the Economic and Monetary Union (EMU), European courts have increased their involvement in the fiscal domain. 80

Fabbrini compares high court judicial decisions in Estonia, France, Germany, Ireland, and Portugal, highlighting that these five member states represent the very diverse political, economic, and legal conditions that characterize the European Union (EU). 81 These countries vary dramatically: not only in size, wealth, and culture, but also in terms of the length of their EU membership and the power available to their supreme courts to review legislation. 82

Drawing from post-Euro-crisis court rulings, Fabbrini identifies a common cause of this increasingly high degree of judicial intervention in fiscal and economic affairs: EU member states’ intergovernmental management of the Euro-crisis. 83 As the dominant decision-making bodies, EU member states’ executive branches reformed the EMU architecture via international agreements, allowing courts to influence fiscal reform. 84 By using most different case logic, Fabbrini emphasizes the common cause of the increase in judicial involvement in economic affairs, thereby increasing the credibility and generalizability of his argument. However, most different case design has important limitations: when selected cases share more than one relevant similarity, this technique cannot, on its own, help the researcher distinguish between them. More generally, qualitative work requires that case selection be combined with within-case analysis, to which we turn next.

III.  Process Tracing: Developing Multiple Empirical Implications

After imagining alternative plausible outcomes and selecting cases, qualitatively oriented scholars trace the events prior to the outcome, parsing their theory into logically interconnected propositions that explain why the outcome occurred. If a legal scholar attributes an outcome to a particular cause, it is reasonable to think that this cause would produce other “traces,” or implications. Using available evidence, this scholar can see whether these expected implications actually occurred, thereby strengthening (or weakening) her explanation of the outcome. Additionally, scholars can weigh the plausibility of these implications against alternative explanations of the outcome. 85

The logic of process tracing should not be unfamiliar to lawyers; similar logic is used to assemble evidence in individual cases. In process tracing, scholars form multiple hypotheses about what caused an outcome, identify implications of each hypothesis, and weigh the hypotheses against available evidence. Similarly, to link a suspect to a crime, a prosecutor identifies a motive and develops a theory connecting a suspect’s motive to the time, place, and method of the crime. The prosecutor examines whether the evidence is more consistent with her theory or alternative theories. Evidence will vary in probative value; for example, eyewitness testimony might be less definitive than DNA evidence. 86 Although lawyers “process trace” when composing legal briefs and establishing narrow causal propositions, legal scholars do not use this logic systematically in law review writing. That is, in brief writing, lawyers often assess how diverse facts contribute to their legal arguments, but in academic writing, we often see less effort spent to collect and assess key facts that would make theoretical propositions plausible.

After developing a theoretical explanation of the outcome, scholars using process tracing must assess how diagnostic evidence increases or decreases the probability that this explanation is true. These pieces of diagnostic evidence are called causal process observations (CPOs) because they elucidate the broader causal mechanism linking the variables. 87 These pieces of evidence differ from the independent observations used in statistical analyses; they do not add breadth but depth, and are logically connected, rather than independent of one another. Different types of CPOs have varying probative value. In Professor David Collier’s language, “doubly decisive” evidence and “smoking gun” evidence have high probative value: doubly decisive evidence supports one theory and discredits alternatives, while smoking gun evidence supports one theory but does not speak to alternatives. 88 In contrast, “straw-in-the-wind” evidence and “hoop” evidence are only mildly helpful. 89

Below we provide two applications of process tracing to show how it can assess different types of causal arguments using various legal sources. We distinguish theoretically between (a) testing a theory with multiple empirical implications connected chronologically, and (b) testing a particular type of chronological connection common in legal scholarship—path-dependent processes 90 —in which early events have unusually large consequences later on.

A.    Process Tracing When Observations Are Linked Temporally

Researchers can effectively use process tracing to evaluate theories with chronologically connected empirical implications. To do so, the researcher breaks down her explanation of an outcome into various sequential, causal propositions, and evaluates these propositions against temporally interlinked observations. In The Strength of a Weak Agency , Professors Nicholas Pedriana and Robin Stryker explain how social movement pressure can expand the capacity of an agency with a small staff, limited budget, and limited jurisdiction. 91 Specifically, they highlight how the NAACP and Legal Defense Fund (LDF) pressured the Equal Employment Opportunity Commission (EEOC) to aggressively interpret Title VII, 92 thereby expanding the agency’s powers. 93 While political leaders and lawyers initially understood Title VII as prohibiting only intentional discrimination, social movement pressure forced an aggressive EEOC litigation strategy, culminating in Griggs v Duke Power Co , 94 which prohibited unintentional discrimination. 95

Pedriana and Stryker’s first proposition involves social movements flooding the EEOC with complaints to demonstrate that the agency’s existing resources and capacity were insufficient. 96 Next, early EEOC leaders disagreed about expanding the agency’s mission, leading the EEOC to pursue interpretations the agency’s leaders understood as very aggressive. 97 This set of propositions has relatively distinctive empirical implications, and helps Pedriana and Stryker distinguish their theory from alternative explanations. One possible alternative is that EEOC leadership, seeking to increase their powers, would have pursued an expansive mandate even without social movement pressure. 98 Or perhaps the premise that the EEOC had an initial narrow mandate is incorrect. 99 Alternatively, perhaps the Supreme Court would have decided Griggs similarly regardless of social movement pressure and EEOC advocacy. 100

To reject the alternative explanation that power-seeking bureaucrats drove EEOC expansion, the authors highlight that the first EEOC chairman, Franklin Delano Roosevelt Jr, was yachting during congressional hearings regarding appropriations for his agency. 101 Roosevelt focused on public relations because he wanted to run for governor of New York, leaving EEOC senior staff unsure about the agency’s central objectives and how to accomplish them. 102

To evaluate their proposition that social movements exposed the EEOC’s ineffectiveness, thereby pressuring the EEOC to adopt an aggressive strategy, Pedriana and Stryker note that the NAACP and the LDF filed mass complaints in the months after Title VII came into force. 103 Jack Greenberg, director of the LDF, publicly stated that “the best way to get it amended [Title VII] is to show it doesn’t work.” 104 Throughout its initial years, the EEOC was continually handling at least four times the number of complaints it was budgeted to handle due to the unrelenting tide of complaints from the LDF and the NAACP. 105 The volume of complaints and social movement leaders’ statements are, in the language of Collier’s classification structure, “smoking gun” evidence. Given this evidence, it would be surprising if the alternate explanation—that social movement pressure had no effect on the EEOC—were true.

To evaluate their proposition that there was a push to expand the EEOC’s mandate, Pedriana and Stryker show that EEOC leadership initially disagreed over whether Title VII covered intentional discrimination and discriminatory effects. 106 Pedriana and Stryker first follow steps that legal scholars normally use: they draw from the text of Title VII, the legislative history of the statute, and statements made by the nonpartisan Bureau of National Affairs. 107 Perhaps recognizing the potential for strategic use of the legislative record, Pedriana and Stryker also draw on EEOC internal communications and staff statements. 108 Although the EEOC later (successfully) challenged employment tests as discriminatory based on statistical evidence of their impact on minority applicants, the EEOC’s general counsel initially stated that “if [the EEOC testing guidelines are] intended as a legal position as to what is meant by professionally developed tests then it is very wide off the mark . . . I cannot conceive arguing this position before a District judge .” 109 Additionally, EEOC Executive Director Herman Edelsberg said that incorporating disparate impact into the guidelines would make them “too ambitious to be a legal document.” 110 Again, this is smoking gun evidence; it would be very surprising if the alternate explanation—that the EEOC’s mandate was unquestionably broad—were true given this evidence.

Pedriana and Stryker demonstrate how legal scholars can develop temporally linked propositions with distinctive empirical signatures, and how evaluating these propositions against available evidence can substantially increase their persuasiveness. We now turn to path-dependent causal claims and explain how best to substantiate them.

B.    Process Tracing When Observations Are Path Dependent

Legal scholars commonly make claims about path depen­dence, processes in which early events have large consequences later on. A HeinOnline search showed that 2,662 articles mentioned path dependence explicitly from 2000 to 2015. Legal interpretation techniques, including rules governing precedents, analogical reasoning, and conventions about interpreting similar language systematically, make early judicial decisions crucial. Below we explain why process tracing can help develop path-dependent claims. 111

What distinguishes path dependence from other claims about event sequence? First, in path-dependent processes, positive feedback loops make early events have bigger consequences than later ones. 112 Second, path-dependent processes have critical junctures, when one option is picked among many; after this choice, it becomes increasingly difficult to return to alternatives. 113 The adoption of the QWERTY keyboard effectively illustrates path dependence. While countless ways of arranging letters on a keyboard were initially possible, once the QWERTY sequence was chosen and adopted by millions of typists, it became nearly impossible to switch to another, more efficient arrangement.

Process-tracing techniques are very useful for identifying feedback loops and critical junctures. 114 In The Lost Promise of Civil Rights , Professor Risa Goluboff explains how the NAACP adopted the now-dominant civil rights litigation strategy and why it concentrated on government-imposed segregation rather than challenging abysmal labor conditions, an alternate strategy championed by the Civil Rights Section (CRS) of the Justice Department. 115 Goluboff theorizes that early legal victories encouraged similar litigation and subsequent victories, creating a positive feedback loop that institutionalized this litigation strategy, making alternative litigation strategies much harder to pursue later on. 116

To establish that an event constitutes a critical juncture, a scholar must demonstrate that there were at least two alternatives available and that, after one alternative was chosen, it became increasingly difficult to return to the other option. Goluboff does this for key decisions in the 1930s and 1940s. 117 She also establishes that, once the NAACP chose its litigation strategy, choices about the cases it selected made it difficult, if not impossible, to change. Initially, the NAACP received both racial discrimination complaints from northern industrial workers and labor discrimination complaints from southern agricultural workers. 118 While the NAACP originally pursued both types of complaints, by the 1940s, the NAACP fashioned a legal strategy around the racial discrimination claims of industrial workers. 119 Multiple factors influenced this decision. The NAACP relied heavily on local counsel, and in the 1940s most black lawyers were in northern cities. 120 Additionally, the NAACP found that “sympathetic judges and amenable lawyers” were scarce in the south, making it “easier to win cases” in the north. 121

Perhaps the biggest critical juncture was the Supreme Court’s decision in Brown , which vindicated the NAACP’s legal strategy and established equal protection as the dominant civil rights lens. 122 Brown is perhaps the most significant US Supreme Court case; the antidiscrimination framework Brown and its progeny represent is common in casebooks and taught across law schools nationally. 123 While establishing the antidiscrimination approach’s dominance is easy, it is challenging for legal scholars to imagine that an alternative vision was possible. Goluboff convincingly establishes this alternate vision in a number of ways. Goluboff develops a plausible, alternate legal vision championed by the CRS: raw legal material for an alternate vision of civil rights, namely, agricultural workers’ horrific complaints, was ample, 124 allowing the CRS to develop a conception of civil rights based on labor and economic discrimination. 125 Additionally, she highlights that the Supreme Court overturned its own precedents with unusual frequency throughout the 1930s and 1940s 126 and presents comments from prominent civil rights lawyers and casebooks exemplifying their perceptions of ambiguity in civil rights doctrine. 127 This is smoking gun evidence because it makes it highly unlikely that the Brown decision was inevitable.

Implications and Conclusions

In place of a conclusion, we speculate on an observation that transformed quantitative research. In a much-cited 1986 piece, Paul Holland argued that some questions can be answered much more easily than others. 128 For example, it is very difficult to ascertain why people commit crimes; however, we can more easily determine whether expanding the police force reduces crime rates. Statistical analysis, Holland argued, has distinct advantages for answering the second type of question, which focuses on measuring the effect of a given variable. 129 The ease and effectiveness with which statistical analyses can answer “effects-driven” questions have led this method and question type to dominate social science research. More and more, social scientists are asking answerable questions with quantitative methods; however, fewer reflect on whether these questions, while answerable, are interesting and contribute to our understanding of the world.

Legal scholars arguably face the opposite problem. Legal scholarship has no shortage of interesting questions. However, many of these critical questions are never answered; legal scholars rarely defend their preferred theories against plausible alternatives effectively. By showcasing a variety of methodological techniques that are well suited to the types of claims and evidence legal scholars typically work with, we hope to move closer to answering the critically important questions legal scholars pose.

• 16 Kristin Luker, Salsa Dancing into the Social Sciences: Research in an Age of Info-Glut 55 (Harvard 2008).
• 17 See Hendrik Hartog, Pigs and Positivism , 1985 Wis L Rev 899, 904–06.
• 18 Id at 905–06, 908–09.
• 19 Id at 912–13.
• 20 See id at 913, 935.
• 21 See Hartog, 1985 Wis L Rev at 912–15, 929–35 (cited in note 17).
• 22 See generally Rachel Kahn Best, et al, Multiple Disadvantages: An Empirical Test of Intersectionality Theory in EEO Litigation , 45 L & Society Rev 991 (2011).
• 23 Id at 999–1000, 1017–19.
• 24 Id at 1009 (noting that the employee wins a clear victory in 15 percent of cases with intersectional bases of discrimination, as opposed to 30 percent of cases with nonintersectional bases of discrimination).
• 25 Id at 999.
• 26 Best, et al, 45 L & Society Rev at 999 (cited in note 22).
• 27 Id at 999 & nn 4–5.
• 28 See id at 1000 & n 8.
• 29 Id at 999.
• 30 See, for example, Paul D. Halliday, Habeas Corpus: From England to Empire 28–29, 319 (Belknap 2010).
• 31 Id at 4.
• 32 Id at 319.
• 33 See id at 5.
• 34 Halliday, Habeas Corpus at 319 (cited in note 30).
• 35 See Sarah Curtis, et al, Approaches to Sampling and Case Selection in Qualitative Research: Examples in the Geography of Health , 50 Soc Sci & Med 1001, 1002 (2000) (discussing the theoretical framework for case selection).
• 36 Katerina Linos, How to Select and Develop International Law Case Studies: Lessons from Comparative Law and Comparative Politics , 109 Am J Intl L 475, 480 (2015).
• 37 See Jason Dietrich, The Effects of Sampling Strategies on the Small Sample Properties of the Logit Estimator , 32 J Applied Stat 543, 544 (2005) (“On average, simple random sampling yields a sample reflecting the true population distributions. . . . For smaller samples, however, there is an increased risk that the model cannot be estimated because of limited variation in either the dependent or independent variables.”).
• 38 Robert L. Nelson and William P. Bridges, Legalizing Gender Inequality: Courts, Markets, and Unequal Pay for Women in America 2 (Cambridge 1999).
• 39 See id at 2–3.
• 40 Id at 102, 105–08.
• 41 Id at 102.
• 42 Nelson and Bridges, Legalizing Gender Inequality at 108 (cited in note 38).
• 44 This last step distinguishes theoretically informed sampling from stratified sampling. In stratified sampling, cases are picked at random within each stratum; in theoretically informed sampling, researchers select cases within each stratum. See id at 109–10.
• 45 Id at 112.
• 46 Nelson and Bridges, Legalizing Gender Inequality at 112–13 (cited in note 38).
• 47 Harry Eckstein, Case Study and Theory in Political Science , in Fred I. Greenstein and Nelson W. Polsby, eds, 7 Handbook of Political Science: Strategies of Inquiry 79, 119 (Addison-Wesley 1975). See also Jack S. Levy, Case Studies: Types, Designs, and Logics of Inference , 25 Conflict Mgmt & Peace Sci 1, 12 (2008).
• 48 See Levy, 25 Conflict Mgmt & Peace Sci at 12 (cited in note 47).
• 49 Rosenberg, The Hollow Hope at 420 (cited in note 5).
• 50 See id at 7.
• 51 Id at 8.
• 52 See Jason Seawright and John Gerring, Case Selection Techniques in Case Study Research: A Menu of Qualitative and Quantitative Options , 61 Polit Rsrch Q 294, 304 (2008).
• 53 For an in-depth description of most similar case selection, see Ran Hirschl, The Question of Case Selection in Comparative Constitutional Law , 53 Am J Comp L 125, 133–39 (2005).
• 54 See generally David S. Law, Judicial Comparativism and Judicial Diplomacy , 163 U Pa L Rev 927 (2015).
• 55 Id at 942.
• 56 Id at 942–43, 949–50.
• 57 Id at 950.
• 58 Law, 163 U Pa L Rev at 953, 962 (cited in note 54).
• 59 See id at 977.
• 60 Id at 953–54.
• 61 Id at 972–73, 1033.
• 62 Law, 163 U Pa L Rev at 1035 (cited in note 54).
• 63 See id at 964, 970–71.
• 64 Id at 1012–13.
• 65 Id at 949–52.
• 66 See generally John Armour, et al, Private Enforcement of Corporate Law: An Empirical Comparison of the United Kingdom and the United States , 6 J Empirical Legal Stud 687 (2009).
• 67 See id at 692.
• 68 Id at 689, 692 (citation omitted).
• 69 Id at 692.
• 70 Armour, et al, 6 J Empirical Legal Stud at 690 (cited in note 66).
• 71 See generally Shauhin A. Talesh, How Dispute Resolution System Design Matters: An Organizational Analysis of Dispute Resolution Structures and Consumer Lemon Laws , 46 L & Society Rev 463 (2012).
• 72 Id at 466–68.
• 73 Id at 464.
• 74 Id at 464–65.
• 75 Talesh, 46 L & Society Rev at 474 (cited in note 71).
• 76 See id at 478–80.
• 77 See id at 478.
• 78 Id at 483–89.
• 79 See Seawright and Gerring, 61 Polit Rsrch Q at 306 (cited in note 52).
• 80 Federico Fabbrini, The Euro-Crisis and the Courts: Judicial Review and the Political Process in Comparative Perspective , 32 Berkeley J Intl L 64, 65 (2014).
• 81 See id at 75–76.
• 83 Id at 65.
• 84 Fabbrini, 32 Berkeley J Intl L at 65 (cited in note 80).
• 85 See Lawrence B. Mohr, The Reliability of the Case Study as a Source of Information , 2 Advances Info Processing Orgs 65, 67–69 (1985).
• 86 However, for a critique of the reliability of DNA evidence, see generally Andrea Roth, Maryland v. King and the Wonderful, Horrible DNA Revolution in Law Enforcement , 11 Ohio St J Crim L 295 (2013).
• 87 David Collier, Understanding Process Tracing , 44 PS: Polit Sci & Polit 823, 826 (2011).
• 88 Id at 825.
• 89 “Straw-in-the-wind” evidence does not prove or disprove a theory, but suggests that its validity is more likely than it would otherwise be. “Hoop” evidence can disprove a theory but cannot independently establish its validity. Id.
• 90 For an excellent example of how to effectively use process tracing, see Tasha Fairfield, Going Where the Money Is: Strategies for Taxing Economic Elites in Unequal Democracies , 47 World Development 42, 46–51 (2013).
• 91 See generally Nicholas Pedriana and Robin Stryker, The Strength of a Weak Agency: Enforcement of Title VII of the 1964 Civil Rights Act and the Expansion of State Capacity, 1965–1971 , 110 Am J Sociology 709 (2004).
• 92 Title VII of the Civil Rights Act of 1964, Pub L No 88-352, 78 Stat 253, codified at 42 USC § 2000e et seq.
• 93 Pedriana and Stryker, 110 Am J Sociology at 710–11, 725–27 (cited in note 91).
• 94 401 US 424 (1971).
• 95 Id at 431 (holding that “[t]he Act proscribes not only overt discrimination but also practices that are fair in form, but discriminatory in operation”).
• 96 Pedriana and Stryker, 110 Am J Sociology at 725 (cited in note 91).
• 97 See id at 721.
• 98 Id at 720.
• 99 See id at 729.
• 100 Pedriana and Stryker, 110 Am J Sociology at 739–40, 748 (cited in note 91).
• 101 Id at 721.
• 103 Id at 725.
• 104 Pedriana and Stryker, 110 Am J Sociology at 725 (cited in note 91) (brackets in original).
• 105 See id.
• 106 See id at 728–30.
• 107 See id at 723, 726.
• 108 See Pedriana and Stryker, 110 Am J Sociology at 723 (cited in note 91).
• 109 Id at 735 (brackets and ellipsis in original).
• 111 For another example of process tracing to establish path dependence, see generally Katerina Linos, Path Dependence in Discrimination Law: Employment Cases in the United States and the European Union , 35 Yale J Intl L 115 (2010).
• 112 See Paul Pierson, Increasing Returns, Path Dependence, and the Study of Politics , 94 Am Polit Sci Rev 251, 251–52 (2000).
• 113 James Mahoney, Path Dependence in Historical Sociology , 29 Theory & Society 507, 513 (2000).
• 114 See Giovanni Capoccia and R. Daniel Kelemen, The Study of Critical Junctures: Theory, Narrative, and Counterfactuals in Historical Institutionalism , 59 World Polit 341, 343, 358–59 (2007).
• 115 See Risa L. Goluboff, The Lost Promise of Civil Rights 175–76 (Harvard 2007).
• 116 See id at 217–37.
• 117 See id at 174–237.
• 118 See id at 81–82.  
• 119 See Goluboff, The Lost Promise at 197 (cited in note 115).
• 120 Id at 187.
• 122 See id at 238–70.
• 123 See, for example, Erwin Chemerinsky, Constitutional Law: Principles and Policies 722–25 (Wolters Kluwer 4th ed 2011).
• 124 See Goluboff, The Lost Promise at 81–84, 175–76 (cited in note 115).
• 125 Id at 112–14 (“The CRS maintained its original commitment to the rights of labor and reworked, rather than rejected, labor rights into its new civil rights practice during the 1940s.”).
• 126 See id at 23.
• 127 Id at 111–12.
• 128 See generally Paul W. Holland, Statistics and Causal Inference , 81 J Am Stat Assn 945 (1986).
• 129 See id at 945–48.

Thanks to Jake Gersen, Todd Henderson, Daryl Levinson, Jens Ludwig, Richard McAdams, Tom Miles, Matthew Stephenson, David Strauss, Adrian Vermeule, Noah Zatz, and participants at a workshop at The University of Chicago Law School for helpful comments.

We are grateful to Susan Bandes, Elizabeth Foote, Jacob Gersen, Brian Leiter, Anup Malani, Richard McAdams, Elizabeth Mertz, Jonathan Nash, Eric Posner, Adam Samaha, Larry Solum, David Strauss, Noah Zatz, and participants in a work-inprogress lunch at The University of Chicago Law School for valuable comments. We are also grateful to the Chicago Judges Project, and in particular to Dean Saul Levmore, for relevant support.

We thank Eric Posner, Richard Posner, Peter Strauss, and Adrian Vermeule for helpful comments. We are also grateful to Rachael Dizard, Casey Fronk, Darius Horton, Matthew Johnson, Bryan Mulder, Brett Reynolds, Matthew Tokson, and Adam Wells for superb research assistance.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Qualitative Methods for Law and Society—A Research Guide

This research guide is intended as a starting point for doctoral researchers in the EUI Department of Law who plan to (or hope to) draw on socio-political, anthropological or historical methodologies as part of their dissertation research. Ultimately, the methods, methodology, and the boundaries of the research project are produced dynamically by the researcher, so this is no more than a starting point, a set of suggestions rather than a book of recipes. All sources listed are available either through the EUI library or free online.

Related Papers

Lisa Webley

Tonny Nyarko

Canon of legal research: LAW AS AN ACADEMIC DISCIPLINE, HANOCH DAGAN: MAPPING LEGAL RESEARCHMATHIAS M. SIEMS AND DAITHI´ MAC SI´THIGH: INTERDISCIPLINARY APPROACH TO LEGAL SCHORLASHIP: A BLEND FROM THE QUALITATIVE PARADIGM, A. LYDIA A. ANKANSAH AND VICTOR CHIMBWANDA

Erasmus Law Review

Danielle Chevalier

This seminar focuses on qualitative methods in the social sciences. It is structured as a survey course, exposing students to a range of issues, rather than intensive training in a single approach. The purpose of the seminar is twofold: First, to provide participants with a broad sense of qualitative research strategies, a better understanding of how to design and carry out research, an awareness of the different logics and trade-offs that distinguish methodologies and methods and an improved capacity to read and evaluate diverse qualitative social science research. Second, to write a dissertation proposal that will be competitive for various external dissertation fellowship funders—such as NSF, Fulbright, SSRC, etc.—and defensible before one’s dissertation committee.

Darren O’Donovan

This chapter will focus upon conceptually mapping the place of socio-legal methodology within legal research. Questions to be addressed include: what are the underlying theories regarding the nature of law and legal argument underpinning this form of scholarship? How do we understand the position of law in relation to the general social sciences? Having located this methodological school, I will then proceed to consider what reasons students or researchers might have for engaging in socio-legal research. This will be achieved by discussing five major strands of socio-legal research and how they seek to make distinctive contributions to knowledge. It will be shown that socio-legal scholarship has challenged doctrinal legal research culture by questioning the assumed centrality of law and legal institutions to many social problems. It has sought to present a more complex understanding of 'how legal rules, doctrines, legal decisions, institutionalised cultural and legal practices work together to create the reality of law in action'. 1 As a result, the proponents of the methodology have successfully challenged legal scholars to display greater policy imagination, by acknowledging law's status as just one form of regulation, and cautioning against overly doctrinal understandings of the discipline.

Juliette Galonnier

This course aims at introducing students to qualitative methods in the social sciences. It highlights the contributions of qualitative research to the study of the State, public policy and institutions, while also exploring its limitations. By the end of the course, students will be familiar with a range of qualitative methods, including interviews, ethnography, archival research, focus groups, international comparison, text analysis and the use of leaked documents. They will read key authors who use qualitative methods to examine the crafting of public policies, the history of institutions or the workings of international organizations. The selected readings focus mostly on Europe, North America and the MENA region. Students will collectively engage with the strengths and challenges of qualitative methods. They will conduct their own qualitative research on one public policy or institution of their choice and will have the opportunity to reflect on the practical obstacles and opportunities that these methods raise on the field.

Kristina Simion

Qualitative and quantitative research can lay the basis for rule of law interventions that are rooted in sound evidence and responsive to local community interests, aspirations, values, and demands. Without grounded knowledge of qualitative and quantitative research, researchers' results can easily be erroneous (as a result of, for example, poorly designed interview protocols and questionnaires). Indeed, it is an unfortunate truth that rule of law interventions are continually critiqued for being planned on the basis of inadequate research and information, and for producing unsatisfactory results. INPROL's new Practitioner's Guide on Qualitative and Quantitative Approaches to Rule of Law Research was drafted to assists practitioners in structuring research. It clarifies common research terminology and concepts, and outlines the steps involved in designing and implementing qualitative and quantitative research. The Guide recognizes that high-quality research is an essential element of the design and evaluation of rule of law programs. It is also a useful way of enhancing a practitioner's personal information needs, as conducting rule of law research can be overwhelming for the practitioner who has little previous experience. Where do you start? What components do you need to factor into your plans? What kind of research do you need to conduct? These difficult questions are even harder to address in a conflict-affected environment, where access to research participants (i.e., the people participating in research) may be difficult; information may be scarce and difficult to evaluate; and the researcher may find it hard to travel because of security risks.

Emilio Dabed

Theoretical approaches and methodological choices in the anthropological study of the “legal” share the assumption that normative phenomena and, more specifically, law is a social product, carrying the traces of the context in which they are produced. In this sense, norms/law can be understood as “metaphoric representation” of their social and political context. At the same time, anthropological legal research has looked abundantly at the ways in which law participates in the creation of social reality. In assessing the “performative” impact of juridical phenomena, or what Bourdieu refers to as the “power of law”, an analysis of the relation between legal processes, discursive practice and political and social changes is imperative. The central argument is, thus, that juridical phenomena “not only reflect but also produce and reinforce social processes”.

Armando Marques-Guedes

RELATED PAPERS

Nihmar Tetembu

Vincent Cantin

J. Mater. Chem. C

Amreen Hussain

Swiss Medical Forum ‒ Schweizerisches Medizin-Forum

Sofia Bashir

Osteoporosis International

Scott Going

Jurnal Warna : Pendidikan dan Pembelajaran Anak Usia Dini

maria evivani

Journal of Pediatric Surgery

Bello B Shehu

A Educação no Brasil e no Mundo: Avanços, Limites e Contradições 5

Maria Regina Momesso

Instituto de Geografia Tropical (IGT)

Alfred Pohlan , Plantation du Beauregard

Jung-Woo Ha

Jack Higginbotham

Francesco Piraino

Hevelyn Rocha

Revista de Administração Contemporânea

Marcelo Machado

Development in Practice

Getachew Mequanent

Hábitat y Sociedad

JUAN FRANCISCO OJEDA RIVERA

maryam javidi

Food and Nutrition Bulletin

Diabetes Research and Clinical Practice

Medhat Shazly

Desvendando a Engenharia: sua abrangência e multidisciplinaridade

Ruth Queiroz

Ian Bolliger

American Journal of Hematology

Krzysztof Warzocha

RePEc: Research Papers in Economics

Christian Larrain

Dermatologic Therapy

Abdelrahman Fathy

See More Documents Like This

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

Empirical Legal Research Resources

• Statistics and Datasets
• Journals and Law Reviews

Selected Treatises: How to Conduct Empirical Legal Research

Representative empirical legal research works.

• Data Analytics Help and Resources at Stanford
• Current Awareness and Training Resources
• Bulk Download Availability

Ask us anything!

Law librarians and library staff are available to help you find your way around the library and its legal materials, provide research support, and assist you in locating materials that are not in our library. You can find the latest information about our hours and services on the library homepage . There are several ways to contact us:

• [email protected]
• Recommend a Resource

In Person : The Borrowing Services Desk and Reference Office are on the main floor of the law library. Online and in-person appointments with the reference staff can be made from the library homepage . 

Borrowing Services Desk: (650) 723-2477 Reference Desk: (650) 725-0800

This page contains a selected list of major treatises on conducting empirical legal research, as well as a selection of books that demonstrate how some scholars use empirical legal research in their writing. You may find more works on the topic using subject headings, such as "legal research -- methodology," in Searchworks , the Stanford Library catalog.

• << Previous: Journals and Law Reviews
• Next: Data Analytics Help and Resources at Stanford >>
• Last Updated: Dec 15, 2023 1:06 PM
• URL: https://guides.law.stanford.edu/empiricallegalresearch

• Stanford Home
• Maps & Directions
• Search Stanford
• Emergency Info
• Terms of Use
• Non-Discrimination
• Accessibility

© Stanford University , Stanford , California 94305 .

Qualitative Approaches to Empirical Legal Research

133  citations

86  citations

65  citations

63  citations

49  citations

33,113  citations

31,251  citations

27,598  citations

4,867  citations

4,786  citations

Related Papers (5)

Ask Copilot

Related papers

Contributing institutions

Related topics

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Archaeology
• Greek and Roman Papyrology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Agriculture
• History of Education
• History of Emotions
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Acquisition
• Language Variation
• Language Families
• Language Evolution
• Language Reference
• Lexicography
• Linguistic Theories
• Linguistic Typology
• Linguistic Anthropology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Modernism)
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Religion
• Music and Culture
• Music and Media
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Science
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Oncology
• Medical Toxicology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Clinical Neuroscience
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Ethics
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Neuroscience
• Cognitive Psychology
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Strategy
• Business History
• Business Ethics
• Business and Government
• Business and Technology
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic Systems
• Economic Methodology
• Economic History
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Theory
• Political Behaviour
• Political Economy
• Political Institutions
• Politics and Law
• Public Administration
• Public Policy
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

38 Quantitative Approaches to Empirical Legal Research

Lee Epstein is the Ethan A. H. Shepley Distinguished University Professor at Washington University in St. Louis.

Andrew D. Martin is Professor of Law and Political Science at Washington University in St. Louis.

• Published: 18 September 2012
• Cite Icon Cite
• Permissions Icon Permissions

This article deals with the objective nuances of empirical research, within the ambit of the quantitative kind. It begins with an overview of conducting empirical legal research, discussing its research design, implementation, and challenges faced. Theorizing in empirical legal scholarship comes in different forms: in some projects theories seek to provide insight into a wide range of phenomena, others are tailored to fit particular situations. In the clarification process the researcher translates abstract notions into concrete ones. To convert the data into an analyzable form, empirical legal researchers make use of a variety of data-generation mechanisms. Researchers can implement random sampling in various ways depending on the nature of the problem. Data analysis enables researchers to compare their overlap. The goal of empirical legal research is to find facts about the unknown. The last step of empirical legal research is to present its results, for which, documentation is a requisite.

Conducting Empirical Legal Research: An Overview 902

Designing Research   905

Collecting Data and Coding Variables   909

Analyzing Data   912

The Last Step: Presenting the Results of Empirical Legal Research   917

T he title of this Chapter seems too wordy. Why call it doing “empirical legal research,” and not simply doing “empirical research”? After all, regardless of whether empirical researchers are addressing a legal question or any other, they follow the same rules—the rules that enable them to draw inferences from the data they have collected (Epstein and King, 2002 ; King et al., 1994 ). What's more, because empirical research in law has methodological concerns that overlap with those in Biology, Chemistry, Economics, Medicine and Public Health, Political Science, Psychology, and Sociology, empirical legal researchers can adopt methods from these other disciplines to suit their own purposes.

On the other hand, in virtually every discipline that has developed a serious empirical research program—law not excepted—scholars discover methodological problems that are unique to the special concerns in that area. Each new data source often requires at least some adaptation of existing methods, and sometimes the development of new methods altogether. There is bioinformatics within Biology, biostatistics and epidemiology within Medicine and Public Health, econometrics within Economics, chemometrics within Chemistry, political methodology within Political Science, psychometrics within Psychology, sociological methodology within Sociology, and so on. As of this writing, there is no “legalmetrics” but that should happen soon enough (though probably not before this Chapter appears in print).

In short, with a few wording substitutions here and there, much of what follows pertains to all empirical research. But much is not all . Recognizing that empirical legal work is unique in various ways, as we describe the research process we also outline some of the field's distinct challenges—most notably, how to communicate complex statistical results to a community lacking in statistical training.

We begin by describing the research process. Then, in sections II-V, we flesh out the various components of the process: designing research, collecting and coding data, analyzing data, and presenting results.

I. Conducting Empirical Legal Research: An Overview

How do scholars implement quantitative empirical research? What challenges do they confront? To begin to formulate responses, consider a legal question at the center of hundreds, perhaps thousands, of lawsuits each year: do employers pay men more than women solely because of their gender? Next consider how researchers who faced absolutely no constraints—i.e., researchers with more powers than Batman, Superman, and Wonder Woman combined—would address this question. If we were the researchers, we would begin by creating a workplace, randomly drawing a worker from the workforce population, randomly assigning a sex (say, male) to the worker, instructing him to enter the workplace, and observing his wage. 2 Next, we would reverse time, and assign the same worker the other sex (female), send her into exactly the same workplace, and observe her wage. If we observed a difference in the wages of our two workers—such that the same employer paid the male version less than the female version—then we might conclude that, yes, gender causes pay inequities.

Unfortunately, researchers aren't superheroes; they usually don't have the power to create a workplace and assign a sex. And they certainly don't have the power to rerun history. This is known as fundamental problem of causal inference (Holland, 1986 : 947). It simply means that researchers can only observe the factual (e.g., a female worker's salary, if in fact the worker was a female) and not the counterfactual (e.g., a male worker's salary, if in fact the worker was female). 3

This is a problem without a solution but scholars have developed various fixes. The gold standard along these lines is a proper experiment—that is, an experiment in which the researcher randomly selects subjects from the population of interest and then randomly assigns the subjects to treatment and control conditions (see Ho et al., 2007 ). Very few experiments in empirical legal studies actually meet the first condition (random selection from the population) but some scholars have tried to meet the second. Jeffrey J. Rachlinski and his colleagues (2006), for example, recruited 113 bankruptcy court judges to participate in an experiment designed to detect whether the race of a party affected the judges' decisions. 4 They asked the judges to read the same case materials but unbeknownst to the judges, the researchers randomly assigned them to a control or treatment group. Those judges in the control group were led to believe that the debtor was white; those in the treatment group were led to believe that the debtor was black. (It turned out that race did not affect the judges' decisions.)

This is a reasonable approach to the fundamental problem of causal inference. But, sadly, it is infeasible for many empirical legal projects—including studies of pay equity (no experiment can assign a sex to workers). It is not even feasible for most analyses of judicial behavior (the Rachlinski et al. study is a notable exception). To provide but one example, suppose we wanted to investigate the extent to which female judges affect the decisions of their male colleagues. No U.S. Court of Appeals would allow us to manipulate the composition of panels so that we could identify a possible gender effect. We could say the same of the other institutions of government. Can you imagine the President of the United States agreeing to nominate two judicial candidates identical in all respects except that one is highly qualified and the other highly unqualified just to enable us to learn whether qualifications affect the confirmation votes of U.S. senators? We can't.

The upshot is that most empirical legal researchers simply do not have the luxury of analyzing data they developed in an experiment (i.e., experimental data). Instead, they must make use of data the world—not they—generated (i.e., observational data): salaries paid to workers by real companies; the decisions of judges in concrete cases; the votes cast by senators over the president's nominee to the federal courts. And this, of course, substantially complicates the task empirical legal researchers confront. While experimental data—generated by random assignment to treatment and control groups—effectively minimize the confounding effects of other variables, the same cannot be said of observational data. For those data, researchers must invoke statistical techniques (discussed below) to accomplish the same thing.

Because observational datasets are so much more common in quantitative empirical legal research, in what follows we focus on strategies for working with them. It is important to keep in mind, however, that other than issues of data generation and control (statistical versus experimental), experimental and observational studies are not altogether different for our purposes. Either way, scholars tend to execute them in four steps: they design their projects, collect and code data, conduct analyses, and present results. 5

Research design largely (though not exclusively) involves the process of moving from the conceptual to the concrete. To return to our example of pay equity, suppose the researcher hypothesizes that once she takes into account the experience of the workers, males earn no more than females. However plausible this hypothesis, the researcher confronts a non-trivial problem in assessing it: how to operationally define the concept of “experience.” Is it years from degree? Years in the workforce? Months in the same job? More generally, before researchers can answer empirical legal questions—actually before they can even collect the first piece of data—they must devise ways to clarify concepts such as experience so that they can observe them. All of this and more appear on that first (metaphorical) slide.

Data collection and coding entails translating information in a way that researchers can make use of it. For a study of pay equity, the researcher may have piles of pay stubs and employee records. Unless the researcher can transform the piles into data she can analyze the study cannot proceed.

Data analysis typically consists of two activities. First, researchers often summarize the data they have collected. If, for example, we collect information on a sample of 50 workers' salaries in a firm with 500 workers, it may be interesting to know the average salary for the men in our sample and the average salary for the women. Second, analysts use data to make inferences—to use facts they know (about the salaries, gender, experience, and so on of the 50 workers in their sample) to learn about facts they do not know (the salaries, gender, experience, and so on of the 500 workers). To perform inference in quantitative studies, researchers employ various statistical methods. Worth noting, though is that use of statistics presupposes that the study is well designed and the data are of a sufficiently high quality. If either the design is poor or the data inadequate, researchers will be unable to reach inferences of high quality. In other words, without a proper research design no statistical method can provide reliable answers; not even the best statistician cannot make lemonade without lemons.

Finally, once empirical legal analysts have drawn inferences from their data, they must be able to communicate their results to a community that may have little (or no) knowledge of even simple statistics. Doing so effectively blends both art and science, and requires careful consideration of both the project and the intended audience.

These are the contours of the research process. Let us now flesh them out to the extent possible given space constraints.

II. Designing Research

It should go without saying that before researchers can design their project, they must have one. To “have a project” usually means that the analyst has a question she wishes to answer and has theorized about possible responses.

Research questions in empirical legal studies come from everywhere and anywhere. Perhaps scholars see a gap in the existing literature or perhaps they think the literature is incomplete or even wrong. Sometimes questions come from current events—whether a new law is having the desired (or any) effect or whether a court decision is efficacious—and sometimes they come from history. A perusal of any socio-legal journal would provide evidence of these and other motivations.

The variation is not unexpected. Empirical legal scholars are a diverse lot, with equally diverse interests. What their questions have in common, though, may be just as important: virtually all are quite conceptual. Consider a variation on the question we asked at the onset:

Do males and females who have the same level of experience earn the same amount of money?

However important this question, it is not one that even the best empirical legal project can ever address. Rather, the question the study will actually answer comes closer to this:

Do males and females who have been in the workforce for the same number of years net the same salary per month?

Note that the first form of the question contains several concepts—“earn” and “experience”—which researchers cannot directly observe. Only by clarifying these concepts, as the second form does, can the researcher empirically answer the question. Because this is more or less true of every empirical project, a major research challenge is to tighten the fit between the question asked and the question actually answered. If it is too loose the researcher cannot, at the end of the day, claim to have answered the question she initially posed. 6

Once analysts have settled on a research question, they usually begin theorizing about possible answers they can use to develop observable implications (sometimes called hypotheses or expectations). 7 A theory is simply a reasonable and precise answer to the research question. An observable implication is a claim about what we would expect to observe in the real world if our theory is right—typically, a claim that specifies a relationship between (or among) a dependent variable (what we are trying to explain) and an independent variable(s) (what our theory suggests explains the dependent variable) (Epstein and King, 2002 : 61–2).

Theorizing is a big topic, one to which we can hardly do justice in this short Chapter. So two observations will have to suffice. First, theorizing in empirical legal scholarship comes in many different forms: in some projects theories are quite big and grand, seeking to provide insight into a wide range of phenomena (e.g., rational choice theory in law and economics); others are simple, small, or tailored to fit particular situations. For the purposes of conducting an empirical study, this distinction may not be very important.

What is important—and this takes us to the second key point—is that the researcher extract observable implications from the theory. The reason is simple. Just as analysts almost never actually answer the question they pose, they almost never directly test their theory. Rather, they only indirectly assess it by evaluating the observable implications that follow from it.

To see the point, return to our question about pay equity between males and females, and consider the following theories and their observable implications.

Difference Theory Owing to discriminatory judgments about worth, employers pay females less than comparable males. Observable Implication All else being equal (e.g., experience), if my theory is correct, we should observe females earning less than males. Efficiency Theory Because labor markets are efficient, any observed differences between male and female workers are a product of experience, quality, productivity, and so on. Observable Implication All else being equal (e.g., experience), if my theory is correct, we should observe females and males earning the same

Note that in neither instance—no matter how good their design, their data, and their methods—will the researchers be able to conclude that their theory is right or wrong (that discriminatory judgments lead to pay inequity or that efficient markets lead to pay equity). All they will be able to say is whether their data are consistent with the observable implications following from their theory.

And even saying that involves hard work. The problem, yet again, is that observable implications are conceptual claims about the relationship between (or among) variables. To evaluate these, researchers must delineate how they actually can observe them in the real world. They must, in short, move from the abstract to the concrete—a task that forms the core of research design and that Figure 1 depicts. Note that in the clarification process the researcher translates abstract notions, such as “experience” and “earnings,” into the far more concrete “years in the workforce” and “gross annual income.” Unlike the abstractions, researchers can observe and measure “years in the workforce” and so on.

The process of clarifying observable implications so that researchers can evaluate them .

Note that in the clarification process the researcher translates abstract notions, such as “experience” and “earnings,” into the far more concrete “years in the workforce” and “gross annual income.” Unlike the abstractions, researchers can observe and measure “years in the workforce” and so on.

But how do analysts evaluate their choices and procedures? Why “years in the workforce” and not “years from degree,” “months in the same position,” or any of the other many plausible measures of experience? Typically, researchers look to the reliability and validity of their measures. Reliability is the extent to which it is possible to replicate a measure, reproducing the same value (regardless of whether it is the right one) on the same standard for the same subject at the same time. Measures of high reliability are preferable to those with lower levels of reliability. Validity is the extent to which a reliable measure reflects the underlying concept being measured. Along these lines, we might consider whether the measure is facially valid, that is, whether it comports with prior evidence or existing knowledge, among other criteria.

There is another test to which many researchers put their measures: robustness checks. Suppose we settled on “years in the workforce” as our measure of experience but believed that “months in the same position” was plausible as well. In our statistical work, we might try both hoping to obtain consistent results regardless of the particular measure. This procedure does not tell us whether “years in the workforce” is a better measure than “years in the same position” but it does help to anticipate a question put to many empirical legal scholars: “what if you had used measure Y instead of measure X? Would your results have been the same?”

III Collecting Data and Coding Variables

Once researchers have designed their project—that is, they have filled out the first slide—they typically turn to collecting and coding their data—the makings of the second slide. By this point, it should go without saying, though we shall say it anyway, that we can hardly scratch the surface of either; both deserve Chapters of their own.

What we can do instead is offer some brief counsel, beginning with data collection—actually, with a crucial step before data collection: determining whether the data the researcher needs already exist in the form she needs it. For decades now, empirical legal scholars have been amassing datasets—some for particular projects and others, the so-called “multi-user” datasets, designed for application to a wide range of problems. Either way, it is entirely possible (even probable in some areas of empirical legal studies) that researchers can locate suitable data without having to invest in costly from-scratch data-collection efforts.

A few examples suffice to make the point. If analysts are interested in cases decided by the U.S. Supreme Court, they should proceed directly to the U.S. Supreme Court Database (〈 http;//supemecourtdatabase.org 〉) This remarkable resource houses scores of variables on Supreme Court cases decided since 1953, including the legal provisions under analysis, the identity of the majority opinion writer, and the votes of the justices. A similar dataset, the U.S. Courts of Appeals Database, exists for cases decided by the U.S. circuit courts (at: 〈 http;//www.cas.sc.edu/poli/juri/ 〉). For the researcher interested in public opinion, the General Social Survey and the American National Election Study (both available via an intuitive interface at: 〈 http;//sda.berkeley.edu/archive.htm 〉) are natural places to look for relevant data. For other types of projects, we recommend visiting the websites of the Inter-University Consortium for Political and Social Research (〈 http;//www.icpsr.umich.edu/ 〉) and the IQSS Dataverse Network 〈 http;//dvn.iq.harvard.edu/dvn/ 〉), both of which serve as repositories for (or have links to) existing datasets. Federal and state governments and agencies too retain enormous amounts of information of interest to empirical legal scholars, including data on population demographics, economic indicators, and court caseloads. Last but not least, experience has taught us that a well-formulated Internet search can unearth datasets that scholars maintain on their own websites.

If the data simply do not exist in an analyzable form, empirical legal researchers can and do make use of a wide variety of data-generation mechanisms. They amass numerical data from structured interviews or surveys, from field research, from public sources, from private papers, and on and on. Each has its strengths and weaknesses (as do archived datasets) and it is the researchers' job to learn, understand, and convey them.

Still, within all this variation, two principles governing the data-collection process apply to most empirical legal research projects. One is simple enough: as a general rule, researchers should collect as much data as resources and time allow because basing inferences on more data rather than less is almost always preferable. To see the point, think about a study designed to study gender pay equity in academia. The more professors included in the study, the more certain the conclusions the analyst can reach. As a practical matter, however, diminishing returns kick in and settling on a sample size (as opposed to including all professors) is good enough. For example, one can estimate a proportion with ±2% margin of error with a random sample of approximately 2400 observations; the number increases dramatically to 9,600 for ±1%. This is why most public opinion surveys query, at most, a couple thousand respondents. As discussed in more detail below, this “margin of error” is sometimes referred to as the “sampling error” or the “confidence interval” (e.g., “CI ±3% ” in examples below).

Second, if researchers cannot collect data on all members of the population of interest (e.g., all professors)—and they rarely can—they must invoke selection mechanisms that avoid selection bias (mechanisms that don't bias their sample for or against their theory). For large- n studies (where n = number of participants) only random probability sampling meets this criterion. 8 A random probability sample involves identifying the population of interest (all professors) and selecting a subset (the sample) according to known probabilistic rules. To perform these tasks, the researcher must assign each member of the population a selection probability and select each person into the observed sample according to these probabilities. (Collecting all the observations is a special case of random selection with a selection probability of 1.0 for every element in the population.) 9

Researchers can implement random sampling in various ways depending on the nature of the problem. For a study of pay equity in the academy, for example, we could draw an equal probability sample—a sample in which all professors have an equal chance of being selected. If, on the other hand, we wanted to include all racial and ethnic groups in our study and worried that our sample, by chance, might not include, say, any American Indians, stratified random sampling may be a better strategy. The idea is to draw separate equal-probability-of-selection random samples within each category of a variable (here, race/ethnicity).

Whatever the procedure (so long as it involves random selection for large- n samples!), the legal researcher will typically end up with piles or computer files of questionnaires, field notes, court cases, and so on. Coding variables is the process of translating the relevant properties or attributes of the world (i.e., variables) housed in the piles and files into a form that the researcher can then analyze systematically (presumably after they have chosen appropriate measures to tap the underlying variables of interest).

Coding is a near-universal task in empirical legal studies. No matter whether their data are quantitative or qualitative, from where their data come, or how they plan to analyze the information they have collected, researchers seeking to make claims or inferences based on observations of the real world must code their data. And yet, despite the common and fundamental role it plays in research, coding typically receives only the briefest mention in most volumes on empirical research; it has received almost no attention in empirical legal studies.

Why this is the case is a question on which we can only speculate, but an obvious response centers on the seemingly idiosyncratic nature of the undertaking. For some projects researchers may be best off coding inductively, that is, collecting their data, drawing a representative sample, examining the data in the sample, and then developing their coding scheme. For others, investigators proceed in a deductive manner, that is, they develop their schemes first and then collect/code their data. For still a third set, a combination of inductive and deductive coding may be most appropriate. 10

Nonetheless, we believe it is possible to offer three generalizations about the process of coding variables. First, regardless of the type of data they collect, the variables they intend to code, or even of whether they plan to code inductively or deductively, at some point empirical legal researchers require a coding schema, that is, a detailing of each variable of interest, along with the values of each variable. For example, in a study of the effect of female judges on the votes of their male colleagues, the variable Vote of the Judge would obviously figure prominently; for this variable we might code three values: the judge voted to “affirm,” to “reverse,” or “other.” With this sort of information in hand, investigators can prepare codebooks—or guides they employ to code their data and that others can use to replicate, reproduce, update, or build on the variables the resulting database contains and any analyses generated from it.

Second, depending on the type of data and variables, developing schema and creating codebooks are not always easy or straightforward tasks. To see this, reconsider the seemingly simple example of the variable Vote of the Judge. We just listed three values (affirm, reverse, and other) but what of a vote “affirming in part and reversing in part”? Should we code this as “other,” even if the judge gave the plaintiff some relief? For that matter, what should we make of the “other” category? Depending on the subjects under analysis, it may be appropriate (meaning that it would be an option exercised infrequently) or not. But our more general point should not be missed: accounting for the values of the variables of interest, even of seemingly simple ones, may be tricky. 11

To be sure, following best practices can help; for example, ensuring that the values of the variables are exhaustive, creating more (rather than fewer) values, establishing that the values of the variables are mutually exclusive, and more generally, pretesting the schema (for more details, see Epstein and Martin, 2005 ). But there is one assumption that all the rules and guidelines make—and this brings us to our third point: researchers must have a strong sense of their project, particularly about the piece of the legal world they are studying and how that piece generated the data they will be coding, as well as the observable implications of the theory that they will be assessing (see, e.g., Babbie, 2007 : 384; Frankfort-Nachmias and Nachmias, 2007 ). Even adhering to simple rules will be difficult, if not impossible, if the researcher lacks a deep understanding of the objects of her study and an underlying theory about whatever feature(s) of their behavior for she wishes to account.

IV. Analyzing Data

If research design is the first overhead slide and collecting and coding data, the second, then data analysis enables researchers to compare their overlap. When the overlap between the observable implications and data is substantial, analysts may conclude that the real world confirms their hunches; if the overlap is negligible, they may go back to the drawing board or even abandon the project altogether.

How do empirical legal scholars perform this task? The answer depends in no small part on their goals. If the goal is to summarize the data they have collected (say, the salaries of all male and female professors at their school), then some simple measures of central tendency (e.g., means, medians) and dispersion (e.g., standard deviations, ranges) might suffice. These will give researchers a feel for the distributions of their variables that, depending on the number of cases, they could not possibly develop from looking at a column of data.

For the vast majority of empirical legal projects, however, making inferences—using facts we know to learn about facts we do not know—is the goal. Rarely do we care much about the, say, 50 individuals or 100 cases in our sample. Rather, we care about what those 50 individuals or 100 cases can tell us about all the employees of the corporation or all the cases. In quantitative research, inferences come in two flavors: descriptive and causal . Descriptive claims themselves can take several forms but some seem quite a kin to data summaries. Suppose, for example, that we collected data on 100 court cases involving employment discrimination and learned that, on average, appellate court panels held for the plaintiff in 40% of the cases. In and of itself this figure of 40% (a summary of the data), probably isn't all that interesting to our readers or us. What we want to learn about is the fraction of all employment discrimination cases in which all courts held for the plaintiff. That is, we want to use what we know (the 100 cases we have collected) to learn about what we do not know (the cases we haven't collected). This is the task of drawing a descriptive inference. We do not perform it by summarizing facts; we make it by using facts we know—the small part of the world we have studied—to learn about facts we do not observe (the rest of the world). Researchers call the “small part” a sample and the “world” a population. (An important part of performing descriptive inference is quantifying the uncertainty we have about that inference. We discuss this in greater detail below.) It is important to keep in mind that when dealing with data coming from a non-probability sampling neither descriptive nor causal inferences can be drawn.

Causal inference too is about using facts we do know to learn about facts we do not know. In fact, a causal inference is the difference between two descriptive inferences —the average value the dependent variable (for example, the fraction of cases decided in favor of the plaintiff ) takes on when a “treatment” is applied (for example, a female judge serves on the panel) and the average value the dependent variable takes on when a “control” is applied (for example, if no female judge sits on the panel). The causal effect —the goal of the process of causal inference—is this difference, the amount the fraction of decisions in favor of the plaintiff increases or decreases when we move from all-male panels to panels with a female.

How do quantitative empirical researchers go about making descriptive or causal claims? Assuming they have appropriately designed their projects and appropriately amassed and coded their data, they make use of statistical inference , which entails examining a small piece of the world (the sample) to learn about the entire world (the population), along with evaluating the quality of the inference they reach. Conceptually, statistical inference is not all that hard to understand; actually we confront such inferences almost every day. When we open a newspaper, we might find the results of a survey showing that 70% (± 5% margin of error) of American voters have confidence in the US president. Or when we read about a scientific study indicating that a daily dose of aspirin helps 60% (95% CI ± 3% ) of Americans with heart disease. (95% CI and ± X% are explained below.) In neither of these instances, of course, did all Americans participate. The pollsters did not survey every voter, and the scientists did not study every person with heart problems. They rather made an inference (in these examples, a descriptive inference) about all voters and all those stricken with heart disease by drawing a sample of voters and of ill people.

But how do the researchers go about making the statistical inference (for example, 70% of all American voters have confidence in the president) and assess its quality (that is, indicate how un certain they are about the 70% figure, as indicated by the ± 5% )? It is one thing to say that 70% of the voters in the sample have confidence in the president (this is summarizing or describing the data); but it is quite another to say that 70% of all voters have confidence (this is the descriptive inference).

To support the first claim, all analysts need do is tally (i.e., summarize ) the responses to their survey. To support (and evaluate) the second, they must (1) draw a random probability sample of the population of interest and (2) determine how certain (or uncertain) they are that the value they observe from their sample of voters (70% ), called the sample statistic , reflects the population of voters, the population parameter .

We already have discussed (1)—drawing a random sample—so we only need reiterate here that this step is crucial. If a sample is biased (for instance, if Democrats had a better chance of being in the pollsters' sample than Republicans), researchers cannot draw accurate conclusions.

Assuming researchers draw a random probability sample, they can move to (2) and make a (descriptive) inference about how well their sample reflects the population. Or, to put it another way, they can convey their degree of uncertainty about the sample statistic. Surveys reported in the press, for example, typically convey this degree of uncertainty as “the margin of error,” which is usually a 95% confidence interval (or 95% CI). When pollsters report the results of a survey—that 70% of the respondents have confidence in the president with a ±5 margin of error—they are supplying the level of uncertainty they have about the sample statistic of 70%. That is, the true fraction of voters who have confidence in the president will be captured in the stated confidence interval in 95 out of 100 applications of the same sampling procedure. The fact that the data come from a random sample is what makes it possible to use the rules of probability to compute these margins of error.

Note that this information does not say exactly where, or whether, the population (parameter) lies within this range. (In fact, the parameter either falls within the interval or not; only an all-knowing researcher would ever know.) What is critical, however, is that if the researcher continues to draw samples from a population of voters, the mean of the samples of voters will eventually equal the mean of the population, and if the researcher creates a specialized bar graph called a histogram showing the distribution of the individual sample means, the resulting shape would resemble a normal distribution. This is what enables researchers to make an inference—here, in the form of a sample statistic and a margin of error—about how all voters (the population) feel about the president by observing a single sample statistic. For the sake of illustration, consider Figure 2 . Here we show the confidence intervals computed from 50 random samples from a population where the known parameter of interest is ten. The 95% confidence intervals are constructed to contain the true parameter 95% of the time. Here in all but two samples the horizontal confidence intervals contain the known parameter value. Of course, in any application we do not know the parameter value (if we did we would not need to perform inference!), but we use confidence intervals that over repeated samples will return the right answer a high percentage of the time.

Confidence intervals for a known population mean ten for fifty random samples from a population .

This pertains to descriptive claims but it is important to draw a statistical inference when performing causal inference as well. Suppose that the average monthly income for the male professors in our sample of employees was $4,200, while for the females it was $3,900, yielding a difference of $300 in this sample. There are two possible explanations for the $300 difference (assuming all else is constant , a phrase we explain below). It might be the case that it is due solely to the particular sample we randomly drew; in other samples from the population the difference might only be $10, or women might make, on average, $250 more than men. It is also possible that in the population, men actually earn more than women.

The process researchers use to make this determination is called hypothesis testing . A hypothesis test tells us whether differences across groups are simply an artifact of sampling (the first possible explanation), or whether meaningful differences exist in the population (the second possibility). In the latter case we would say the difference is statistically significant . All statistical significance means is that sampling alone cannot explain the observed difference, and as such, it is likely that differences exist in the population. One would conclude a relationship is statistically insignificant when the difference in the sample can be explained by sampling alone.

In addition to statistical significance, it is important to consider the substantive significance of any finding. A $1,000 per month difference in salary is certainly large; an $8 per month difference is not. Both could be statistically significant, but only the first would be substantively significant. Accordingly, it is crucial for empirical researchers to compute and report the size of the differences—in addition to reporting the results of hypothesis tests—so that the reader can ascertain whether the findings are substantively important. In the following section we recommend using graphics to report these differences.

But before turning to data displays, one final topic deserves some attention: the assumption of “all else being constant” or “all things being equal.” This assumption takes us back to a point we made at the onset; namely, when working with data generated by the world, most of the time “all else is constant” or “all things being equal” is untenable. It is quite possible, for example, that male professors in our sample do not have the same experience as females. Thus, just naïvely comparing the average salaries across the two groups would not provide a reliable causal inference.

Today, there are two approaches commonly used for making causal inferences from observational data. One type of analysis is multiple regression analysis, and related regression models (such as logistic regression). Regression models work by allowing the researcher to hold all other measured variables constant while assessing the relationship of interest. In this example, we could see whether the difference in salaries persisted by controlling for experience. Regression models have been used for decades and are the most common tool in empirical legal research. For many types of research they work quite well, but they do require some strong assumptions about the relationship between the key causal variable and the outcome variable of interest (see Imai, 2005 ).

Another set of methods called matching methods is becoming more popular in applied statistics. These cutting-edge tools are making their way into empirical legal studies (Epstein et al. 2005 ; Greiner 2008 ), and for many reasons we predict that their use will increase in the coming decades. The idea, to return to the example of pay equity, is to match most-similar male and female professors, and then compute differences between the matched observations. Once researchers have made the matches, these methods allow them to treat observational data as if it were experimental.

Regardless of whether one uses regression analysis or matching to control for alternative explanations, a causal inference is just a statistical inference about a difference. At bottom what researchers want to know is whether observed differences in a sample represent the same differences in a population.

V. The Last Step: Presenting the Results of Empirical Legal Research 12

Just as scholars have been improving methods for causal inference, they have been working on approaches to convey the results of their studies. These developments should be of particular interest to quantitative empirical legal scholars who often must communicate their findings to judges, lawyers, and policy-makers—in other words, to audiences who have little or no training in statistics. Too often, though, analysts fail to take advantage of the new developments thus missing an opportunity to speak accessibly to their community.

To see the problem, consider an example adapted from a study that seeks to explain the votes cast by U.S. senators on Supreme Court nominees (Epstein et al., 2006 ). 13 Briefly, the authors operate under the assumption that electorally minded senators vote on the basis of their constituents' “principal concerns in the nomination process” (Cameron et al., 1990 : 528). These concerns primarily (though not exclusively) center on whether a candidate for the Court is (1) qualified for office and (2) ideologically proximate to the senator (i.e., to his or her constituents). Consequently, the two key causal variables in their statistical model are (1) the degree to which a senator perceives the candidate as qualified for office and (2) the ideological distance between the senator and the candidate, such that the more qualified the nominee and the closer the nominee is to the senator on the ideological spectrum, the more likely the senator is to cast a yea vote. Also following from the extant literature, the researchers control for two other possible determinants of senators' votes: whether the president was “strong” in the sense that his party controlled the Senate and he was not in his fourth year of office; and whether a senator is of the same political party as the president.

To assess the extent to which these variables help account for senators' votes, the researchers employed logistic regression, a common tool in legal scholarship when the dependent variable is binary. Table 1 displays the results, and they seem to lend support to the researchers' hypothesis. For example, the ⋆ on the coefficient for lack of qualifications variable tells us that a statistically significant relationship exists between qualifications and voting: the lower a nominee's qualifications, the higher the likelihood that a senator will vote against the nominee.

On the other hand, tables of this sort (which run rampant in empirical legal scholarship) are not just ugly and off-putting to most readers; they communicate virtually no information of value either to the audience or even to the researchers themselves. Most lawyers, judges, and even law professors do not understand terms such as “statistical significance,” much less “logit coefficient.”

How might empirical legal scholars improve their data presentations? Adhering to three general principles would be a good start. First, we recommend that analysts communicate substance, and not only statistics. Reconsider this statement:

In looking at Table 1 , we see that the coefficient on the variable lack of qualifications of - 4.11 is “statistically significant.”

This is not wrong but the emphasis on the coefficient is more than off-putting; it fails to convey useful information. In fact, all we learn from the -4.11 coefficient on lack of qualifications is that, controlling for all other factors, as we move from the most qualified to the most unqualified nominee we move down 4.11 on a logit scale. To make matters worse, because the logit scale is nonlinear, moving down 4.11 units will result in different probabilities of a yea vote depending on where we start on the scale.

Because few of their readers would understand what any of this means, it is no wonder many empirical legal scholars simply say “the coefficient on lack of qualifications is statistically significant at the.01 level.” But this too isn't an informative statement to many readers; it isn't even informative to readers with statistical training (a very small fraction of the legal community). It tells us is that qualified candidates are more likely to receive a yea vote than unqualified candidates but not how much more likely. 0.2 times more likely? 2 times? Or perhaps even 4 times? We probably wouldn't be very impressed, for example, if all else being equal, the predicted probability of a senator voting for a very qualified candidate was 0.11 and for a very unqualified candidate was 0.14. Certainly, a quantity such as a predicted probability is what matter most to readers of empirical legal scholarship. But it is not one that they can learn from a tabular display of logit coefficients.

This is why we recommend supplying readers with a quantity of interest; that is, replace “In looking at Table 1 , we see that the coefficient on the variable lack of qualifications of - 4.11 is statistically significant” with:

Other things being equal, 14 when a nominee is perceived as highly unqualified the likelihood of a senator casting a yea vote is only about 0.24 . That probability increases to 0.92 when the nominee is highly qualified.

Statements of this sort are easy to understand even by the most statistically challenged members of the legal community.

Second, we suggest that when they perform inference, researchers convey their uncertainty. To see the point, think about the statement above—that the likelihood of a senator casting a yea vote is only about 0.24 when the candidate is unqualified. This figure of 0.24 represents the researchers' “best guess” about the likelihood of a senator voting yea based on qualifications. But we know that error or uncertainty exists around that best guess. It is simply a fact of statistical analysis that we can never be certain about our guesses because they themselves are based on estimates.

Most quantitative empirical legal scholars appreciate this fact and supply the error surrounding their estimated coefficients . Statements such as this are not uncommon:

In looking at Table 1 , the coefficient on the variable lack of qualifications (- 4.11 with a standard error of 0.22) is statistically significant at the.01 level.

True, this conveys uncertainty in the form of a standard error around the estimate but of what value is it? None, it turns out, because all the error value supplies is an estimate of the standard deviation of the estimated coefficient—which, standing alone, is of interest to no one, readers and scientists alike. 15

One possible fix is for empirical legal scholars to follow other disciplines and report far-more-meaningful 95% (or even 99% ) confidence intervals rather than (or in addition to) standard errors. In the case of lack of qualifications, the values of that interval are a lower bound of - 4.54 and an upper bound of - 3.69.

This interval comes closer than the standard error to conveying useful information: the researchers' best guess about the coefficient on lack of qualifications is - 4.11 but they are “95% certain” that it is in the range of - 4.54 to - 3.69. Because 0 is not in this range (the confidence interval), the researchers and their readers can safely reject the null hypothesis of no relationship between the nominees' qualifications and senators' votes.

But even denoting the confidence interval around a coefficient would not be making the most of the model's results. When researchers say they are “95% certain” that the true logit coefficient lies between - 4.54 to - 3.69, they lose half their audience. What we recommend instead is combining the lesson here of relating uncertainty with the first principle of conveying substantive information:

Other things being equal, when a nominee is perceived as highly unqualified the likelihood of a senator casting a yea vote is only about 0.24 (±0.05). That predicted probability increases to 0.93, (±0.02) when the nominee is highly qualified.

Now readers need no specialized knowledge about standard errors or even confidence intervals to understand the results of the study—including uncertainty about the results. They can easily see that the researchers' best guess about the predicted probability of yea vote for a highly unqualified candidate is 0.24, though it could be as low as 0.19 or as high 0.29. Such accessible communication creates a win-win for empirical legal researchers and their audience: both are now in a far better position to evaluate the study's conclusions.

Our final recommendation is that analysts graph their data and results. With this, we are trying to convey two ideas. One is just a general point: if the goal is to give readers a feel for patterns or trends in the data, graphs are superior to tables—even for small amounts of data. Figure 3 provides an example from the project on Supreme Court nominees.

To be sure, if we looked at the table long enough some of the patterns we observe in the figure would emerge but it takes a lot more cognitive work on the part of the reader. Plus, it is unlikely that readers of empirical legal studies need such specific, precise information as in the table. So in most instances graphic displays can convey the right information without losing much.

The second idea, more relevant to the communication of results (rather than data, as in Figure 3 ), is that figures enable analysts to combine the first two principles we set out above (substance and uncertainty) across many values. Think about it this way: while substantive claims of the form “When a nominee is perceived as highly unqualified the likelihood of a senator casting a yea vote is only about 0.24 (±0.05)” may be informative, they exclude a lot of information—the values in between “highly unqualified” and “highly qualified.” To provide these quantities, we could generate a long series of statements such as

Other things being equal, when a nominee is perceived as highly unqualified the likelihood of a senator casting a yea vote is 0.24 (±0.05).

Other things being equal, when a nominee is perceived as about average on the qualifications scale, the likelihood of a senator casting a yea vote is 0.83 (±0.03).

Other things being equal, when a nominee is perceived as highly qualified the likelihood of a senator casting a yea vote is 0.93 (±0.02).

Tables versus Figures .

Both the table and the figure provide information on the ideological distance between Senator Edward Kennedy (D-Mass.) and ten recent Supreme Court nominees. Juxtaposed against the table, the dot plot provides a more visually and cognitively appealing solution to the problem of providing the reader with information about variables of interest.

But graphing the results is a far more parsimonious, pleasing, and, for the readers of empirical legal work, cognitively less demanding approach. Underscoring these points is Figure 4 . Here the reader gets a real sense of the (1) results and (2) uncertainty across the values of qualifications without having to sift through a long series of claims.

Even better, and usually necessary in multivariate analysis, is to bring in other variables of interest, as Figure 5 does. Here, we've juxtaposed qualifications against another variable: ideology, when senators and nominees are ideologically very close and when they are very distant. Specifically, in the two panels we show the probability of a senator casting a yea vote across the range of lack of qualifications and when we set ideological distance at its minimum and maximum levels. In both panels we depict our uncertainty, in the form of 95% confidence intervals, with vertical lines.

This display, we believe, is a good example of what we mean by parsimony. It conveys a great deal of information—actually it encodes 66 pieces of information—quite efficiently or at least more efficiently than the 66 sentences it would have taken to describe each and every result depicted in the two panels and certainly more accessibly than a table of logit coefficients.

The effect of qualifications on Senate votes over Supreme Court nominees, from Black (1937) through Alito (2006) .

The figure shows the predicted probability of a senator casting a yea vote over the range of lack of qualifications (0 is the most qualified), when we set ideological distance at its mean and the other variables in the statistical model at 0. The small vertical bars are 95% confidence intervals. Created using S-Post.

Little more than a decade ago, implementing a graph of the sort depicted in Figure 4 would have been quite the chore: estimating the confidence intervals, in particular, was not possible for most empirical legal scholars. But now, because contemporary software packages use simulations (repeated sampling of the model parameters from their sampling distribution) to produce estimates of quantities of interest (e.g., predicted probabilities), generating assessments of error (e.g., confidence intervals) is quite easy. 16 Moreover, using the software requires no additional assumptions beyond those the researcher already has made to perform statistical inference.

Once researchers have prepared their results for presentation (and, ultimately, publication), their work would seem to be done. And, for the most part it is. But we in the empirical legal community should demand that they take one final step: archive their data and documentation. So doing ensures that empirical legal scholars adhere to the replication standard : Another researcher should be able to understand, evaluate, build on, and reproduce the research without any additional information from the author (King, 1995 ). This rule does not actually require anyone to replicate the results of an article or book; it only requires that researchers provide information—in the article or book or in some other publicly available or accessible form—sufficient to replicate the results in principle.

The effect of qualifications on Senate votes over Supreme Court nominees, from Black (1937) through Alito (2006), when the ideological distance between the Senator and nominee is very close (minimum) and very distant (maximum) and all other variables in the statistical model are set at 0 .

Why is such documentation a requisite step in conducting empirical research (regardless of whether the work is qualitative or quantitative in nature)? Epstein and King 2002 supply two answers. The first centers on the ability of outsiders to evaluate the research and its conclusions. In a broad sense, the point of the replication standard is to ensure that a published work stands alone so that readers can consume what it has to offer without any necessary connection with, further information from, or beliefs about the status or reputation of the author. The replication standard keeps empirical inquiry above the level of ad hominum attacks on or unquestioning acceptance of arguments by authority figures. The second reason is straightforward enough: as this Chapter has (hopefully!) made clear, the analyst's procedures may, and in most instances do, influence the outcomes they report. Readers deserve an opportunity to evaluate the researchers' choices, not to mention their data.

Designing research, collecting and coding data, analyzing data, and presenting results represent the four chief tasks of quantitative empirical legal scholarship, and we have tried to explain some of the basics. But readers should keep in mind that mastering the four requires far more than we can possibly convey here; it requires training. That is why PhD programs in the social sciences offer (at the least) a one-semester course on each.

Reading some of the books and articles we cite below would be a good start for legal scholars wishing to learn more—but only a start. To develop a full appreciation for the research process, we strongly recommend that readers contact their local social science departments.

Babbie, E. ( 2007 ). The Practice of Social Research , (11th edn.), Belmont, CA: Thomson.

Google Scholar

Google Preview

Boyd, C.L., Epstein, L., and Martin, A.D. ( 2010 ). “ Untangling the Causal Effects of Sex and Judging, ” American Journal of Political Science 54: 389–411.

Cameron, C.M., Cover, A.D., and Segal, J.A. ( 1990 ). “ Senate Voting on Supreme Court Nominees: A Neo-Institutional Model, ” American Political Science Review 85: 525–34.

Epstein, L. and King, G. ( 2002 ). “ The Rules of Inference, ” University of Chicago Law Review 69: 191–209.

Epstein, L. and Martin, A.D. ( 2005 ). “Coding Variables,” in K. Kempf-Leonard (ed.), The Encyclopedia of Social Measurement , San Diego: Academic Press.

Epstein, L., Ho, D.E., King, G., and Segal, J.A. ( 2005 ). “ The Supreme Court During Crisis, ” NYU Law Review 80: 1–116.

Epstein, L., Lindstadt, R., Segal, J.A., and Westerland, C. ( 2006 ). “ The Changing Dynamics of Senate Voting on Supreme Court Nominees, ” Journal of Politics 68: 296–307.

Epstein, L., Martin, A.D., and Schneider, M. ( 2006 ). “ On the Effective Communication of the Results of Empirical Studies, Part I, ” Vanderbilt Law Review 59: 1811–71.

Epstein, L., Martin, A.D., and Boyd, C. ( 2007 ). “ On the Effective Communication of the Results of Empirical Studies, Part II, ” Vanderbilt Law Review 60: 798–846.

Frankfort-Nachmias, C. and Nachmias, D. ( 2007 ). Research Methods in the Social Sciences , New York: Worth.

Gelman, A. et al. ( 2002 ). “ Let's Practice What We Preach: Turning Tables into Graphs, ” The American Statistician 56: 121.

Greiner, D.J. ( 2008 ). “ Causal Inference in Civil Rights Litigation, ” Harvard Law Review 122: 533.

Ho, D.E. et al. ( 2007 ). “ Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference, ” Political Analysis 15: 199.

Holland, P.W. ( 1986 ). “ Statistics and Causal Inference, ” Journal of American Statistical Association 81: 945–70.

Imai, K. ( 2005 ). “ Do Get-Out-The-Vote Calls Reduce Turnout: The Importance of Statistical Methods for Field Experiments, ” American Political Science Review 99: 283–300.

King, G. ( 1995 ). “ Replication, Replication, Replication, ” PS: Political Science and Politics 28: 443–99.

King, G., Keohane, R.O., and Verba, S. ( 1994 ). Designing Social Inquiry , Princeton University Press.

King, G., Tomz, M., and Wittenberg, J. ( 2000 ). “ Making the Most of Statistical Analyses, ” American Journal of Political Science 44: 50.

Neyman, J. ( 1935 ). “ Statistical Problems in Agricultural Experimentation, ” Journal of the Royal Statistical Society 2: 107–154.

Rachlinski, J.A., Guthrie, C., and Wistrich, A.J. ( 2006 ). “ Inside the Bankruptcy Judge's Mind, ” Boston University Law Review 86: 1227–65.

Rubin, D.B. ( 1973 ). “ Matching to Remove Bias in Observational Studies, ” Biometrics 29: 159–83.

Rubin, D.B. ( 1974 ). “ Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies, ” Journal of Educational Psychology 6: 688–701.

For research support, we thank the National Science Foundation, Northwestern University School of Law, and the Center for Empirical Research in the Law at Washington University. For their very helpful comments, we thank the editors of this volume. We adapt some of the material in this Chapter from Epstein and King 2002 ; Epstein and Martin 2005 ; Epstein, Martin, and Boyd (2007) ; Epstein, Martin, and Schneider (2006) ; and Epstein and Martin's annual workshop, Conducting Empirical Legal Research .

Though it should be obvious, for this hypothetical we are assuming that the employer is assigning wages intentionally, not randomly.

For a more formal accounting of this type of analysis, many scholars have adopted a potential outcomes framework—posited by Neyman 1935 and Rubin ( 1973 , 1974 ), thoroughly reviewed in Holland 1986 , and recently applied in the social sciences by Imai 2005 , Epstein et al. 2005 , and Boyd, et al. (2010) .

Their research tested for other biases as well, including anchoring and framing.

These are indeed the key components, and in the Sections to follow we describe them in order, from designing research to conducting analyses. Nonetheless, empirical legal scholars rarely regard their research as following a singular, mechanical process from which they can never deviate. Quite the opposite: scholars must have the flexibility of mind to overturn old ways of looking at the world, to ask new questions, to revise their blueprints as necessary, and to collect more (or different) data than they might have intended. On the other hand, being flexible does not mean that researchers do or should do ad or post hoc adjustment of theories to fit idiosyncrasies. Adjustments made to harmonize theory with data, of course, do not constitute any confirmation of the theory at all. While it is fine to use data to create theory, investigators know they must consult a brand new data set, or completely different and previously unanticipated testable consequences of the theory in the same data set, before concluding that data confirm their theory. For more on the idea of research as a “dynamic process conforming to fixed standards,” see Epstein and King 2001 .

How to ensure a good fit? We turn to this question when we tackle the subject of measurement.

Some might argue that these steps are unnecessary in research motivated purely by policy concerns. Not so. Because the statistical methods we describe momentarily are designed to test hypotheses, the researcher should, well, develop some hypotheses to test.

For advice on small- n studies, see Epstein and King 2002 : 112–13); King et al. 1994 : 124–8).

Dealing with data collected on a population raises some foundational statistical issues. One approach is to argue that an observed population is a “sample” from possible histories, and as such, traditional inferential statistics can be used. Another option is to simply summarize the data and not report measures of uncertainty. The ideal approach, from our perspective, is to adopt a Bayesian approach and treat the parameters as random variables, not the data.

Some writers associate inductive coding with research that primarily relies on qualitative data and deductive coding, with quantitative research. Given the [typically] dynamic nature of the processes of collecting data and coding, however, these associations do not always or perhaps even usually hold. Indeed, it is probably the case that most researchers, regardless of whether their data are qualitative or quantitative, invoke some combination of deductive and inductive coding.

More generally, the relative ease (or difficulty) of the coding task varies according to the types of data with which the researcher is working, the level of detail for which the coding scheme calls, and the amount of pretesting the analyst has conducted.

We draw material in this section from Epstein et al. (2007) ; Epstein et al. (2006) ; Gelman, et al. (2002) ; King et al. (2000) .

Since publication of their study, Epstein et al. have updated their dataset (available at: 〈 epstein.law.northwestern.edu/research/Bork.html 〉). We rely on the updated data.

We use the term “other things being equal” to signify that all variables in the model (other than the variable interest, here qualifications) are fixed at particular values. In this example, we set ideological distance at its mean and strong president and same party at 0.

Its value, rather, lies in computing confidence intervals.

King et al.'s (2000) Clarify is an example. It uses the Monte Carlo algorithm for the simulations, and can be implemented via the Clarify plug-in for Stata.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.