grey literature in research

Grey literature: An important resource in systematic reviews

Affiliation.

• 1 Nuffield Department of Primary Care Health Sciences, Centre for Evidence-Based Medicine, Department for Continuing Education, The University of Oxford, Oxford, UK.
• PMID: 29266844
• DOI: 10.1111/jebm.12265

Systematic reviews aid the analysis and dissemination of evidence, using rigorous and transparent methods to generate empirically attained answers to focused research questions. Identifying all evidence relevant to the research questions is an essential component, and challenge, of systematic reviews. Grey literature, or evidence not published in commercial publications, can make important contributions to a systematic review. Grey literature can include academic papers, including theses and dissertations, research and committee reports, government reports, conference papers, and ongoing research, among others. It may provide data not found within commercially published literature, providing an important forum for disseminating studies with null or negative results that might not otherwise be disseminated. Grey literature may thusly reduce publication bias, increase reviews' comprehensiveness and timeliness and foster a balanced picture of available evidence. Grey literature's diverse formats and audiences can present a significant challenge in a systematic search for evidence. However, the benefits of including grey literature may far outweigh the cost in time and resource needed to search for it, and it is important for it to be included in a systematic review or review of evidence. A carefully thought out grey literature search strategy may be an invaluable component of a systematic review. This narrative review provides guidance about the benefits of including grey literature in a systematic review, and sources for searching through grey literature. An illustrative example of a search for evidence within grey literature sources is presented to highlight the potential contributions of such a search to a systematic review. Benefits and challenges of grey literature search methods are discussed, and recommendations made.

Keywords: evidence; grey literature; publication bias; search; systematic review.

© 2017 Chinese Cochrane Center, West China Hospital of Sichuan University and John Wiley & Sons Australia, Ltd.

• Langson Library
• Science Library
• Grunigen Medical Library
• Law Library
• Connect From Off-Campus
• Accessibility
• Gateway Study Center

Email this link

Systematic reviews & evidence synthesis methods.

• Schedule a Consultation / Meet our Team
• What is Evidence Synthesis?
• Types of Evidence Synthesis
• Evidence Synthesis Across Disciplines
• Finding and Appraising Existing Systematic Reviews
• 0. Develop a Protocol
• 1. Draft your Research Question
• 2. Select Databases
• 3. Select Grey Literature Sources
• 4. Write a Search Strategy
• 5. Register a Protocol
• 6. Translate Search Strategies
• 7. Citation Management
• 8. Article Screening
• 9. Risk of Bias Assessment
• 10. Data Extraction
• 11. Synthesize, Map, or Describe the Results
• Open Access Evidence Synthesis Resources

Grey Literature

Grey literature is literature produced by individuals or organizations outside of commercial and/or academic publishers. This can include information such as government reports, conference proceedings, graduate dissertations, unpublished clinical trials, and much more. The sources you select will be informed by your research question and field of study, but should likely include, at a minimum, theses and dissertations.

Why Search the Grey Literature?

The intent of an evidence synthesis is to synthesize all available evidence that is applicable to your research question. There is a strong bias in scientific publishing toward publishing studies that show some sort of significant effect. Meanwhile, many studies and trials that show no effect end up going unpublished. But knowing that an intervention had no effect is just as important as knowing that it did have an effect when it comes to making decisions for practice and policy-making. While not peer-reviewed, grey literature represents a valuable body of information that is critical to consider when synthesizing and evaluating all available evidence.

Searching the Grey Literature

Finding grey literature and searching it systematically is challenging. But there are a few approaches that you can take to add some structure to your search of this type of information:

• Refer to grey literature sources used for related evidence syntheses. Refer to both published evidence syntheses and registered protocols.
• Ask experts in the field for relevant grey literature sources. If you are an expert, include important grey literature sources, and ask colleagues for their recommendations.
• Search databases that specialize in grey literature: See the "Grey Literature Sources" box at the bottom of this page for more information.
• Search for theses and dissertations: There are a number of databases dedicated to theses and dissertations, which you can search using your search terms. See the "Grey Literature Sources" box at the bottom of this page for links to these resources.
• Search clinical trials: There may be clinical trials being conducted that are relevant to your research question, but that haven't been published yet or never were published. See the "Grey Literature Sources" box at the bottom of this page for links to these resources.
• Identify government agencies and international and non-governmental organizations that might publish technical papers and reports on your topic. Search their websites or any online libraries that they may provide. See the "Grey Literature Sources" box at the bottom of this page for links to some examples.
• Search conference proceedings and newsletters: Identify professional organizations that have and/or conferences at which researchers might be presenting work related to your topic. Search those conference proceedings or newsletters on the organization's website or by contacting organizational boards for access to past proceedings that may not be online. See the "Grey Literature Sources" box at the bottom of this page for some examples.
• Contact known researchers in the field to determine if there are any ongoing or unpublished studies that s/he may be aware of.
• Search professional and trade magazines. Professional magazines contain literature that is written by professionals in the field for other professionals in the field, but that may not be about research. Trade magazines contain advertisements and news very specific to a topic or industry.

How to Manage the Grey Literature Search

• Identify and record the sources you will search. The sources you search will be informed by your research question and where you expect to find information related to your question.
• Document where you are searching and your search strategies, including document resource name, URL, search terms, and date searched.
• Collect citation information as you go.
• Adhere to your established inclusion and exclusion criteria when selecting sources.

See below for guidance documents specific to grey literature searching.

Grey Literature Sources

• Grey Literature Databases
• Theses and Dissertations
• Clinical Trials
• Public Policy
• Conference Proceedings
• Preprint Repositories
• Other Resources
• NY Academy of Medicine Grey Literature Report This report is a bimonthly publication of The New York Academy of Medicine (NYAM) alerting readers to new grey literature publications in health services research and selected public health topics. The database platform is keyword searchable and serves as an archive for the cataloged reports.
• WHO Library Database The WHO library database includes governing documents, reports and technical documentation.
• Global Think Tanks A database of reports, papers, and other resources published by NGOs, think tanks, IGOs and research centers from across the world. It also includes a directory of thousands of organizations involved in public and social policy in most subject area. A module in the Policy Commons database.
• MedNar MedNar searches across more than 60 medical research sources, including commercial databases, medical societies, NIH resources, and other government resources.
• Global Index Medicus The Global Index Medicus (GIM) provides worldwide access to biomedical and public health literature produced by and within low-middle income countries. The main objective is to increase the visibility and usability of this important set of resources. The material is collated and aggregated by WHO Regional Office Libraries on a central search platform allowing retrieval of bibliographical and full text information.
• National Grey Literature Collection - UK Funded by Health Education England this holds the catalogue for the hard copy grey literature collection of the former North Grey Literature Service. It is collecting an increasing range of electronic Grey Literature.

• Networked Digital Library of Theses and Dissertations (NDLT) Free international resource for theses and dissertations.
• Center for Research Libraries A resource for institutions outside of the U.S. and Canada.
• EThOS Electronic Theses Online Service contains doctoral theses from candidates in the UK.
• Open Access Theses and Dissertations Resource for finding open access graduate theses and dissertations published around the world. Metadata (information about the theses) comes from over 1100 colleges, universities, and research institutions. OATD currently indexes over 4 million theses and dissertations.
• DART European - Digital Access to Research Theses Europe The aim of DART is to provide researchers with a single European Portal for the discovery of Electronic Theses and Dissertations (ETDs). Note that only theses available in electronic form are included.
• Australia New Zealand Clinical Trials Registry Online register of clinical trials being undertaken in Australia, New Zealand and elsewhere. The ANZCTR includes trials from the full spectrum of therapeutic areas of pharmaceuticals, surgical procedures, preventive measures, lifestyle, devices, treatment and rehabilitation strategies, and complementary therapies.
• ClinicalTrials.gov The US registry for clinical trials. Includes new, ongoing, and completed human clinical trials both in the US and countries around the world.
• Cochrane Central Register of Controlled Trials A highly concentrated source of reports of randomized and quasi-randomized controlled trials. The majority of CENTRAL records are taken from bibliographic databases (mainly MEDLINE and Embase), but records are also derived from other published and unpublished sources.
• EU Clinical Trials Register The European Union Clinical Trials Register allows you to search for protocol and results information on interventional clinical trials that are conducted in the European Union (EU) and the European Economic Area (EEA) and clinical trials conducted outside the EU / EEA that are linked to European paediatric-medicine development.
• WHO International Clinical Trials Registry Platform ICTRP provides access to a central database containing the trial registration data sets provided by numerous international registries. It also provides links to the full original records.
• Wikipedia Clinical Trial Registries List This Wikipedia entry contains a list of links to individual clinical trial registries by country.
• WHO Institutional Repository for Information Sharing (IRIS) Institutional WHO database of intergovernmental policy documents and technical reports. Can search by IRIS by region (Africa, Americas, Eastern Mediterranean, Europe, South-East Asia, Western Pacific).
• Health Research Web A wiki with descriptors of national health research systems, ethics review committees, local and regional policies and research priorities and more.

This list of conference proceedings is not comprehensive, but are ideas of where to search. We recommend that you search proceedings for the three most relevant conferences for your evidence synthesis.

• OCLC PapersFirst OCLC PapersFirst is an OCLC index of papers presented at conferences worldwide.
• BIOSIS Previews A subcollection of Web of Science, BIOSIS Previews searches across journals, meetings, patents, and books in the life sciences and biomedical fields.

• arXiv arXiv is a free distribution service and an open-access archive for scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.
• medRxiv medRxiv (pronounced "med-archive") is a free online archive and distribution server for complete but unpublished manuscripts (preprints) in the medical, clinical, and related health sciences.
• PsyArXiv A free preprint service for the psychological sciences. It is maintained by The Society for the Improvement of Psychological Science.
• OSF Preprints A general preprint repository that covers architecture, business, engineering, life sciences, physical sciences and mathematics, arts and humanities, education, law, medicine and health sciences and social and behavioral sciences, among others.
• TechRxiv Hosted by IEEE
• Engineering Archive Created in 2016. Accepts all types of manuscripts, including but not limited to systematic reviews, hypotheses, ‘negative’ results, and data and methods papers.Includes 2 major types of articles: (1) Preprints of articles in parallel to traditional journal submissions. (2) Previously published articles that you wish to make Open Access
• Preprints.org This is a multidiscipline platform providing preprint service that is dedicated to making early versions of research outputs permanently available and citable. We post original research articles and comprehensive reviews, and papers can be updated by authors as long as the updated content has not been published online.
• Grey Matters: A practical search tool for evidence-based medicine This online manual provides a thorough list of sources for grey literature in medicine and a helpful checklist to help systematize your process.
• Searching the grey literature: A handbook for searching reports, working papers, and other unpublished research Searching the Grey Literature is for librarians and information professionals interested in learning more about grey literature. This book will aid with crafting a grey lit search successfully, from start to finish. Many types of librarians will find the content of this book useful, particularly those in health or social science.
• Duke University Medical Center Guide to Resource for Searching the Grey Literature Resources for trial registries, pharmacological studies, conference abstracts, government document and more.
• Gray Literature Resources for Agriculture Evidence Syntheses This is a comprehensive list of agricultural grey literature sources, one of several evidence synthesis worksheets and tools provided to help in non-medical evidence synthesis development and implementation.
• Searching for studies: A guide to information retrieval for Campbell Systematic Reviews This document gives comprehensive guidance in searching for evidence syntheses, and offers a robust list of grey literature sources for social science disciplines.
• Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions: AHRQ and the Effective Health Care Program An introduction to clinical resources and strategies in approaching systematic reviews.

Video: Grey literature (3:10 minutes)

• << Previous: 2. Select Databases
• Next: 4. Write a Search Strategy >>
• Last Updated: Feb 14, 2024 4:20 PM
• URL: https://guides.lib.uci.edu/evidence-synthesis

Off-campus? Please use the Software VPN and choose the group UCIFull to access licensed content. For more information, please Click here

Software VPN is not available for guests, so they may not have access to some content when connecting from off-campus.

A Guide to Evidence Synthesis: 3. Select Grey Literature Sources

• Meet Our Team
• Our Published Reviews and Protocols
• What is Evidence Synthesis?
• Types of Evidence Synthesis
• Evidence Synthesis Across Disciplines
• Finding and Appraising Existing Systematic Reviews
• 0. Develop a Protocol
• 1. Draft your Research Question
• 2. Select Databases
• 3. Select Grey Literature Sources
• 4. Write a Search Strategy
• 5. Register a Protocol
• 6. Translate Search Strategies
• 7. Citation Management
• 8. Article Screening
• 9. Risk of Bias Assessment
• 10. Data Extraction
• 11. Synthesize, Map, or Describe the Results
• Evidence Synthesis Institute for Librarians
• Open Access Evidence Synthesis Resources

Video: Grey literature (3:10 minutes)

The Grey Literature Background

What is grey literature  .

Grey (or gray) literature is literature produced by individuals or organizations outside of commercial and/or academic publishers. This can include information such as government reports, conference proceedings, graduate dissertations, unpublished clinical trials, and much more.  The sources you select will be informed by your research question and field of study, but should likely include, at a minimum, theses and dissertations.

Why Search the Grey Literature

Why search the grey literature  .

The intent of an  evidence synthesis  is to synthesize  all available evidence  that is applicable to your research question. There is a strong bias in scientific publishing toward publishing studies that show some sort of significant effect.  Meanwhile, many studies and trials that show no effect end up going unpublished.  But knowing that an intervention had no effect is just as important as knowing that it did have an effect when it comes to making decisions for practice and policy-making. While not peer-reviewed, grey literature represents a valuable body of information that is critical to consider when synthesizing and evaluating all available evidence.

How to Search the Grey Literature

How Do I Search the Grey Literature?  

Finding grey literature and searching it systematically is challenging.  But there are a few approaches that you can take to add some structure to your search of this type of information:

• Refer to grey literature sources used for related evidence syntheses: Refer to both published evidence syntheses and registered protocols. 
• Ask experts in the field for relevant grey literature sources: If you are an expert, include important grey literature sources, and ask colleagues for their recommendations.
• Search  databases that specialize in grey literature:  See the "Grey Literature Sources" box at the bottom of this page for more information.
• Search for theses and dissertations :   There are a number of databases dedicated to theses and dissertations, which you can search using your search terms. See the "Grey Literature Sources" box at the bottom of this page for links to these resources. 
• Search clinical trials:  There may be clinical trials being conducted that are relevant to your research question, but that haven't been published yet or never were published. See the "Grey Literature Sources" box at the bottom of this page  for links to these resources. 
• Identify government agencies and international and non-governmental organizations  that might publish technical papers and reports on your topic.  Search their websites or any online libraries that they may provide.  See the "Grey Literature Sources" box at the bottom of this page  for links to some examples.
• Search conference proceedings and newsletters :  Identify professional organizations that have and/or conferences at which researchers might be presenting work related to your topic. Search those conference proceedings or newsletters on the organization's website or by contacting organizational boards for access to past proceedings that may not be online. See the "Grey Literature Sources" box at the bottom of this page for some examples.
• Contact known researchers  in the field to determine if there are any ongoing or unpublished studies that s/he may be aware of.
• Search professional and trade magazines: Professional magazines contain literature that is written by professionals in the field for other professionals in the field, but that may not be about research. Trade magazines contain advertisements and news very specific to a topic or industry.

How to Manage the Grey Literature Search

How do i manage the grey literature search.

• Identify and record  the sources you will search. The sources you search will be informed by your research question and where you expect to find information related to your question.
• Document where you are searching and your search strategies, including document resource name, URL, search terms, and date searched.
• Collect citation information as you go.
• Adhere to your established inclusion and exclusion criteria when selecting sources.

See below for guidance documents specific to grey literature searching.

Grey Literature Sources Header

Grey literature resources, grey literature sources.

• Grey Literature Databases
• Theses and Dissertations
• Clinical Trials
• Public Policy
• Conference Proceedings
• Preprint Repositories
• Other Resources
• WHO Library Database The WHO library database includes governing documents, reports and technical documentation.
• MedNar MedNar searches across more than 60 medical research sources, including commercial databases, medical societies, NIH resources, and other government resources.
• Global Index Medicus The Global Index Medicus (GIM) provides worldwide access to biomedical and public health literature produced by and within low-middle income countries. The main objective is to increase the visibility and usability of this important set of resources. The material is collated and aggregated by WHO Regional Office Libraries on a central search platform allowing retrieval of bibliographical and full text information.
• NY Academy of Medicine Grey Literature Report (Last Updated 2016) This report is a bimonthly publication of The New York Academy of Medicine (NYAM) alerting readers to new grey literature publications in health services research and selected public health topics. The database platform is keyword searchable and serves as an archive for the cataloged reports. Although it is no longer updated, existing reports can be found through the NYAM catalog.
• ProQuest Dissertations and Theses With more than 2 million entries, PQD&T is the single central, authoritative resource for information about doctoral dissertations and master's theses.
• Networked Digital Library of Theses and Dissertations (NDLT) NDLT is a free international resource for theses and dissertations.
• Center for Research Libraries (CRL) CRL is a resource for institutions outside of the U.S. and Canada.
• OCLC WorldCat Dissertations and Theses OCLC WorldCat Dissertations and Theses "provides fast and convenient access to the dissertations and theses available in OCLC member libraries. Many theses are available electronically, at no charge, directly from the publishing institution." This database can be searched in FirstSearch, which provides Basic, Advanced, and Expert search options. Access may be limited to members of participating institutions.
• Guide to Theses and Dissertations Resources This guide provides information on finding and accessing both Cornell and non-Cornell theses and dissertations, including many resources for non-US literature.
• EThOS (offline) Electronic Theses Online Service (EThOS) contains doctoral theses from candidates in the UK and is maintained by the British Library. Due to the October 2023 cyber attack, this resource is currently unavailable and is estimated to return in 2025.
• Australia New Zealand Clinical Trials Registry Online register of clinical trials being undertaken in Australia, New Zealand and elsewhere. The ANZCTR includes trials from the full spectrum of therapeutic areas of pharmaceuticals, surgical procedures, preventive measures, lifestyle, devices, treatment and rehabilitation strategies, and complementary therapies.
• ClinicalTrials.gov The US registry for clinical trials. Includes new, ongoing, and completed human clinical trials both in the US and countries around the world.
• Cochrane Central Register of Controlled Trials A highly concentrated source of reports of randomized and quasi-randomized controlled trials. The majority of CENTRAL records are taken from bibliographic databases (mainly MEDLINE and Embase), but records are also derived from other published and unpublished sources.
• EU Clinical Trials Register The European Union Clinical Trials Register allows you to search for protocol and results information on interventional clinical trials that are conducted in the European Union (EU) and the European Economic Area (EEA) and clinical trials conducted outside the EU / EEA that are linked to European paediatric-medicine development.
• WHO International Clinical Trials Registry Platform ICTRP provides access to a central database containing the trial registration data sets provided by numerous international registries. It also provides links to the full original records.
• Wikipedia Clinical Trial Registries List This Wikipedia entry contains a list of links to individual clinical trial registries by country.
• World Bank ​The World Bank now makes all of their publications openly available online.
• WHO Institutional Repository for Information Sharing (IRIS) Institutional WHO database of intergovernmental policy documents and technical reports. Can search by IRIS by region (Africa, Americas, Eastern Mediterranean, Europe, South-East Asia, Western Pacific).
• Health Research Web Health Research Web is an interactive resource containing information about national health research systems, ethics review committees, local and regional policies and research priorities and more. It is built by user contributors, and the data may be incomplete.

This list of conference proceedings is not comprehensive, but are ideas of where to search. We recommend that you search proceedings for the three most relevant conferences for your evidence synthesis.

• OCLC PapersFirst OCLC PapersFirst is an OCLC index of papers presented at conferences worldwide.
• BIOSIS Previews A subcollection of Web of Science, BIOSIS Previews searches across journals, meetings, patents, and books in the life sciences and biomedical fields.
• arXiv arXiv is a free distribution service and an open-access archive for scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.
• medRxiv medRxiv (pronounced "med-archive") is a free online archive and distribution server for complete but unpublished manuscripts (preprints) in the medical, clinical, and related health sciences.
• bioRxiv Another complementary archive to arXiv, bioRxiv is a preprint repository that where authors can share unpublished preprints in the life sciences (including biology, ecology, neuroscience and more).
• OSF Preprints OSF Preprints is a general preprint repository that covers architecture, business, engineering, life sciences, physical sciences and mathematics, arts and humanities, education, law, medicine and health sciences and social and behavioral sciences, among others.
• Grey Matters: A practical search tool for evidence-based medicine Grey Matters is an online manual that provides a thorough list of sources for grey literature in medicine and a helpful checklist to help systematize your process. (Requires free account)
• Searching the grey literature: A handbook for searching reports, working papers, and other unpublished research Searching the Grey Literature is for librarians and information professionals interested in learning more about grey literature. This book will aid with crafting a grey lit search successfully, from start to finish. Many types of librarians will find the content of this book useful, particularly those in health or social science.
• Duke University Medical Center Guide to Resource for Searching the Grey Literature Scroll down to the bottom of this Duke University page to find resources for trial registries, pharmacological studies, conference abstracts, government documents and more.
• Gray Literature Resources for Agriculture Evidence Syntheses This OSF-hosted sheet is a comprehensive list of agricultural grey literature sources, one of several evidence synthesis worksheets and tools provided to help in non-medical evidence synthesis development and implementation.
• Searching for studies: A guide to information retrieval for Campbell Systematic Reviews This document gives comprehensive guidance in searching for evidence syntheses, and offers a robust list of grey literature sources for social science disciplines.
• << Previous: 2. Select Databases
• Next: 4. Write a Search Strategy >>
• Last Updated: Apr 19, 2024 12:37 PM
• URL: https://guides.library.cornell.edu/evidence-synthesis

USF Libraries Hours by campus

Libraries locations.

• Libraries Hours
• Outages & Maintenance Alerts

RESEARCH TOOLS

• Subject & Course Guides
• USF Libraries Catalog
• Quicksearch All-in-one-search
• Citing Sources
• Find my Librarian

GUIDES / HOW-TO

• Tutorials & Workshops
• Finding Books and Articles
• Finding Reserves
• Checking Out & Renewing
• Reserve a Study Room
• Additional Help Topics
• star Other Services
• For Faculty
• For Graduate Students
• For Undergrads
• Requesting Books & Articles (ILL)
• Textbook Affordability (TAP)
• Library Instruction
• Laptop Checkout
• Schedule Research Help
• Geographic Information Systems
• Data Management Planning
• Copyright & Intellectual Property
• Scholarly Publishing
• Other Services

COLLECTIONS

• What are Collections?
• Special Collections
• Digital Collections
• Digital Heritage & Humanities
• Digital Commons @ USF
• Oral Histories
• Online Exhibitions
• Printing in the Library
• IT Help Desk
• Digital Media Commons (DMC)
• Writing Studio
• Office of Development
• Office for Undergraduate Research
• Directions to the Library
• Library Info & Floor Maps
• Connect From Off Campus
• Renew Materials Online
• Check UBorrow Status
• Printing Help
• Report a Problem
• About the USF Libraries

Systematic Reviews for Social Sciences

• Systematic Reviews
• Mapping Review
• Mixed Methods Review
• Rapid Review
• Scoping Review
• Develop the Question
• Define Inclusion/Exclusion Criteria
• Develop A Review Protocol
• Create Search Strategies
• Select Studies
• Extract Data
• Assess the Quality of Studies
• Synthesize Data and Write the Report
• Review Tools
• Librarian Involvement
• Grey Literature

What is Grey Literature?

Grey Literature is a field in library and Information science that deals with the production, distribution, and access to multiple document types produced on all levels of government, academia, business, and organization in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body.

Types of grey literature include theses and dissertations, conference proceedings, newsletters, reports, government documents, informal communications, translations, census data, research reports, technical reports, standards, patents, videos, clinical trials and practice guidelines, eprints, preprints, wiki articles, emails, blogs, listserv archives, and repository content.

For more recommendations of subject-specific grey literature resources beyond what is presented below, consult the USF Libraries Subject Guide most closely related to your topic.

Theses and Dissertations

• Dissertations & Theses @ University of South Florida - FCLA This link opens in a new window This database gives access to the dissertations and theses produced by students at USF.
• Networked Digital Library of Theses and Dissertations This link opens in a new window Features the Networked Digital Library of Theses and Dissertations (NDLTD), an initiative to improve graduate education, increase the sharing of knowledge, and help universities build their information infrastructure. NDLTD is a project team at Virginia Polytechnic Institute and State University in Blacksburg.
• ProQuest Dissertations and Theses, A&I This link opens in a new window This database is the world’s most comprehensive collection of dissertations and theses from around the world, spanning from 1743 to the present day. -- Simultaneous Users: Unlimited
• ProQuest Dissertations and Theses [PQDT] Global This link opens in a new window ProQuest Dissertations and Theses: Global (PQDTGlobal) is the world's most comprehensive collection of full-text dissertations and theses. As the official digital dissertations archive for the Library of Congress and as the database of record for graduate research, PQDTGlobal includes millions of searchable citations to dissertations and theses together with over a million full-text dissertations that are available for download in PDF format.
• WorldCat Dissertations and Theses [OCLC FirstSearch] This link opens in a new window Provides access to to over 5 million records of dissertations and theses available in OCLC member libraries catalogs in the WorldCat database. Many theses are available electronically, at no charge, directly from the publishing institution.

Conference Proceedings

Abstracts, poster presentations, and presented papers are considered grey literature because they may never be seen other than by those who attended the conference. however, they can be a good source of information about studies, particularly studies that never made it to publication..

• AIP Conference Proceedings This link opens in a new window AIP Conference Proceedings report findings presented at many of the most important scientific meetings around the world. Published proceedings are valuable as topical status reports providing quick access to information before it appears in the traditional journal literature. USF has perpetual access to the American Institute of Physics Journal Digital Archives.
• Conference Proceedings Citation Index [1990 Present] This link opens in a new window Conference Proceedings Citation Index, accessed via Web of Science Core Collection, helps researchers access the published literature from the most significant conferences, symposia, seminars, colloquia, workshops, and conventions worldwide. This resource offers a complete view of conference proceedings and their impact on global research, allowing researchers to use cited reference searching to track emerging ideas and new research beyond what is covered in the journal literature. -- USF's subscription includes: -- - Science - Social Sciences & Humanities -- Simultaneous Users: Unlimited
• PapersFirst [OCLC FirstSearch] This link opens in a new window Indexes over two million papers presented at conferences worldwide. Also covers congresses, expositions, workshops, symposiums, and other meetings. -- Simultaneous Users: Unlimited
• Proceedings [OCLC FirstSearch] This link opens in a new window Contains citations to proceedings of every congress, symposium, conference, exposition, workshop and meeting received at The British Library from October 1993 to the present. -- Simultaneous Users: Unlimited

Grey Literature Databases

• DOE Pages For access to journal articles and peer-reviewed accepted manuscripts resulting from DOE research funding, please visit the Department of Energy Public Access Gateway for Energy and ScienceBeta (DOE PAGESBeta), which is accessible and searchable at no charge to users.
• OpenGrey / SIGLE System for Information on Grey Literature in Europe (SIGLE) links to over 700,000 bibliographical references of grey literature produced in Europe and allows you to export records and locate the documents.
• Social Science Research Network This link opens in a new window The SSRN eLibrary consists of an abstract database containing abstracts of scholarly working papers and forthcoming papers and an electronic paper collection of downloadable full text documents in pdf format.
• National Criminal Justice Reference Service [NCJRS] Abstracts Database This link opens in a new window This database is published by the Office of Justice Programs, U.S. Department of Justice's National Criminal Justice Reference Service, an information clearinghouse for research, policy, and practice related to criminal and juvenile justice, and drug control. The NCJRS Abstracts Database contains summaries of U.S. and international publications, including federal, state, and local government reports, books, research reports, journal articles, audiovisual presentations, and unpublished research.
• PAIS Index This link opens in a new window This resource covers issues in the public debate through selective coverage of a wide variety of international sources including journal articles, books, government documents, statistical directories, grey literature, research reports, conference papers, web content, and more. PAIS International is the current file covering 1972 to present, and PAIS Archive includes the content from printed volumes published 1915-1976
• National Technical Reports Library (NTIS)   The NTIS serves as the largest central resource for government-funded scientific, technical, engineering, and business related information available today. For more than 60 years NTIS has assured businesses, universities, and the public timely access to approximately 3 million publications covering over 350 subject areas.
• ProQuest Congressional [Congressional Publications] This link opens in a new window Comprehensive online collection of primary source congressional publications and legislative research materials covering all topics, including government, current events, politics, economics, business, science and technology, international relations, social issues, finance, insurance, and medicine. Finding aid for congressional hearings, committee prints, committee reports and documents from 1970-present, and the daily Congressional Record from 1985-present. Compiled legislative histories from 1969-present.

Other Resources

• Grey Matters Grey Matters is an open access tool for national, international websites, clinical practice guideline producers, regulatory agencies, health economic resources, etc.
• OpenDOAR OpenDOAR is an authoritative directory of academic open access repositories.
• Grey Source GreySource provides examples of grey literature to the average net-user and in so doing profiles organizations responsible for its production and/or processing. GreySource identifies the hyperlink directly embedded in a resource, thus allowing immediate and virtual exposure to grey literature.
• Social Systems Evidence The types of syntheses in Social Systems Evidence include evidence briefs for policy, overviews of systematic reviews, systematic reviews, systematic reviews in progress (i.e. protocols for systematic reviews), and systematic reviews being planned (i.e. registered titles for systematic reviews). Social Systems Evidence also contains a continuously updated repository of economic evaluations in these same domains.
• << Previous: Librarian Involvement
• Next: Learn More >>
• Last Updated: Aug 31, 2023 4:34 PM
• URL: https://guides.lib.usf.edu/systematicreviews

Systematic Reviews and Meta-Analyses: Grey Literature

• Get Started
• Exploratory Search
• Where to Search
• How to Search

Grey Literature

• What about errata and retractions?
• Eligibility Screening
• Critical Appraisal
• Data Extraction
• Synthesis & Discussion
• Assess Certainty
• Share & Archive

This page provides more information about grey literature.

Conference Proceedings

Grey Literature is a broad term that varies across discipline. In very general terms, grey literature is anything other than peer-reviewed, empirical research. Examples of grey literature include: unpublished research, government publications, theses and dissertations, statistics and datasets, news articles, trade and popular magazines, blogs, podcasts, social media posts (e.g., tweets), slide decks, or factsheets. Including grey literature in your systematic review can help mitigate publication bias.

According to  GreyNet, 2013 :

Grey Literature...deals with the production, distribution, and access to multiple document types produced on all levels of government, academics, business, and organization in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body.

According to Paez, 2017 :

Publication bias refers to the propensity for only studies reporting positive findings to be published, and may skew the results of the meta-analysis and systematic review. As gray literature may describe neutral or negative results, including it with commercially published research may provide a more balanced understanding of the evidence and a more accurate effect sizes ...excluding unpublished studies may compromise the validity and reliability of meta-analyses and the specificity of systematic reviews.

Should you include grey literature? 

Can a type (or several types) of grey literature  help to answer your research question ? Do you have time for additional searches and processing additional material? If so, you should consider including grey literature in your review!

However, not all questions will benefit from grey literature and you cannot include  all  kinds of grey literature.

Wilson (2009) states:

"Ignoring the grey literature on the basis that it is of inferior methodological quality is empirically and logically invalid . Furthermore, the use of publication status as a proxy measure for methodological quality abdicates the important role of carefully evaluating the robustness of the evidence to others using unknown criteria (i.e., the journal peer-review process)."

Ćurković M, Košec A. Bubble effect: including internet search engines in systematic reviews introduces selection bias and impedes scientific reproducibility. BMC Med Res Methodol. 2018 Nov 13;18(1):130. doi: 10.1186/s12874-018-0599-2. PMID: 30424741; PMCID: PMC6234590.

Godin, K., Stapleton, J., Kirkpatrick, S.I.  et al.   Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada.   Syst Rev   4 , 138 (2015). https://doi.org/10.1186/s13643-015-0125-0

Gusenbauer M, Haddaway NR. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources . Res Synth Methods. 2020 Mar;11(2):181-217. doi: 10.1002/jrsm.1378. Epub 2020 Jan 28. PMID: 31614060; PMCID: PMC7079055

Mahood, Q., Van Eerd, D., & Irvin, E. (2014). Searching for grey literature for systematic reviews: Challenges and benefits: MAHOOD ET AL . Research Synthesis Methods , 5 (3), 221–234. https://doi.org/10.1002/jrsm.1106

Paez A.  Grey literature: An important resource in systematic reviews . J Evid Based Med. 2017 Dec 21. doi: 10.1111/jebm.12265. Epub ahead of print. PMID: 29266844.

Check out  Appendix I  for an annotated  list of grey lit sources!

Scherer, R.W., Saldanha, I.J.  How should systematic reviewers handle conference abstracts? A view from the trenches .   Syst Rev   8,  264 (2019). https://doi-org.ezproxy.lib.vt.edu/10.1186/s13643-019-1188-0

Wilson, D.B. Missing a critical piece of the pie: simple document search strategies inadequate for systematic reviews .  J Exp Criminol   5,  429–440 (2009). https://doi-org.ezproxy.lib.vt.edu/10.1007/s11292-009-9085-5

Conference proceedings are some of the most common types of grey literature to consider in systematic reviews. Many of the databases you'll search  will  include conference proceedings . But there are databases that contain only conference proceedings such as  ProceedingsFirst , as well as databases containing the conference proceedings for a specific conference organization (which may require affiliation to access). If you plan to include a specific type of grey literature, like conference proceedings, it is not enough to include what you happen to find through your primary search. You should also search purposefully for this type of literature by targeting focused databases, just as you did when searching for peer-reviewed empirical research.

According to  PRISMA-S, Item 4:

Including conference proceedings in a systematic review search helps  minimize bias ...specify the conference names , the dates of conferences included, and the method used to search the proceedings...

Scherer & Saldanha, 2019  offer a more pragmatic approach , suggesting reviewers consider their capacity and corpus:

If available evidence is  sparse or conflicting , it may be worthwhile to include conference abstracts. If results from conference abstracts are included, then it is necessary to make diligent attempts to  contact the authors  of the abstract and  examine study registers  and  published protocols  to obtain further and confirmatory information on methods and results.

This is important as, in a Cochrane Network Meta-Analysis,  Hackenbroich, Kranke, Meybohm, & Weibel (2022)  found:

Most conference abstracts on PONV were insufficiently reported regarding review's narrow inclusion criteria and could not be included in NMA. The resource-intensive search and evaluation of abstracts did not substantially extent the full-text evidence base of the review, given the few adequately reported abstracts. Conferences should oblige authors to adhere to CONSORT for abstracts.

• << Previous: How to Search
• Next: What about errata and retractions? >>
• Last Updated: Apr 12, 2024 12:41 PM
• URL: https://guides.lib.vt.edu/SRMA

• Methodology
• Open access
• Published: 29 September 2016

Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: critical reflections on three case studies

• Jean Adams   ORCID: orcid.org/0000-0002-5733-7830 1 , 2 , 3 ,
• Frances C. Hillier-Brown 3 , 4 ,
• Helen J. Moore 3 , 4 ,
• Amelia A. Lake 3 , 4 , 5 ,
• Vera Araujo-Soares 2 , 3 ,
• Martin White 1 , 2 , 3 &
• Carolyn Summerbell 3 , 4  

Systematic Reviews volume  5 , Article number:  164 ( 2016 ) Cite this article

55k Accesses

242 Citations

83 Altmetric

Metrics details

Grey literature includes a range of documents not controlled by commercial publishing organisations. This means that grey literature can be difficult to search and retrieve for evidence synthesis. Much knowledge and evidence in public health, and other fields, accumulates from innovation in practice. This knowledge may not even be of sufficient formality to meet the definition of grey literature. We term this knowledge ‘grey information’. Grey information may be even harder to search for and retrieve than grey literature.

On three previous occasions, we have attempted to systematically search for and synthesise public health grey literature and information—both to summarise the extent and nature of particular classes of interventions and to synthesise results of evaluations. Here, we briefly describe these three ‘case studies’ but focus on our post hoc critical reflections on searching for and synthesising grey literature and information garnered from our experiences of these case studies. We believe these reflections will be useful to future researchers working in this area.

Issues discussed include search methods, searching efficiency, replicability of searches, data management, data extraction, assessing study ‘quality’, data synthesis, time and resources, and differentiating evidence synthesis from primary research.

Conclusions

Information on applied public health research questions relating to the nature and range of public health interventions, as well as many evaluations of these interventions, may be predominantly, or only, held in grey literature and grey information. Evidence syntheses on these topics need, therefore, to embrace grey literature and information. Many typical systematic review methods for searching, appraising, managing, and synthesising the evidence base can be adapted for use with grey literature and information. Evidence synthesisers should carefully consider the opportunities and problems offered by including grey literature and information. Enhanced incentives for accurate recording and further methodological developments in retrieval will facilitate future syntheses of grey literature and information.

Peer Review reports

Public health researchers may want to include ‘grey literature’ in evidence syntheses for at least three reasons. Firstly, including grey literature can reduce the impact of publication bias as studies with null findings are less likely to be published in peer-reviewed journals [ 1 ]. Secondly, grey literature can provide useful contextual information on how, why, and in whom complex public health interventions are effective [ 2 – 6 ]. Finally, syntheses of grey literature can help applied researchers and practitioners understand what interventions exist for a particular problem, the full range of evaluations (if any) that have been conducted, and where further intervention development and evaluation is needed.

Numerous definitions of grey literature exist. These tend to focus on the fact that it is not controlled by commercial organisations, making it difficult to search for and retrieve [ 7 – 9 ]. One common definition restricts grey literature to literature ‘protected by intellectual property rights, of sufficient quality to be collected and preserved by library holdings or institutional repositories’ [ 10 ]. Other definitions are more inclusive and propose that, given the growth of new forms of media, grey literature should not be restricted to written ‘literature’ [ 4 ].

Much knowledge and evidence in applied settings, such as public health practice, accumulates from innovation in practice [ 7 ]. This may include the rationale for why new approaches were tried; what changes, if any, were made to previous approaches and why; what was done and how; and what happened. In some cases, this may be accompanied by more formal process evaluation and, most rarely, outcome evaluation (Hillier-Brown F, Summerbell C, Moore H, Wrieden W, Abraham C, Adams J, Adamson A, Araujo-Soares V, White M, Lake A. A description of interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England: a systematic mapping and evidence synthesis. BMC public health 2015, unpublished). Interventions and evaluations that were primarily conducted as part of, or to inform, practice may be particularly unlikely to be described in peer-reviewed publications or even formally documented in reports available to others in electronic or hard copy. Information on these activities may, instead, be stored in more private or informal spaces such as meeting notes, emails, or even just in people’s memories. This information is likely to be of insufficient formality to meet the definition of grey literature. We term this ‘grey information’.

The phrase ‘grey information’ has been used previously to extend the concept of grey literature to a wider range of sources [ 11 ], but it is not widely used and we are not aware of a previously stated clear definition. The term ‘grey data’ [ 12 ] has also been used specifically to describe user-generated web content—something that we feel is more formal and public than ‘grey information’ but less formal than ‘grey literature’. Table  1 describes defining aspects and examples of the three terms: grey literature, grey data, and grey information.

Systematically identifying grey literature and grey data is not a straightforward task [ 5 , 7 – 9 , 12 , 13 ]. Systematically identifying ‘grey information’ is likely to be even more challenging. A number of case studies have been published describing procedures for searching and retrieving ‘grey literature’ in public health contexts [ 13 , 14 ]. These tend to adopt relatively similar approaches including searching databases of peer-reviewed and grey literature; conducting structured searches of relevant websites and search engines; and contacting relevant experts [ 5 , 8 , 9 , 13 ].

On three occasions, various authors of this article have attempted to systematically search for and synthesise public health grey literature and information. Here, we briefly describe our experiences of these three case studies and then critically reflect on these. ‘Critical reflection’ is a concept most often associated with adult learning and professional development. Although poorly and diversely defined, critical reflection is generally associated with post hoc examination of experiences in an attempt to improve future practice [ 15 , 16 ]. Our aim was to provide insights on searching for and synthesising grey literature and information that may be useful for future researchers.

The aims, methods, results, and conclusions of our three case studies are summarised in Table  2 .

Review 1: the health, social, and financial impacts of welfare right advice delivered in healthcare settings [ 17 ]

Our first review included grey literature alongside peer-reviewed literature in a systematic review of the health, social, and financial impacts of welfare rights advice delivered in healthcare settings [ 17 ]. In part, this systematic review was conducted in preparation for an application for funding for a randomised controlled trial of the impacts of welfare rights advice on health in older people [ 18 , 19 ]. Thus, we were interested both in the extent and findings of other research. We conducted a quantitative synthesis of the average financial impacts of welfare rights advice and a narrative synthesis of other quantitative and qualitative findings.

As expected, less than half of the evaluations of welfare right advice included in the review were published in peer-reviewed journals. The remainder were published in reports published by delivery organisations, universities, other research organisations, and service and research funders.

Review 2: adult cooking skill interventions in England [ 20 ]

Our second attempt to review grey literature explored the nature, content, and range, but not effects, of interventions seeking to enhance adult cooking skills delivered in England [ 20 ]. Our intention was to identify the most sustainable and theoretically promising of these to take forward for more formal evaluation. Similar to other reviews [ 21 ], our synthesis focused on categorising interventions according to delivery setting and training model and summarising the training delivered, throughput, setup and running costs, funding, and behaviour change techniques used [ 22 ].

This review focused entirely on grey literature and information and did not include any searching for peer-reviewed literature. A scoping review of peer-reviewed outcome evaluations of adult cooking skill interventions was conducted in parallel [ 23 ].

Review 3: interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England (Hillier-Brown F, Summerbell C, Moore H, Wrieden W, Abraham C, Adams J, Adamson A, Araujo-Soares V, White M, Lake A. A description of interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England: a systematic mapping and evidence synthesis. BMC public health 2015, unpublished)

Finally, we recently completed a review of interventions aiming to promote healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by food outlets in England (Hillier-Brown F, Summerbell C, Moore H, Wrieden W, Abraham C, Adams J, Adamson A, Araujo-Soares V, White M, Lake A. A description of interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England: a systematic mapping and evidence synthesis. BMC public health 2015, unpublished). This explored the nature and range of interventions implemented and summarised evaluation findings. We used a narrative approach to evidence synthesis, characterising interventions, identifying issues of design and delivery, and summarising evaluation findings on process, acceptability, cost, and impact. Our intention in this case was to use the findings to inform development and evaluation of new interventions based on the most promising features of previous ones.

Whilst this review did include searches of peer-reviewed literature, these only identified one included study—although two other relevant peer-reviewed papers were identified using other methods. As in review 2, a linked review of peer-reviewed evidence was conducted in parallel [ 24 ].

Results and discussion

Whilst there is much useful guidance available on evidence synthesis in general [ 25 – 27 ], and searching for and synthesising grey literature in particular [ 5 , 8 , 9 , 13 , 28 – 30 ], one size rarely fits all. Throughout, and in common with the best research practice, our methods were guided by our aims.

In evidence syntheses, the sensitivity, specificity, and type of information retrieved by searches is highly dependent on the search strategy used. As described above and in Table  2 , we used a variety of different methods to search for information across all three reviews. We reflect on some of the issues raised below and summarise some of our conclusions in Table  3 .

Search methods

In all three cases, and as recommended by others [ 5 , 8 , 9 , 13 , 28 – 30 ], we used a wide variety of methods to search for relevant grey literature and information. Across our three examples, we searched trial databases (e.g. www.isrctn.com ), grey literature databases (e.g. www.opengrey.eu ), websites of relevant organisations (e.g. charities with an interest in social inclusion in review 1), and a popular internet search engine (i.e. www.google.co.uk ).

We also contacted those working in the areas we were interested in. We sent both personalised requests to key informants, as well as more generic requests to professional organisations and groups, using a variety of methods. In reviews 1 and 3, researchers working in relevant fields were contacted via email and requests for information were sent to relevant email lists, posted on online bulletin boards, and published in the ‘professional press’ (e.g. newsletters of professional organisations). In reviews 2 and 3, we also attempted to contact relevant individuals working in all local public health departments in England. In review 3, we incorporated social media into our search strategy.

Review 1 was conducted in 2005 when social media and social networks were less well established than they are now. To target large networks of professionals in this case, we published requests for help in the ‘professional press’. By the time review 3 was conducted, in 2014, social media had become an important space for professional networking. We posted numerous tweets requesting relevant information and asking those who saw them to repost (i.e. ‘retweet’) them to their own networks—in order to increase the potential number of people who saw these requests. Many of these tweets tagged (i.e. ‘@mentioned’) relevant professional organisations. We are not aware of previous reviewers using social media to identify grey (or peer-reviewed) literature or information.

This transition in methods from reviews 1–3 over just less than a decade reflects how information storage and sharing has changed over this time in the UK. At the same time, information storage and sharing patterns may vary internationally. Search methods need to adapt to local and international trends in information systems, and researchers should be flexible to this.

Searching efficiency

As with ‘typical’ systematic reviews [ 31 , 32 ], our searching sacrificed specificity for sensitivity. Searches yielded many results that did not meet our inclusion criteria. The resource and scientific implications of the trade-off between search specificity and sensitivity have been widely discussed in the systematic review literature [ 33 , 34 ].

Previous case studies have described very different ‘hit’ rates associated with different grey literature search strategies. In a review of interventions to promote walking and cycling, requests for help emailed to key informants added little to database searching [ 35 ]. Whereas, in a review of behaviour change interventions published only in grey literature, 70 % of items included in the final synthesis came from key informants [ 5 ]. Similarly, we found that different methods for locating information were differentially effective across our three reviews. In review 1, generic requests sent to email lists and published in the professional press were particularly useful. On a number of occasions, these requests were passed through a number of people before someone responded with relevant information—further adding to the time taken to conduct searching that is discussed below. Perhaps, similarly, in review 3, Twitter requests were particularly valuable. These were widely retweeted, vastly increasing the pool of potential viewers, but this appeared to be a much quicker process than cascading of email requests and requests in the professional press.

The efficiency of different search methods are, at least partly, dependent on the quality of the search strategy used. Simple comparisons, such as those described above, are not necessarily fair. Nor is it clear if the differences in efficiency are predictable. If the efficiency of different approaches to searching could be predicted in advance, this could help reviewers to focus their resources.

Our resources were most limited in review 2, and it became evident early in searching that we would not be able to complete a comprehensive search for all adult cooking skill interventions in England. We made a pragmatic decision to focus instead on identifying intervention types—based on delivery context and training model. As others have done [ 13 ], we borrowed the concept of ‘data saturation’ from qualitative research and stopped searching when we felt we were not identifying any new intervention types. We felt that sacrificing sensitivity in this case did not compromise our ability to meet our aims.

Using others to target searching

In reviews 2 and 3, we attempted to ask all local public health departments in England what relevant projects they were aware of. We are not aware of any peer-reviewed publications which report the efforts of other evidence synthesisers, or indeed primary researchers, who have attempted to systematically contact all local public health departments across one country in this way. That said, we recognise that the gathering of data on the activity and type of public health interventions conducted at various geographic levels is a relatively common activity. To facilitate this, we identified named individuals and contact email addresses for those with relevant roles using internet searches and telephone calls. This was a time-consuming task in itself. The requests for information we sent specifically asked recipients to pass our enquiries onto those they felt were best placed to respond. As with other email requests (see above), there were examples where messages had been passed through a number of individuals before someone responded.

Replicability of searches

Whilst in all cases, we had clear plans describing what we felt were comprehensive, systematic, and replicable approaches to information searching, it is hard to claim that these led to replicable results. Certainly, it would be possible for future investigators to replicate our search methods, but it is unlikely that these would lead to the same results on replication, as would be expected when using electronic databases. On two different occasions, different people would be likely to see calls for information on social networks or in the professional press. Even if the same people did see requests for help on different occasions, many other contextual factors may influence how likely they were to respond or pass them onto those most likely to be able to respond.

As time passes and grey literature and information becomes lost or forgotten, potential respondents’ ability to provide usable information may also decline. Whilst contacting both those currently and previously in posts as key informants may, theoretically, reduce this problem, it may not be practically possible. Others have highlighted the problem of replicability in relation to internet searching, particularly using search engines such as www.google.com which returns results based on, amongst other things, recent popularity [ 8 , 9 ].

The conclusion that searching for grey literature and information can be systematic, but not necessarily replicable, reinforces the importance of using many overlapping searching approaches. This maximises the chances that any particular piece of relevant information will be found.

Developing the ‘best’ search methods

Whilst our search methods were similar to, and built on, those described by others as well as on ‘best practice’ guidance [ 5 , 8 , 9 , 13 , 28 – 30 ], it is difficult to be sure what the ‘best’ search methods for retrieving grey literature and information are. Whilst it is possible to validate search approaches in peer-reviewed literature against a ‘gold standard’ of hand searching [ 31 , 32 ], no similar gold standard exists for grey literature and information: there is no definitive repository against which other search methods can be compared. This makes it difficult to ever be sure that all relevant information has been found or validate new search methods.

Data management

In all three reviews, we found data management to be challenging. Technology now allows fairly straightforward integration of academic databases and reference management software—both of which facilitate information organisation and record keeping. Such workflows are not well developed for grey literature and information. Developing clear filing and recording systems, using simple spreadsheets, helped us to keep track of where and how information had been identified.

However, we found it harder to capture other aspects of our searches. For example, whilst tools like NCapture allow social media content to be imported in NVivo for qualitative analysis, they do not necessarily provide a useful facility for capturing how many people (and who) ‘retweeted’ a particular tweet. It is even harder to capture when requests for information are circulated using more private methods such as email. For these reasons, we are not able to provide accurate estimates of how many people saw our requests for information.

Data extraction

In all three reviews, we developed and used data extraction forms to record information. In review 1, we adopted a similar approach to data extraction used in many ‘typical’ systematic reviews—if information was not provided in the written report we obtained, we assumed this information was missing. However, systematic review guidance encourages reviewers to attempt to minimise missing data by contacting authors of original papers [ 25 ]. We adopted an approach much more similar to this in reviews 2 and 3. In fact, many data extraction forms in these reviews were completed during telephone calls or following email conversations with key informants. To maximise accuracy, in review 3, we asked informants to check electronic versions of completed data extraction forms. As there is often little or no documentary evidence to refer back to, ensuring data extraction forms are as accurate and complete as possible is particularly important in reviews of grey literature and information. This reflects and reinforces the fact that much information on interventions in public health practice is not well documented and can be ‘temporary’: once the relevant individuals move to new posts, and interventions recede into the past, individual and institutional memories are likely to fade. This further contributes to the limited replicability of this sort of grey information searching.

Despite the efforts we made in reviews 2 and 3 to speak with those directly responsible for intervention design and delivery, we were often not able to obtain the information we intended to capture. For example, of 102 interventions identified in review 3, we were not able to obtain any information beyond a programme name in 27 cases. In most, if not all, cases, our failure to obtain information appeared to be because such information was not documented or easily obtainable. For example, many of the costs of public health interventions in everyday practice are unclear even to those responsible for them. Whilst the cost of additional staff may be explicit, costs for office space for those staff might be absorbed by organisations and so be much more implicit.

The problem of limited data availability is common to all types of evidence synthesis, but others have noted it as a particular problem when synthesising grey literature [ 7 , 13 ]. When attempting to synthesise the extent of public health practice, it may be important to be aware of the types of information that are and are not important to practitioners and easy for them to record and hence are likely to be documented. For example, service throughput appears to be much more likely to be documented than outcomes of interventions (Hillier-Brown F, Summerbell C, Moore H, Wrieden W, Abraham C, Adams J, Adamson A, Araujo-Soares V, White M, Lake A. A description of interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England: a systematic mapping and evidence synthesis. BMC public health 2015, unpublished) [ 17 ].

Risk of bias and value of information

The risk of bias of any piece of information is dependent, in part, on the question it is being used to answer. In review 2, and part of review 3, our aims were to describe the nature and range of particular classes of interventions. The risk of bias of individual pieces of grey literature and information in this case is likely to be low—there is little reason why such information would be misrecorded. In contrast, in review 1 and part of review 3, we aimed to synthesise evaluation findings. The risk of bias of grey literature and information in this case may be likely to be higher. Indeed, in reviews 1 and 3, we described some aspects of evaluation methods relating to risk of bias. In both cases, we concluded that the majority of studies were methodologically weak and at high risk of bias.

Evaluations found in grey literature and information may be at high risk of bias for a number of reasons. In public health practice, evaluations are often conducted by the same practitioners who developed and delivered an intervention. This results in an inherent conflict of interest which may increase risk of bias. In public health practice, resources for evaluation are often limited, limiting the scope of what is possible [ 36 ]. Furthermore, the interest of funders and practitioners is often on throughput rather than outcomes [ 37 ], limiting the scope of what is necessary. Whilst many evaluations included in our reviews were at high risk of bias in terms of conclusions about effects on outcomes, they may well have been fit for the purpose for which they were conducted.

Methods for assessing risk of bias in controlled trials are well established [ 38 , 39 ], and tools for other types of study are becoming available [ 40 – 42 ]. However, these approaches may be too narrow in perspective for grey literature and information. Realist synthesis takes a researcher-driven ‘value of information’ approach to assessing studies, rather than the more familiar protocol-driven risk of bias approach used in ‘typical’ systematic reviews. In the value of information paradigm, individual studies are included if the information they provide is considered relevant and rigorous enough to help contribute to answering the research question [ 6 , 43 , 44 ]. Whilst this requires researchers to make judgements about what is ‘relevant and rigorous enough’, it may result in inclusion of more potentially useful grey literature and information than stricter approaches which exclude studies based on risk of bias assessments.

Data synthesis

Many approaches to data synthesis in the context of systematic reviewing and evidence synthesis have now been described, and these are not limited to quantitative meta-analysis [ 25 , 45 ]. Although we performed a quantitative synthesis in review 1, this focused on the economic benefits of welfare rights advice to recipients (which could be summarised in £/week). We were not able to summarise health and social implications so simply and used narrative syntheses for these.

In review 3, in an attempt to capture all the data available to us, we adopted a three tiered approach to synthesis. Firstly, we listed all relevant interventions that we found ( n  = 102; tier 1). Secondly, for those interventions for which we had further information on content and delivery, we summarised this using a standard template ( n  = 75; tier 2). Finally, we summarised the results of any evaluations of included interventions ( n  = 30; tier 3). Interventions in each tier were nested within each other such that all interventions were included in the tier 1 synthesis, but only a subset of these were included in tier 2, and only a subset of those in tier 2 were included in the tier 3.

These differences in synthesis approach reflect both the contrasting aims of different reviews and how flexible and responsive researchers should be to the realities of data availability within grey literature and grey information.

Time and resources

Systematic reviews can be time and resource intensive. In ‘typical’ systematic reviews, preliminary scoping reviews can help reviewers estimate the size of a full review and resources required [ 46 ]. ‘Rapid reviews’ of peer-reviewed literature offer the hope and potential for conducting much quicker evidence syntheses that arrive at the same conclusions as full reviews [ 47 – 49 ]. Unfortunately, there is no clear equivalent of scoping or rapid reviews in relation to grey literature and information. As others have noted, searching for less formally archived information is, almost by nature, time-consuming and inefficient [ 5 , 8 , 50 ].

Encouraging public health practitioners to deposit intervention documents and information in online repositories (e.g. www.ncdlinks.org ) could enable more efficient information retrieval on current and recent practice [ 7 ]. But the utility of such databases relies heavily on their coverage, and previous attempts to ensure high coverage have been varying in their success [ 51 ]. With few obvious current incentives for busy practitioners to deposit information in these repositories, it is not necessarily clear how they could be made more useful. Further attention could be given to developing such incentives. In addition, developing better searching and retrieval methods should also facilitate syntheses of grey literature and information, particularly using more sophisticated methods for internet searching such as text analytics or data mining [ 7 , 52 ]. However, if grey information is not recorded in a searchable way (e.g. is retained only on private networks or in memory), this is also only a partial solution. Action is required to improve both information deposition and information retrieval.

Differentiating evidence synthesis from primary research

Although we approached and considered all three of our case studies as evidence syntheses, they could be considered as verging on primary research. This is particularly the case for reviews 2 and 3 where we made attempts to contact all local authorities in England and collect unpublished information via telephone or email interviews with key informants. Contacting authors is encouraged in ‘typical’ systematic reviews, particularly to collect information that may be incompletely recorded in published outputs [ 25 ]. This type of contact is not routinely considered primary research, as the contact is limited to clarifying or augmenting existing published information—rather than collecting entirely new data. However, in many cases in reviews 2 and 3, no published information was available to clarify or augment meaning that these reviews could, perhaps, be considered as collecting new data.

This grey area between evidence synthesis and primary research is particularly important in terms of research ethics. In general, research ethics committee review is not required for evidence synthesis projects because they do not involve research participants [ 53 ]. In line with this, we did not obtain research ethics committee review for any of the case studies described. It is not clear at what point ‘key informants’ become ‘research participants’ and hence when the type of evidence synthesis we conducted in reviews 2 and 3 becomes primary research that does require research ethics committee review. Further consideration, and clarification, of this issue by research ethics organisations would be helpful. In the meantime, and as has been previously proposed, it may be judicious for researchers proposing to conduct this type of work to at least discuss it informally with their local research ethics committee before proceeding [ 54 ].

We propose the term ‘grey information’ to capture a wide range of documented and undocumented information that may be excluded by common definitions of ‘grey literature’. Information on applied public health research questions relating to the nature and range of public health interventions, and many evaluations of these interventions, may be predominantly, or only, held in grey literature and grey information. Evidence syntheses on these topics need, therefore, to embrace grey literature and information. Many standard systematic review methods for searching, appraising, managing, and synthesising the evidence base can be adapted for use with grey literature and information. Evidence synthesisers should carefully consider the opportunities and problems offered by including grey literature and information. Further action to improve both information deposition and retrieval would facilitate more efficient and complete syntheses of grey literature and information.

Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev. 2007;(2):Mr000010. http://www.ncbi.nlm.nih.gov/pubmed/17443631 .

Craig P, Dieppe P, Macintyre S, Mitchie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:979–83.

Article   Google Scholar  

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O’Cathain A, Tinati T, Wight D, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

Pappas C, Williams I. Grey literature: its emerging importance. Journal of Hospital Librarianship. 2011;11(3):228–34.

Franks H, Hardiker NR, McGrath M, McQuarrie C. Public health interventions and behaviour change: reviewing the grey literature. Public Health. 2012;126(1):12–7.

Article   CAS   PubMed   Google Scholar  

Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist synthesis: an introduction. In: ESRC research methods programme. Manchester: University of Manchester; 2004.

Google Scholar  

Turner AM, Liddy ED, Bradley J, Wheatley JA. Modeling public health interventions for improved access to the gray literature. J Med Libr Assoc. 2005;93(4):487–94.

PubMed   PubMed Central   Google Scholar  

Benzies KM, Premji S, Hayden KA, Serrett K. State-of-the-evidence reviews: advantages and challenges of including grey literature. Worldviews Evid Based Nurs. 2006;3(2):55–61.

Article   PubMed   Google Scholar  

Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Research Synthesis Methods. 2014;5(3):221–34.

Schopfel J. Towards a prague definition of grey literature. In: Twelfth International Conference on Grey Literature: Transparency in Grey Literature. Prague, Czech Republic: DANS; 2010.11–26.

Chavez TA, Perrault AH, Reehling P, Crummett C. The impact of grey literature in advancing global karst research: an information needs assessment for a globally distributed interdisciplinary community. Publishing Research Quarterly. 2007;23(1):3–18.

Banks M. Blog posts and tweets: the next frontier for grey literature. In: Darace D, Schopfel J, editors. Grey literature in library and information studies. Berlin: DeGruyter Saur; 2010. p. 217–26.

Godin K, Stapleton J, Kirkpatrick SI, Hanning RM, Leatherdale ST. Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Syst Rev. 2015;4:138.

Article   PubMed   PubMed Central   Google Scholar  

Canadian Agency for Drugs and Technologies in Health. Grey Matters: A Practical Search Tool for Evidence-based Medicine. In: Health CAfDaT, editor. 2014.

Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: a systematic review. Advances in health sciences education : theory and practice. 2009;14(4):595–621.

White S, Fook J, Gardner F. Critical reflection : a review of contemporary literature and understandings. In: White S, Fook J, Gardner F, editors. Critical reflection in health and social care. Maidenhead: Open University Press; 2006. p. 3–20.

Adams J, White M, Moffatt S, Howel D, Mackintosh J. A systematic review of the health, social and financial impacts of welfare rights advice delivered in healthcare settings. BMC Public Health. 2006;6:doi: 10.1186/1471-2458-1186-1181 .

White M, Howell D, Deverill M, Moffatt S, Lock K, McColl E, Milne E, Rubin G, Aspray T. Randomised controlled trial, economic and qualitative process evaluation of domiciliary welfare rights advice for socio-economically disadvantaged older people recruited via primary health care: the Do-Well study. Newcastle upon Tyne: Newcastle University; 2013.

Haighton C, Moffatt S, Howel D, McColl E, Milne E, Deverill M, Rubin G, Aspray T, White M. The Do-Well study: protocol for a randomised controlled trial, economic and qualitative process evaluations of domiciliary welfare rights advice for socio-economically disadvantaged older people recruited via primary health care. BMC Public Health. 2012;12:382.

Adams J, Simpson E, Penn L, Adamson A, White M. Research to support the evaluation and implementation of adult cooking skills interventions in the UK: phase 1 report. Newcastle upon Tyne: Public Health Research Consortium; 2011.

Bardus M, van Beurden SB, Smith JR, Abraham C. A review and content analysis of engagement, functionality, aesthetics, information quality, and change techniques in the most popular commercial apps for weight management. Int J Behav Nutr Phys Act. 2016;13(1):1–9.

Abraham C, Michie S. A taxonomy of behaviour change techniques used in interventions. Health Psychol. 2008;27:379–87.

Rees R, Hinds K, Dickson K, O’Mara-Eves A, Thomas J. Communities that cook: a systematic review of the effectiveness and appropriateness of interventions to introduce adults to home cooking. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2012.

Hillier-Brown FC, Moore HJ, Lake AA, Adamson AJ, White M, Adams J, Araujo-Soares V, Abraham C, Summerbell CD. The effectiveness of interventions targeting specific out-of-home food outlets: protocol for a systematic review. Syst Rev. 2014;3(1):1–5.

Higgins J, Green S (eds.). Cochrane Handbook for Systematic Reviews of Interventions version 5.1. [updated March 2011]. Available from www.cochrane-handbook.org : The Cochrane Collaboration; 2011.

Systematic reviews: CRD’s guidance for undertaking reviews in health care. York: Centre for Reviews and Dissemination, University of York; 2009.

Hammerstrøm K, Wade A, Jørgensen A-MK. Searching for studies: a guide to information retrieval for Campbell systematic reviews. 2010.

Grey literature: home [ http://guides.mclibrary.duke.edu/greyliterature ]. Accessed 19 Sep 2016.

Grey literature for health sciences: getting started [ http://guides.library.ubc.ca/greylitforhealth ]. Accessed 19 Sep 2016.

How to find…: Grey literature [ http://libguides.library.curtin.edu.au/c.php?g=202368&p=1332723 ]. Accessed 19 Sep 2016.

Montori VM, Wilczynski NL, Morgan D, Haynes RB. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2005;330(7482):68.

Haynes RB, McKibbon KA, Wilczynski NL, Walter SD, Werre SR, for the Hedges Team. Optimal search strategies for retrieving scientifically strong studies of treatment from Medline: analytical survey. BMJ. 2005;330:1179.

Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62(9):944–52.

Boluyt N, Tjosvold L, Lefebvre C, Klassen TP, Offringa M. Usefulness of systematic review search strategies in finding child health systematic reviews in MEDLINE. Arch Pediatr Adolesc Med. 2008;162(2):111–6.

Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health. 2005;59(9):804–8.

Roberts K, Cavill N, Rutter H. Standard evaluation framework for weight management interventions. London: National Obesity Observatory; 2009.

Department for Health. The public health outcomes framework for England, 2013-2011. London: Department of Health; 2012.

Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savović J, Schulz KF, Weeks L, Sterne JAC. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Lundh A, Gøtzsche PC. Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies. BMC Med Res Methodol. 2008;8(1):1–9.

Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007;36(3):666–76.

Thomas B, Ciliska D, Dobbins M, Micucci S. A process for systematically reviewing the literature: providing the research evidence for public health nursing interventions. Worldviews Evid Based Nurs. 2004;1(3):176–84.

Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, Petticrew M, Altman DG. Evaluating non-randomised intervention studies. Health technology assessment (Winchester, England) . 2003;7(27):iii-x, 1-173.

Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: realist syntheses. BMC Med. 2013;11(1):21.

Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10 Suppl 1:21–34.

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Medical Research Methodology. 2009;9(59). doi: 10.1186/1471-2288-1189-1159 .

Armstrong R, Hall BJ, Doyle J, Waters E. ‘Scoping the scope’ of a cochrane review. J Public Health. 2011;33(1):147–50.

Schünemann HJ, Moja L. Reviews: rapid! rapid! rapid! …and systematic. Syst Rev. 2015;4(1):1–3.

Polisena J, Garritty C, Kamel C, Stevens A, Abou-Setta AM. Rapid review programs to support health care and policy decision making: a descriptive analysis of processes and methods. Syst Rev. 2015;4(1):1–7.

Featherstone RM, Dryden DM, Foisy M, Guise J-M, Mitchell MD, Paynter RA, Robinson KA, Umscheid CA, Hartling L. Advancing knowledge of rapid reviews: an analysis of results, conclusions and recommendations from published review articles examining rapid reviews. Syst Rev. 2015;4(1):1–8.

Cook AM, Finlay IG, Edwards AGK, Hood K, Higginson IJ, Goodwin DM, Normand CE, Douglas H-R. Efficiency of searching the grey literature in palliative care. J Pain Symptom Manage. 2001;22(3):797–801.

Kothari A, Hovanec N, Hastie R, Sibbald S. Lessons from the business sector for successful knowledge management in health care: a systematic review. BMC Health Services Research. 2011;11(173). doi: 10.1186/1472-6963-11-173 .

Lefebvre C, Glanville J, Wieland LS, Coles B, Weightman AL. Methodological developments in searching for studies for systematic reviews: past, present and future? Syst Rev. 2013;2(1):1–9.

World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA . 2013;310(20):2191-2194.

Vergnes JN, Marchal-Sixou C, Nabet C, Maret D, Hamel O. Ethics in systematic reviews. J Med Ethics. 2010;36(12):771–4.

Download references

Acknowledgements

Not applicable

This paper presents independent research funded by the National Institute for Health Research (NIHR)’s School for Public Health Research (SPHR) with support from Durham and Newcastle Universities and the NIHR Collaboration for Leadership in Applied Health Research and Care of the South West Peninsula (PenCLAHRC). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health.

The School for Public Health Research (SPHR) is funded by the National Institute for Health Research (NIHR). SPHR is a partnership between the Universities of Sheffield, Bristol, Cambridge, and UCL; The London School for Hygiene and Tropical Medicine; The Peninsula College of Medicine and Dentistry; the LiLaC collaboration between the Universities of Liverpool and Lancaster and Fuse; The Centre for Translational Research in Public Health, a collaboration between Newcastle, Durham, Northumbria, Sunderland, and Teesside Universities.

FHB, CDS, HJM, WLW, AA, VAS, and AAL are members of Fuse; JA and MW are funded by the Centre for Diet and Activity Research (CEDAR). Fuse and CEDAR are UKCRC Public Health Research Centre of Excellence. Funding for Fuse and CEDAR from the British Heart Foundation, Cancer Research UK, Economic and Social Research Council, Medical Research Council, and the National Institute for Health Research, under the auspices of the UK Clinical Research Collaboration, is gratefully acknowledged.

The funders played no role in the design of the study, analysis, and interpretation of data, writing the manuscript, or the decision to submit for publication.

Availability of data and materials

This manuscript does not refer to any new data. Of the three case studies that form the focus for the discussion, two have either been previously published and are now in the public domain [ 17 , 20 ] and one is under review (Hillier-Brown F, Summerbell C, Moore H, Wrieden W, Abraham C, Adams J, Adamson A, Araujo-Soares V, White M, Lake A. A description of interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England: a systematic mapping and evidence synthesis. BMC public health 2015, unpublished).

Authors’ contributions

JA and MW (with additional co-authors) conducted case study 1 [ 17 ]; JA and MW (with additional co-authors) conducted case study 2 [ 20 ]; FHB, CS, HJM, JA, VAS, MW, and ALA (with additional co-authors) conducted case study 3 (Hillier-Brown F, Summerbell C, Moore H, Wrieden W, Abraham C, Adams J, Adamson A, Araujo-Soares V, White M, Lake A. A description of interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England: a systematic mapping and evidence synthesis. BMC public health 2015, unpublished). FHB, CS, HJM, JA, VAS, MW, and ALA contributed to the development of the ideas described in this paper. JA led the writing of the current manuscript. FHB, CS, HJM, JA, VAS, MW, and ALA provided the critical comments on previous drafts of the manuscript and agreed to submit the final version. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Ethics approval and consent to participate.

This is not a primary piece of research, no participants were recruited, and ethical approval was not required.

Author information

Authors and affiliations.

Present address: Centre for Diet and Activity Research, MRC Epidemiology Unit, University of Cambridge, CB2 0QQ, Cambridge, UK

Jean Adams & Martin White

Institute of Health and Society, Newcastle University, NE2 4AA, Newcastle, UK

Jean Adams, Vera Araujo-Soares & Martin White

Fuse, The Centre for Translational Research in Public Health, Newcastle, UK

Jean Adams, Frances C. Hillier-Brown, Helen J. Moore, Amelia A. Lake, Vera Araujo-Soares, Martin White & Carolyn Summerbell

School of Medicine, Pharmacy and Health, Durham University, TS17 3BA, Stockton-on-Tees, UK

Frances C. Hillier-Brown, Helen J. Moore, Amelia A. Lake & Carolyn Summerbell

Centre for Public Policy & Health, Durham University, TS17 6BH, Stockton-on-Tees, UK

Amelia A. Lake

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jean Adams .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Adams, J., Hillier-Brown, F.C., Moore, H.J. et al. Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: critical reflections on three case studies. Syst Rev 5 , 164 (2016). https://doi.org/10.1186/s13643-016-0337-y

Download citation

Received : 23 May 2016

Accepted : 13 September 2016

Published : 29 September 2016

DOI : https://doi.org/10.1186/s13643-016-0337-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Grey literature
• Grey information
• Systematic review
• Evidence synthesis
• Public health
• Interventions

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Reference management. Clean and simple.

What is grey literature? [with examples]

What's the importance of grey literature?

What are some of the drawbacks of grey literature, what are some examples of grey literature, how do you identify grey literature, where can you search for grey literature, are there any other ways to find grey literature, use search engines, go directly to organizations' websites, contact researchers directly, frequently asked questions about grey literature, related articles.

Grey literature refers to materials and research published specifically outside of the traditional commercial, academic publishing, and distribution channels.

It is produced by organizations on all levels of government, academia, business, and industry and either published informally or remains unpublished. Not every grey literature database will have undergone the peer review process due to this informal setup, but the information contained therein can still prove valuable to researchers and academics alike - if they are able to find it.

Since grey literature usually comes from multiple sources, finding the right resource can prove to be challenging. It's not as simple as visiting a library. With no indexes or formal organization, grey literature won't appear in academic sources like books or journals. But the question still remains, what is grey literature and why is it important to academic publishing?

• Grey literature is often the best source for up-to-date research on certain topics. This is because it doesn't have to go through the often lengthy process of publication before it can be reviewed.
• In the same breath, grey literature databases can have a scope that includes recently emerging areas of research.
• Scientific publishing, for instance, has a strong bias toward publishing research that shows a significant positive correlation. Many studies show no effect and can go unpublished, but negative results are just as important to note. Accessing grey literature fills this knowledge gap, and generates a more balanced picture of the available evidence.
• The reports produced in grey literature are often more detailed in nature and can contain raw data sets not available anywhere else - the caveat is that they come in tons of different formats and document types, making the data slightly more difficult to find.
• Grey literature can help researchers to reduce publication bias.
• Most grey literature is free (one exception is materials produced by market research firms, who often charge for access). This makes it a lot more accessible to everyone.
• The information in grey literature hasn't gone through the usual publishing channels, so it isn't rigorously and formally peer-reviewed. This means you'll need to carry out your own checks to assess its reliability.
• When grey literature isn't published, it's free from publication bias. However, it may carry its own set of biases. The organization that distributed the grey literature can have its own political, social, or financial motivations for promoting unreliable research. Make sure to cross-check information from grey literature against studies found elsewhere.
• Grey literature is produced from a great variety of sources, so it's usually not indexed or organized in any way. That can make it very difficult to locate what you're looking for.
• Librarians often have a difficult time acquiring grey literature and making it accessible.

Publication types of grey literature are almost endless, and include:

• Government reports
• Technical reports
• Unpublished clinical trials
• Conference abstracts
• Graduate dissertations and theses
• Newsletters
• White papers
• Working papers
• Dissertations

Websites, manuals, statistics, and datasets are technically also types of grey literature that you may come across during the research process.

Grey literature can be found in so many contexts it can be tough to list them all. Essentially, if you didn't find it in a renowned journal or published book, there's a good chance it's grey literature. Luckily, if you know how to find academic sources, you should be able to identify grey literature in the wild.

Here are some other indicators that you may be looking at a piece of grey literature:

• It was difficult for you to find
• It hasn't been widely distributed
• It contains more information on the subject than you've seen elsewhere, as there weren't publication length stipulations
• It was produced and disseminated quickly
• It hasn't been archived

It can be difficult to develop a systematic approach to finding grey literature for academic purposes. Unlike scholarly journals that are often available in electronic formats, grey literature databases will be all over the map in terms of formats and accessibility.

You should prepare any search terms you're going to use, identify the ones that are most applicable, and record the sources as you search. If you have a research question, use it to guide where you look for the sources you're going to search.

There are also many grey literature databases that specialize in cataloging grey literature:

There are methods that you can make use of to target your search for grey literature.

• Restrict content to only governments or organizations by typing in your search term and then either 'site:.org' or 'site:.gov'.
• You can take it further and limit your search to e.g. UK government sites only by typing 'site:.gov.uk'.
• Restrict content to a specific file type by typing in your search term and then e.g. 'filetype:pdf' or 'filetype:doc'.
• If you're looking for grey literature from a specific country, use their version of Google, for example, Google Australia or Google UK .
• If you'd like a fresh take, try search engine DuckDuckGo , as it doesn't track location or user searches. This means it doesn't filter results based on your user profile.

Identify organizations that might publish technical papers and reports on the subject you're looking for. These could be government agencies, non-profits, research institutes, pressure groups, international organizations, or professional associations.

For example, the World Health Organization (WHO) has several online collections of grey literature and the World Bank makes all of their publications available online for free.

Search your chosen organizations' websites, looking for sections of the site with names such as 'Documents', 'Reports' or 'Library'.

Research known researchers in your field of interest. You can then contact them to kindly ask if they're aware of any unpublished or ongoing studies - more often than not they will be happy to point you in the right direction.

Grey literature is materials and research that have been published specifically outside of the traditional commercial or academic publishing and distribution channels. The term 'grey' refers to the uncertainty of the status of this information.

Yes, preprints are considered grey literature, as they have not undergone the peer-review process.

PubMed is a research database, not a publication type. Material indexed by major databases such as PubMed is generally not considered grey literature.

Books and other scholarly publications have to pass through a formal publishing process and are therefore not considered grey literature.

Yes, dissertations are generally considered grey literature and might be hard to obtain in some cases.

• Subject guides
• Grey Literature
• What is grey literature

Grey Literature: What is grey literature

• Finding grey literature
• Evaluating grey literature
• Grey Sources

Other useful guides

• Citing and referencing
• Literature reviews

​Grey literature is "Information produced on all levels of government, academics, business and industry in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body."

CGL Luxembourg definition , 1997-expanded in New York, 2004

Grey literature is published informally or non-commercially, or remains unpublished. It can appear in many forms, including government reports, statistics, patents, conference papers and even non-written resources such as posters and infographics.

Grey literature usually has not been peer reviewed, but may still be good, reliable information. It can thus be invaluable for your research. It is produced from a variety of sources, and is usually not indexed or organised, often making it difficult to locate.

What is grey literature?

A short video explaining grey literature from Western Universities library

Examples of grey literature

For a comprehensive listing of grey literature types view the greynet.org table .

Common grey literature producers

Why use grey literature.

Grey literature is used in research because:

• It will often be more current than traditionally published sources, with a better coverage of emergent research areas. 
• A literature search that accesses only “black literature” will likely miss information vital to research, especially in health sciences fields.
• It can be a better source of information on policies and programs.
• Using grey literature may help minimise publication bias in systematic reviews.
• Grey literature is often a good source of raw data and data sets to which you wouldn’t otherwise have access.
• << Previous: Home
• Next: Finding grey literature >>

• Library Catalogue

Grey literature: What it is & how to find it

On this page, what is grey literature, where to search for grey literature, using google to search for grey literature, google custom searches, google scholar, other sources.

• By geographic coverage
• Statistics and data sources

Further resources and strategies for searching for grey literature

Grey literature is information produced outside of traditional publishing and distribution channels, and can include reports, policy literature, working papers, newsletters, government documents, speeches, white papers, urban plans, and so on.

This information is often produced by organizations "on the ground" (such as government and inter-governmental agencies, non-governmental organisations, and industry) to store information and report on activities, either for their own use or wider sharing and distribution, and without the delays and restrictions of commercial and academic publishing. For that reason, grey literature can be more current than literature in scholarly journals.

However, because grey literature (usually) does not go through a peer review process, the quality can vary a great deal. Be sure to critically evaluate your source .

For an extensive list of the types of documents that might be considered grey literature, see Document types in grey literature (from GreyNet International ).

To search across the websites of governments and organizations, use the resources below.

Another way to find grey literature is to identify organizations that might be publishing this type of information on topics that you're interested in and then to search their websites — paying close attention to website sections with names like "Documents", "Reports", and "Library". Consider government agencies, non-profits, professional associations, research institutes, and other organisations, based on the research you have already done. 

Use Google's site limits to only search a government or institution's site or top-level domain, for instance use  site:gov.bc.ca to limit your search to British Columbia government sites. This technique paired with keywords in Google is a quick way to locate grey literature. Example search: site:gov.bc.ca housing affordable

For additional advanced searching techniques see  Search tips for Google and Google Scholar , our quick overview of basic and more advanced techniques.

Type your search terms into these pre-built searches  to limit results to specific organisation types. Use Search tips for Google and Google Scholar  to create your search strings. 

NGO Search NGO Search is a Google Custom Search that searches across hundreds non-governmental organization (NGO) websites. NGO Search is a project of the International Documents Taskforce (IDTF) and the Government Documents Roundtable (GODORT) of the American Library Association (ALA).  This is a spreadsheet of NGOs included in the project . 

IGO Search International governmental organizations (IGOs) are organizations made up of more than one national government—examples include NATO (North Atlantic Treaty Organization) and the WHO (World Heath Organization). The governments are the members. IGO Search is a Google Custom Search that searches across IGO websites. It is a project of the International Documents Taskforce (IDTF) and the Government Documents Roundtable (GODORT) of the American Library Association (ALA). Here is the growing spreadsheet of IGO urls  included in the search. 

There is some grey literature in Google Scholar — mainly theses and dissertations in institutional repositories. Be sure to access  Google Scholar via the Library's website  to ensure you see the "Where can I get this?" links that will take you to the full text SFU offers. 

Use  Search tips for Google and Google Scholar  to create your search strings. 

Policy Commons   This database brings together grey literature -- reports, working papers, policy briefs, data sources, and media -- from NGOS, think tanks, and research centers around the world.  

 By geographic coverage

Canada Commons Includes some full text public policy documents from Canadian institutes, think-tanks, and research groups.

CHODARR (Community Health Online Digital Archive & Research Resource) An archived virtual library of community-based and government publications related to health and social welfare policy, including grey literature, with emphasis on Vancouver-area resources and organisations. Note: This source is no longer being updated. You can browse the individual collections, or use Google's site search function to search across it. Example search in Google:  site:chodarr.org AIDS 

International

OpenGrey  (Europe) A search portal for European grey literature. It covers Science, Technology, Biomedical Science, Economics, Social Science and the Humanities.

 Statistics and data sources

Statistical resources: Data & statistics information (SFU Library) Use these resources for processed data drawn from national agencies, intergovernmental organizations, and other official sources of statistics.

Microdata resources: Data & statistics information (SFU Library) Use microdata when you need to do analysis, or generate visualizations such as tables and maps. 

For more about the value of grey literature and further strategies for finding it, you can also search the Library Catalogue for terms such as  grey literature methodology .

• Research Guides

Grey Literature

• What is Grey Literature? Activities
• Why Use Grey Literature?
• Types of Grey Literature

Sources of Grey Literature

• Searching for Grey Literature
• Evaluating Grey Literature Activity
• How to Incorporate & Cite Grey Literature

This guide includes content adapted with permission from the University of Illinois Library .

Vanderbilt Resources for Grey Literature

• Dissertations & Theses

The following are relevant databases the Vanderbilt Libraries are subscribed to.

• Dissertations & Theses Full Text This link opens in a new window Dissertations and theses from around the world in business, health & medicine, history, literature & language, science & technology, social sciences, and the arts. Includes Vanderbilt University dissertations and theses. more... less... Coverage: 1743-present.
• EThOS: Electronic Theses Online Service This link opens in a new window UK’s national thesis service. National aggregated record of all doctoral theses awarded by UK Higher Education institutions. more... less... Coverage: 1800 - present.
• Vanderbilt University Electronic Theses and Dissertations (ETD) This link opens in a new window Vanderbilt University theses and dissertations. more... less... Coverage: 1999 to present. User Limit: none.
• Budget of the United States Government This link opens in a new window Budget message of the President, information about the President's budget proposals for a given fiscal year, and other budgetary publications. more... less... Coverage: Previous year, estimates for current year, projections for five years, and historical information from 1940.
• CIA World Factbook This link opens in a new window Central Intelligence Agency's database: information on the history, people, government, economy, geography, communications, transportation, military, and transnational issues for countries, territories, and dependencies. more... less... Coverage: Current year.
• CQ Congress Collection This link opens in a new window Public policy issues, information about members of Congress (1981-), vote analysis, and dictionary and encyclopedia-type entries on legislative terms and the legislative process.
• EDGAR This link opens in a new window US government database of public company filings collected by the U.S. Securities and Exchange Commission. Full-text of company annual 10-K reports, quarterly 10-Q reports, proxy statements, prospectus, etc. more... less... Vendor Tutorials: http://www.sec.gov/edgar/quickedgar.htm
• ERIC (ProQuest version) This link opens in a new window Journal articles, conferences, meetings, government documents, theses, dissertations, reports, audiovisual media, bibliographies, directories, books and monographs on education-related literature. more... less... Coverage: 1966 to present.
• FAOSTAT This link opens in a new window Statistical database on food & agriculture. Includes production, trade, food balance sheets, fertilizer and pesticides, land use and irrigation, forest products, fishery products, population, agricultural machinery, and food aid shipments.
• FDsys - Federal Digital System This link opens in a new window Government Printing Office's access to publications of the Federal Government.
• GPO (Government Printing Office) Catalog, 1976- This link opens in a new window All types of US government documents, including Congressional reports, hearings, debates, and records; judiciary materials; and documents issued by executive departments (Defense, State, Labor, Office of the President, etc.).
• Access World News Research Collection This link opens in a new window Research diverse perspectives, topics and trends that align with curricular areas such as Political Science, English, Sociology, Humanities, Business, International Studies and more. Features reliable, credible information from a wide variety of international, national and local news sources.
• CQ Researcher This link opens in a new window Coverage of the most important issues of the day, written by experienced journalists, footnoted and professionally fact-checked. Articles include an overview, historical background, chronology, pro/con feature, and resources for additional research. more... less... Coverage: 1991 to present.
• News (Gale OneFile) This link opens in a new window National, regional, and local newspapers; images; radio and television broadcasts and transcripts. more... less... Coverage: 1977 to present.
• Newspapers.com Library Edition - Southeast Collection This link opens in a new window Historical newspapers archive containing thousands of regional, state, and small local newspapers from the southeastern U.S.
• ProQuest News and Newspapers This link opens in a new window Direct link to the newspaper section of ProQuest. Includes Alt-Press Watch, Ethnic NewsWatch, GenderWatch, Latin American Newsstand, Canadian Newsstand Complete, ProQuest Newsstand, ProQuest Historical Newspapers. Chicago Tribune, Christian Science Monitor, Los Angeles Times, New York Times, Wall Street Journal, Washington Post, Times of London, and The Tennessean. more... less... All current Vanderbilt University students, faculty, and staff have access, both on and off campus.
• Tennessean (2002-present) This link opens in a new window Nashville's local and regional newspaper: The Tennessean. more... less... Coverage: Nov. 7, 2002, to date, with incomplete coverage back to Jan. 1, 1999.
• ProQuest Statistical Insight This link opens in a new window Federal, state, and international statistical, economic, demographic, and geographic information. National income and production statistics, retail and wholesale trade data, interest rates, economic indicators, and social and demographic statistics, plus data on energy, environmental protection, agriculture, fisheries, crime, etc. more... less... Coverage: From the U.S. government, 1973 to present; from international intergovernmental, 1983 to present.
• Statista This link opens in a new window Quantitative data, statistics, infographics, and related information compiled from market research institutes, trade publications, scientific journals, and government databases.
• Statistical Abstract of the United States (ProQuest) This link opens in a new window Extensive selection of statistics for the United States, with selected data for regions, divisions, states, metropolitan areas, cities, and foreign countries from reports and records of government and private agencies. more... less... Access Note: If prompted to select an account, select Vanderbilt University.
• Global Think Tanks This link opens in a new window Curated, high quality policy reports, briefs, analyses, working papers, and datasets produced by the world’s leading policy experts, think tanks, research centers, IGOs, and NGOs.
• Nexis Uni (formerly LexisNexis Academic) This link opens in a new window News and newspaper stories, federal and state court cases, US law reviews and journals, company profiles and directories. Company Dossier; data on public figures. more... less... Access Note: Personal account required for some functionality. User Limit: Unlimited.
• PAIS International This link opens in a new window Public Affairs Information Service: journal articles, books, government documents, statistical directories, grey literature, research reports, conference papers, web content, etc. covering issues in the public debate. more... less... Coverage: PAIS International is the current file covering 1972 to present, and PAIS Archive includes the content from printed volumes published 1915-1976. User Limit: Unlimited.
• ProQuest Central This link opens in a new window Scholarly journals, newspapers, dissertations, and business information. Includes: ABI/INFORM Complete, Accounting & Tax, Banking Information Source, Canadian Newsstand Complete, CBCA Complete, Hoover's Company Profiles, OxResearch, Pharmaceutical News Index, Asian Business & Reference, Biology Journals, Career and Technical Education, Computing, Criminal Justice, Education Journals, European Business, Family Health, Health & Medical Complete, Health Management, Military Collection, Newsstand, Nursing & Allied Health Source, Psychology Journals, Religion, Research Library, Science Journals, Social Science Journals, Telecommunications, Snapshots. more... less... All current Vanderbilt University students, faculty, and staff have access, both on and off campus.
• ProQuest Social Sciences Premium Collection This link opens in a new window International literature in the social sciences: politics, sociology, social services, anthropology, criminology and education. IBSS, PAIS and Sociological Abstracts. Abstracts, indexing and full text of articles, books, chapters, dissertations, working papers. • Applied Social Sciences Index and Abstracts (ASSIA) • Criminal Justice Database • Education Database • ERIC • International Bibliography of the Social Sciences (IBSS) • Linguistics and Language Behavior Abstracts (LLBA) • PAIS Index • Political Science Database • PTSDpubs • Social Science Database • Social Services Abstracts • Sociological Abstracts • Sociology Database • Worldwide Political Science Abstracts
• Web of Science This link opens in a new window Citation database from journals and international conference proceedings. Cross-search of the Web of Science Core Collection, BIOSIS Previews, and the SciELO Citation Index (Latin America, Spain, Portugal, the Caribbean and South Africa). more... less... All current Vanderbilt University students, faculty, and staff have access, both on and off campus.

Tools for Searching for Grey Literature

• Directories
• Custom Google Search Bars
• General Sources
• FIND POLICY A side project of Transparify, search think tank sites grouped by topic and location.
• Open Think Tank Directory A global collection of 2700 think tanks and related organizations.
• GreySource GreySource provides examples of grey literature to the average net-user and in so doing profiles organizations responsible for its production and/or processing. GreySource identifies the hyperlink directly embedded in a resource, thus allowing immediate and virtual exposure to grey literature.
• Grey Guide portal and repository for resources in the field of grey literature.
• OpenGrey A search portal for European grey literature. It covers Science, Technology, Biomedical Science, Economics, Social Science and the Humanities.
• World Association of Non-Governmental Organizations (WANGO) Worldwide NGO Directory Search for NGO's based on location.
• OpenDOAR OpenDOAR is the quality-assured, global Directory of Open Access Repositories. You can search and browse through thousands of registered repositories based on a range of features, such as location, software or type of material held
• UNESCO Database of Non-Governmental Organizations (NGOs) Contains the profiles of non-governmental organizations that have official relations with UNESCO.
• UN NGO Branch - Integrated Civil Society Organizations System The integrated Civil Society Organizations (iCSO) System, developed by the Department of Economic and Social Affairs (DESA), facilitates interactions between civil society organizations and DESA. The system provides online registration of general profiles for civil society organizations, including address, contacts, activities and meeting participation.
• Harvard Kennedy School of Government Think Tank Search A Google Custom Search of more than 1200 think tanks and research centers.
• Non-Governmental Organizations Custom searches to find information from NGOs created by Government Documents Round Table (GODORT)
• JSTOR Open Research Reports JSTOR hosts a growing curated collection of more than 45,000 open research reports from 175 policy institutes from around the world. These publications are freely accessible to everyone on JSTOR and discoverable as their own content type alongside journals, books, and primary sources. We update research reports on our platform each month as they become available through contributing institutes.Research reports provide current analysis on many of today’s most discussed and debated issues from a diversity of ideological and international perspectives representing 39 countries and 29 languages. A sample of topics would include climate change, border security, fake news, cybersecurity, electric vehicles, artificial intelligence, energy policy, gender issues, terrorism, remote learning, and various public health issues, including COVID-19.
• Government Documents
• Non-Scholarly Academic
• GovInfo GovInfo is a service of the United States Government Publishing Office (GPO), which is a Federal agency in the legislative branch. GovInfo provides free public access to official publications from all three branches of the Federal Government.
• United Nations Digital Library The online catalog of UN documents and publications. The Digital Library includes UN documents, voting data, speeches, maps, and open access publications. The platform provides access to UN-produced materials in digital format and bibliographic records for print UN documents starting in 1979. System features include linked data between related documentation such as resolutions, meeting records and voting, and refining of searches by UN body, agency or type of document.
• Social Science Research Network (SSRN) SSRN, formerly known as Social Science Research Network, is an open access research platform used to share early-stage research, evolve ideas, measure results, and connect scholars around the world.

Grey Literature by Topic

• Public Health
• Governance and Conflict
• Migration and Culture
• Science, Technology and Enviroment
• Development and Aid
• The Community Health Online Digital Archive & Research Resource (CHODARR) An archived virtual library of community-based and government publications related to health and social welfare policy, including grey literature, with emphasis on Vancouver-area resources and organizations. Note: This source is no longer being updated. You can browse the individual collections, or use Google's site search function to search across it. Example search in Google: site:chodarr.org AIDS
• PolicyArchive PolicyArchive is a comprehensive digital library of public policy research containing over 30,000 documents.
• Migration Policy Institute Reports, fact sheets, policy briefs, articles, commentaries and more on several topics related to Migration.
• National Technical Reports Library The National Technical Information Service acquires, indexes, abstracts, and archives the largest collection of U.S. government-sponsored technical reports in existence. The NTRL offers online, free and open access to these authenticated government technical reports
• Science.gov Science.gov searches over 60 databases and over 2,200 scientific websites to provide users with access to more than 200 million pages of authoritative federal science information including research and development results.
• USAID Development Experience Clearinghouse Search 50 years worth of USAID project records and other related documents on a variety of topics.
• The World Bank Research and Publications Data, research and reports on topics related to poverty and development.
• Eldis Eldis supports free and open access to useful and relevant research on global development challenges.
• << Previous: Types of Grey Literature
• Next: Searching for Grey Literature >>
• Last Updated: Apr 16, 2024 2:18 PM
• URL: https://researchguides.library.vanderbilt.edu/greyliterature

• University of Michigan Library
• Research Guides

Grey Literature

• Strategies for Finding Grey Literature
• Conferences
• Clinical Trials/Regulatory Data
• Reports & Reviews
• Other Resources

What's on this page

This page provides strategies for finding grey literature. Grey literature is an important part of the evidence base for health services research but can be difficult to search in a systematic way. You'll find three core strategies for identifying this type of content.   See the  Grey Literature Research Guide for many other resources

#1 Going directly to organization   l  #2 Searching Portals Aggregating Grey Literature |  #3 Google Advanced Searches  

Strategy #2: Portals

Below are some resources that can help you identify organizations and/or search directly for grey literature.

• Overton.io This link opens in a new window Provides the world’s largest collection of policy documents, guidelines, think tank publications and working papers.
• Dimensions Plus This link opens in a new window Brings together various research-related data sources, such as grant information, books, journal articles, conference proceedings, and patents.
• Grey Literature Producing Organizations A list of grey literature producing organizations from the New York Academy of Medicine.
• Kennedy School of Government Think Tank Search Searches more than 700 think tanks and research centers. searches the websites of institutions that generate public policy research, analysis, and activity. These sites are affiliated with universities, governments, advocacy groups, foundations, non-governmental organizations, and businesses.
• NLM Digital Collections National Library of Medicine's free online repository of biomedical resources including books, manuscripts, still images, videos, and maps. The content in Digital Collections is freely available worldwide and, unless otherwise indicated, in the public domain.
• Policy Atlas (Harvard Catalyst) Helps identify data sources that can be used for your policy-relevant research or to discover new sources for health data
• Policy Commons Policy Commons provides more than 2.8 million+ policy reports, briefs, analyses, and datasets from IGOs, NGOs, and think tanks.

Strategy #3: Google Advanced Searches

Google offers a great way to track down grey literature resources. Use the advanced search limits below to focus your search.

Limit to a domain:

• site:org  

Find Related Sites:

• related:urban.org

Limit to filetype:

• filetype:pdf (or doc, ppt)

Search for words in a URL:

• inurl:rand.org

Search for sites that link to site of interest:

• link to:academyhealth.org 

Find quick information about a site:

• info:kff.org

Strategy #1: Going Directly to Organization's Website

Below are some top non-governmental organizations & federal agencies working in the health sciences. Since there is a seemingly endless number of organizations that produce grey literature, searching portals & using Google Advanced searches to identify names can be an effective way to find grey literature when you don't know where to begin.

Non-Governmental Organizations

• Academy Health
• Advisory Board
• American Hospital Association
• Center for Advancing Health
• Center for Health Equity Research & Promotion collaborations
• Center for Responsive Politics: Open Secrets
• Commonwealth Fund more... less... A private foundation whose mission is "to promote a high performing health care system." Browse the publications, maps & data and surveys. Search the entire site. Example: "health care" AND (racial OR ethnic).
• Cross Cultural Health Care Program more... less... The mission of the Cross Cultural Health Care Program is to serve as a bridge between communities and health care institutions to ensure full access to quality health care that is culturally and linguistically appropriate.
• Dartmouth Atlas of Health Care more... less... A project of the Dartmouth Institute for Health Policy and Clinical Practice, this site offers data national, regional and some local data on health care issues with mapping tools and publications.
• Henry J. Kaiser Family Foundation
• Joint Center for Political and Economic Studies
• National Center for Cultural Competence more... less... From Georgetown University's Center for Child & Human Development.
• National Quality Forum more... less... The National Quality Forum is a nonprofit organization that works to improve the quality of American healthcare.
• New York Academy of Medicine
• Prevention Institute
• Public Health Law Research
• RAND Corporation more... less... The RAND Corporation is a nonprofit institution that focuses on improving policy surrounding various issues through research and analysis.
• Robert Wood Johnson Foundation more... less... This is the largest philanthropy in the U.S. devoted to issues of public health. The web site offers research reports on issues related to health care and health policy.
• Urban Institute more... less... The Urban Institute works to foster sound public policy and effective government through research and analysis.

Government Agencies (Federal)

• Agency for Healthcare Research & Quality (AHRQ)
• Centers for Disease Control (CDC)
• Department of Energy
• Department of Health & Human Services (DHHS)
• Health Resources & Services Administration (HRSA)
• National Institutes of Health
• National Science Foundation

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Springer Nature - PMC COVID-19 Collection

The high scholarly value of grey literature before and during Covid-19

Kayvan kousha.

Statistical Cybermetrics Research Group, School of Mathematics and Computer Science, University of Wolverhampton, Wulfruna Street, Wolverhampton, WV1 1LY UK

Mike Thelwall

Matthew bickley, associated data.

The shared data provides citation counts, types and descriptions of 340 grey literature URLs with the most Scopus citations from Covid-19 articles. It also includes four graphs showing the number of Scopus articles with at least one citation to 17 grey literature sources between 2005 and October 2021 ( https://doi.org/10.6084/m9.figshare.16810471.v1 ).

New academic knowledge in journal articles is partly built on peer reviewed research already published in journals or books. Academics can also draw from non-academic sources, such as the websites of organisations that publish credible information. This article investigates trends in the academic citing of this type of grey literature for 17 health, media, statistics, and large international organisations, with a focus on Covid-19. The results show substantial and steadily increasing numbers of citations to all 17 sites, with larger increases from 2019 to 2020. In 2020, Covid-19 citations to these websites were particularly common for news organisations, the WHO, and the UK Office for National Statistics, apparently for up-to-date information in the rapidly changing circumstances of the pandemic. Except for the UN, the most cited URLs of each organisation were not traditional report-like grey literature but were other types, such as news stories, data, statistics, and general guidance. The Covid-19 citations to most of these websites originated primarily from medical research, commonly for coronavirus data and statistics. Other fields extensively cited some of the non-health websites, as illustrated by social science (including psychology) studies often citing UNESCO. The results confirm that grey literature from major websites has become even more important within academia during the pandemic, providing up-to-date information from credible sources despite a lack of academic peer review. Researchers, reviewers, and editors should accept that it is reasonable to cite this information, when relevant, and evaluators should value academic work that supports these non-academic outputs.

Supplementary Information

The online version of this article (10.1007/s11192-022-04398-3) contains supplementary material, which is available to authorized users.

Introduction

References in academic research publications fulfil different needs, not all of which can be adequately served by other academic publications. Their primary purposes are to acknowledge previous contributions and support the argument within a paper that the new research is valid and important enough to be published (Small, 2004 ). The sections of an article may be supported by references in different ways, however (Thelwall, 2019 ). Peer reviewed articles are often cited to justify the validity of a method, identify a research gap to be filled, and introduce theory underpinning the study (e.g., Spiegel-Rosing, 1977 ). In contrast, the following list illustrates some purposes that might be better served by citing documents (including web pages) that have not been formally published and peer reviewed.

• Statistics to justify the importance of a topic (e.g., Swales, 2011 ), such as the death toll of Covid-19 in a country or the world.
• Practical issues addressed by the research, such as national guidelines for Covid-19 social distancing.
• Influential non-academic documents analysed by the research, such as United Nations Food and Agricultural Organisation policy recommendations for protecting the food supply chain during pandemic social distancing restrictions.
• Facts relevant to the context of a study, such as a news story about a politician’s speech in an article about political responses to the pandemic.
• Non-academic data analysed in a study, such as datasets from the UK’s Inter-Departmental Business Register collected by the government through its official functions, used to assess the impact of the pandemic on different business types.

An important advantage of non-academic citations is that they can provide a bridge to the external world and ground academic research in practical concerns. Thus, it seems possible that as academia in some countries increasingly conducts research with societal impact (e.g., as assessed in the UK Research Excellence Framework impact case studies: Watermeyer & Hedgecoe, 2016 ), the prevalence of non-academic citations should increase. The Covid-19 pandemic seems to have accelerated this by partly refocusing the attention of academia on immediate practical issues, such as identifying characteristics of the virus and illness (Di Girolamo & Reynders, 2020 ; Kousha & Thelwall, 2020 ). The pandemic may therefore have led to an increase in the citing of non-academic sources, reflecting an enhanced value for grey literature within academia. Identifying such changes may help authors, reviewers, and research evaluators to judge the appropriateness of non-academic sources of reference for academic publications.

Previous analyses of the prevalence of citations to grey literature in academic research have tended to have a relatively narrow scope because of the lack of a universal database of citations to grey literature. For example, studies have investigated citations to UK government publications (Bickley et al., 2020 ), biotechnology articles citing patents (Glänzel & Zhou, 2011 ), grey literature cited by six nursing journals (Woods et al., 2020 ), and grey literature cited in impact case studies (Kousha et al., 2021 ). The limited nature of previous studies reflects the difficulty in identifying citations to grey literature. Small scale investigations can manually check reference lists in sets of articles for grey literature publications (Woods et al., 2020 ), presumably capturing all relevant sources. Moreover, analyses of specific subsets of the grey literature may be able to define Scopus queries to identify relevant citations (e.g., those including patent terminology). It is also possible to identify documents that cite online grey literature by running Scopus queries for URLs in reference lists and automatically extracting the URL in references that seem likely to be for grey literature (e.g., for YouTube: Kousha, et al., 2012 ). This facility makes it possible to run large-scale grey literature studies based on finding references that include distinctive URLs (e.g., www.india.gov.in/ *). Although such results are restricted to online grey literature, there seems to be a shift to putting all information free online and so an increasing (but unknown) proportion of the grey literature can now be found in this way. For example, government publications are increasingly online, as are major patent databases, such as that of the USA ( www.uspto.gov ). This improves the validity of heuristics that use URLs in reference lists as a device to find grey literature citations.

This article assesses the extent to which grey literature from a set of large organisations has been cited in academic publications to assess whether citations to grey literature are common and whether they have become more prevalent during Covid-19. The focus is on citations to a set of large organisations’ URLs rather than citations to all grey literature because it is not yet practical to automatically identify all citations to grey literature with URLs because many published academic articles may also be cited with a variety of URLs (from publishers, institutions, personal home pages, repositories). This article also discusses the practical implications for policymakers.

Grey literature background

The widest definition of grey literature is that it is text-based output that is not formally published by commercial or academic publishers (i.e., not a book or academic journal: Carpenter, 2005 ). This includes, but is not limited to, webpages, patents, preprints, white papers, and social media (Aloia & Naughton,  2016 ; Schöpfel, 2006 ). The more formal and credible side of grey literature includes relatively substantial documents, such as unpublished reports, theses, conference proceedings, and government documents (Pejšová & Vaska, 2011 ). The difference between formal report-like grey literature and other types has been blurred by online publishing. For example, whereas previously a government might produce annual documents summarising an aspect of national statistics, this information might now be published instead, or in addition, as a series of webpages. As an example of this, one webpage contains current statistics on staff disability in UK higher education ( www.hesa.ac.uk/data-and-analysis/staff/table-5 ), which represents the end result of a large scale data collection and collation operation but is reported as a web page rather than as a PDF report with associated methods details and commentary.

The importance of formal grey literature can be illustrated by the main differences between it and standard academic outputs. Research published in books or journal articles typically has a longer delay between analysis and publication (Pappas & Williams, 2011 ), and in some fields, such as biomedical sciences, only a third of research reported in meeting abstracts ever gets formally published (von Elm et al., 2003 ). This shows that a significant proportion of research undertaken is never made available in an academic journal or book. Even if the research is published, there may still be substantial publication delays. This suggests that much research is unavailable to be cited, at least as a standard peer reviewed article or book. Of course, not all formal grey literature is of good quality or easily and freely accessible (Batt et al., 2004 ). Moreover, since peer review is a cornerstone of academic knowledge, it would be inappropriate to cite unrefereed documents in many contexts.

Even though formal grey literature has been produced for a long period (Alberani et al., 1990 ), its digitisation and the mass production of online information (Savić, 2018 ) reflects an ever-expanding scope for usage. For example, many healthcare policy guidelines from institutions such as the World Health Organization (WHO; Nove et al., 2017 ) and Centres for Disease Control and Prevention (CDC; Shrivastava & Mahajan, 2021 ), are primarily published as online webpages or PDF documents. Evaluating the citations to grey literature outputs can also help to assess the academic influence of the cited organisations, or the success of their in-house publication strategy (Bickley et al., 2020 ).

Research questions

This study investigates how different sources of grey literature (in the wider sense) were cited in academic articles before (2005–2019) and during (2020–2021) the coronavirus pandemic and assesses how non-academic sources helped Covid-19 research. For this, 17 grey literature sources were chosen, covering health care, news, data and statistics, and large international organisations. This ad hoc selection includes an illustrative range of different non-academic sources that could support research articles. It is not practical to cover all grey literature sources and there seems to be no systematic way to select a range of different types of source, so the 17 sites were instead manually selected to cover prominent websites of different relevant types. The research questions are as follows.

• How frequently are the 17 chosen sources of grey literature cited in Scopus indexed journal articles during 2005–2021, is there a trend, and has Covid-19 influenced any trend?
• Which types of grey literature documents are cited in Covid-19 research?
• What are the broad topics of Covid-19 articles citing grey literature?

Grey literature sources

Seventeen websites hosting academic-relevant grey literature were selected for analysis, to include health care, news, statistics, and international organisations (Table ​ (Table1). 1 ). The selections were based on prior experience analysing grey literature citations.

Seventeen sources of grey literature selected in the study and their scope of activities

Scopus URL citation search for grey literature

Elsevier's Scopus citation database was used to count URL citations (i.e., documents with references containing relevant URLs) to the 17 grey literature websites. Scopus was selected instead of Clarivate’s Web of Science because it has wider coverage of academic journals and supports searches to identify cited URLs in the references of academic publications. All data collection from Scopus was carried out during the 13th of October 2021.

For the Scopus URL citation searches, the main web addresses of the seventeen grey literature sources were searched using the “Reference Website” Scopus field, such as through the queries WEBSITE("who.int/*") or WEBSITE("un.org/*") . The query results were restricted to journal articles published between January 2005 and October 2021 to estimate the role of these non-academic web sources in formal research communication before (2005–2019) and during the Covid-19 pandemic (2020–2021). Two domains were searched in Scopus for the BBC ( bbc.co.uk and bbc.com ) and the Guardian ( theguardian.com and guardian.co.uk ) to locate citations to both of their main websites (see an example below).

WEBSITE ("bbc.co.uk/*") OR WEBSITE ("bbc.com/*") AND PUBYEAR > 2004 AND (LIMIT-TO (DOCTYPE,"ar")) AND (LIMIT-TO (SRCTYPE, "j"))

Because Scopus increases in size annually and to allow the analysis of the partial year 2021 data, the numbers of URL citations to grey literature were normalised by dividing by the number of Scopus journal articles for each year to examine their growth as a percentage. For instance, although there were more citations to The European Union website (europa.eu) from Scopus articles published during January to December 2020 (35,599) than from articles published during January to October 2021 (34,475), the normalised results were higher in 2021 (0.016) than in 2020 (0.014).

Additional searches were conducted to locate citations to the 17 grey literature websites from coronavirus-related research published between January 2020 and October 2021, as shown in the example below for WHO. For this, the following set of distinctive terms was searched for in titles, abstracts or keywords: covid, covid-19, SARS-CoV-2, 2019-nCoV, coronavirus, "corona virus" .

WEBSITE ("who.int/*") AND TITLE-ABS-KEY (covid) OR TITLE-ABS-KEY ("SARS-CoV-2") OR TITLE-ABS-KEY ("2019-nCoV") OR TITLE-ABS-KEY (coronavirus) OR TITLE-ABS-KEY ("corona virus") OR TITLE-ABS-KEY ("2019-nCoV") AND (LIMIT-TO (PUBYEAR, 2021) OR LIMIT-TO (PUBYEAR, 2020)) AND (LIMIT-TO (DOCTYPE, "ar")) AND (LIMIT-TO (SRCTYPE, "j")).

To assess the broad fields of the Covid-19 articles citing each website, related subcategories were merged into four subjects using the predefined Scopus SUBJAREA search field for Health Care ( MEDI OR NURS OR VETE OR DENT OR HEAL OR MULT ), Life Sciences ( AGRI OR BIOC OR IMMU OR NEUR OR PHAR ), Physical Sciences ( CENG OR CHEM OR COMP OR EART OR ENER OR ENGI OR ENVI OR MATE OR MATH OR PHYS ) and Social Sciences ( ARTS OR BUSI OR DECI OR ECON OR PSYC OR SOCI ). The query below was used to identify URL citations to the United Nations (un.org) from Covid-19 articles in Social Science subjects between 2020 and October 2021. For example, the query retrieved the article “Pandemics, tourism and global change: a rapid assessment of COVID-19” published in the Journal of Sustainable Tourism in 2020, citing statistics about air transport passengers.

WEBSITE("who.int/*") AND TITLE-ABS-KEY(covid) OR TITLE-ABS-KEY("COVID-19") OR TITLE-ABS-KEY("SARS-CoV-2") OR TITLE-ABS-KEY("2019-nCoV") OR TITLE-ABS-KEY(coronavirus) OR TITLE-ABS-KEY("corona virus") OR TITLE-ABS-KEY("2019-nCoV") AND SUBJAREA(ARTS OR BUSI OR DECI OR ECON OR PSYC OR SOCI) AND (LIMIT-TO (SRCTYPE, "j")) AND (LIMIT-TO (DOCTYPE,"ar")) AND (LIMIT-TO (PUBYEAR,2021) OR LIMIT-TO (PUBYEAR,2020)).

Content analysis of cited grey literature by Covid-19 research

A content analysis was conducted on the 340 grey literature URLs with the most citations from Covid-19 articles to identify the characteristics of cited grey literature sources. For this, the free Webometric Analyst software (lexiurl.wlv.ac.uk) was used to automatically convert Scopus CSV outputs to text files (see “Tab-Sep”, “Convert CSV to Tab-separated”) and to extract the URL citations to each grey literature source from the references of the citing Scopus articles found (see “Citation” menu, “Extract URLs from Scopus Web”).

The frequencies of the cited URLs were calculated after manual checking of the 20 URLs with the most citations for each grey literature website, excluding general citations to their homepages (e.g., who.int/ or fao.org/). The first author checked the contents of the 20 URLs with most Scopus citations from Covid-19 articles (20 × 17 = 340) to assess what type of information was commonly cited in academic articles and whether the cited webpages were relevant to Covid-19. After examining the data for common themes, the cited HTML and PDF webpages were classified into the following five broad categories.

• Data and statistics This category includes webpages providing any kind of data, statistics, survey results, census, situation report or numeric indicators, graphs, charts, maps, or other numerical formats. This could be about Covid-19 (e.g., WHO global Covid-19 situation) or not (e.g., FAO Food Price Index).
• Policy document or report This class includes webpages reflecting policy, strategies or agendas of international organisations or governments. These were mainly about towards different development plans such as for health care, science, socio-economic actions and education. For instance, this class includes UN reports on the impact of COVID-19 on children, EU discharge criteria for confirmed COVID-19 cases, and the universal declaration of human rights.
• News This category includes news stories, features and analysis from news agencies, organisations or other institutions to inform the general public about an event or situation. Examples include a WHO director briefing about Covid-19, an FDA announcement about first treatments for COVID-19 and BBC news stories about reducing air pollution and CO2 during the pandemic.
• Clinical guideline or trial This category contains clinical documents providing recommendations, observations or trials for treatments, diagnosis, or management of diseases or other medical conditions. Examples include WHO Covid-19 clinical management guidance, the NICE (National Institute for Health and Care Excellence) guideline for post-traumatic stress disorder and a trial of remdesivir in adults with severe Covid-19.
• General guidance This category includes webpages providing general guidance, advice, questions and answers. It includes guidance for certifying deaths due to Covid-19, tips and guidance for washing hands to protect against Covid-19 and breastfeeding during the pandemic

Normalised comparison of citations to grey literature

Figures  1 , ​ ,2, 2 , ​ ,3 3 and ​ and4 4 show the annual proportions of Scopus articles with citations to the seventeen health care, international organisation, news, and statistics websites during 2005–2021. The results show that the proportions of articles citing these grey literature sources have grown over time, although there are also differences between the websites. In particular, the WHO and European Union websites had attracted the most citations from academic articles. There is also in many cases a greater increase during the pandemic (2020–2021) and the increase is greatest for WHO, the European Union, the US Center for Health Statistics, and most news websites. For instance, the proportion of academic citations to WHO during the pandemic in 2020 and 2021 was 1.3 and 1.5 times higher than before the pandemic in 2019. Tables A1–A4 in the online appendix report the number of Scopus articles with at least one citation to the 17 websites during 2005–2021 ( https://doi.org/10.6084/m9.figshare.16810471.v1 ).

The percentage of Scopus articles citing four health care websites between 2005 and October 2021

The proportion of Scopus articles citing five large organisation websites between 2005 and October 2021

The proportion of Scopus articles citing four news websites between 2005 and October 2021

The proportion of Scopus articles citing three statistics websites between 2005 and October 2021

Academic use of grey literature for coronavirus research

Figure  5 shows the percentage of citations to the 17 grey literature sources from Covid-19 and non-Covid-19 articles published between January 2020 and October 2021. To identify citations from non-Covid-19 articles, the AND NOT operator was used to exclude coronavirus relevant articles. 1 Overall, 16% of the citations found were from Covid-19 research. Nevertheless, there are some differences between the 17 non-academic sources. A quarter of the citations to WHO (24.9%) and Office for National Statistics (23.2%) documents were from Covid-19 research, reflecting the greater importance of this organisation for the pandemic. Three of the news sources also attracted an above average share citations from Covid-19 articles (CNN: 26.4%, BBC: 24.5%, New York Times: 23.3%), suggesting that coronavirus information disseminated by news agencies was often useful for research. In the following section, the types of grey literature sources with the most citations from articles is discussed.

The percentage of citations to grey literature sources from Covid-19 and non-Covid-19 journal articles between January 2020 and October 2021

Figure  6 shows the broad subjects of the Covid-19 articles citing the 17 websites. Just under half (48%) of the Medicine and Health Science citations were from Covid-19 research, followed by Social Sciences (24.6%), Physical Sciences (14.4%) and Life Sciences (13%). This suggests that grey literature has been useful for coronavirus research outside the medical sciences, but at a lower rate. For instance, Covid-19 Social Science research was particularly likely to cite UNESCO (55%), the UN (44%), and the World Bank (43%) in addition to the Guardian (52%), BBC (45%), CNN (44%) and the New York Times (42%).

The percentage of citations to grey literature sources from Covid-19 journal articles between January 2020 and October 2021 by broad subject

Types of grey literature cited in Covid-19 research

A content analysis of 340 grey literature URLs with the most Scopus citations from Covid-19 articles (20 per website) found that 81.8% (278) were Covid-relevant. The types of information usually cited varied between the sources. For instance, news (37%), data and statistics (29%) and general health guidance (25%) from the WHO website were most cited by Covid-19 research, whereas clinical guidelines or trials were predominantly cited from NICE (98%), ClinicalTrials.gov (100%), and FDA (89%). With one exception (the UN), the majority of the most cited webpages were not traditional grey literature (policy documents, reports, clinical guidelines or trials) but were newer types of grey literature.

Surprisingly, the New York Times (69%) and the BBC (42%) were often cited for data and statistics rather than for news stories. Similarly, data and statistics from four of the international organisations had also attracted many citations from coronavirus research: World Bank (79%), UNESCO (77%), EU (51%) and FAO (38%). In contrast, policy documents or reports from the UN had received the most citations (53%), followed by FAO (25%) and EU (14%). Unsurprisingly, nearly all citations to statistics websites were to data, statistics, survey results, census information, or indicators: The UK Office for National Statistics (97%), The US Census Bureau (95%), and The National Center for Health Statistics (90%). Overall, it seems that data and statistics published by many non-academic sources had an important role in supporting Covid-19 research. The citation counts, types and description of 340 grey literature URLs with the most Scopus citations from Covid-19 articles are available online ( https://doi.org/10.6084/m9.figshare.16810471.v1 ).

This study is primarily limited by the restriction to 17 websites rather than covering all types of grey literature. Since the results differed substantially between websites, any other selection of websites might have given substantially different results overall. Thus, the results for individual sources provide only a weak indication of the likely pattern for others. Future studies may assess the role of national grey literature sources within academia (e.g., India, Brazil, China and Russia). For instance, out of all Scopus citations to Chinese Government websites (“.gov.cn/*") during the pandemic in 2020 (11,290) and 2021 (13,138) about 22 and 17% were from COVID-19 relevant articles (2227 and 2507 respectively) Similarly, about 11 and 16% of Scopus citations from journal articles to Indian Government websites (".gov.in/*") in 2020 and 2021 were from COVID-19 related research, suggesting that non-academic information published by governments can be valuable in research communication.

The almost universally increasing citing of all 17 websites (Figs.  1 , ​ ,2, 2 , ​ ,3, 3 , ​ ,4) 4 ) is perhaps surprising, despite the arguments in the Introduction about the increasing prevalence of online information and the increasing need to make research relevant to society. The online availability arguments do not seem relevant to news websites, for example, which were established many years ago (e.g., 1997 for the BBC: BBC, 2007 ) and do not seem to be expanding. In contrast, an increased rate of citation to government websites might be expected as they increase the amount of material that they put in the public domain (Bickley et al., 2020 ). The almost universally increasing citation rates therefore seem to reflect a greater willingness to cite non-academic sources, rather than an increase in the citable content. This may be due to the increased academic focus on societal impact (e.g., Watermeyer & Hedgecoe, 2016 ) and consequent need to reference external events and data, as also suggested in the Introduction. The apparent greater willingness to cite web sources may also partly stem from an easing of early fears that such citations were unhelpful because webpages could be deleted or changed (Germain, 2000 ; Wagner et al., 2009 ). Even though the issue of disappearing URLs is still relevant, authors and referees may have seen enough web citations in published research that they are rarely noticed, except perhaps in exceptional cases. They may also accept disappearing URLs for citations that contextualise a study rather than underpinning its data, methods or theory. The Internet Archive is increasingly used to fix URLs, when necessary, but still only by a minority of articles citing URLs (e.g., 3195 Scopus articles referenced it in 2020, according to: WEBSITE(" www.archive.org/ *") AND PUBYEAR = 2020 AND (LIMIT-TO(DOCTYPE, "ar"))). Another possible explanation for the increased citing of URLs is that online publication is standard now, with many articles being online-only, where URLs may be seen as a natural part of the format.

Although causal connections have not been directly investigated, the influence of the pandemic is clear from some of the trends. The sharp increase in citing WHO during the pandemic is expected but it is still perhaps surprising that over 3% of articles in Scopus cited it in 2021: one article out of every 31. Despite WHO not publishing formally peer reviewed documents on its website, its staff are clearly international academic medical experts, and its work is presumably internally peer reviewed or checked thoroughly by academic medical experts and so the core information on its website might reasonably be thought to have been through a form of quality control that is at least comparable to that of a rigorous academic journal. It is not clear whether this applies to the WHO Global Covid-19 situation dashboard ( https://covid19.who.int ), however, which had attracted the most Scopus citations (3427) of any page from the selected grey literature sources, providing authoritative global data and statistics and other information about Covid-19.

The trends for the other three medical websites examined are also plausible. The FDA published important Covid-19 information and Covid-19 presumably prompted a dramatic increase in rapid clinical trials to test vaccines and treatments for Covid-19 as a novel virus needing urgent medical interventions. For example, the FDA’s “Instructions for using real-time RT-PCR Covid-19 test” (165 Citations: https://www.fda.gov/media/134922/download ) had attracted many Scopus citations. In contrast, there did not seem to be a pandemic citation increase for NICE guidelines. Whilst NICE publishes many guidelines on the treatment of conditions for UK health professionals based on expert group reviews of the literature, it also produced a series of “Covid-19 rapid guidelines” (e.g., 19 by October 2021: NICE, 2021 ). These guidelines are signposted on the NICE website but are hosted on a different international collaborative site, app.MAGICapp.org. Articles do not seem to be citing MAGICapp.org (e.g., it has 18 citations in 2020) instead of NICE, so it seems that the rapid guidelines are either rarely cited or that other NICE guidelines are less cited to balance their citations. For example, a NICE Covid-19 rapid guideline for managing children, young people and adults (NG159 and updated version NG191: https://www.nice.org.uk/guidance/ng191 ) was cited 213 times by Scopus articles.

The remaining site types (news, international organisation, statistics) all seemed to have extra citations due to the pandemic. These citations partly originated from non-medical research engaging with aspects of the pandemic and citing information about its social context, such as UNESCO statistics about Covid-19’s impact on education and school closures (683 citations), death statistics involving Covid-19 in England and Wales from the Office for National Statistics (455 citations), or The New York Times Coronavirus Vaccine Tracker (145 citations). Thus, these increases presumably reflect a more immediate connection between academic research and societal needs during the pandemic. They also reflect the wide variety of non-academic sources that have helped academic contributions to addressing the needs of the pandemic such as the World Bank Indicator (643 citations), the UN 2019 Revision of World Population Prospects (167 citations) or FAO statistics about food production, security, trade, and prices (67 citations).

In terms of the type of webpage cited, whilst traditional grey literature in the form of reports published online were sometimes cited from international organisations (mainly FAO, UN), news or media posts, or statistics were far more commonly cited (Fig.  7 ). Thus, academic research does not seem to be extensively citing reports that have not been through academic peer review but is instead citing other types of content, presumably for facts about the pandemic, medical information, or government actions. This partly aligns with the open science movement aims to encourage researchers to share more non-refereed content, such as preprints, software, and data. For example, the FORCE11 declaration on data sharing stresses the need to formally cite data used (to credit the authors) (Martone, 2014 ). This practice may have helped to normalise citing data so that non-academic data creators (e.g., Census Bureau, FAO) are also frequently cited formally when their data is used.

The percentage of type of top 20 websites with most citations from Covid-19 articles during January 2020 to October 2021 across different grey literature sources

Conclusions

The results show that citing grey literature is common in academic research, with a single organisation, WHO, now being cited in over 3% of all papers. Citations to grey literature are common in all broad areas of science, and the usual targets are not reports but data, statistics, news stories and other webpage types. Moreover, there seems to be an increasing use of grey literature citations in academic research, including to all 17 of the organisations examined here. This increase seems to have been accelerated by Covid-19 for nearly all organisations. Two partial causes may be increased online publishing and increased acceptance of citing non-refereed and online sources, perhaps as a by-product of the open access movement. Another partial cause may be the increased importance of societal relevance for research, leading to a need to cite documentation produced by important organisations, facts about the world (such as in news stories), or to analyse (and cite) real-world data.

From the perspective of research evaluation, the results suggest that citing grey literature has become an accepted and standard part of research. Whilst this is concerning from a knowledge integrity perspective, because the cited URLs could disappear, it is positive in terms of the suggested closer connections between academic research and society, and a disappearing URL may not be critical for context-setting citations (e.g., Covid-19 citing death rates to emphasise the importance of pandemic-relevant studies). The increase in citing grey literature may reflect a greater tendency of scholarship to directly engage with societal problems, such as the pandemic, and should not therefore be discouraged unless the integrity of an article is compromised. Thus, research evaluators should be accepting of the potential value of grey literature citations from academic publications. In parallel with this, academic contributions to cited grey literature produced by non-academic organisations, such as WHO, should also be valued for their contribution to research as well as for the direct societal impact of the organisation. This should include not just reports but also general guidance and carefully collected data, statistics and facts.

Below is the link to the electronic supplementary material.

Data availability

1 An example of query used to identify citations to FAO from non-Covid-19 articles: WEBSITE ("fda.gov/*") AND NOT TITLE-ABS-KEY (covid) AND NOT TITLE-ABS-KEY ("SARS-CoV-2") AND NOT TITLE-ABS-KEY ("2019-nCoV") AND NOT TITLE-ABS-KEY (coronavirus) AND NOT TITLE-ABS-KEY (" corona virus ") AND NOT TITLE-ABS-KEY ("2019-nCoV") AND (LIMIT-TO (DOCTYPE, "ar")) AND (LIMIT-TO ( SRCTYPE, "j")) AND (LIMIT-TO (PUBYEAR, 2021) OR LIMIT-TO (PUBYEAR, 2020)).

Contributor Information

Kayvan Kousha, Email: [email protected] .

Mike Thelwall, Email: [email protected] .

Matthew Bickley, Email: [email protected] .

• Alberani V, De Castro Pietrangeli P, Mazza AM. The use of grey literature in health sciences: A preliminary survey. Bulletin of the Medical Library Association. 1990; 78 (4):358–363. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Aloia D, Naughton R. Share #GreyLit: Using social media to communicate grey literature. Grey Journal (TGJ) 2016; 12 (2):97–103. [ Google Scholar ]
• Batt K, Fox-Rushby JA, Castillo-Riquelme M. The costs, effects and cost-effectiveness of strategies to increase coverage of routine immunizations in low- and middle-income countries: Systematic review of the grey literature. Bulletin of the World Health Organization. 2004; 82 (9):689–696. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• BBC (2007). 10 years of the BBC News website. http://news.bbc.co.uk/1/hi/in_depth/629/629/7057140.stm
• Bickley MS, Kousha K, Thelwall M. Can the impact of grey literature be assessed? An investigation of UK government publications cited by articles and books. Scientometrics. 2020; 125 (3):1425–1444. doi: 10.1007/s11192-020-03628-w. [ CrossRef ] [ Google Scholar ]
• Carpenter BB. GrayLit network: A science portal of technical papers. Reference Reviews. 2005; 19 (5):40–41. doi: 10.1108/09504120510604292. [ CrossRef ] [ Google Scholar ]
• Di Girolamo N, Reynders RM. Characteristics of scientific articles on COVID-19 published during the initial 3 months of the pandemic. Scientometrics. 2020; 125 (1):795–812. doi: 10.1007/s11192-020-03632-0. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Germain CA. URLs: Uniform resource locators or unreliable resource locators. College & Research Libraries. 2000; 61 (4):359–365. doi: 10.5860/crl.61.4.359. [ CrossRef ] [ Google Scholar ]
• Glänzel W, Zhou P. Publication activity, citation impact and bi-directional links between publications and patents in biotechnology. Scientometrics. 2011; 86 (2):505–525. doi: 10.1007/s11192-010-0269-6. [ CrossRef ] [ Google Scholar ]
• Kousha K, Thelwall M. COVID-19 publications: Database coverage, citations, readers, tweets, news, Facebook walls, Reddit Posts. Quantitative Science Studies. 2020; 1 (3):1068–1091. doi: 10.1162/qss_a_00066. [ CrossRef ] [ Google Scholar ]
• Kousha K, Thelwall M, Abdoli M. The role of online videos in research communication: A content analysis of youtube videos cited in academic publications. Journal of the American Society for Information Science and Technology. 2012; 63 (9):1710–1727. doi: 10.1002/asi.22717. [ CrossRef ] [ Google Scholar ]
• Kousha K, Thelwall M, Abdoli M. Which types of online evidence show the non-academic benefits of research? Websites cited in UK impact case studies. Quantitative Science Studies. 2021 doi: 10.1162/qss_a_00145. [ CrossRef ] [ Google Scholar ]
• Martone M. Data citation synthesis group: Joint declaration of data citation principles. FORCE11; 2014. [ Google Scholar ]
• NICE (2021). COVID-19 products: A list of all our products on COVID-19. https://www.nice.org.uk/guidance/conditions-and-diseases/respiratory-conditions/covid19/products?ProductType=Guidance&Status=Published
• Nove A, Cometto G, Campbell J. Assessing the health workforce implications of health policy and programming: How a review of grey literature informed the development of a new impact assessment tool. Human Resources for Health. 2017 doi: 10.1186/s12960-017-0252-x. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Pappas C, Williams I. Grey literature: Its emerging importance. Journal of Hospital Librarianship. 2011; 11 (3):228–234. doi: 10.1080/15323269.2011.587100. [ CrossRef ] [ Google Scholar ]
• Pejšová P, Vaska M. An analysis of current grey literature document typology. Grey Journal (TGJ) 2011; 7 (2):72–80. [ Google Scholar ]
• Savić D. Rethinking the role of grey literature in the fourth industrial revolution. Grey Journal (online) 2018; 14 (2018):7–14. [ Google Scholar ]
• Schöpfel J. Observations on the future of grey literature. Grey Journal (TGJ) 2006; 2 (2):67–76. [ Google Scholar ]
• Shrivastava R, Mahajan P. Analysis of the usage and diversity of grey literature in addiction research: A study. Collection and Curation. 2021; 40 (3):93–99. doi: 10.1108/CC-12-2019-0046. [ CrossRef ] [ Google Scholar ]
• Small H. On the shoulders of Robert Merton: Towards a normative theory of citation. Scientometrics. 2004; 60 (1):71–79. doi: 10.1023/B:SCIE.0000027310.68393.bc. [ CrossRef ] [ Google Scholar ]
• Spiegel-Rosing I. Science studies: Bibliometric and content analysis. Social Studies of Science. 1977; 7 (1):97–113. doi: 10.1177/030631277700700111. [ CrossRef ] [ Google Scholar ]
• Swales JM. Aspects of article introductions (No. 1) University of Michigan Press; 2011. [ Google Scholar ]
• Thelwall M. Should citations be counted separately from each originating section? Journal of Informetrics. 2019; 13 (2):658–678. doi: 10.1016/j.joi.2019.03.009. [ CrossRef ] [ Google Scholar ]
• von Elm E, Costanza MC, Walder B, Tramèr MR. More insight into the fate of biomedical meeting abstracts: A systematic review. BMC Medical Research Methodology. 2003; 3 (12):1. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Wagner C, Gebremichael MD, Taylor MK, Soltys MJ. Disappearing act: Decay of uniform resource locators in health care management journals. Journal of the Medical Library Association: JMLA. 2009; 97 (2):122. doi: 10.3163/1536-5050.97.2.009. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Watermeyer R, Hedgecoe A. Selling ‘impact’: Peer reviewer projections of what is needed and what counts in REF impact case studies. A retrospective analysis. Journal of Education Policy. 2016; 31 (5):651–665. doi: 10.1080/02680939.2016.1170885. [ CrossRef ] [ Google Scholar ]
• Woods S, Phillips K, Dudash A. Grey literature citations in top nursing journals: A bibliometric study. Journal of the Medical Library Association: JMLA. 2020; 108 (2):262–269. doi: 10.5195/jmla.2020.760. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

An official website of the United States government

Here's how you know

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Literature Search: Databases and Gray Literature

The literature search.

• A systematic review search includes a search of databases, gray literature, personal communications, and a handsearch of high impact journals in the related field.  See our list of recommended databases and gray literature sources on this page.
• a comprehensive literature search can not be dependent on a single database, nor on bibliographic databases only.
• inclusion of multiple databases helps avoid publication bias (georaphic bias or bias against publication of negative results).
• The Cochrane Collaboration recommends PubMed, Embase and the Cochrane Central Register of Controlled Trials (CENTRAL) at a minimum.     
• NOTE:  The Cochrane Collaboration and the IOM recommend that the literature search be conducted by librarians or persons with extensive literature search experience. Please contact the NIH Librarians for assistance with the literature search component of your systematic review. 

Cochrane Library

A collection of six databases that contain different types of high-quality, independent evidence to inform healthcare decision-making. Search the Cochrane Central Register of Controlled Trials here.

European database of biomedical and pharmacologic literature.

PubMed comprises more than 21 million citations for biomedical literature from MEDLINE, life science journals, and online books.

Largest abstract and citation database of peer-reviewed literature and quality web sources. Contains conference papers.

Web of Science

World's leading citation databases. Covers over 12,000 of the highest impact journals worldwide, including Open Access journals and over 150,000 conference proceedings. Coverage in the sciences, social sciences, arts, and humanities, with coverage to 1900.

Subject Specific Databases

APA PsycINFO

Over 4.5 million abstracts of peer-reviewed literature in the behavioral and social sciences. Includes conference papers, book chapters, psychological tests, scales and measurement tools.

CINAHL Plus

Comprehensive journal index to nursing and allied health literature, includes books, nursing dissertations, conference proceedings, practice standards and book chapters.

Latin American and Caribbean health sciences literature database

Gray Literature

• Gray Literature  is the term for information that falls outside the mainstream of published journal and mongraph literature, not controlled by commercial publishers
• hard to find studies, reports, or dissertations
• conference abstracts or papers
• governmental or private sector research
• clinical trials - ongoing or unpublished
• experts and researchers in the field     
• Library catalogs
• Professional association websites
• Google Scholar  - Search scholarly literature across many disciplines and sources, including theses, books, abstracts and articles.
• Dissertation Abstracts - dissertation and theses database - NIH Library biomedical librarians can access and search for you.
• NTIS  - central resource for government-funded scientific, technical, engineering, and business related information.
• AHRQ  - agency for healthcare research and quality
• Open Grey  - system for information on grey literature in Europe. Open access to 700,000 references to the grey literature.
• World Health Organization  - providing leadership on global health matters, shaping the health research agenda, setting norms and standards, articulating evidence-based policy options, providing technical support to countries and monitoring and assessing health trends.
• New York Academy of Medicine Grey Literature Report  - a bimonthly publication of The New York Academy of Medicine (NYAM) alerting readers to new gray literature publications in health services research and selected public health topics. NOTE: Discontinued as of Jan 2017, but resources are still accessible.
• Gray Source Index
• OpenDOAR - directory of academic repositories
• International Clinical Trials Registery Platform  - from the World Health Organization
• Australian New Zealand Clinical Trials Registry
• Brazilian Clinical Trials Registry
• Chinese Clinical Trial Registry - 
• ClinicalTrials.gov   - U.S.  and international federally and privately supported clinical trials registry and results database
• Clinical Trials Registry  - India
• EU clinical Trials Register
• Japan Primary Registries Network  
• Pan African Clinical Trials Registry

info This is a space for the teal alert bar.

notifications This is a space for the yellow alert bar.

Research Process

• Brainstorming
• Explore Google This link opens in a new window
• Explore Web Resources
• Explore Background Information
• Explore Books
• Explore Scholarly Articles
• Narrowing a Topic
• Primary and Secondary Resources
• Academic, Popular & Trade Publications
• Scholarly and Peer-Reviewed Journals
• Grey Literature
• Clinical Trials
• Evidence Based Treatment
• Scholarly Research
• Database Research Log
• Search Limits
• Keyword Searching
• Boolean Operators
• Phrase Searching
• Truncation & Wildcard Symbols
• Proximity Searching
• Field Codes
• Subject Terms and Database Thesauri
• Reading a Scientific Article
• Website Evaluation
• Article Keywords and Subject Terms
• Cited References
• Citing Articles
• Related Results
• Search Within Publication
• Database Alerts & RSS Feeds
• Personal Database Accounts
• Persistent URLs
• Literature Gap and Future Research

Web of Knowledge

• Annual Reviews
• Systematic Reviews & Meta-Analyses
• Finding Seminal Works
• Exhausting the Literature
• Finding Dissertations
• Researching Theoretical Frameworks
• Research Methodology & Design
• Tests and Measurements
• Organizing Research & Citations This link opens in a new window
• Scholarly Publication
• Learn the Library This link opens in a new window

Additional Information

• Grey Literature Guides A directory of grey literature research guides, repositories, journals and conference proceedings, and training resources.

Grey Literature Defined

Grey literature is literature produced by government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers. A widely quoted definition of grey literature is "literature which is not readily available through normal book-selling channels, and therefore difficult to identify and obtain" ( Wood, 1982 ). Grey literature includes theses and dissertations, conference papers and proceedings, reports (such as white papers, working papers, internal documentation), government documents, technical notes and specifications, proposals, datasets/statistics, policies/procedures, patents, unpublished trial data, regulatory data, speeches, urban plans, test instruments, pre-prints, company information, social media, and more. A full list of  Document Types in Grey Literature is provided by GreyNet International.

It is crucial to note where the term “grey literature” derives from. Grey literature comes from the uncertainty of the status of this information. However, in cases where there may not be much information on a topic in peer-reviewed research, grey literature may prove a very valuable source of information and introduce alternate viewpoints. Additional benefits to using grey literature include:

• Grey literature can be published much more quickly since it does not have to be subjected to the lengthy peer-review process. Results of studies may appear in gray literature 12 to 18 months before being published via traditional channels.
• Online information from organizations may be updated more frequently than traditional published journals/books.
• Some grey literature may contain more depth—for example, a dissertation may include some raw data not published in a journal article that author goes on to write.
• Grey literature may provide a broader overview of an issue/topic, such as a white paper or fact sheet.
• Minimizes reporting or publication biases; grey literature is more likely to include negative results - e.g. clinical trials.

Locating Grey Literature

Grey literature may present a number of challenges for the researcher, making it difficult to identify and find. Consider the following when searching for grey literature:

• May not be widely disseminated
• May not be published online or not stable online (URL/website may change)
• Older documents may not be archived
• Format and citation information may be inconsistent
• Volume of material may be overwhelming and consuming
• May not have an international standard book number (ISBN) or an international standard serial number (ISSN)
• Typically not peer-reviewed and quality of evidence varies

Review the Evaluating Information Guide to help determine if a resource is appropriate for use in your research. The AACODS checklist is designed to enable evaluation and critical appraisal of grey literature.

The techniques below outline suggestions for locating select types of grey literature, as well as searching more broadly online. Please contact the Library with any questions.

Grey Literature Types

• Conference Proceedings
• Dissertations & Theses
• Government Documents
• Research Reports
• Social Media

Clinical trials are a form of grey literature  and can inform current research conducted by organizations, Federal agencies like the National Institutes of Health, academic institutions, and individual health care providers. These studies investigate the effectiveness of new treatments, interventions, drugs, procedures, and devices in order to improve health outcomes for a specific population. 

Several Library databases and online resources provide access to full-text clinical trials. See our Clinical Trials Guide for detailed instructions.

If you are unable to locate the full text for a particular clinical trial, you may submit an Interlibrary Loan request for the material.

A conference proceeding is the published record of a conference, congress, symposium, or other meeting sponsored by professional association or society. Proceedings typically include abstracts or reports of papers presented by the participants. When the entire  text of the papers presented is included, they are called transactions. For more information, see the FAQ: Are conference proceedings peer reviewed?

Conference papers and proceedings can be challenging to find because they may take several years to be published or may not be published at all. They can be published in various formats including books, abstracts, and journal articles; and, they may be deposited only in an author or institutional repository.

Some Library databases, like ProQuest , make it easier to search for these types of documents by allowing you to limit your search to Source type. Other databases that provide conference proceedings include IEEE Xplore, ACM Digital Library, LearnTechLib, CINAHL, and ERIC. These databases are available from the A-Z Databases page.

Previously published dissertations and theses can be a great source of inspiration for your own dissertation topic. You can access millions of full-text dissertations and theses from the Library. Go to the Library’s Dissertation Resources page to see a list of databases. You can specifically access dissertations from alumni via ProQuest Dissertations and Theses @ Northcentral University. Or you can explore the 3 million full-text dissertations available through ProQuest Dissertations and Theses Global.

For additional information and resources, see the Finding Dissertations Guide .

Government documents are an important primary source of information on a wide range of issues. Most government documents can be found through official government websites.

• Catalog of U.S. Government Publications Finding tool for federal publications that includes descriptive information for historical and current publications as well as direct links to the full document, when available.
• Data.gov The home of the U.S. Government's open data. Here you will find data, tools, and resources to conduct research, develop web and mobile applications, design data visualizations, and more.
• Electronic Code of Federal Regulations The codification of the general and permanent rules published in the Federal Register by the departments and agencies of the Federal Government produced by the Office of the Federal Register (OFR) and the Government Publishing Office.
• Federal Registrar Official journal of the federal government of the United States that contains government agency rules, proposed rules, and public notices.
• govinfo Provides free public access to official publications from all three branches of the Federal Government.
• HathiTrust A collection of U.S. Federal Documents as identified via the HathiTrust Federal Government Documents Registry.
• MetaLib Federated search engine that searches multiple U.S. Federal government databases, retrieving reports, articles, and citations while providing direct links to selected resources available online.
• Pueblo.GPO.gov Operated by the U.S. Government Publishing Office Pueblo Distribution Center (US GPO PDC) to support free federal publication distribution activities.
• U.S. Government Bookstore Allows users to search for and order Government information products currently available from the Information Dissemination area of the U.S. Government Publishing Office.
• U.S. Government Printing Office (GPO) Keeping America Informed as the official, digital, and secure source for producing, preserving, and distributing official Federal Government publications and information products for Congress, Federal agencies, and the American public.
• USA.gov USA.gov is the official U.S. portal to federal, state, local, tribal, and international government information and services.

A preprint is an author's working version of a research manuscript prior to publication. In most cases, they can be considered final drafts. Preprints may also be referred to as working papers or unpublished manuscripts. Preprints have not gone through the peer-review process, nor have they been improved upon by the publisher (e.g., formatting, copy-editing, technical enhancements). It is very important to critically evaluate preprint publications. Because the peer review process can take three to six months, preprints are a way to provide valuable research results and discussion ahead of publication. They are also a great way to locate information on emerging or rapidly changing research topics.

Preprints are published electronically and made publicly available on large databases or preprint repositories. The techniques below describe how to locate preprints in the Libray and online. Note when searching for preprints in Library databases, you do not want to select the peer-reviewed journal limiter.

EBSCO Business Source Complete

In Business Source Complete , select the Advanced Search screen. Scroll down to Publication Type and select Working Paper .

ProQuest Central

In ProQuest Central , select the Advanced Search screen. Scroll down to the Source Type limiter and select Working Papers .

PubMed Central

In PubMed Central, select the Advanced Search Builder . Enter the preprint[filter] command along with your topic keywords.

After conducting a search in Web of Knowledge (Web of Science), check the Early Access box under the Quick Filters on the left-hand side of the screen.

Preprint Repositories

• Advance: a SAGE preprints community Allows researchers within the fields of humanities and social sciences to post their work online and free of charge.
• arXiv Open access to more than a million e-prints in Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance and Statistics.
• Authorea Browse over 20,000 multi-disciplinary research preprints.
• bioRxiv Free online archive and distribution service for unpublished preprints in the life sciences.
• ChemRxiv Free submission, distribution, and archive service for unpublished preprints in chemistry and related areas.
• Figshare Repository where users can make all of their research outputs available in a citable, shareable and discoverable manner.
• medRxiv Free online archive and distribution server for complete but unpublished manuscripts (preprints) in the medical, clinical, and related health sciences.
• OSF Preprints (Open Science Framework) Free, open platform to support research and enable collaboration.
• PLOS Preprints Nonprofit, Open Access publisher empowering researchers to accelerate progress in science and medicine by leading a transformation in research communication.
• Preprints Preprints is a multidisciplinary preprint platform that accepts articles from all fields of science and technology.
• PsyArXiv A free preprint service for the psychological sciences.
• Research Square Research Square is a multidisciplinary preprint and author services platform.
• Rxivist Combines biology preprints from bioRxiv and medRxiv with data from Twitter to help researchers find the papers being discussed in their field.

Research reports contain the results of research projects, investigations, and surveys, and are usually published by the funder or the body undertaking the research. They can be found by searching the websites of subject associations and research organizations in addition to NU Library’s databases.

For example, you may limit your NavigatorSearch results to reports by using the limiters on the left-hand side of your results screen.

Social media and other Web 2.0 applications have become a valuable medium for for experts, governments and academics to publish new information and analysis outside of the traditional publishing arrangement. These sites may be useful as a way to disseminate information/results, follow experts and trending topics, and as a forum for exchange of ideas.

For a thorough overview of Web 2.0 grey literature sources, please see the Grey Literature: Grey 2.0 Guide from New York University.

Grey Literature Repositories

Grey literature repositories are curated collections of grey literature sources. You can use these collections to search or browse for resources for use in your research.

• BASE One of the world's most voluminous search engines especially for academic web resources. Note: Not exclusively grey literature.
• CORE World’s largest collection of open access research papers. Note: Not exclusively grey literature.
• Directory of Open Access Journals (DOAJ) Provides full-text, quality controlled scientific and scholarly, peer-reviewed journals in many subjects and languages. Note: Not exclusively grey literature.
• Grey Literature Publishers List The report is a publication produced by the The New York Academy of Medicine between 1999 - 2016, alerting readers to new grey literature publications in health services research and selected urban health topics.
• Grey Literature Report The report is a bimonthly publication of The New York Academy of Medicine alerting readers to new grey literature publications in health services research and selected urban health topics.
• GreySource Provides examples of grey literature and in so doing profiles organizations responsible for its production and/or processing.
• OAIster Searches across thousands of libraries and scholarly websites for open access content. Note: Not exclusively grey literature.
• OpenGrey Search portal for European grey literature covering Science, Technology, Biomedical Science, Economics, Social Science and the Humanities.
• Science.gov Gateway to U.S. Federal Science. Search over 60 databases and over 2,200 selected websites from 15 federal agencies, for research and development results.
• Google Custom Search
• Domain Limiter
• Document Type Limiter

There are a couple of Google custom searches that narrow your results to a specific type of organizational website. This may be helpful step when searching for grey literature. Enter your search terms into these pre-built basic search boxes to see your filtered results.

• NGO Search Google Custom Search Engine that searches across hundreds non-governmental organization (NGO) websites.
• IGO Search Google Custom Search Engine that searches across IGO websites--International governmental organizations made up of more than one national government. IGO examples include NATO (North Atlantic Treaty Organization) and the WHO (World Health Organization).

Google allows you to limit your search results to a particular domain (e.g., .edu, or .gov). This can be particularly helpful if you are looking for grey literature as you are more likely to find reliable content from organizations rather than commercial content.

Look for the "site or domain" box in Google's Advanced Search options and enter the domain you'd like to search, as shown below.

You can also do this by adding site:.edu (or .org, .gov, etc) to the end of your search terms in any Google search box. For example, to find articles about “ethical leadership” published on government websites, enter the terms "ethical leadership" site:.gov, as shown below.

Google also allows you to search for results with a specific file type (example: .pdf, .doc, .ppt, .xls). This is also helpful when looking for grey literature as government and organizational reports are more likely to be available as PDF or Word documents. Datasets may be provided as Excel files.

Look for the "file type" box in Google's Advanced Search options and select the file type you would like to search, as shown below.

You can also do this by adding filetype:pdf (or .doc, .ppt, .xls, etc.) to the end of your search terms in any Google search box. For example, to find PDF documents about “ethical leadership” enter the terms "ethical leadership" filetype:pdf, as shown below.

Was this resource helpful?

• << Previous: Scholarly and Peer-Reviewed Journals
• Next: Clinical Trials >>
• Last Updated: Apr 14, 2024 12:14 PM
• URL: https://resources.nu.edu/researchprocess

© Copyright 2024 National University. All Rights Reserved.

Privacy Policy | Consumer Information

Conducting a Multivocal Systematic Literature Review About Compliance with the Brazilian Law for General Data Protection

• First Online: 20 April 2024

Cite this chapter

• Roberta Cláudia de Jesus Bordalo 4 ,
• Hugo do Val F. Fernandes 4 &
• Mônica Ferreira da Silva 4  

A Multivocal Systematic Literature Review (MSLR) is a form of Systematic Literature Review (SLR) that includes gray literature in addition to formal literature. In this chapter, the objective was to instantiate the MSLR to research information security and the Brazilian General Data Protection Law (called LGPD) in the public health area, based on the steps proposed by Kitchenham and Charters ( Guidelines for performing Systematic Literature Reviews in Software Engineering , 2007) for an SLR and in the steps for inclusion of gray literature proposed by Garousi et al. (Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Inf. Softw. Technol. 106: 101–121. https://doi.org/10.1016/j.infsof.2018.09.006 , 2019). Thus, it was possible to identify formal works that have studied the level of compliance and the challenges to adapt to the LGPD in the Brazilian public health sector, as well as gray literature publications that help the reader to know the current scenario of public health from the perspective of the LGPD, from the beginning of its processing (May 2016) to the start of sanctions (August 2021). The MSLR adds value by listing the main publications on the subject studied and exposing the state of the art and practice, both in the academic and informal spheres. The MSLR proved to be a valuable tool in understanding a topic that permeates academic interest in an expressive way, since the method is suitable for recent topics that have not yet been formally explored in the academy.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

V. Garousi, M. Felderer, e M. V. Mäntylä. 2019. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Inf. Softw. Technol. 106: 101–121. https://doi.org/10.1016/j.infsof.2018.09.006 .

B. Kitchenham e S. Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering , 57.

Google Scholar  

H. do V. F. Fernandes. 2021. Desafios da Saúde Pública Brasileira com a Proteção de Dados: uma revisão do estado de conformidade do setor de saúde pública brasileiro com a LGPD. http://hdl.handle.net/11422/15777 . Accessed 29 Mar 2023.

T. F. Galvão, T. de S. A. Pansani, e D. Harrad. 2015. Principais itens para relatar Revisões sistemáticas e Meta-análises: A recomendação PRISMA. Epidemiol. e Serviços Saúde 24: 335–342. https://doi.org/10.5123/S1679-49742015000200017 .

J. Adams et al., 2016. Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: Critical reflections on three case studies. Syst. Rev. 5: 1–11. https://doi.org/10.1186/s13643-016-0337-y .

Article   Google Scholar  

Download references

Acknowledgments

We would like to thank the Grupo de Pesquisas em Fatores Humanos e Tecnologia da Informação (HumânITas/UFRJ), which has as part of its objectives the study of protocol methods of literature review, for their conceptual contributions and critical feedback for improving the theme and strengthening of collaborative bonds.

Author information

Authors and affiliations.

Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil

Roberta Cláudia de Jesus Bordalo, Hugo do Val F. Fernandes & Mônica Ferreira da Silva

You can also search for this author in PubMed   Google Scholar

Editor information

Editors and affiliations.

University of Minho, Braga, Portugal

Sílvia Araújo

Micaela Aguiar

University of Brest, Brest, France

Liana Ermakova

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Bordalo, R.C.d.J., Fernandes, H.d.V.F., Silva, M.F.d. (2024). Conducting a Multivocal Systematic Literature Review About Compliance with the Brazilian Law for General Data Protection. In: Araújo, S., Aguiar, M., Ermakova, L. (eds) Digital Humanities Looking at the World . Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-031-48941-9_14

Download citation

DOI : https://doi.org/10.1007/978-3-031-48941-9_14

Published : 20 April 2024

Publisher Name : Palgrave Macmillan, Cham

Print ISBN : 978-3-031-48940-2

Online ISBN : 978-3-031-48941-9

eBook Packages : Social Sciences Social Sciences (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Open access
• Published: 13 December 2023

Arts and creativity interventions for improving health and wellbeing in older adults: a systematic literature review of economic evaluation studies

• Grainne Crealey 1 ,
• Laura McQuade 2 ,
• Roger O’Sullivan 2 &
• Ciaran O’Neill 3  

BMC Public Health volume  23 , Article number:  2496 ( 2023 ) Cite this article

1152 Accesses

22 Altmetric

Metrics details

As the population ages, older people account for a larger proportion of the health and social care budget. A significant body of evidence suggests that arts and creativity interventions can improve the physical, mental and social wellbeing of older adults, however the value and/or cost-effectiveness of such interventions remains unclear.

We systematically reviewed the economic evidence relating to such interventions, reporting our findings according to PRISMA guidelines. We searched bibliographic databases (MEDLINE, EMBASE, Econlit and Web of Science and NHSEED), trial registries and grey literature. No language or temporal restrictions were applied. Two screening rounds were conducted independently by health economists experienced in systematic literature review. Methodological quality was assessed, and key information extracted and tabulated to provide an overview of the published literature. A narrative synthesis without meta-analysis was conducted.

Only six studies were identified which provided evidence relating to the value or cost-effectiveness of arts and creativity interventions to improve health and wellbeing in older adults. The evidence which was identified was encouraging, with five out of the six studies reporting an acceptable probability of cost-effectiveness or positive return on investment (ranging from £1.20 to over £8 for every £1 of expenditure). However, considerable heterogeneity was observed with respect to study participants, design, and outcomes assessed. Of particular concern were potential biases inherent in social value analyses.

Conclusions

Despite many studies reporting positive health and wellbeing benefits of arts and creativity interventions in this population, we found meagre evidence on their value or cost-effectiveness. Such evidence is costly and time-consuming to generate, but essential if innovative non-pharmacological interventions are to be introduced to minimise the burden of illness in this population and ensure efficient use of public funds. The findings from this review suggests that capturing data on the value and/or cost-effectiveness of such interventions should be prioritised; furthermore, research effort should be directed to developing evaluative methods which move beyond the confines of current health technology assessment frameworks, to capture a broader picture of ‘value’ more applicable to arts and creativity interventions and public health interventions more generally.

PROSPERO registration

CRD42021267944 (14/07/2021).

Peer Review reports

The number and proportion of older adults in the population has increased in virtually every country in the world over past decades [ 1 ]. In 2015, there were around 901 million people aged 60 years and over worldwide, by 2030, this will have increased to 1.4 billion [ 2 ]. An ageing population is one of the greatest successes of public health but it has implications for economies in numerous ways: slower labour force growth; working-age people will have to make greater provisions in welfare payments for older people who are no longer economically active; provisions for increased long-term care; and, society must adjust to the changing needs, expectations and capabilities of an expanding group of its citizens.

The Covid-19 pandemic shone an uncompromising light on the health and social care sector, highlighting the seriousness of gaps in policies, systems and services. It also focused attention on the physical and mental health consequences of loneliness and social isolation. To foster healthy ageing and improve the lives of older people, their families and communities, sustained and equitable investment in health and wellbeing is required [ 3 ]. The prevailing model of health and social care which is based ostensibly on formal care provision is unlikely to be sustainable over the longer term. New models, which promote healthy ageing and recognise the need for increasing reliance on self-care are required, as will be evidence of their effectiveness, cost-effectiveness and scalability.

Arts and creativity interventions (ACIs) can have positive effects on health and well-being, as several reviews have shown [ 4 , 5 ]. For older people, ACI’s can enhance wellbeing [ 6 , 7 , 8 , 9 ], quality of life [ 10 , 11 ] and cognitive function [ 12 , 13 , 14 , 15 , 16 ]. They can also foster social cohesion [ 17 , 18 , 19 ] and reduce social disparities and injustices [ 20 ]; promote healthy behaviour; prevent ill health (including enhancing well-being and mental health) [ 21 , 22 , 23 , 24 , 25 ], reducing cognitive decline [ 26 , 27 ], frailty [ 28 , 29 , 30 , 31 , 32 , 33 ] and premature mortality [ 34 , 35 , 36 , 37 , 38 ]); support people with stroke [ 39 , 40 , 41 , 42 ]; degenerative neurological disorders and dementias and support end of life care [ 43 , 44 ]. Moreover, ACIs can benefit not only individuals, but also others, such as supporting the well-being of formal and informal carers, enriching our knowledge of health, and improving clinical skills [ 4 , 5 ].

The benefits of ACIs have also been acknowledged at a governmental level by those responsible for delivering health and care services: The UK All-Party Parliamentary Special Interest group on Arts, Health and Wellbeing produced a comprehensive review of creative intervention for health and wellbeing [ 45 ]. This report contained three key messages: that the arts can keep us well, aid recovery and support longer better lived lives; they can help meet major challenges facing health and social care; and that the arts can save money for the health service and social care.

Despite robust scientific evidence and governmental support, no systematic literature review has collated the evidence with respect to the value, cost or cost-effectiveness of such interventions. Our objective was to assess the economic impact of ACIs aimed at improving the health and wellbeing of older adults; to determine the range and quality of available studies; identify gaps in the evidence-base; and guide future research, practice and policy.

A protocol for this review was registered at PROSPERO, an international prospective register of systematic reviews (Registration ID CRD42021267944). We used pre-determined criteria for considering studies to include in the review, in terms of types of studies, participant and intervention characteristics.

The review followed the five-step approach on how to prepare a Systematic Review of Economic Evaluations (SR-EE) for informing evidence-based healthcare decisions [ 46 , 47 , 48 ]. Subsequent to developing and registering the protocol, the International Society for Pharmacoeconomic Outcomes and Research (ISPOR) published a good practice task force report for the critical appraisal of systematic reviews with costs and cost-effectiveness outcomes (SR-CCEOs) [ 49 ]. This was also used to inform the conduct of this review.

Eligibility criteria

Full economic evaluations are regarded as the optimal type of evidence for inclusion in a SR-EE [ 46 ], hence cost-minimisation analyses (CMA), cost-effectiveness analyses (CEA), cost-utility analyses (CUA) and cost–benefit analyses (CBA) were included. Social value analyses were also included as they are frequently used to inform decision-making and commissioning of services within local government. Additionally, they represent an important intermediate stage in our understanding of the costs and consequences of public health interventions, where significant challenges exist with regard to performing full evaluations [ 50 , 51 , 52 , 53 ].

Development of search strategies

The population (P), intervention (I), comparator (C) and outcomes (O) (PICO) tool provided a framework for development of the search strategy. Studies were included if participants were aged 50 years or older (or if the average age of the study population was 50 years or over). Interventions could relate to performance art (dance, singing, theatre, drama etc.), creative and visual arts (painting, sculpture, art making and design), or creative writing (writing narratives, poetry, storytelling). The intervention had to be active (for example, creating art as opposed to viewing art; playing an instrument as opposed to listening to music). The objective of the intervention had to be to improve health and wellbeing; it had to be delivered under the guidance of a professional; delivered in a group setting and delivered on more than one occasion. No restrictions were placed on the type of comparator(s) or the type of outcomes captured in the study. We deliberately limited the study to professionally led activities to provide a sharper distinction between social events where arts and creativity may occur and arts and creativity interventions per se. We set no language restriction nor a restriction on the date from which studies were reported.

Search methods

PRESS (peer-review electronic search strategies) guidelines informed the design our search strategy [ 54 , 55 ] and an information specialist adapted the search terms (outlined in Table S 1 ) for the following electronic bibliographic databases: MEDLINE, PubMed, EMBASE, Econlit and Web of Science and NHSEED. We also inspected references of all relevant studies; and searched trials registers (ClinicalTrials.gov). Search terms used included cost, return on investment, economic, arts, music, storytelling, dancing, writing and older adult as well as social return on investment (SROI). The last search was performed on 09/11/2022. As many economic evaluations of ACIs (especially SROIs) are commissioned by government bodies or charitable organisations, a search of the grey literature was undertaken.

Handling searches

A PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow chart was used to document study selection, illustrating the numbers of records retrieved and selection flow through the screening rounds [ 56 , 57 , 58 ]; all excluded records (with rationale for exclusion) were documented.

Selection of studies

Two screening rounds were conducted independently by two health economists experienced in undertaking reviews (GC, CO’N). The first round screened the title and abstract of articles based on the eligibility criteria; those selected at this stage entered a second round of full text screening with eligibility based on the inclusion and exclusion criteria. Any disagreements were discussed among the two reviewers, with access to a third reviewer available to resolve disagreements, though this proved unnecessary.

Data extraction and management

Two reviewers extracted relevant information independently using an proforma developed specifically for the purposes of this study, which included all 35 items suggested by Wijnen et al. (2016) [ 48 ]. Information was extracted in relation to the following factors: (1) general information including study title, author, year, funding source, country, setting and study design; (2) recruitment details, sample size, demographic characteristics (age, gender) and baseline health data (diagnosis, comorbidities); (3) interventions, effectiveness and cost data; (4) type of economic evaluation, perspective, payer, beneficiary, time horizon, measure of benefit and scale of intervention; (5) quality assessment, strength of evidence, any other important information; (6) results; (7) analysis of uncertainty and (8) conclusions. The quality assessment/risk of bias checklists were included in the data extraction proforma, and picklists were used to enhance uniformity of responses. The data extraction form was piloted by two reviewers (GC and CON) on one paper and discussion used to ensure consistent application thereafter.

Assessment of study quality

Two reviewers (GC & CON) independently assessed study quality, with recourse to a third reviewer for resolution of differences though this proved unnecessary. Quality assessment was based on the type of economic evaluation undertaken. Full and partial trial-based economic evaluations were assessed using the CHEC-extended checklist [ 59 ]. SROI analyses were assessed using a SROI-specific quality framework developed for the purpose of systematic review [ 60 ].

Data analysis methods

Due to the small number of evaluations detected, possible sources of heterogeneity and a lack of consensus on appropriate methods for pooling cost-effectiveness estimates [ 61 ] a narrative synthesis analysis was undertaken.

Database searches returned 11,619 records; from this, 402 duplicates were removed leaving 11,214 reports. From these 113 reports were assessment against the inclusion and exclusion criteria resulting in 4 studies for inclusion in the review. Over 40 websites were searched for relevant content returning 2 further studies for inclusion. The PRISMA 2020 diagram is presented in Fig.  1 . A high sensitivity search strategy was adopted to ensure all relevant studies were identified, resulting in a large number of studies being excluded at the first stage of screening.

PRISMA 2020 flow diagram for new systematic reviews which include searches of databases, registers and other sources

A total of six studies were identified; key characteristics are presented in Table 1 . Identified studies were published between 2011 and 2020. Two studies used a health technology assessment (HTA) framework alongside clinical trials [ 62 , 63 ] to assess the cost-effectiveness of community singing interventions. Both evaluations scored highly on the CHEC-extended checklist (Table 2 ), with findings reported in line with the CHEERS (Consolidated Health Economic Estimation Reporting Standards) checklist 2022 [ 64 ].

Four further studies employed an SROI framework to assess art and/or craft interventions: two studies were published in the peer-reviewed literature [ 65 , 66 ] and a further two in the grey literature [ 67 , 68 ]. All four adhered closely to the suggested steps for performing an SROI and consequently secured high scores (Table 3 ). No quality differential was discerned between those studies published in the academic literature when compared with those from the grey literature.

Five of the studies were undertaken in the UK [ 63 , 66 , 67 , 68 , 69 ] and one in the US [ 63 ]. Four of the studies were designed for older adults with no cognitive impairment [ 62 , 63 , 67 , 68 ]; one was designed for participants with or without dementia [ 65 ], and another was specifically for older adults with dementia and their caregivers [ 66 ]. Three of the studies were delivered in a community setting [ 62 , 63 , 67 ], two in care homes [ 65 , 68 ] and one across a range of settings (hospital, community and residential) [ 66 ]. The length and duration of the ACIs varied; some lasted 1–2 h (with multiple classes available to participants) [ 65 ], whereas others were structured programmes with sessions lasting 90 min over a 14-week period [ 62 ]. The number of participants included in studies varied; the largest study contained data from 390 participants [ 63 ], whereas other studies measured engagement using numbers of care homes or housing associations included [ 67 , 68 ].

Costs were captured from a narrower perspective (i.e., the payer—health service) for those economic evaluations which followed a health technology assessment (HTA) framework [ 62 , 63 ]. Costs associated with providing the programme and health and social care utilisation costs were captured using cost diaries. Valuation of resource usage was in line with the reference case specified for each jurisdiction.

Social value analyses included in the review [ 65 , 66 , 67 , 68 ] captured a broader picture of cost; programme provision costs included were similar in nature to those identified using an HTA framework, however, the benefits captured went beyond the individual to capture costs to a wide range of stakeholders such as family members, activity co-ordinations and care home personnel. Costs were apportioned using financial proxies from a range of sources including HACT Social Value Bank [ 69 ] and market-based valuation methods.

The range of outcomes captured and valued across HTAs and SROIs was extensive: including, but not limited to, wellbeing, quality of life, physical health, cognitive functioning, communication, control over daily life choices, engagement and empowerment, social isolation, mobility, community inclusion, depressive symptoms, sadness, anxiety, loneliness, positive affect and interest in daily life. In the programmes assessed using an HTA framework, outcomes were captured using standardised and validated instruments, for both control and intervention groups across multiple time points. Statistical methods were used to assess changes in outcomes over time. Programmes assessed using SROI relied primarily on qualitative methods (such as reflective diaries and in-depth interviews) combined with routinely collected administrative data.

The evidence from the singing interventions was encouraging but not conclusive. The ‘Silver Song Club’ programme [ 62 ] reported a 64% probability of being cost-effective at a willingness-to-pay threshold of £30,000. This study was also included in the Public Health England (PHE) decision tool to support local commissioners in designing and implementing services to support older people’s healthy ageing, reporting a positive societal return on investment [ 70 ]. Evidence from the ‘Community of Voices’ trial [ 63 ] suggested that although intervention group members experienced statistically significant improvements in loneliness and interest in life compared to control participants, no significant group differences were observed for cognitive or physical outcomes or for healthcare costs.

A positive return on investment was reported by all social value analyses undertaken. The ‘Imagine Arts’ programme, reported a positive SROI of £1.20 for every £1 of expenditure [ 65 ]. A higher yield of between £3.20-£6.62 for each £1 invested was reported in the ‘Dementia and Imagination’ programme [ 66 ]. The ‘Craft Café’ programme, reported an SROI of £8.27 per £1 invested [ 68 ], and the ‘Creative Caring’ programme predicted a SROI of between £3 to £4 for every £1 spent [ 67 ]. The time period over which return on investment was calculated differed for each evaluation from less than one year to 4 years.

The primary finding from our review concerns the paucity of evidence relating to the value, cost and/or cost-effectiveness of ACIs aimed at improving health and wellbeing in this population. Despite few restrictions being applied to our search, only six studies were found which met our inclusion criteria. This is not indicative of research into ACIs in this population, as evidenced by the identification of ninety-three studies where arts and creativity interventions were found to support better health and wellbeing outcomes in another recent review [ 5 ]. An alternative explanation is that funders do not see the added value of undertaking such evaluations in this area. That is, for funders, the cost of evaluating an ACIs is likely to be deemed unjustified given the relatively small welfare loss a misallocation of resources to them might produce. While at first glance this may seem reasonable, it disadvantages ACIs in competing with other interventions for funding and arguably exposes an implicit prejudice in the treatment of interventions from which it may be difficult to extract profit in general. That is, the paucity of evidence, may reflect inherent biases within our political economy that favour the generation of marketable solutions to health issues from which value can be appropriated as profit. Pharmaceuticals are an obvious example of such solutions, where the literature is replete with examples of evaluations sponsored by pharmaceutical companies or where public funds are used to test the claims made by pharmaceutical companies in respect of the value of their products. If the potential of ACIs to improve health and well-being is to be robustly established, ACIs must effectively compete for funding with other interventions including those from pharma. This requires a larger, more robust evidence base than is currently available and investment in the creation of such an evidence base. As there is currently no ‘for-profit’ industry to generate such an evidence base, public funding of evaluations will be central to its creation.

Our second finding concerns the values reported in the meagre evidence we did find. In five of the six studies we identified, evidence indicated that ACIs targeted at older people offered value for money [ 62 , 65 , 66 , 67 , 68 ]. One study provided mixed evidence [ 63 ], however, in this study a ‘payer’ perspective was adopted when applying an HTA framework which, by virtue of the perspective adopted, excluded a range of benefits attributable to ACIs and public health interventions more generally. Among the four studies that adopted a SROI approach, estimated returns per £1 invested ranged from £1.20 to £8.27. Given the evident heterogeneity among studies in terms of context and methods, care is warranted in comparing estimates with each other or with other SROIs. Care is also required in accepting at face value the estimates reported given methodological issues that pertain to the current state of the art with respect to SROI. With these caveats in mind noted, the values reported for ACIs using the SROI approach are comparable with those from other SROI studies in other contexts including those as diverse as a first aid intervention [ 71 ], investment in urban greenways [ 72 ] and the provision of refuge services to those experiencing domestic violence [ 73 ] (a return on investment of £3.50-£4, £2.88-£5.81 and £4.94 respectively). Similarly, with respect to the study that adopted a cost-effectiveness approach, Coulton and colleagues (2015) reported a 64% probability of the intervention being cost-effective at a threshold of £30,000 [ 62 ]. Again, it is difficult to compare studies directly, but this is similar to that reported for interventions as diverse as a falls prevention initiative [ 74 ] and the treatment of depression using a collaborative approach [ 75 ] both in the UK. That the evidence base is meagre notwithstanding, there is, in other words, a prima facie case that ACIs are capable of offering value for money when targeted at older persons.

Our third finding relates to the state of the art with respect to SROIs in this area. Over the past 40 years, considerable time, effort and resources have been expended in the development of cost-effectiveness techniques in health and social care. While considerable heterogeneity can exist around their conduct, national guidance exists in many jurisdictions on the conduct of cost-effectiveness analyses (CEA) – such as the NICE reference case in the UK [ 76 ]– as well as in the reporting of these as set out in the CHEERS 2022 guidance [ 64 ]. This has helped raise the quality of published evaluations and the consistency with which they are reported. Despite the existence of a step-by-step guidance document on how to perform SROIs [ 77 ] which outlines how displacement effects, double counting, effect attribution and drop-off should be addressed, a significant body of work still remains to ensure that the methodology addresses a range of known biases in a robust manner. Where there is no comparator to the intervention being evaluated (as was the case in the SROIs reported here) it may be difficult to convince funders that the implicit incremental costs and benefits reported are indeed incremental and attributable to the intervention. Equally, where a comparator is present, greater consensus and standardisation is required regarding the identification, generation and application of, for example, financial proxies. Currently, SROI ratios combine value across a wide range of stakeholders, which is understandable if the objective is to capture all aspects of social benefit generated. This ratio, however, may not reflect the priorities and statutory responsibilities of healthcare funders. Whist all of the aforementioned issues can be addressed, investment is required to develop the SROI methodology further to more closely meet the needs of commissioning bodies.

Notwithstanding these challenges, social value analyses play a pivotal role within the procurement processes employed by government, local authorities and other non-departmental public bodies and should not be dismissed simply because the ‘burden of proof’ falls short of that required to secure remuneration within the health sector. As most SROIs are published in the grey literature, this means they often avoid peer scrutiny prior to publication and the potential quality assurance this can offer. It is noteworthy however that two of the SROIs included in this review [ 65 , 66 ] were published in the academic literature, suggesting that the academic community are engaging with this method which is to be applauded.

Moving forward, it is unlikely we will be able to meet all of the health and wellbeing needs of our ageing population solely in a primary or secondary care setting. New models of care are required, as are new models of funding to support interventions which can be delivered in non-healthcare settings. New hybrid models of evaluation will be required to provide robust economic evidence to assist in the allocation of scarce resources across health and non-healthcare settings; such evaluative frameworks must have robust theoretical underpinnings and be capable of delivering evidence from a non-clinical setting in a timely and cost-effective manner.

In the absence of a definitive evaluation framework for ACIs being currently available, we have a number of recommendations. First, and most importantly, all impact assessments should have a control group or credible counterfactual. This is currently not required when performing an SROI making it difficult to determine if all of the benefits ascribed to an intervention are in fact attributable. This recommendation is in line with the conclusion of a report by the London School of Economics [ 78 ] for the National Audit Office (NAO) which concluded that ‘any impact evaluation (and subsequent value for money calculation) requires construction of a counterfactual’. Second, a detailed technical appendix should accompany all impact assessments to allow independent review by a subject specialist. While this would assist peer review, it would allow providing greater transparency where peer review was not undertaken prior to publication. Furthermore, it would enable recalculation of SROI ratios to exclude ‘value’ attributable to stakeholders which are not relevant to a particular funder. Third, equity considerations should be addressed explicitly in all evaluations (this is currently not required in HTAs). Fourth, both costs and outcomes should be captured from a ‘broad’ perspective (adopting a ‘narrow’ healthcare perspective may underestimate the full economic impact), with non-healthcare sector costs being detailed as part of the analysis. Finally, data should be collected post-implementation to ensure that resources continue to be allocated efficiently.

As with any review, there are limitations which should be noted. A search of the grey literature was included as evaluations of applied public health interventions are not always reported in the academic literature. Systematically identifying grey literature and grey data can be problematic [ 79 , 80 , 81 , 82 , 83 ] as it is not collected, organised or stored in a consistent manner. Hence it is possible that we have not identified all relevant studies. Furthermore, as applied public health interventions can be performed in a non-healthcare setting we included SROIs in our review of economic evaluations. Current guidance on the systematic review of economic evaluations has been developed primarily for review of HTA as opposed to public health interventions and hence SROIs would be excluded, or if included would score poorly due to the inherent biases arising from no comparator or counterfactual being included.

This systematic review found that participation in group-based arts and creativity programmes was generally cost-effective and/or produced a positive return on investment whilst having a positive impact on older people’s physical, psychological, and social health and wellbeing outcomes. Unfortunately, the small number of studies identified, coupled with differences in methods used to assess economic impact hinders our ability to conclusively determine which types of art and creativity-based activities are more cost-effective or represent best value for money.

As well as the need for a greater focus on prevention of poor health as we age, new hybrid models of healthcare delivery are necessary to meet the needs of our ageing population. These models will integrate traditional medical care with other services such as home health aides (some of which may include artificial intelligence), telemedicine and social support networks. Alongside these, ACIs have the potential to provide a low cost, scalable, easily implementable and cost-effective solution to reduce the burden of illness in this age group and support healthy ageing.

Evidence on the cost-effectiveness of a range of ACIs is of utmost importance for policy and decision makers as it can both inform the development of policies that support the provision of ACIs in the context of ageing, but also identify the most cost-effective approaches for delivering such interventions. The development of hybrid models of evaluation, capable of capturing cost-effectiveness and social value, is becoming increasingly necessary as healthcare delivery for this age group moves beyond the realms of primary and secondary care and into the community. The development and refinement of such models will ensure a more comprehensive assessment of the impact of a diverse range of interventions providing a more nuanced understanding of the impact of an intervention. This will help inform decision making and ensure interventions are implemented in a cost-effective and socially beneficial manner.

Availability of data and materials

All data generated or analysed during this study are included in the published article and its supplementary information files.

United Nations, Department of Economic and Social Affairs, Population Division. World Population Prospects: The 2015 Revision, Key Findings and Advance Tables. Working Paper No. ESA/P/WP.241. 2015.

Office for National Statistics. Living longer: how our population is changing and why it matters. https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/ageing/articles/livinglongerhowourpopulationischangingandwhyitmatters/2018-08-13#how-do-incomes-of-older-people-compare-with-younger-ages . 2018. Accessed 07/12/2022

Dyakova M, Hamelmann C, Bellis MA, Besnier E, Grey CNB, Ashton K, Schwappach A, Clar C. Investment for health and well-being: a review of the social return on investment from public health policies to support implementing the Sustainable Development Goals by building on Health 2020 [Internet]. Copenhagen: WHO Regional Office for Europe; 2017.

Google Scholar  

Fancourt D, Finn S. What is the evidence on the role of the arts in improving health and well-being? A scoping review. Copenhagen: WHO Regional Office for Europe; 2019.

McQuade L, O’Sullivan R. Examining arts and creativity in later life and its impact on older people’s health and wellbeing: a systematic review of the evidence. Perspect Publ Health. 2023;0(0). https://doi.org/10.1177/17579139231157533

Skingley A, De’Ath S, Napleton L. Evaluation of Edna: arts and dance for older people. Work Older People. 2016;20(1):46–56.

Article   Google Scholar  

Brustio PR, Liubicich ME, Chiabrero M, et al. Dancing in the golden age: a study on physical function, quality of life, and social engagement. Geriatr Nurs. 2018;39(6):635–9.

Article   PubMed   Google Scholar  

Beauchet O, Bastien T, Mittelman M, et al. Participatory art-based activity, community-dwelling older adults and changes in health condition: results from a pre-post intervention, single-arm, prospective and longitudinal study. Maturitas. 2020;134:8–14.

Article   CAS   PubMed   Google Scholar  

Roswiyani R, Hiew CH, Witteman CLM, et al. Art activities and qigong exercise for the well-being of older adults in nursing homes in Indonesia: a randomized controlled trial. Aging Ment Health. 2020;24(10):1569–78.

Shanahan J, Bhriain ON, Morris ME, et al. Irish set dancing classes for people with Parkinson’s disease: the needs of participants and dance teachers. Complement Ther Med. 2016;27:12–7.

Garcia Gouvêa JA, Antunes MD, Bortolozzi F, et al. Impact of senior dance on emotional and motor parameters and quality of life of the elderly. Rev Rene. 2017;18(1):51–8.

Sun J, Zhang N, Buys N, et al. The role of Tai Chi, cultural dancing, playing a musical instrument and singing in the prevention of chronic disease in Chinese older adults: a mind–body meditative approach. Int J Ment Health Pr. 2013;15:227–39.

Fu MC, Belza B, Nguyen H, et al. Impact of group-singing on older adult health in senior living communities: a pilot study. Arch Gerontol Geriatr. 2018;76:138–46.

Feng L, Romero-Garcia R, Suckling J, et al. Effects of choral singing versus health education on cognitive decline and aging: a randomized controlled trial. Aging-us. 2020;12(24):24798–816.

Seinfeld S, Figueroa H, Ortiz-Gil J, et al. Effects of music learning and piano practice on cognitive function, mood and quality of life in older adults. Front Psychol. 2013;4:810.

Article   PubMed   PubMed Central   Google Scholar  

MacRitchie J, Breaden M, Milne AJ, et al. Cognitive, motor and social factors of music instrument training programs for older adults’ improved wellbeing. Front Psychol. 2020;10:2868.

Freeman WJI. A neurobiological role of music in social bonding. In: Wallin N, Merkur B, Brown S, editors. The origins of music. Cambridge: MIT Press; 2000. http://escholarship.org/uc/item/9025x8rt .

Huron D. Is music an evolutionary adaptation? Ann N Y Acad Sci. 2001;930(1):43–61. https://doi.org/10.1111/j.1749-6632.2001.tb05724.x .

Tarr B, Launay J, Dunbar RIM. Music and social bonding: “self–other” merging and neurohormonal mechanisms. Front Psychol. 2014;5:1096. https://doi.org/10.3389/fpsyg.2014.01096 .

Cain M, Lakhani A, Istvandity L. Short and long term outcomes for culturally and linguistically diverse (cald) and at-risk communities in participatory music programs: a systematic review. Arts Health. 2016;8(2):105–24. https://doi.org/10.1080/17533015.2015.1027934 .

Martin L, Oepen R, Bauer K, Nottensteiner A, Mergheim K, Gruber H, et al. Creative arts interventions for stress management and prevention – a systematic review. Behav Sci (Basel). 2018;8(2):pii:E28. https://doi.org/10.3390/bs8020028 .

Linnemann A, Wenzel M, Grammes J, Kubiak T, Nater UM. Music listening and stress in daily life: a matter of timing. Int J Behav Med. 2018;25(2):223–30. https://doi.org/10.1007/s12529-017-9697-5 .

Linnemann A, Strahler J, Nater UM. The stress-reducing effect of music listening varies depending on the social context. Psychoneuroendocrinology. 2016;72:97–105. https://doi.org/10.1016/j.psyneuen.2016.06.003 .

Panteleeva Y, Ceschi G, Glowinski D, Courvoisier DS, Grandjean DM. Music for anxiety? meta-analysis of anxiety reduction in non-clinical samples. Psychol Music. 2017;46(4):473–87. https://doi.org/10.1177/0305735617712424 .

Fancourt D, Tymoszuk U. Cultural engagement and incident depression in older adults: evidence from the English longitudinal study of ageing. Br J Psychiatry. 2018;214(4):225–9. https://doi.org/10.1192/bjp.2018.267 .

Balbag MA, Pedersen NL, Gatz M. Playing a musical instrument as a protective factor against dementia and cognitive impairment: a population-based twin study. Int J Alzheimer’s Dis. 2014;2014:836748. https://doi.org/10.1155/2014/836748 .

Porat S, Goukasian N, Hwang KS, Zanto T, Do T, Pierce J, et al. Dance experience and associations with cortical gray matter thickness in the aging population. Dement Geriatr Cogn Dis Extra. 2016;6(3):508–17. https://doi.org/10.1159/000449130 .

Federici A, Bellagamba S, Rocchi MBL. Does dance-based training improve balance in adult and young old subjects? a pilot randomized controlled trial. Aging Clin Exp Res. 2005;17(5):385–9 PMID: 16392413.

Alpert PT, Miller SK, Wallmann H, Havey R, Cross C, Chevalia T, et al. The effect of modified jazz dance on balance, cognition, and mood in older adults. J Am Acad Nurse Pract. 2009;21(2):108–15. https://doi.org/10.1111/j.1745-7599.2008.00392.x .

Jeon MY, Bark ES, Lee EG, Im JS, Jeong BS, Choe ES. The effects of a Korean traditional dance movement program in elderly women. Taehan Kanho Hakhoe Chi. 2005;35(7):126876 (in Korean). PMID: 16418553.

Eyigor S, Karapolat H, Durmaz B, Ibisoglu U, Cakir S. A randomized controlled trial of Turkish folklore dance on the physical performance, balance, depression and quality of life in older women. Arch Gerontol Geriatr. 2009;48(1):84–8. https://doi.org/10.1016/j.archger.2007.10.008 .

Noopud P, Suputtitada A, Khongprasert S, Kanungsukkasem V. Effects of Thai traditional dance on balance performance in daily life among older women. Aging Clin Exp Res. 2018;31(7):961–7. https://doi.org/10.1007/s40520-018-1040-8 .

Trombetti A, Hars M, Herrmann FR, Kressig RW, Ferrari S, Rizzoli R. Effect of musicbased multitask training on gait, balance, and fall risk in elderly people: a randomized controlled trial. Arch Intern Med. 2011;171(6):525–33. https://doi.org/10.1001/archinternmed.2010.446 .

Hyyppä MT, Mäki J, Impivaara O, Aromaa A. Individual-level measures of social capital as predictors of all-cause and cardiovascular mortality: a population-based prospective study of men and women in Finland. Eur J Epidemiol. 2007;22(9):589–97. https://doi.org/10.1007/s10654-007-9153-y .

Hyyppä MT, Mäki J, Impivaara O, Aromaa A. Leisure participation predicts survival: a population-based study in Finland. Health Promot Int. 2006;21(1):5–12. https://doi.org/10.1093/heapro/dai027 .

Lennartsson C, Silverstein M. Does engagement with life enhance survival of elderly people in Sweden? the role of social and leisure activities. J Gerontol B Psychol Sci Soc Sci. 2001;56(6):S335–42. https://doi.org/10.1093/geronb/56.6.s335 .

Sundquist K, Lindström M, Malmström M, Johansson SE, Sundquist J. Social participation and coronary heart disease: a follow-up study of 6900 women and men in Sweden. Soc Sci Med. 1982;58(3):615–22. https://doi.org/10.1016/s0277-9536(03)00229-6 .

Väänänen A, Murray M, Koskinen A, Vahtera J, Kouvonen A, Kivimäki M. Engagement in cultural activities and cause-specific mortality: prospective cohort study. Prev Med. 2009;49(2–3):142–7. https://doi.org/10.1016/j.ypmed.2009.06.026 .

Särkämö T, Soto D. Music listening after stroke: beneficial effects and potential neural mechanisms. Ann N Y Acad Sci. 2012;1252(1):266–81. https://doi.org/10.1111/j.1749-6632.2011.06405.x .

Särkämö T, Pihko E, Laitinen S, Forsblom A, Soinila S, Mikkonen M, et al. Music and speech listening enhance the recovery of early sensory processing after stroke. J Cogn Neurosci. 2010;22(12):2716–27. https://doi.org/10.1162/jocn.2009.21376 .

Särkämö T, Ripollés P, Vepsäläinen H, Autti T, Silvenno HM, Salli E, et al. Structural changes induced by daily music listening in the recovering brain after middle cerebral artery stroke: a voxel-based morphometry study. Front Hum Neurosci. 2014;8:245. https://doi.org/10.3389/fnhum.2014.00245 .

Särkämö T, Tervaniemi M, Laitinen S, Forsblom A, Soinila S, Mikkonen M, et al. Music listening enhances cognitive recovery and mood after middle cerebral artery stroke. Brain. 2008;131(3):866–76. https://doi.org/10.1093/brain/awn013 .

Fancourt D, Steptoe A, Cadar D. Cultural engagement and cognitive reserve: museum attendance and dementia incidence over a 10-year period. Br J Psychiatry. 2018;213(5):661–3. https://doi.org/10.1192/bjp.2018.129 .

Fancourt D, Steptoe A, Cadar D. Cultural engagement predicts changes in cognitive function in older adults over a 10 year period: findings from the English longitudinal study of ageing. Sci Rep. 2018;8(1):10226. https://doi.org/10.1192/bjp.2018.129 .

All Party Parliamentary group on arts, health and wellbeing. Creative health: the arts for health and wellbeing. 2017.

van Mastrigt GA, Hiligsmann M, Arts JJ, Broos PH, Kleijnen J, Evers SM, Majoie MH. How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: a five-step approach (part 1/3). Expert Rev Pharmacoecon Outcomes Res. 2016;16(6):689–704. https://doi.org/10.1080/14737167.2016.1246960 . Epub 2016 Nov 2 PMID: 27805469.

Thielen FW, Van Mastrigt G, Burgers LT, Bramer WM, Majoie H, Evers S, Kleijnen J. How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3). Expert Rev Pharmacoecon Outcomes Res. 2016;16(6):705–21. https://doi.org/10.1080/14737167.2016.1246962 . Epub 2016 Nov 2 PMID: 27805466.

Wijnen B, Van Mastrigt G, Redekop WK, Majoie H, De Kinderen R, Evers S. How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3). Expert Rev Pharmacoecon Outcomes Res. 2016;16(6):723–32. https://doi.org/10.1080/14737167.2016.1246961 . Epub 2016 Oct 21 PMID: 27762640.

Mandrik OL, Severens JLH, Bardach A, Ghabri S, Hamel C, Mathes T, Vale L, Wisløff T, Goldhaber-Fiebert JD. Critical appraisal of systematic reviews with costs and cost-effectiveness outcomes: an ISPOR good practices task force report. Value Health. 2021;24(4):463–72. https://doi.org/10.1016/j.jval.2021.01.002 . PMID: 33840423.

Kelly MP, McDaid D, Ludbrook A, Powell J: Economic appraisal of public health interventions. http://www.cawt.com/Site/11/Documents/Publications/Population%20Health/Economics%20of%20Health%20Improvement/Economic_appraisal_of_public_health_interventions.pdf

Weatherly H, Drummond M, Claxton K, Cookson R, Ferguson B, Godfrey C, Rice N, Sculpher M, Sowden A. Methods for assessing the cost-effectiveness of public health interventions: key challenges and recommendations. Health Policy. 2009;93(2–3):85–92. https://doi.org/10.1016/j.healthpol.2009.07.012 . Epub 2009 Aug 25 PMID: 19709773.

Payne K, McAllister M, Davies LM. Valuing the economic benefits of complex interventions: when maximising health is not sufficient. Health Econ. 2012. https://doi.org/10.1002/hec.2795 .

Edwards RT, Charles JM, Lloyd-Williams H. Public health economics: a systematic review of guidance for the economic evaluation of public health interventions and discussion of key methodological issues. BMC Public Health. 2013;24(13):1001. https://doi.org/10.1186/1471-2458-13-1001.PMID:24153037;PMCID:PMC4015185 .

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol. 2015;68(6):617–26. https://doi.org/10.1016/j.jclinepi.2014.11.025 . Epub 2015 Feb 7 PMID: 25766056.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6. https://doi.org/10.1016/j.jclinepi.2016.01.021 . Epub 2016 Mar 19 PMID: 27005575.

Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA Statement. PLoS Med. 2008;6:e1000097. https://doi.org/10.1371/journal.pmed.1000097 .

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;21(339):b2700. https://doi.org/10.1136/bmj.b2700.PMID:19622552;PMCID:PMC2714672 .

Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA Group. PLoS Med. 2009;6(7):e1000097 Evers S, Goossens M, De Vet H, et al. Criteria list for assessment of methodological quality of economic evaluations: consensus on health economic criteria. Int J Technol Assess Health Care. 2005;21(02):240–245.

Evers S, Goossens M, de Vet H, van Tulder M, Ament A. Criteria list for assessment of methodological quality of economic evaluations: Consensus on Health Economic Criteria. Int J Technol Assess Health Care. 2005;21(2):240–5 PMID: 15921065.

Hutchinson CL, Berndt A, Gilbert-Hunt S, George S, Ratcliffe J. Valuing the impact of health and social care programmes using social return on investment analysis: how have academics advanced the methodology? A protocol for a systematic review of peer-reviewed literature. BMJ Open. 2018;8(12):e022534. https://doi.org/10.1136/bmjopen-2018-022534 . PMID:30530579;PMCID:PMC6303612.

Higgins J, Green S. Cochrane handbook for systematic reviews of interventions version 5.1. 0. Chichester: The Cochrane Collaboration; 2013.

Coulton S, Clift S, Skingley A, Rodriguez J. Effectiveness and cost-effectiveness of community singing on mental health-related quality of life of older people: randomised controlled trial. Br J Psychiatry. 2015;207(3):250–5. https://doi.org/10.1192/bjp.bp.113.129908 . Epub 2015 Jun 18 PMID: 26089304.

Johnson JK, Stewart AL, Acree M, Nápoles AM, Flatt JD, Max WB, Gregorich SE. A community choir intervention to promote well-being among diverse older adults: results from the community of voices trial. J Gerontol B Psychol Sci Soc Sci. 2020;75(3):549–59. https://doi.org/10.1093/geronb/gby132 . PMID:30412233;PMCID:PMC7328053.

Husereau D, Drummond M, Augustovski F, de Bekker-Grob E, Briggs AH, Carswell C, Caulley L, Chaiyakunapruk N, Greenberg D, Loder E, Mauskopf J, Mullins CD, Petrou S, Pwu RF, Staniszewska S, CHEERS 2022 ISPOR Good Research Practices Task Force. Consolidated Health Economic Evaluation Reporting Standards 2022 (CHEERS 2022) statement: updated reporting guidance for health economic evaluations. Value Health. 2022;25(1):3–9. https://doi.org/10.1016/j.jval.2021.11.1351 . PMID: 35031096.

Bosco A, Schneider J, Broome E. The social value of the arts for care home residents in England: a social return on investment (SROI) analysis of the imagine arts programme. Maturitas. 2019;124:15–24. https://doi.org/10.1016/j.maturitas.2019.02.005 . Epub 2019 Mar 13 PMID: 31097173.

Jones C, Windle G, Edwards RT. Dementia and imagination: a social return on investment analysis framework for art activities for people living with dementia. Gerontologist. 2020;60(1):112–23. https://doi.org/10.1093/geront/gny147 . PMID: 30476114.

Social Value Lab and Impact Arts Craft Café: creative solutions to isolation and loneliness; Social return on investment. 2011. http://www.socialvaluelab.org.uk/wp-content/uploads/2013/05/CraftCafeSROI.pdf

MB associates. Make my day: the impact of Creative Caring in older people’s care homes. 2013. https://www.suffolkartlink.org.uk/wp-content/uploads/2014/10/CreativeCarersSROIReport_Nov2013.pdf

HACT. n.d. UK Social Value Bank. Retrieved December 11, 2023. from https://hact.org.uk/tools-and-services/uk-social-value-bank/ .

The Older Adults’ NHS and social care return on investment tool. Project report. Public health England. December 2019. Last accessed 27/03/2023.

British Red Cross – Valuing First Aid Education. 2018. https://socialvalueuk.org/wp-content/uploads/2018/12/Valuing-First-Aid-Education-Social-Return-on-Investment-Report-on-the-value-of-First-Aid-Education-Assured-Report.pdf . Accessed 17/02/2023

Hunter R, Dallat M, Tully M, O’Neill C, Heron L, Kee F. Social return on investment analysis of an urban greenway. Cities and Health. 2020. https://doi.org/10.1080/23748834.2020.1766783 .

NEF Consulting. Refuge: A social return on investment evaluation. 2016. https://socialvalueuk.org/wp-content/uploads/2017/04/Refuge-SROI-2016.pdf Accessed 17/02/2022

Corbacho B, Cockayne S, Fairhurst C, Hewitt CE, Hicks K, Kenan AM, Lamb SE, MacIntosh C, Menz HB, Redmond AC, Rodgers S, Scantlebury A, Watson J, Torgerson DJ, on behalf of the REFORM study. Cost-Effectiveness of a Multifaceted Podiatry Intervention for the Prevention of Falls in Older People: The REducing Falls with Orthoses and a Multifaceted Podiatry Intervention Trial Findings. Gerontology. 2018;64(5):503–12. https://doi.org/10.1159/000489171 . Epub 2018 Jun 26 PMID: 29945150.

Green C, Richards DA, Hill JJ, Gask L, Lovell K, Chew-Graham C, Bower P, Cape J, Pilling S, Araya R, Kessler D, Bland JM, Gilbody S, Lewis G, Manning C, Hughes-Morley A, Barkham M. Cost-effectiveness of collaborative care for depression in UK primary care: economic evaluation of a randomised controlled trial (CADET). PLoS ONE. 2014;9(8):e104225. https://doi.org/10.1371/journal.pone.0104225.PMID:25121991;PMCID:PMC4133193 .

National Institute for Health and Care Excellence (NICE). NICE health technology evaluations: the manual. 2022. Retrieved 27 March, 2023 from  https://www.nice.org.uk/process/pmg36/chapter/introduction-to-health-technology-evaluation

NEF Consulting. SSE – Beatrice SROI framework – guidance document. https://www.sse.com/media/svnn5jpk/sroi-methodology-guidance-nef-consulting.pdf . Accessed 17/02/2022

Gibbons S, McNally S, Overman H. Review of Government Evaluations: A report for the NAO. London: National Audit Office; 2013.

Turner AM, Liddy ED, Bradley J, Wheatley JA. Modeling public health interventions for improved access to the gray literature. J Med Libr Assoc. 2005;93(4):487–94 PMID: 16239945; PMCID: PMC1250325.

PubMed   PubMed Central   Google Scholar  

Benzies KM, Premji S, Hayden KA, Serrett K. State-of-the-evidence reviews: advantages and challenges of including grey literature. Worldviews Evid Based Nurs. 2006;3(2):55–61. https://doi.org/10.1111/j.1741-6787.2006.00051.x . PMID: 17040510.

Franks H, Hardiker NR, McGrath M, McQuarrie C. Public health interventions and behaviour change: reviewing the grey literature. Public Health. 2012;126(1):12–7. https://doi.org/10.1016/j.puhe.2011.09.023 . Epub 2011 Nov 29 PMID: 22130477.

Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods. 2014;5(3):221–34. https://doi.org/10.1002/jrsm.1106 . Epub 2013 Dec 6 PMID: 26052848.

Godin K, Stapleton J, Kirkpatrick SI, Hanning RM, Leatherdale ST. Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Syst Rev. 2015;22(4):138. https://doi.org/10.1186/s13643-015-0125-0 . PMID:26494010;PMCID:PMC4619264.

Download references

Acknowledgements

We would like to thank Ms. Louise Bradley (Information Resource Officer, Institute of Public Health) for her assistance in refining search strategies and literature search.

This study was supported by the Institute of Public Health (IPH), 200 South Circular Road, Dublin 8, Ireland, D08 NH90. This study was a collaboration between two health economists (GC, CO’N) and two members of staff from the funding organisation (LM, RO’S). Input from IPH staff was fundamental in defining the scope of work and research question, refining search terms and review and editing of the manuscript. Staff from IPH were not involved in quality assurance or review of papers included in the manuscript.

Author information

Authors and affiliations.

Clinical Costing Solutions, Belfast, BT15 4EB, UK

Grainne Crealey

Institute of Public Health, 200 South Circular Road, Dublin 8, D08 NH90, Ireland

Laura McQuade & Roger O’Sullivan

Centre for Public Health, Institute of Clinical Sciences, Royal Victoria Hospital, Belfast, BT12 6BA, UK

Ciaran O’Neill

You can also search for this author in PubMed   Google Scholar

Contributions

LMcQ and ROS were involved in defining the scope of work, refining the research question, provision of subject specific (public health) context, review of search strategy, review & editing of manuscript. CON and GC were involved in refining the research question and search strategy, provision of health economics and systematic reviewing expertise, review of returned reports, original draft preparation, review, editing and submission of manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ciaran O’Neill .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

: Table S1. Search strategy for electronic databases and grey literature.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Crealey, G., McQuade, L., O’Sullivan, R. et al. Arts and creativity interventions for improving health and wellbeing in older adults: a systematic literature review of economic evaluation studies. BMC Public Health 23 , 2496 (2023). https://doi.org/10.1186/s12889-023-17369-x

Download citation

Received : 23 April 2023

Accepted : 28 November 2023

Published : 13 December 2023

DOI : https://doi.org/10.1186/s12889-023-17369-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Economic evaluation
• Older adults

BMC Public Health

ISSN: 1471-2458

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Open access
• Published: 16 April 2024

Approaches to locum physician recruitment and retention: a systematic review

• Nathan Ferreira 1 ,
• Odessa McKenna 1 ,
• Iain R. Lamb 2 ,
• Alanna Campbell 3 ,
• Lily DeMiglio 2 &
• Eliseo Orrantia   ORCID: orcid.org/0000-0003-3023-8100 2  

Human Resources for Health volume  22 , Article number:  24 ( 2024 ) Cite this article

172 Accesses

2 Altmetric

Metrics details

A robust workforce of locum tenens (LT) physicians is imperative for health service stability. A systematic review was conducted to synthesize current evidence on the strategies used to facilitate the recruitment and retention of LT physicians. English articles up to October 2023 across five databases were sourced. Original studies focusing on recruitment and retention of LT’s were included. An inductive content analysis was performed to identify strategies used to facilitate LT recruitment and retention. A separate grey literature review was conducted from June–July 2023. 12 studies were retained. Over half (58%) of studies were conducted in North America. Main strategies for facilitating LT recruitment and retention included financial incentives (83%), education and career factors (67%), personal facilitators (67%), clinical support and mentorship (33%), and familial considerations (25%). Identified subthemes were desire for flexible contracts (58%), increased income (33%), practice scouting (33%), and transitional employment needs (33%). Most (67%) studies reported deterrents to locum work, with professional isolation (42%) as the primary deterrent-related subtheme. Grey literature suggested national physician licensure could enhance license portability, thereby increasing the mobility of physicians across regions. Organizations employ five main LT recruitment facilitators and operationalize these in a variety of ways. Though these may be incumbent on local resources, the effectiveness of these approaches has not been evaluated. Consequently, future research should assess LT the efficacy of recruitment and retention facilitators. Notably, the majority of identified LT deterrents may be mitigated by modifying contextual factors such as improved onboarding practices.

Peer Review reports

Introduction

The shortage of a sustainable and robust physician workforce is a significant healthcare issue across most of the world [ 1 ]. In regions that face persistent challenges in physician availability, the continuity of the healthcare system heavily relies on locum tenens (LT) physicians, commonly referred to as “locums”. These healthcare providers work in a temporary capacity to fill vacancies or provide coverage for permanent physicians [ 2 , 3 ]. Their importance was highlighted during the COVID-19 pandemic as the lack of locums resulted in the suspension of hospital services and emergency department closures due to insufficient staffing [ 4 ].

Physician recruitment strategies primarily focus on filling permanent positions with minimal emphasis on attracting locum providers [ 5 ]. However, strategies aimed at facilitating the recruitment of permanent physicians may not effectively attract locums given fundamental distinctions in their employment preferences and priorities. LTs, for instance, are motivated by factors such as seeking greater autonomy, working part-time, transitioning into partial retirement, and supplementing income [ 6 , 7 ]. Their attraction to working as a locum may be due to advantages including reduced administrative burdens, lower workplace stress, and flexibility for maintaining a desired work–life balance [ 6 , 7 ]. Additional advantages include competitive salaries comparable to permanent positions without a long-term commitment, travel and accommodation stipends, subsidized malpractice insurance, and lower overhead expenses [ 6 , 7 ].

Governments and communities invest substantial financial resources to attract locums in order to sustain healthcare service delivery [ 6 , 8 ]. As such, existing research has investigated locum recruitment and retention factors [ 6 , 7 , 9 , 10 ]. Despite the important role that locums play in sustaining operational healthcare systems, particularly during periods of health human resources strain, there is a lack of consolidated of evidence on the recruitment of LT physicians. Consequently, there is a need for the synthesis of current research on facilitators used in the recruitment and retention of LTs. This will serve to better inform the development of comprehensive, evidence-based recruitment guidelines tailored specifically to LT physicians. Therefore, this study systematically reviewed existing literature to identify and synthesize the approaches used to recruit locum physicians. Ensuing results will provide valuable guidance to policymakers and healthcare organizations, aiding in the development of evidence-based recruitment policies and practices to address the unique needs of locum physicians.

This systematic review was performed in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 11 ]. This research protocol was registered in the PROSPERO database (CRD42022339666).

Search strategy

Between April 26th and April 27th, 2022 we performed a systematic search of the electronic databases Ovid MEDLINE, Cochrane Database of Systematic Reviews, PsycINFO, CINAHL and Web of Science-Core Collection. Examples of the medical subject headings (MeSH) applied include “Contract Services”, “Career Choice”, “Personnel Staffing and Scheduling”, “Personnel Loyalty” and “Physician Incentive Plans”. This initial search has since been followed by an updated search in October of 2023 prior to submission for publication. Keywords were used to collect non-indexed material and those terms not captured by MeSH, such as “locum”. No limits were applied to the searches. Articles not available in English were excluded. Secondary research (e.g., meta‐analyses, dissertations, systematic reviews, case reports, commentaries, grey literature) were excluded from the scholarly search. Reference lists of included studies were searched for additional articles. Details of the scholarly search strategy appear in Additional file 1 : Appendix S1. This search strategy was developed in collaboration with a librarian and peer-reviewed by a second librarian.

Between June 12th and July 16th, 2023 we performed an iterative systematic hand-search of grey literature. This included public search engines (e.g., Google), grey literature repositories (e.g., OpenGrey), health care quality organizations, and data facilities across five countries, Canada, United States of America, United Kingdom, Australia, and India. Examples of the search terms and headings applied include “Locum”, “Contract”, “Temporary”, and “Locum Physician''. Search parameters were restricted to include only articles published in the year 1990 or later. For database searches information beyond the first 150 or 250 search results were not incorporated in the analysis. For full search histories please see Additional file 1 : Appendix S3.

Selection and screening process

A modified version of the PICO (population, intervention, comparison, outcomes) framework was used (Table  1 ) [ 12 ]. We included original qualitative, quantitative or mixed-methods studies focused on recruitment and retention initiatives specific to locums in any country across clinical settings. There was variability in how studies defined locum physicians (Table  2 ). Articles focusing on recruitment and retention of non-locum physicians and healthcare workers without an MD designation (with the exception of medical students training in a MD programme) were excluded, including articles that combined both populations in which individual data for locums could not be extracted. Articles that exclusively incorporated the recruitment and retention of locums in the interpretive context such that locum recruitment and retention initiatives were not prospectively mentioned in the study framework or methodology were excluded.

Retrieved articles were managed using Covidence online systematic review software (Veritas Health Innovation, Melbourne, Australia). Two reviewers independently performed title and abstract screening for relevance. Full texts were then reviewed against eligibility criteria (Table  1 ). In both stages of screening, discussion was used to resolve disagreements. Remaining discrepancies were resolved by a third reviewer.

Data extraction and synthesis

Data extraction took place within Covidence using two independent reviewers. A template was developed and piloted for two studies to ensure reviewer agreement prior to utilization. Outstanding conflicts were resolved by a third author. Extraction parameters included study design, participant characteristics, context of locum assignment, and strategies used to recruit and retain locums. Two authors (NF and OM) performed an inductive content analysis to characterize recurring patterns of the locum recruitment and retention strategies discussed in each paper included in the systematic analysis. Following the identification of these strategies, they were grouped into broader, overarching themes relevant to LT recruitment and retention. Methodologic quality of each study was assessed using the Mixed Methods Appraisal Tool (MMAT) [ 13 ]; two authors (NF and OM) conducted the appraisal independently and any discrepancies in appraisal were resolved by discussion with a third author (EO). Authors of included studies were contacted if data were missing.

Our initial search identified 5390 citations. After the removal of duplicates ( n  = 812), 4578 studies’ titles and abstracts were screened. Following this stage, 242 articles were screened using full-text, and 230 were excluded from the review. Twelve studies [ 2 , 3 , 7 , 8 , 9 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ] fulfilled inclusion criteria and were retained for data extraction. The PRISMA flow diagram detailing the screening procedure is displayed in Fig.  1 . Articles reporting data from the same participant population at separate time points are reported together.

PRISMA flow diagram detailing the selection process

Study characteristics are summarized in Table  3 . Most ( n  = 11, 92%) studies [ 2 , 3 , 7 , 8 , 9 , 13 , 14 , 15 , 17 , 18 , 19 ] were published within the last 20 years of our search. Four (33%) studies [ 7 , 9 , 13 , 19 ] were from the United States and four (33%) [ 8 , 14 , 15 , 16 ] were from the United Kingdom. A smaller portion ( n  = 3, 25%) [ 2 , 3 , 17 ] originated in Canada. One (8%) study [ 18 ] was based in India. Quantitative studies [ 3 , 7 , 9 , 13 , 15 , 16 , 17 ] ( n  = 7, 58%) were cross-sectional ( n  = 6, 50%) [ 3 , 7 , 9 , 13 , 15 , 16 ] or pre–post study ( n  = 1, 8%) [ 17 ] in design. Four (33%) studies [ 8 , 14 , 18 , 19 ] used a qualitative design, including semi-structured interviews ( n  = 2, 17%) [ 18 , 19 ], focus groups ( n  = 1, 8%) [ 14 ], and content analysis ( n  = 1, 8%) [ 8 ]. One (8%) study [ 2 ] adopted a mixed-methods design.

The majority of studies [ 2 , 7 , 8 , 9 , 13 , 15 , 16 , 19 ] ( n  = 8, 67%) specified clinical setting, but did not indicate whether it was rural or urban (Table  3 ). A variety of specialities were reported among locum populations. The majority ( n  = 9, 75%) [ 2 , 3 , 7 , 13 , 14 , 15 , 16 , 18 , 19 ] included primary care physicians, and over half ( n  = 5, 42%) [ 7 , 9 , 13 , 18 , 19 ] included specialists. Three (25%) studies [ 7 , 13 , 19 ] reported a subspecialist population. Two (17%) studies [ 2 , 17 ] included physicians in training, with one (8%) [ 2 ] involving resident physicians and another (8%) [ 17 ] medical students. One (8%) study [ 8 ] did not indicate the specialty of the physician population. A total of six (50%) studies [ 7 , 13 , 14 , 15 , 16 , 19 ] reported years of physician practice experience.

A diversity of locum recruitment approaches were reported across studies, with some ( n  = 2, 17%) [ 7 , 8 ] using more than one method. Four (33%) studies [ 7 , 8 , 9 , 13 ] used a third-party recruitment agency, two (17%) [ 8 , 14 ] used a locum bank, word-of-mouth, or personal networks [ 16 ], informal means [ 7 ], and an unspecified novel recruitment software were each reported once [ 8 ] (8%). Four studies [ 2 , 15 , 18 , 19 ] (33%) did not report a specific method.

Quality assessment

The 2018 version of the MMAT was used to appraise the quality of retained articles [ 20 ]. Overall, nine (75%) of the articles [ 7 , 8 , 13 , 14 , 16 , 17 , 18 , 19 ] met 75–100% of the evaluated criteria, representing high quality. Three (25%) studies [ 2 , 9 , 15 ] met 50–75% of the evaluated criteria, representing moderate quality. Further details regarding the assessment of quality of retained articles appear in Additional file 1 : Appendix S2. Grey literature was assessed using the AACODS Checklist [ 21 ]. Additional information on the appraisal of grey literature can be found in Additional file 1 : Appendix S3 and in the supplemental content titled “Grey Literature Search Strategy, Data Extraction, and Evaluation”.

Facilitators of locum recruitment and retention

Six locum recruitment and retention themes were identified across retained studies (Table  4 ). Five overarching themes emerged for factors that facilitated LT recruitment and retention: financial incentives, familial considerations, educational or career-based factors, personal facilitators, and mentorship/clinical support. One theme focused on deterrents of locum work.

Ten (83%) studies [ 2 , 3 , 7 , 8 , 9 , 13 , 14 , 15 , 16 , 18 ] reported financial incentives with nine individual subthemes identified (Table  5 ). A significant portion ( n  = 4, 33%) of studies’ [ 2 , 14 , 15 , 16 ] did not provide specific details about the nature of the financial incentives provided. Four (33%) of the studies’ [ 2 , 7 , 8 , 13 ] financial incentives referred to an increase in income. Reimbursement for locum travel and lodging was reported three times (25%) [ 3 , 7 , 9 ]. Reimbursement for medical licensure ( n  = 2, 17%) and provision of malpractice insurance ( n  = 2, 17%) were also reported [ 7 , 9 ]. Augmented pay for challenging work conditions [ 18 ], payment assistance for continuing medical education (CME) [ 18 ], supplementation of retirement income [ 13 ], and guaranteed income [ 3 ] were all reported once (8%) each.

Three (25%) studies [ 2 , 7 , 14 ] reported familial considerations as facilitators to recruitment and retention which included accommodating family (17%) [ 2 , 14 ], school accessibility (8%) [ 14 ], and unspecified (8%) [ 7 ]. Furthermore, eight (67%) studies [ 2 , 3 , 7 , 8 , 13 , 14 , 16 , 17 ] reported education or career-based incentives. A total of 13 subthemes related to educational and career-based factors facilitating recruitment and retention were reported (Table  5 ), which included pre-permanent practice scouting [ 2 , 7 , 13 , 17 ] and temporary or transition in employment [ 2 , 7 , 13 , 16 ] both reported four (33%) times. Freedom from administrative responsibilities and transitioning into retirement were reported three (25%) times [ 7 , 13 , 16 ]. Avoiding commitment [ 2 , 16 ], increasing skills and competencies [ 2 , 8 ], and a desire to take on part-time employment [ 7 , 13 ] was reported twice (17%). The remaining career-based facilitators to recruitment and retention were each reported once (8%), including acquiring cross-provincial locum medical licensure [ 3 ], facilitation of hospital credentialing and medical licensure [ 7 ], gaining exposure to running a medical practice [ 2 ], accessing novel CME opportunities [ 3 ], assistance with maintaining medical knowledge [ 14 ], and accessing peer-facilitated educational support [ 14 ].

A total of eight (67%) studies [ 2 , 7 , 8 , 13 , 14 , 15 , 16 , 17 ] reported using personal factors as facilitators of LT recruitment and retention. Within this category, five subthemes were identified (see Table  5 ). Seven (58%) reported using flexible contracts (e.g., suitable availability, work schedule flexibility, and work–life balance) [ 2 , 7 , 8 , 13 , 14 , 15 , 16 ]. Having the ability to travel and experience new communities (locum tourism) was reported five (42%) times [ 2 , 7 , 13 , 16 , 17 ]. Three (25%) studies [ 7 , 14 , 15 ] reported unspecified personal incentives including stress relief [ 14 ], structured support [ 14 ], facility amenities [ 7 ], working conditions [ 7 ], personal safety [ 15 ], and overall facility quality [ 7 ]. Compatibility with post and convenience of the assignment were reported once (8%) each [ 8 ].

Four (33%) studies [ 14 , 15 , 16 , 19 ] reported recruitment and retention facilitators involving mentorship and clinical support with four subthemes. Having a network of supportive colleagues [ 14 , 15 ] and a chance to become familiar with the practice before arrival [ 16 , 19 ] were reported twice (17%) each. Intentional relationship building, whereby the seasoned colleague met with the incoming locum to ensure comfort in the practice was reported once (8%) [ 19 ]. Availability of a back-up physician for support was reported once (8%) [ 19 ].

A total of eight (67%) studies [ 2 , 8 , 9 , 14 , 16 , 17 , 18 , 19 ] addressed deterrents of or barriers to locum work, encompassing a total of 19 reported subthemes. Professional isolation ( n  = 5, 42%) [ 2 , 8 , 14 , 18 , 19 ] was reported most frequently followed by work unpredictability (33%) [ 2 , 8 , 16 , 19 ]. Insufficient patient continuity of care was reported three (25%) times [ 2 , 8 , 16 ]. The following deterrents/barriers were each reported twice (17%): inadequate employee onboarding and orientation [ 8 , 19 ], demanding locum work [ 2 , 17 ], poor job security [ 16 , 18 ], lack of information to make an informed decision about accepting the job post [ 8 , 9 ], and a lack of career advancement [ 16 , 18 ]. The following deterrents/barriers were reported just once (8%): excessive travelling [ 16 ], low patient volume [ 2 ], administrative burden [ 18 ], difficulty accessing time-off [ 18 ], inadequate housing [ 18 ], challenging working conditions [ 18 ], exclusion from pension plans [ 16 ], lack of equitable pay [ 18 ], low salary [ 18 ], feeling distanced from CME and limitations in staying up-to-date [ 14 ], and perceptions of inferior professional status by colleagues [ 16 ].

Facilitators of locum recruitment and retention within grey literature

Grey literature findings closely mirrored the facilitators and deterrents found in peer-reviewed literature. A notable exception captured in the ‘education and career’ theme involved the potential benefits of implementing a national physician licensure, which was absent in the primary literature but present in nearly a third ( n  = 27, 26%) of the grey literature.

Interpretation

We identified 12 English language studies that explored the recruitment and retention of locums in Canada, USA, UK, and India over a 30-year period. Finance, education, and personal factors were the most used LT recruitment strategies while family considerations and clinical/mentorship support were less frequently cited. However, almost all studies [ 2 , 3 , 7 , 8 , 13 , 14 , 15 , 16 ] ( n  = 8, 67%) reviewed reported using a combination of these recruitment approaches. While there is a paucity of evidence on whether employing multiple approaches leads to improved LT recruitment, utilizing a range of methods may still be a reasonable strategy. This approach prevents organizations from becoming overly reliant on a single approach and enables them to adapt their strategy more easily as required to maintain LT recruitment, retention and service. Further, as physicians choose locum positions based on different priorities, utilizing multiple strategies provides a range of incentives with wider appeal.

Across the five LT recruitment strategies, the diverse range of unique approaches used indicates there is no one-size-fits-all method. This suggests that organizations develop their own specific approach tailored to their available resources, location, and the anticipated needs of the LT physicians they aim to recruit. For instance, certain recruitment strategies incentives such as back-up availability, network of supportive colleagues and access to CME may not be feasible for some organizations given their size, location, and resource constraints. This may lead to the development of alternative recruitment approaches and/or increased emphasis on other strategies. Notably, we found that common recruitment and retention strategies used elsewhere, such as providing competitive salaries, were extensively used in the recruitment and retention of locums. However, approaches that seem to be specifically designed to address the unique requirements and preferences of locum physicians were also employed, such as offering reimbursements for travel and accommodation, providing support for family-related needs, offering flexible scheduling, and facilitating access to leisure activities. Although the effectiveness of these strategies is poorly defined, their implementation suggests that organizations recognize that conventional recruitment and retention approaches, effective in the broader health workforce, may not adequately address the unique aspects and challenges associated with the transient and temporary nature of locum work. For example, incentives like competitive compensation, while valued, might not be as appealing to those seeking the flexibility of short-term work assignments or lifestyle benefits. Therefore, acknowledging the appeal of locum work, creating incentives that emphasize these benefits, and addressing the related challenges are likely to enhance recruitment and retention efforts.

The finding that showed sites employed a wide range of recruitment and retention approaches highlights the complexity of this process. However, implementing such a wide range of strategies makes it challenging to identify the most effectives. Consequently, future work should identify optimal recruitment strategies within diverse health contexts and organizational structures. This would enable organizations to streamline their approach, maximizing recruitment success while efficiently utilizing their resources. This may be particularly valuable in resource poor healthcare environments where strategic asset allocation is essential.

Numerous factors were cited as deterrents of locum work, indicating that physicians’ decision to work as a LT is influenced by a variety of considerations. Although some of the cited deterrents were addressed by recruitment strategies, it is unclear whether these approaches were effective. In the studies reviewed, professional isolation and work assignment predictability were the two most cited deterrents to locum work appearing in 42% and 33% of studies, respectively. As temporary workers, there are inherent challenges in developing rapport with colleagues. Moreover, providing coverage introduces uncertainties regarding work schedules and conditions (e.g., hours worked, frequency and duration of assignments). Together these factors can contribute to lower job satisfaction, which may result in a decreased willingness to work as a locum. As a result, recruitment strategies should consider measures to address these deterrents. The wide range of deterrents emphasizes the importance for healthcare organizations to adopt comprehensive recruitment strategies that recognize and respond to the various unique needs of LT physicians. Further, many of these deterrents may be addressed by improving locum onboarding and job conditions, such as enhancing infrastructure quality and minimizing social isolation.

It is important to recognize information on locum recruitment and retention extends beyond peer-reviewed articles to include the grey literature. These non-academic resources contain potential insights into practical approaches for recruitment and retention, thus underscoring the need to evaluate the grey literature in this field. Interestingly, our review of the grey literature generally aligns with the facilitators and deterrents of locum work identified in this systematic review apart from support for a national physician license. Such a measure would enhance the portability of licensure, allowing improved mobility of physicians across regions, reducing administrative burdens and the time required for obtaining proper licensing, hospital privileges, and contractual agreements. This, in turn, may reduce barriers to locum recruitment and more effectively facilitate the transition of locums to their temporary place of practice. This finding, which was not identified in the systematic review, again reiterates the importance of assessing the grey literature to gain a comprehensive understanding on the current strategies being used for recruiting and retaining locum physicians.

Importantly, the success of LT physician recruitment relies on a collaborative effort that extends beyond responsibility of individual healthcare organizations. This is particularly important considering that facilitators of LT recruitment and retainment, such as remuneration, fall beyond the scope of health teams. Therefore, the various stakeholders in health human resources, including educational institutions, regulatory bodies, and professional associations, all play a role in LT recruitment efforts. Recognizing and embracing this shared responsibility will be crucial in fostering a robust and sustainable healthcare workforce that incorporates LT physicians.

Limitations

In the systematic and grey literature reviews, a comprehensive set of keywords related to locum recruitment and retention were used (as detailed in Additional file 1 : Appendices S1 and S3). However, some search terms, such as region-specific terminology used to describe locums, were not included. As a result, it is possible that relevant resources may have been missed during the literature search. However, the use of diverse keywords related to locum recruitment and retention would have captured relevant studies thus reducing the likelihood that relevant resources were missed. As described in the literature search strategies, date limitations were applied to both the systematic and grey literature searches, and only select databases were searched. Therefore, there is a possibility that relevant publications or grey literature produced outside of these date ranges or databases might have been missed. The quality of synthesized evidence was moderate as most of the retained quantitative studies were cross-sectional [ 3 , 7 , 9 , 13 , 15 , 16 ] (50%) or mixed-methods [ 2 ] (8%). Only one study adopted a pre–post study design [ 17 ], which is fraught with internal validity issues. Remaining studies [ 8 , 14 , 18 , 19 ] were qualitative and not inherently generalizable to broad populations. None of the studies were intervention-based, making it difficult to draw conclusions about the effectiveness of various recruitment and retention strategies. Inconsistent reporting on locum (LT) gender limited conclusions regarding differences in motivations for LT practice. Geographies of included studies reported were likely influenced by the methodological choice to include English only articles, limiting the generalizability of the presented findings to other regions. Further, the mix of qualitative and quantitative sources make it challenging to comprehend the cumulative size of the physician population raising each issue, and the relative significance of each issue compared to others.

Conclusions

This systematic review synthesized existing knowledge pertaining to international locum physician recruitment and retention strategies. Locum physicians are essential to the delivery of quality healthcare services across Canada and other parts of the world. We demonstrate that organizations employ five main LT recruitment strategies and deploy these in a variety of ways. Though these may be incumbent on local resources, more concerning is that the effectiveness of these approaches has not been tested. Given the present financial challenges within the global healthcare landscape there is a need to better understand recruitment and retention strategies of LTs so this limited resource can be used most effectively. Findings merit future research into the effectiveness of LT recruitment approaches via prospective methodologies.

Availability of data and materials

All data generated or analysed during this study are included in this published article [and its supplementary information files].

Li JH, Scott A, McGrail M, Humphreys J, Witt J. Retaining rural doctors: doctors’ preferences for rural medical workforce incentives. Soc Sci Med. 2014;121:56–64.

Article   PubMed   Google Scholar  

Myhre DL, Konkin J, Woloschuk W, Szafran O, Hansen C, Crutcher R. Locum practice by recent family medicine graduates. Can Fam Physician. 2010;56(5):E183–90.

PubMed   PubMed Central   Google Scholar  

Rourke JTB, Incitti F, Rourke LL, Kennard MA. Keeping family physicians in rural practice—solutions favoured by rural physicians and family medicine residents. Can Fam Physician. 2003;49:1142–9.

Duong D. Why are emergency departments closing? Can Med Assoc J (CMAJ). 2022;194(33):E1138–9.

Article   Google Scholar  

Abelsen B, Strasser R, Heaney D, Berggren P, Sigurðsson S, Brandstorp H, et al. Plan, recruit, retain: a framework for local healthcare organizations to achieve a stable remote rural workforce. Hum Resour Health. 2020;18(1):63–63.

Article   PubMed   PubMed Central   Google Scholar  

Ferguson J, Tazzyman A, Walshe K, Bryce M, Boyd A, Archer J, et al. “You’re just a locum”: professional identity and temporary workers in the medical profession. Sociol Health Illn. 2021;43(1):149–66. https://doi.org/10.1111/1467-9566.13210 .

Alonzo AA, Simon AB. Have stethoscope, will travel: contingent employment among physician health care providers in the United States. Work Employment Soc. 2008;22(4):635–54.

Theodoulou I, Reddy AM, Wong J. Is innovative workforce planning software the solution to NHS staffing and cost crisis? An exploration of the locum industry. BMC Health Serv Res. 2018;18:1–13.

DiMeglio M, Furey W, Laudanski K. Content analysis of locum tenens recruitment emails for anesthesiologists. BMC Health Serv Res. 2018;18:1–7.

Waldie AC. Put out the welcome mat for locums. Can Med Assoc J (CMAJ). 1998;158(8):1009–1009.

Google Scholar  

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Int J Surg. 2021;88:105906–105906.

Larkin J, Foley L, Smith SM, Harrington P, Clyne B. The experience of financial burden for people with multimorbidity: a systematic review of qualitative research. Health Expect. 2021;24(2):282–95.

Simon AB, Alonzo AA. The demography, career pattern, and motivation of Locum tenens physicians in the United States. J Healthc Manag. 2004;49(6):363–75.

PubMed   Google Scholar  

Jenson CM, Hutchins AJ, Rowlands G. Is small-group education the key to retention of sessional GPs? Educ Prim Care. 2006;17(3):218–26.

Jenson C, Reid F, Rowlands G. Locum and salaried general practitioners: an exploratory study of recruitment, morale, professional development and clinical governance. Educ Primary Care. 2008;19(3):285–302. https://doi.org/10.1080/14739879.2008.11493685 .

McKevitt C, Morgan M, Hudson M. Locum doctors in general practice: motivation and experiences. Br J Gen Pract. 1999;49(444):519–21.

CAS   PubMed   PubMed Central   Google Scholar  

Woloschuk W, Tarrant M. Does a rural educational experience influence students’ likelihood of rural practice? Impact of student background and gender. Med Educ. 2002;36(3):241–7.

Rajbangshi PR, Nambiar D, Choudhury N, Rao KD. Rural recruitment and retention of health workers across cadres and types of contract in north-east India: a qualitative study. WHO South East Asia J Public Health. 2017;6(2):51–9.

Lagoo J, Berry W, Henrich N, Gawande A, Sato L, Haas S. Safely practicing in a new environment: a qualitative study to inform physician onboarding practices. Jt Comm J Qual Patient Saf. 2020;46(6):314–20. https://doi.org/10.1016/j.jcjq.2020.03.002 .

Hong QN, Pluye P, Fàbregues S, Bartlett G, Boardman F, Cargo M, et al. Mixed methods appraisal tool (MMAT), version 2018; 2018.

Tyndall J. AACODS checklist. Flinders University; 2010. https://www.library.sydney.edu.au/research/systematic-review/downloads/AACODS_Checklist.pdf

Download references

Acknowledgements

We gratefully acknowledge the help of Jennifer Dumond, Education Services Librarian at NOSM University, for peer-reviewing the search strategy.

Supported by Northern Ontario Academic Medicine Association (NOAMA) Academic Funding Plan (AFP) Innovation Fund Project #A-22-07.

Author information

Authors and affiliations.

Faculty of Medicine, University of Ottawa, Ottawa, ON, K1N 6N5, Canada

Nathan Ferreira & Odessa McKenna

Division of Clinical Sciences, Northern Ontario School of Medicine (NOSM) University, Marathon, ON, P0T 2E0, Canada

Iain R. Lamb, Lily DeMiglio & Eliseo Orrantia

Northern Ontario School of Medicine (NOSM) University, Sudbury, ON, P3E 2C6, Canada

Alanna Campbell

You can also search for this author in PubMed   Google Scholar

Contributions

N.F and O.M were responsible for experimental design, the acquisition, analysis, interpretation of data and writing of the manuscript. A.C was responsible for acquisition, analysis, interpretation of data. I.R.L was responsible for interpretation of data, and the writing and revising of the manuscript. L.D and E.O were responsible for conception and experimental design, data analysis, data interpretation and revising the manuscript. All authors approve of the final version of the manuscript and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. All persons designated as authors qualify for authorship, and all those who qualify for authorship are listed.

Corresponding author

Correspondence to Eliseo Orrantia .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1. appendix s1.

: Search Strategy. Appendix S2 : Mixed-methods Appraisal Tool (MMAT) quality assessment of included studies. Appendix S3 : Grey Literature Search Strategy, Data Extraction, and Evaluation.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Ferreira, N., McKenna, O., Lamb, I.R. et al. Approaches to locum physician recruitment and retention: a systematic review. Hum Resour Health 22 , 24 (2024). https://doi.org/10.1186/s12960-024-00906-z

Download citation

Received : 29 November 2023

Accepted : 02 April 2024

Published : 16 April 2024

DOI : https://doi.org/10.1186/s12960-024-00906-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Workforce stability
• Health human resources
• Recruitment strategies
• Retention strategies

Human Resources for Health

ISSN: 1478-4491

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]