grey literature in research meaning

• Research Guides

Grey Literature

• What is Grey Literature? Activities
• Why Use Grey Literature?
• Types of Grey Literature
• Sources of Grey Literature
• Searching for Grey Literature
• Evaluating Grey Literature Activity
• How to Incorporate & Cite Grey Literature

Librarian for Psychology, Human Development, & Special Education

Librarian for Human and Organizational Development

Librarian for Education

This guide includes content adapted with permission from the University of Illinois Library .

What is Grey Literature?

Describing grey literature in its entirety is more challenging than outlining what it isn't. Grey literature encompasses various media, resources, documents, and data that diverge from the conventional academic or commercial publishing pathways, often termed "white literature." If a resource lacks publication in a scholarly journal, it likely falls within the realm of grey literature. Unlike white literature, grey Literature doesn't undergo peer review and typically avoids publication in books or scholarly journals. Today, most grey literature is disseminated digitally through PDFs, web pages, blog posts, and multimedia content. Unlike academic publishing, grey literature authors are not required to possess extensive field experience, though recognized experts or organizations often author the most credible pieces. Grey literature spans a broad spectrum—it's not necessarily always factual or nonfactual, nor is it constrained to a purely professional or casual tone.

• A local government's report on the city's water quality
• The front page news story from a national news company***
• An engineering PhD candidate's dissertation on keyword latency  
• The state government's crime statistics from the last year
• A pamphlet from the Forest Service on the history of a national park
• A report from an international non-governmental organization (NGO) on modern slavery
• A blog post from a well-known academic on a new theory concerning her field
• Tweets from protestors involved in the Arab Spring demonstrations 
• Court transcripts and other legal documents

All these items can be helpful in research, but using, evaluating, and finding good grey literature can be challenging. This guide is designed to help users understand and navigate the complexities of this special literature. 

***Some do not consider news grey literature because it comes from a commercial publishing system. Still, others do because it is not peer-reviewed or necessarily written by experts in the field. This guide includes news media as grey literature because it is not peer-reviewed and can be evaluated using the same methods as grey literature.

What is "Scholarly" and "Peer Reviewed"?

Scholarly sources (also referred to as academic, peer-reviewed, or refereed sources) are written by experts in a particular field and keep others interested in that field up to date on the most recent research, findings, and news.

When a source has been peer-reviewed, it has undergone the review and scrutiny of a review board of colleagues in the author’s field. They evaluate this source as part of the body of research for a particular discipline and make recommendations regarding its publication in a journal, revisions before publication, or, in some cases, reject its publication.

In essence, when a work is scholarly and peer-reviewed, the work of evaluating a resource is done by the publisher, and the resource user does not have to spend too much time evaluating it themselves. This is extremely useful as it ensures the information is factual. However, the process of peer review is far from perfect; it has its own set of biases and issues with diversity; it is a lengthy process, and scholarly and peer-reviewed works are often expensive to access.

When resources are not peer-reviewed, the work of evaluating the resources falls almost entirely on the user. Grey literature is typically only reviewed for accuracy by their organization, and the process varies widely from organization to organization if they have one at all. The users reading the information cannot be sure if anyone has reviewed the facts presented in grey literature and if the organizations' biases have distorted the facts.  

Activity: What is Grey Literature?

Use the interactive module below to test your knowledge of grey literature!

Activity: Grey Literature Drag and Drop

Grey literature: An important resource in systematic reviews

Affiliation.

• 1 Nuffield Department of Primary Care Health Sciences, Centre for Evidence-Based Medicine, Department for Continuing Education, The University of Oxford, Oxford, UK.
• PMID: 29266844
• DOI: 10.1111/jebm.12265

Systematic reviews aid the analysis and dissemination of evidence, using rigorous and transparent methods to generate empirically attained answers to focused research questions. Identifying all evidence relevant to the research questions is an essential component, and challenge, of systematic reviews. Grey literature, or evidence not published in commercial publications, can make important contributions to a systematic review. Grey literature can include academic papers, including theses and dissertations, research and committee reports, government reports, conference papers, and ongoing research, among others. It may provide data not found within commercially published literature, providing an important forum for disseminating studies with null or negative results that might not otherwise be disseminated. Grey literature may thusly reduce publication bias, increase reviews' comprehensiveness and timeliness and foster a balanced picture of available evidence. Grey literature's diverse formats and audiences can present a significant challenge in a systematic search for evidence. However, the benefits of including grey literature may far outweigh the cost in time and resource needed to search for it, and it is important for it to be included in a systematic review or review of evidence. A carefully thought out grey literature search strategy may be an invaluable component of a systematic review. This narrative review provides guidance about the benefits of including grey literature in a systematic review, and sources for searching through grey literature. An illustrative example of a search for evidence within grey literature sources is presented to highlight the potential contributions of such a search to a systematic review. Benefits and challenges of grey literature search methods are discussed, and recommendations made.

Keywords: evidence; grey literature; publication bias; search; systematic review.

© 2017 Chinese Cochrane Center, West China Hospital of Sichuan University and John Wiley & Sons Australia, Ltd.

Reference management. Clean and simple.

What is grey literature? [with examples]

What's the importance of grey literature?

What are some of the drawbacks of grey literature, what are some examples of grey literature, how do you identify grey literature, where can you search for grey literature, are there any other ways to find grey literature, use search engines, go directly to organizations' websites, contact researchers directly, frequently asked questions about grey literature, related articles.

Grey literature refers to materials and research published specifically outside of the traditional commercial, academic publishing, and distribution channels.

It is produced by organizations on all levels of government, academia, business, and industry and either published informally or remains unpublished. Not every grey literature database will have undergone the peer review process due to this informal setup, but the information contained therein can still prove valuable to researchers and academics alike - if they are able to find it.

Since grey literature usually comes from multiple sources, finding the right resource can prove to be challenging. It's not as simple as visiting a library. With no indexes or formal organization, grey literature won't appear in academic sources like books or journals. But the question still remains, what is grey literature and why is it important to academic publishing?

• Grey literature is often the best source for up-to-date research on certain topics. This is because it doesn't have to go through the often lengthy process of publication before it can be reviewed.
• In the same breath, grey literature databases can have a scope that includes recently emerging areas of research.
• Scientific publishing, for instance, has a strong bias toward publishing research that shows a significant positive correlation. Many studies show no effect and can go unpublished, but negative results are just as important to note. Accessing grey literature fills this knowledge gap, and generates a more balanced picture of the available evidence.
• The reports produced in grey literature are often more detailed in nature and can contain raw data sets not available anywhere else - the caveat is that they come in tons of different formats and document types, making the data slightly more difficult to find.
• Grey literature can help researchers to reduce publication bias.
• Most grey literature is free (one exception is materials produced by market research firms, who often charge for access). This makes it a lot more accessible to everyone.
• The information in grey literature hasn't gone through the usual publishing channels, so it isn't rigorously and formally peer-reviewed. This means you'll need to carry out your own checks to assess its reliability.
• When grey literature isn't published, it's free from publication bias. However, it may carry its own set of biases. The organization that distributed the grey literature can have its own political, social, or financial motivations for promoting unreliable research. Make sure to cross-check information from grey literature against studies found elsewhere.
• Grey literature is produced from a great variety of sources, so it's usually not indexed or organized in any way. That can make it very difficult to locate what you're looking for.
• Librarians often have a difficult time acquiring grey literature and making it accessible.

Publication types of grey literature are almost endless, and include:

• Government reports
• Technical reports
• Unpublished clinical trials
• Conference abstracts
• Graduate dissertations and theses
• Newsletters
• White papers
• Working papers
• Dissertations

Websites, manuals, statistics, and datasets are technically also types of grey literature that you may come across during the research process.

Grey literature can be found in so many contexts it can be tough to list them all. Essentially, if you didn't find it in a renowned journal or published book, there's a good chance it's grey literature. Luckily, if you know how to find academic sources, you should be able to identify grey literature in the wild.

Here are some other indicators that you may be looking at a piece of grey literature:

• It was difficult for you to find
• It hasn't been widely distributed
• It contains more information on the subject than you've seen elsewhere, as there weren't publication length stipulations
• It was produced and disseminated quickly
• It hasn't been archived

It can be difficult to develop a systematic approach to finding grey literature for academic purposes. Unlike scholarly journals that are often available in electronic formats, grey literature databases will be all over the map in terms of formats and accessibility.

You should prepare any search terms you're going to use, identify the ones that are most applicable, and record the sources as you search. If you have a research question, use it to guide where you look for the sources you're going to search.

There are also many grey literature databases that specialize in cataloging grey literature:

There are methods that you can make use of to target your search for grey literature.

• Restrict content to only governments or organizations by typing in your search term and then either 'site:.org' or 'site:.gov'.
• You can take it further and limit your search to e.g. UK government sites only by typing 'site:.gov.uk'.
• Restrict content to a specific file type by typing in your search term and then e.g. 'filetype:pdf' or 'filetype:doc'.
• If you're looking for grey literature from a specific country, use their version of Google, for example, Google Australia or Google UK .
• If you'd like a fresh take, try search engine DuckDuckGo , as it doesn't track location or user searches. This means it doesn't filter results based on your user profile.

Identify organizations that might publish technical papers and reports on the subject you're looking for. These could be government agencies, non-profits, research institutes, pressure groups, international organizations, or professional associations.

For example, the World Health Organization (WHO) has several online collections of grey literature and the World Bank makes all of their publications available online for free.

Search your chosen organizations' websites, looking for sections of the site with names such as 'Documents', 'Reports' or 'Library'.

Research known researchers in your field of interest. You can then contact them to kindly ask if they're aware of any unpublished or ongoing studies - more often than not they will be happy to point you in the right direction.

Grey literature is materials and research that have been published specifically outside of the traditional commercial or academic publishing and distribution channels. The term 'grey' refers to the uncertainty of the status of this information.

Yes, preprints are considered grey literature, as they have not undergone the peer-review process.

PubMed is a research database, not a publication type. Material indexed by major databases such as PubMed is generally not considered grey literature.

Books and other scholarly publications have to pass through a formal publishing process and are therefore not considered grey literature.

Yes, dissertations are generally considered grey literature and might be hard to obtain in some cases.

University Library, University of Illinois at Urbana-Champaign

Grey Literature

• What is Grey Literature? Activities
• Why use Grey Literature?
• Evaluating Grey Literature Activity
• Types of Grey Literature
• Grey Literature's Role in Global Studies
• How to Incorporate & Cite Grey Literature in your Work
• Searching for Grey Literature
• Sources for Grey Literature

What is Grey Literature?

It is easier to describe what Grey Literature is not than to describe fully what it is. Grey Literature is a category that includes media, resources, documents, data, etc. that was not produced by traditional academic or commercial publishing systems, which is often referred to as White Literature. More simply put, if a resource was not published in an scholarly journal it is likely considered Grey Literature. Unlike White Literature, Grey Literature is not peer reviewed and is not typically published in books or scholarly journals. In fact, most Grey Literature today is distributed via digital formats as PDFs, web pages, blog posts and multimedia content. There is not a requirement for authors of Grey Literature to have substantial experience in the field they are writing in, as with academic publishing, but the best Grey Literature is written by authors or organizations who are well-known experts in the field. Grey Literature is not always factual and not always nonfactual.; it is not required to be entirely professional or casual. There is a wide spectrum of Grey Literature!

• A local government's report on the city's water quality
• The front page news story from a national news company***
• A engineering PhD candidate's Dissertation on keyword latency  
• The state government's crime statistics from the last year
• A pamphlet from the Forest Service on the history of a National Park
• A report from an international Non-Governmental Organization (NGO) on modern slavery
• A blog post from a well-known academic on a new theory concerning her field
• Tweets from protestors involved in the Arab Spring demonstrations 
• Court transcripts and other legal documents

All these items can be useful in research, but using, evaluating, and finding good Grey Literature can be a challenge. This guide is designed to help users understand and navigate the complexities of this special literature. 

***Some people do not consider news to be Grey Literature because it comes from a commercial publishing system but others do consider the news to be Grey Literature because it is not peer reviewed or necessarily written by experts in the field. This guide includes news media as Grey Literature because it is not peer-reviewed and it can be evaluated with the same methods as Grey Literature.

What does it mean to not be "Scholarly" and "Peer Reviewed"?

Scholarly sources (also referred to as academic, peer-reviewed, or refereed sources) are written by experts in a particular field and serve to keep others interested in that field up to date on the most recent research, findings, and news. When a source has been peer-reviewed , it has undergone the review and scrutiny of a review board of colleagues in the author’s field. They evaluate this source as part of the body of research for a particular discipline and make recommendations regarding its publication in a journal, revisions prior to publication, or, in some cases, reject its publication.

In essence, when a work is scholarly and peer-reviewed, the work of evaluating a resource is done by the publisher and the user of the resource does not have to spend too much time evaluating it themselves. This is extremely useful to users as it ensures that the information we are using is factual. However, the process of Peer Review is far from perfect; it has its own set of biases and issues with diversity, it is a lengthy process and scholarly and peer-reviewed works are often expensive to access. To learn more about the peer-review process, please visit this Library Guide: "Peer Review: An Introduction" . 

When resources are not peer-reviewed, this means that the work of evaluating the resources falls almost entirely on the user.   Grey literature is typically only reviewed for accuracy by their own organization and the process for doing so varies widely from organization to organization, if they have one at all. The users reading the information cannot be sure if anyone has reviewed the facts presented in grey literature and if the organizations' biases have distorted the facts.  

Keep in mind that not every document found in scholarly journals is peer-reviewed, "White" Literature. Many journals publish book reviews, letters from the editors, event notifications etc that are actually Grey Literature!

• Next: What is Grey Literature? Activities >>
• Last Updated: Sep 27, 2023 10:43 AM
• URL: https://guides.library.illinois.edu/c.php?g=1310347

• Subject guides
• Grey Literature
• What is grey literature

Grey Literature: What is grey literature

• Finding grey literature
• Evaluating grey literature
• Grey Sources

Other useful guides

• Citing and referencing
• Literature reviews

​Grey literature is "Information produced on all levels of government, academics, business and industry in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body."

CGL Luxembourg definition , 1997-expanded in New York, 2004

Grey literature is published informally or non-commercially, or remains unpublished. It can appear in many forms, including government reports, statistics, patents, conference papers and even non-written resources such as posters and infographics.

Grey literature usually has not been peer reviewed, but may still be good, reliable information. It can thus be invaluable for your research. It is produced from a variety of sources, and is usually not indexed or organised, often making it difficult to locate.

What is grey literature?

A short video explaining grey literature from Western Universities library

Examples of grey literature

For a comprehensive listing of grey literature types view the greynet.org table .

Common grey literature producers

Why use grey literature.

Grey literature is used in research because:

• It will often be more current than traditionally published sources, with a better coverage of emergent research areas. 
• A literature search that accesses only “black literature” will likely miss information vital to research, especially in health sciences fields.
• It can be a better source of information on policies and programs.
• Using grey literature may help minimise publication bias in systematic reviews.
• Grey literature is often a good source of raw data and data sets to which you wouldn’t otherwise have access.
• << Previous: Home
• Next: Finding grey literature >>

Grey Literature: Home: What is Grey Literature?

• Home: What is Grey Literature?
• Identifying Grey Literature
• How and Where to find Grey Literature

What is Grey Literature?

Grey literature is "Information produced on all levels of government, academics, business and industry in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body."

- As defined by the Grey Literature Network Service, 2004

Simply put, Grey Literature consists of materials that are unpublished or have been published outside of the typical commercial or academic publishing environment . 

It is often more difficult to find and obtain than the published literature, and may include such things as conference procedings, policy studies and reports, white papers, annual reports, draft legislation, think-tank reports. 

How is Grey Literature used?

Grey literature can be used in a wide variety of ways, depending on the type. Most offer data and results which can supplement original research. Many detail policies or procedures which can be used to improve existing methods.

For the most part, grey literature can be used in the same way as scholarly literature: by setting a foundation on which other types of research can be conducted.

Examples of Grey Literature

Examples of grey literature include: 

• conference abstracts
• presentations
• proceedings
• government publications
• reports  (such as white papers, working papers, internal documentation)
• dissertations & theses
• regulatory data
• unpublished trial data
• policies & procedures

for more information on types of grey literature, visit the Document Types in Grey Literature page  from GreyNet International .

"Types of Literature" by Katy Kavanagh Webb is licensed under  CC BY-NC-SA 4.0

Why look for it if Grey Literature isn't peer-reviewed?

Gray literature can be an important source of data and information. Though not published in the traditional academic outlets, it is produced by researchers in the field. It can be made available more quickly and without the rigid format of academic publishing. Gray literature can also offer greater detail than other types of literature. 

In addition, it can reduce positive  publication bias - negative results are often reported in the grey literature but not in published work.

What is Publication Bias?

Within scientific publishing, the outcome of a paper can often affect the decision by a journal or authority to publish it. Publishers often show a strong bias towards publishing studies which show some sort of significant effect over studies which do not show the expected outcome. This is known as publication bias. 

Publication bias can have a serious impact on the existing literature, since studies with and without significant findings are usually conducted in the same manner. Since studies without significant findings are much less likely to get published, it reduces the impact that the study could have on a given discipline. However, knowing that an intervention had no effect is just as important as knowing that it did have an effect when it comes to making decisions for practice and policy-making. 

Thus, grey literature can be critical when conducting meta-analyses.

• Next: Identifying Grey Literature >>
• Last Updated: Mar 28, 2024 9:37 AM
• URL: https://libraryguides.fullerton.edu/grey_literature

This site is maintained by Pollak Library .

To report problems or comments with this site, please contact [email protected] . © California State University, Fullerton. All Rights Reserved.

Web Accessibility

CSUF is committed to ensuring equal accessibility to our users. Let us know about any accessibility problems you encounter using this website. We'll do our best to improve things and get you the information you need.

• Adobe Reader
• Microsoft Viewers
• Report An ATI Issue
• Accessible @ CSUF

Evidence Synthesis: Finding, appraising and synthesizing research

• Getting Started
• Steps in Evidence Synthesis
• Methodology and Documentation
• Searching the published literature
• Searching the Gray Literature
• Managing the process

The gray literature: What is it and how do I search it?

Why search the gray literature if it hasn't been peer-reviewed?   There is a strong bias in scientific publishing toward publishing studies that show some sort of significant effect.  Meanwhile, many studies and trials that show no effect end up going unpublished.  But knowing that an intervention had no effect is just as important as knowing that it did have an effect when it comes to making decisions for practice and policy-making.  Thus, the grey literature can be critical.

How do I search the gray literature if it's unpublished and not in the scholarly databases?  Its true--finding gray literature and searching it systematically is challenging.  But there are a few approaches that you can take to add some structure to your search of this type of information:

• Identify and record the sources you will search, including some indication of search terms used if appropriate.  The sources you search will be informed by your research question and where you expect to find information related to your question.
• Try searching databases that specialize in gray literature like OpenMD , MedNar and BASE . OpenGrey , a previously available resource for gray literature, is also searchable from a locally maintained SQL database.  For more resources, see the box below.
• Conference proceedings :  Identify professional organizations that have conferences at which researchers might be presenting work related to your topic.  Search those conference proceedings on the organization's website or by contacting organizational boards for access to past proceedings that may not be online.
• Theses and dissertations :  There are a number of databases dedicated to theses and dissertations, which you can search using your search terms.  See the box below for links to these resources. 
• Identify government agencies, and international and non-governmental organizations , that might publish technical papers and reports on your topic.  Search their websites or any online libraries that they may provide.  For example, the WHO has a number of searchable online collections and the World Bank now makes all of their publications openly available online.
• Contact known researchers in the field to determine if there are any ongoing or unpublished studies that s/he may be aware of.

Image credit:  Stack 'em high, by Antony Theobald via Flickr.com, https://flic.kr/p/6WW7E2, CC BY-NC-ND 2.0

Guidance documents for gray literature searching

• Grey Matters: A practical search tool for evidence-based medicine This online manual provides a thorough list of sources for gray literature in medicine and a helpful checklist to help systematize your process.
• Googling the Greys: Tips for Searching Beyond Health Databases and Turning Information into Insights This presentation by Sarah Bonato, a health science librarian at the Centre for Addiction and Mental Health in Toronto, ON, provides an excellent guide to searching Google effectively to find gray literature.
• Duke University Medical Center Guide to Resource for Searching the Gray Literature A more thorough guide to gray literature, including resources for trial registries, pharmacological studies, conference abstracts, government document and more.

Databases of grey literature

• BASE BASE is one of the world's most voluminous search engines especially for academic web resources. BASE provides more than 240 million documents from more than 8,000 content providers. You can access the full texts of about 60% of the indexed documents for free (Open Access). BASE is operated by Bielefeld University Library.
• MedNar Mednar is a free, medically-focused deep web search engine that uses Explorit Everywhere!, an advanced search technology by Deep Web Technologies. As an alternative to Google, Mednar accelerates your research with a search of authoritative public and deep web resources, returning the most relevant results to one easily navigable page.
• NY Academy of Medicine Grey Literature Report This report was a bimonthly publication of The New York Academy of Medicine (NYAM) alerting readers to new grey literature publications in health services research and selected public health topics. The report is no longer being maintained but some of the resources are still accessible.
• OpenMD OpenMD.com is a search engine that makes high-quality medical information easily accessible to everyone. OpenMD searches billions of documents from government agencies, global health organizations, medical journals, and reference sites.

Sources for searching theses and dissertations

• ProQuest Dissertations and Theses With more than 2 million entries, PQD&T is the single, central, authoritative resource for information about doctoral dissertations and master's theses.

Resources for Clinical Trials

• Australia New Zealand Clinical Trials Registry Online register of clinical trials being undertaken in Australia, New Zealand and elsewhere. The ANZCTR includes trials from the full spectrum of therapeutic areas of pharmaceuticals, surgical procedures, preventive measures, lifestyle, devices, treatment and rehabilitation strategies and complementary therapies.
• ClinicalTrials.gov The US registry for clinical trials. Includes new, ongoing and completed human clinical trials both in the US and in countries around the world.
• Cochrane CENTRAL Register A highly concentrated source of reports of randomised and quasi-randomised controlled trials. The majority of CENTRAL records are taken from bibliographic databases (mainly MEDLINE and Embase), but records are also derived from other published and unpublished sources.
• EU Clinical Trials Register The European Union Clinical Trials Register allows you to search for protocol and results information on interventional clinical trials that are conducted in the European Union (EU) and the European Economic Area (EEA) and clinical trials conducted outside the EU / EEA that are linked to European paediatric-medicine development.
• WHO International Clinical Trials Registry Platform (ICTRP) This Clinical Trials Search Portal provides access to a central database containing the trial registration data sets provided by numerous international registries. It also provides links to the full original records.
• Wikipedia clinical trial registries list This Wikipedia entry contains a list of links to individual clinical trial registries by country.
• << Previous: Searching the published literature
• Next: Managing the process >>
• Last Updated: Feb 26, 2024 3:48 PM
• URL: https://guides.library.cmu.edu/systematic_reviews

Grey Literature: What is grey literature and why use it?

• What is grey literature and why use it?
• Tools for finding grey literature
• Sources of grey literature
• Statistics This link opens in a new window

What is grey literature?

The Fourth International Conference on Grey Literature (GL '99) in Washington, DC, in October 1999 defined grey literature as: "That which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers."

Grey literature  is research which is either unpublished or published through channels other than commercial publishing. Most of it is not found in bibliographic databases. You should consider what type of literature you are interested in before you begin searching. Not all grey literature is relevant to all researchers and it will depend on the subject studied (clinical trials will not be relevant to research about the environment for example but research reports may well be relevant). 

Types of grey literature

• Government agencies
• Research institutes
• Researchers
• Organisations
• Associations
• Campaign and pressure groups

Why include grey literature in your research?

Why should you use grey literature?

• It can record findings in niche or emerging research areas not found elsewhere
• To provide a comprehensive, balanced and unbiased picture of the research on a particular topic – negative results are more often included in grey literature so including unpublished material is a way of counteracting positive bias
• It can be more current than mainstream published literature and its findings may not find their way through into journals or books until years later after passing through lengthy peer review and editorial processes
• It can be more detailed than the literature published through commercial channels
• It can be a good source of raw data
• Enables the viewpoints of individuals less likely to publish in journals to be heard
• Provides valuable insight of a sector or industry perspective 

Evaluating grey literature

Although grey literature can be very valuable and useful for your projects, it has often been through less thorough processes of reviewing and quality assurance. It is therefore crucial to carefully consider whether the grey literature you are using is trustworthy, reliable and accurate.

One tool you can use to help with evaluating grey literature is the  AACODS Checklist for appraising grey literature

Acknowledgements

Based on University of Wolverhampto n,  University of Exeter  and University College Cork guides to grey literature

• Next: Tools for finding grey literature >>
• Last Updated: Sep 12, 2023 5:29 PM
• URL: https://libguides.qmu.ac.uk/grey-literature

• Interlibrary Loan and Scan & Deliver
• Course Reserves
• Purchase Request
• Collection Development & Maintenance
• Current Negotiations
• Ask a Librarian
• Instructor Support
• Library How-To
• Research Guides
• Research Support
• Study Rooms
• Research Rooms
• Partner Spaces
• Loanable Equipment
• Print, Scan, Copy
• 3D Printers
• Poster Printing
• OSULP Leadership
• Strategic Plan

Gray Literature: Beyond Peer Review

• Introduction
• Scientific literature
• International This link opens in a new window
• Federal This link opens in a new window
• Oregon This link opens in a new window
• Non-governmental organizations (NGOs)
• Theses & dissertations
• Using Generative AI in Research

Where can I get more help?

Contact the OSU Libraries Information Desk

• Call us at 541-737-7293
• Text us at 541-526-7655
• Chat with a librarian
• Sign up for online library workshops
• Visit the Valley Library information desk

Gray Literature

What is it.

Gray literature is an important source of information that consists of government, academic, and business information that is shared outside of traditional academic publishing channels. It generally isn't peer reviewed.

It includes:

• working papers or pre-prints
• government/NGO publications
• technical/industry reports
• conference proceedings/posters
• presentation slides
• newsletters
• dissertations/theses
• oral presentations

Why use it?

• It provides access to new ideas and perspectives.
• It allows you to broadly explore a topic and understand the less formal conversations happening around it.
• It may help counterbalance publication bias (the increased likelihood of reporting positive results).
• It may be more current than peer reviewed material, since it may not be peer reviewed and authors may be able to update it.
• It may provide a more comprehensive view of emerging research areas.
• Government agencies, non-government organization, and businesses are often important stakeholders in research issues.

"Peer review" refers to the process where researchers submit a paper they have written to a journal. The journal editor then sends the article to the author's peers , other researchers and scholars who are in the same discipline. These reviewers determine if the article should be published based on the quality of the research, including the validity of the data, the conclusions the authors' draw, and the originality of the research. While peer review is important for validating research, it also takes a great deal of time.

Initial Tools

When performing a "what is out there"-style search, Google will return the largest number of results. However, it will also return a lot of misinformation and irrelevant material. For more control over your search results, try the Google Advanced Search or searching with the following terms:

• site:*.edu or site:*.gov This tells Google you only want search results from websites with a .edu (educational) or .gov (government) address. It doesn't guarantee the material is useful, but it can help narrow your results to material that's more conducive to research.
• filetype:pdf This tells Google you're only interested in PDF files. The PDF format is commonly used for articles, handouts, syllabi, and organizational documents.
• "phrases in quotes" This tells Google you want the exact phrase contained within the quotes. It's helpful when searching for specific concepts that include common phrases

For example, the search for [site:*.edu filetype:pdf "social emotional learning] (without the square brackets) will return all of the PDFs that Google has indexed from educational websites that contain the phrase "social emotional learning".

Google Scholar

Google Scholar contains journal and conference papers, theses and dissertations, academic books, pre-prints, abstracts, technical reports and other scholarly literature that's available on the internet. It also contains material from the OSU Libraries database subscriptions if you sign in with your ONID. For more information on Google Scholar, check out OSU's Google Scholar LibGuide

As with Google, using the Advanced Search or using search strings like author: provide more focused results.

Databases dedicated just to gray literature do exist. For example:

• OpenGrey.eu : A directory of grey literature produced in Europe.
• OpenDOAR : A global directory of open access repositories

However, gray lit is often stored among peer-reviewed articles. Many of OSU Libraries' databases contain gray lit.

Most databases provide advanced search tools or a "refine search" option to limit your results to specific formats or document types. For example, some databases allow searches specifically for conference proceedings, dissertations, or government documents. Another way to look for gray lit is to filter out everything marked "peer-reviewed". These are a few databases that contain a mix of gray lit and peer-reviewed articles:

The Theses & Dissertations guide has more information on theses, dissertations, and ScholarsArchive@OSU .

Evaluating Gray Literature

Gray literature is usually not peer reviewed. It may not be appropriate for certain assignments or research activities. Moreover, academic disciplines place different amounts of value on gray literature. When including gray lit in your research, consider these steps: 

• Pay attention to the types of sources referenced in your field. Do your peers mention blog posts? Dissertations? Government reports? If so, that's a good sign you could do it too.
• Is the depth of information appropriate for your research? If you're looking for three peer-reviewed articles, gray lit probably won't help. However, if you're performing a broad exploration of a topic and its ongoing scholarly conversations, gray lit may be just what you need.

Critically evaluating any source is important, but it's especially important with gray literature since you don't have the safety net of peer review. Who created the literature? What are their biases? Are they respected in their field?

For more information on evaluating information, check out the SIFT method or the CRAAP test .

• Next: Scientific literature >>
• Last Updated: Feb 8, 2024 3:15 PM
• URL: https://guides.library.oregonstate.edu/graylit

Contact Info

121 The Valley Library Corvallis OR 97331–4501

Phone: 541-737-3331

Services for Persons with Disabilities

In the Valley Library

• Oregon State University Press
• Special Collections and Archives Research Center
• Undergrad Research & Writing Studio
• Graduate Student Commons
• Tutoring Services
• Northwest Art Collection

Digital Projects

• Oregon Explorer
• Oregon Digital
• ScholarsArchive@OSU
• Digital Publishing Initiatives
• Atlas of the Pacific Northwest
• Marilyn Potts Guin Library  
• Cascades Campus Library
• McDowell Library of Vet Medicine

• Langson Library
• Science Library
• Grunigen Medical Library
• Law Library
• Connect From Off-Campus
• Accessibility
• Gateway Study Center

Email this link

Systematic reviews & evidence synthesis methods.

• Schedule a Consultation / Meet our Team
• What is Evidence Synthesis?
• Types of Evidence Synthesis
• Evidence Synthesis Across Disciplines
• Finding and Appraising Existing Systematic Reviews
• 0. Develop a Protocol
• 1. Draft your Research Question
• 2. Select Databases
• 3. Select Grey Literature Sources
• 4. Write a Search Strategy
• 5. Register a Protocol
• 6. Translate Search Strategies
• 7. Citation Management
• 8. Article Screening
• 9. Risk of Bias Assessment
• 10. Data Extraction
• 11. Synthesize, Map, or Describe the Results
• Open Access Evidence Synthesis Resources

Grey Literature

Grey literature is literature produced by individuals or organizations outside of commercial and/or academic publishers. This can include information such as government reports, conference proceedings, graduate dissertations, unpublished clinical trials, and much more. The sources you select will be informed by your research question and field of study, but should likely include, at a minimum, theses and dissertations.

Why Search the Grey Literature?

The intent of an evidence synthesis is to synthesize all available evidence that is applicable to your research question. There is a strong bias in scientific publishing toward publishing studies that show some sort of significant effect. Meanwhile, many studies and trials that show no effect end up going unpublished. But knowing that an intervention had no effect is just as important as knowing that it did have an effect when it comes to making decisions for practice and policy-making. While not peer-reviewed, grey literature represents a valuable body of information that is critical to consider when synthesizing and evaluating all available evidence.

Searching the Grey Literature

Finding grey literature and searching it systematically is challenging. But there are a few approaches that you can take to add some structure to your search of this type of information:

• Refer to grey literature sources used for related evidence syntheses. Refer to both published evidence syntheses and registered protocols.
• Ask experts in the field for relevant grey literature sources. If you are an expert, include important grey literature sources, and ask colleagues for their recommendations.
• Search databases that specialize in grey literature: See the "Grey Literature Sources" box at the bottom of this page for more information.
• Search for theses and dissertations: There are a number of databases dedicated to theses and dissertations, which you can search using your search terms. See the "Grey Literature Sources" box at the bottom of this page for links to these resources.
• Search clinical trials: There may be clinical trials being conducted that are relevant to your research question, but that haven't been published yet or never were published. See the "Grey Literature Sources" box at the bottom of this page for links to these resources.
• Identify government agencies and international and non-governmental organizations that might publish technical papers and reports on your topic. Search their websites or any online libraries that they may provide. See the "Grey Literature Sources" box at the bottom of this page for links to some examples.
• Search conference proceedings and newsletters: Identify professional organizations that have and/or conferences at which researchers might be presenting work related to your topic. Search those conference proceedings or newsletters on the organization's website or by contacting organizational boards for access to past proceedings that may not be online. See the "Grey Literature Sources" box at the bottom of this page for some examples.
• Contact known researchers in the field to determine if there are any ongoing or unpublished studies that s/he may be aware of.
• Search professional and trade magazines. Professional magazines contain literature that is written by professionals in the field for other professionals in the field, but that may not be about research. Trade magazines contain advertisements and news very specific to a topic or industry.

How to Manage the Grey Literature Search

• Identify and record the sources you will search. The sources you search will be informed by your research question and where you expect to find information related to your question.
• Document where you are searching and your search strategies, including document resource name, URL, search terms, and date searched.
• Collect citation information as you go.
• Adhere to your established inclusion and exclusion criteria when selecting sources.

See below for guidance documents specific to grey literature searching.

Grey Literature Sources

• Grey Literature Databases
• Theses and Dissertations
• Clinical Trials
• Public Policy
• Conference Proceedings
• Preprint Repositories
• Other Resources
• NY Academy of Medicine Grey Literature Report This report is a bimonthly publication of The New York Academy of Medicine (NYAM) alerting readers to new grey literature publications in health services research and selected public health topics. The database platform is keyword searchable and serves as an archive for the cataloged reports.
• WHO Library Database The WHO library database includes governing documents, reports and technical documentation.
• Global Think Tanks A database of reports, papers, and other resources published by NGOs, think tanks, IGOs and research centers from across the world. It also includes a directory of thousands of organizations involved in public and social policy in most subject area. A module in the Policy Commons database.
• MedNar MedNar searches across more than 60 medical research sources, including commercial databases, medical societies, NIH resources, and other government resources.
• Global Index Medicus The Global Index Medicus (GIM) provides worldwide access to biomedical and public health literature produced by and within low-middle income countries. The main objective is to increase the visibility and usability of this important set of resources. The material is collated and aggregated by WHO Regional Office Libraries on a central search platform allowing retrieval of bibliographical and full text information.
• National Grey Literature Collection - UK Funded by Health Education England this holds the catalogue for the hard copy grey literature collection of the former North Grey Literature Service. It is collecting an increasing range of electronic Grey Literature.

• Networked Digital Library of Theses and Dissertations (NDLT) Free international resource for theses and dissertations.
• Center for Research Libraries A resource for institutions outside of the U.S. and Canada.
• EThOS Electronic Theses Online Service contains doctoral theses from candidates in the UK.
• Open Access Theses and Dissertations Resource for finding open access graduate theses and dissertations published around the world. Metadata (information about the theses) comes from over 1100 colleges, universities, and research institutions. OATD currently indexes over 4 million theses and dissertations.
• DART European - Digital Access to Research Theses Europe The aim of DART is to provide researchers with a single European Portal for the discovery of Electronic Theses and Dissertations (ETDs). Note that only theses available in electronic form are included.
• Australia New Zealand Clinical Trials Registry Online register of clinical trials being undertaken in Australia, New Zealand and elsewhere. The ANZCTR includes trials from the full spectrum of therapeutic areas of pharmaceuticals, surgical procedures, preventive measures, lifestyle, devices, treatment and rehabilitation strategies, and complementary therapies.
• ClinicalTrials.gov The US registry for clinical trials. Includes new, ongoing, and completed human clinical trials both in the US and countries around the world.
• Cochrane Central Register of Controlled Trials A highly concentrated source of reports of randomized and quasi-randomized controlled trials. The majority of CENTRAL records are taken from bibliographic databases (mainly MEDLINE and Embase), but records are also derived from other published and unpublished sources.
• EU Clinical Trials Register The European Union Clinical Trials Register allows you to search for protocol and results information on interventional clinical trials that are conducted in the European Union (EU) and the European Economic Area (EEA) and clinical trials conducted outside the EU / EEA that are linked to European paediatric-medicine development.
• WHO International Clinical Trials Registry Platform ICTRP provides access to a central database containing the trial registration data sets provided by numerous international registries. It also provides links to the full original records.
• Wikipedia Clinical Trial Registries List This Wikipedia entry contains a list of links to individual clinical trial registries by country.
• WHO Institutional Repository for Information Sharing (IRIS) Institutional WHO database of intergovernmental policy documents and technical reports. Can search by IRIS by region (Africa, Americas, Eastern Mediterranean, Europe, South-East Asia, Western Pacific).
• Health Research Web A wiki with descriptors of national health research systems, ethics review committees, local and regional policies and research priorities and more.

This list of conference proceedings is not comprehensive, but are ideas of where to search. We recommend that you search proceedings for the three most relevant conferences for your evidence synthesis.

• OCLC PapersFirst OCLC PapersFirst is an OCLC index of papers presented at conferences worldwide.
• BIOSIS Previews A subcollection of Web of Science, BIOSIS Previews searches across journals, meetings, patents, and books in the life sciences and biomedical fields.

• arXiv arXiv is a free distribution service and an open-access archive for scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.
• medRxiv medRxiv (pronounced "med-archive") is a free online archive and distribution server for complete but unpublished manuscripts (preprints) in the medical, clinical, and related health sciences.
• PsyArXiv A free preprint service for the psychological sciences. It is maintained by The Society for the Improvement of Psychological Science.
• OSF Preprints A general preprint repository that covers architecture, business, engineering, life sciences, physical sciences and mathematics, arts and humanities, education, law, medicine and health sciences and social and behavioral sciences, among others.
• TechRxiv Hosted by IEEE
• Engineering Archive Created in 2016. Accepts all types of manuscripts, including but not limited to systematic reviews, hypotheses, ‘negative’ results, and data and methods papers.Includes 2 major types of articles: (1) Preprints of articles in parallel to traditional journal submissions. (2) Previously published articles that you wish to make Open Access
• Preprints.org This is a multidiscipline platform providing preprint service that is dedicated to making early versions of research outputs permanently available and citable. We post original research articles and comprehensive reviews, and papers can be updated by authors as long as the updated content has not been published online.
• Grey Matters: A practical search tool for evidence-based medicine This online manual provides a thorough list of sources for grey literature in medicine and a helpful checklist to help systematize your process.
• Searching the grey literature: A handbook for searching reports, working papers, and other unpublished research Searching the Grey Literature is for librarians and information professionals interested in learning more about grey literature. This book will aid with crafting a grey lit search successfully, from start to finish. Many types of librarians will find the content of this book useful, particularly those in health or social science.
• Duke University Medical Center Guide to Resource for Searching the Grey Literature Resources for trial registries, pharmacological studies, conference abstracts, government document and more.
• Gray Literature Resources for Agriculture Evidence Syntheses This is a comprehensive list of agricultural grey literature sources, one of several evidence synthesis worksheets and tools provided to help in non-medical evidence synthesis development and implementation.
• Searching for studies: A guide to information retrieval for Campbell Systematic Reviews This document gives comprehensive guidance in searching for evidence syntheses, and offers a robust list of grey literature sources for social science disciplines.
• Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions: AHRQ and the Effective Health Care Program An introduction to clinical resources and strategies in approaching systematic reviews.

Video: Grey literature (3:10 minutes)

• << Previous: 2. Select Databases
• Next: 4. Write a Search Strategy >>
• Last Updated: Feb 14, 2024 4:20 PM
• URL: https://guides.lib.uci.edu/evidence-synthesis

Off-campus? Please use the Software VPN and choose the group UCIFull to access licensed content. For more information, please Click here

Software VPN is not available for guests, so they may not have access to some content when connecting from off-campus.

• Methodology
• Open access
• Published: 29 September 2016

Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: critical reflections on three case studies

• Jean Adams   ORCID: orcid.org/0000-0002-5733-7830 1 , 2 , 3 ,
• Frances C. Hillier-Brown 3 , 4 ,
• Helen J. Moore 3 , 4 ,
• Amelia A. Lake 3 , 4 , 5 ,
• Vera Araujo-Soares 2 , 3 ,
• Martin White 1 , 2 , 3 &
• Carolyn Summerbell 3 , 4  

Systematic Reviews volume  5 , Article number:  164 ( 2016 ) Cite this article

55k Accesses

242 Citations

83 Altmetric

Metrics details

Grey literature includes a range of documents not controlled by commercial publishing organisations. This means that grey literature can be difficult to search and retrieve for evidence synthesis. Much knowledge and evidence in public health, and other fields, accumulates from innovation in practice. This knowledge may not even be of sufficient formality to meet the definition of grey literature. We term this knowledge ‘grey information’. Grey information may be even harder to search for and retrieve than grey literature.

On three previous occasions, we have attempted to systematically search for and synthesise public health grey literature and information—both to summarise the extent and nature of particular classes of interventions and to synthesise results of evaluations. Here, we briefly describe these three ‘case studies’ but focus on our post hoc critical reflections on searching for and synthesising grey literature and information garnered from our experiences of these case studies. We believe these reflections will be useful to future researchers working in this area.

Issues discussed include search methods, searching efficiency, replicability of searches, data management, data extraction, assessing study ‘quality’, data synthesis, time and resources, and differentiating evidence synthesis from primary research.

Conclusions

Information on applied public health research questions relating to the nature and range of public health interventions, as well as many evaluations of these interventions, may be predominantly, or only, held in grey literature and grey information. Evidence syntheses on these topics need, therefore, to embrace grey literature and information. Many typical systematic review methods for searching, appraising, managing, and synthesising the evidence base can be adapted for use with grey literature and information. Evidence synthesisers should carefully consider the opportunities and problems offered by including grey literature and information. Enhanced incentives for accurate recording and further methodological developments in retrieval will facilitate future syntheses of grey literature and information.

Peer Review reports

Public health researchers may want to include ‘grey literature’ in evidence syntheses for at least three reasons. Firstly, including grey literature can reduce the impact of publication bias as studies with null findings are less likely to be published in peer-reviewed journals [ 1 ]. Secondly, grey literature can provide useful contextual information on how, why, and in whom complex public health interventions are effective [ 2 – 6 ]. Finally, syntheses of grey literature can help applied researchers and practitioners understand what interventions exist for a particular problem, the full range of evaluations (if any) that have been conducted, and where further intervention development and evaluation is needed.

Numerous definitions of grey literature exist. These tend to focus on the fact that it is not controlled by commercial organisations, making it difficult to search for and retrieve [ 7 – 9 ]. One common definition restricts grey literature to literature ‘protected by intellectual property rights, of sufficient quality to be collected and preserved by library holdings or institutional repositories’ [ 10 ]. Other definitions are more inclusive and propose that, given the growth of new forms of media, grey literature should not be restricted to written ‘literature’ [ 4 ].

Much knowledge and evidence in applied settings, such as public health practice, accumulates from innovation in practice [ 7 ]. This may include the rationale for why new approaches were tried; what changes, if any, were made to previous approaches and why; what was done and how; and what happened. In some cases, this may be accompanied by more formal process evaluation and, most rarely, outcome evaluation (Hillier-Brown F, Summerbell C, Moore H, Wrieden W, Abraham C, Adams J, Adamson A, Araujo-Soares V, White M, Lake A. A description of interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England: a systematic mapping and evidence synthesis. BMC public health 2015, unpublished). Interventions and evaluations that were primarily conducted as part of, or to inform, practice may be particularly unlikely to be described in peer-reviewed publications or even formally documented in reports available to others in electronic or hard copy. Information on these activities may, instead, be stored in more private or informal spaces such as meeting notes, emails, or even just in people’s memories. This information is likely to be of insufficient formality to meet the definition of grey literature. We term this ‘grey information’.

The phrase ‘grey information’ has been used previously to extend the concept of grey literature to a wider range of sources [ 11 ], but it is not widely used and we are not aware of a previously stated clear definition. The term ‘grey data’ [ 12 ] has also been used specifically to describe user-generated web content—something that we feel is more formal and public than ‘grey information’ but less formal than ‘grey literature’. Table  1 describes defining aspects and examples of the three terms: grey literature, grey data, and grey information.

Systematically identifying grey literature and grey data is not a straightforward task [ 5 , 7 – 9 , 12 , 13 ]. Systematically identifying ‘grey information’ is likely to be even more challenging. A number of case studies have been published describing procedures for searching and retrieving ‘grey literature’ in public health contexts [ 13 , 14 ]. These tend to adopt relatively similar approaches including searching databases of peer-reviewed and grey literature; conducting structured searches of relevant websites and search engines; and contacting relevant experts [ 5 , 8 , 9 , 13 ].

On three occasions, various authors of this article have attempted to systematically search for and synthesise public health grey literature and information. Here, we briefly describe our experiences of these three case studies and then critically reflect on these. ‘Critical reflection’ is a concept most often associated with adult learning and professional development. Although poorly and diversely defined, critical reflection is generally associated with post hoc examination of experiences in an attempt to improve future practice [ 15 , 16 ]. Our aim was to provide insights on searching for and synthesising grey literature and information that may be useful for future researchers.

The aims, methods, results, and conclusions of our three case studies are summarised in Table  2 .

Review 1: the health, social, and financial impacts of welfare right advice delivered in healthcare settings [ 17 ]

Our first review included grey literature alongside peer-reviewed literature in a systematic review of the health, social, and financial impacts of welfare rights advice delivered in healthcare settings [ 17 ]. In part, this systematic review was conducted in preparation for an application for funding for a randomised controlled trial of the impacts of welfare rights advice on health in older people [ 18 , 19 ]. Thus, we were interested both in the extent and findings of other research. We conducted a quantitative synthesis of the average financial impacts of welfare rights advice and a narrative synthesis of other quantitative and qualitative findings.

As expected, less than half of the evaluations of welfare right advice included in the review were published in peer-reviewed journals. The remainder were published in reports published by delivery organisations, universities, other research organisations, and service and research funders.

Review 2: adult cooking skill interventions in England [ 20 ]

Our second attempt to review grey literature explored the nature, content, and range, but not effects, of interventions seeking to enhance adult cooking skills delivered in England [ 20 ]. Our intention was to identify the most sustainable and theoretically promising of these to take forward for more formal evaluation. Similar to other reviews [ 21 ], our synthesis focused on categorising interventions according to delivery setting and training model and summarising the training delivered, throughput, setup and running costs, funding, and behaviour change techniques used [ 22 ].

This review focused entirely on grey literature and information and did not include any searching for peer-reviewed literature. A scoping review of peer-reviewed outcome evaluations of adult cooking skill interventions was conducted in parallel [ 23 ].

Review 3: interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England (Hillier-Brown F, Summerbell C, Moore H, Wrieden W, Abraham C, Adams J, Adamson A, Araujo-Soares V, White M, Lake A. A description of interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England: a systematic mapping and evidence synthesis. BMC public health 2015, unpublished)

Finally, we recently completed a review of interventions aiming to promote healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by food outlets in England (Hillier-Brown F, Summerbell C, Moore H, Wrieden W, Abraham C, Adams J, Adamson A, Araujo-Soares V, White M, Lake A. A description of interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England: a systematic mapping and evidence synthesis. BMC public health 2015, unpublished). This explored the nature and range of interventions implemented and summarised evaluation findings. We used a narrative approach to evidence synthesis, characterising interventions, identifying issues of design and delivery, and summarising evaluation findings on process, acceptability, cost, and impact. Our intention in this case was to use the findings to inform development and evaluation of new interventions based on the most promising features of previous ones.

Whilst this review did include searches of peer-reviewed literature, these only identified one included study—although two other relevant peer-reviewed papers were identified using other methods. As in review 2, a linked review of peer-reviewed evidence was conducted in parallel [ 24 ].

Results and discussion

Whilst there is much useful guidance available on evidence synthesis in general [ 25 – 27 ], and searching for and synthesising grey literature in particular [ 5 , 8 , 9 , 13 , 28 – 30 ], one size rarely fits all. Throughout, and in common with the best research practice, our methods were guided by our aims.

In evidence syntheses, the sensitivity, specificity, and type of information retrieved by searches is highly dependent on the search strategy used. As described above and in Table  2 , we used a variety of different methods to search for information across all three reviews. We reflect on some of the issues raised below and summarise some of our conclusions in Table  3 .

Search methods

In all three cases, and as recommended by others [ 5 , 8 , 9 , 13 , 28 – 30 ], we used a wide variety of methods to search for relevant grey literature and information. Across our three examples, we searched trial databases (e.g. www.isrctn.com ), grey literature databases (e.g. www.opengrey.eu ), websites of relevant organisations (e.g. charities with an interest in social inclusion in review 1), and a popular internet search engine (i.e. www.google.co.uk ).

We also contacted those working in the areas we were interested in. We sent both personalised requests to key informants, as well as more generic requests to professional organisations and groups, using a variety of methods. In reviews 1 and 3, researchers working in relevant fields were contacted via email and requests for information were sent to relevant email lists, posted on online bulletin boards, and published in the ‘professional press’ (e.g. newsletters of professional organisations). In reviews 2 and 3, we also attempted to contact relevant individuals working in all local public health departments in England. In review 3, we incorporated social media into our search strategy.

Review 1 was conducted in 2005 when social media and social networks were less well established than they are now. To target large networks of professionals in this case, we published requests for help in the ‘professional press’. By the time review 3 was conducted, in 2014, social media had become an important space for professional networking. We posted numerous tweets requesting relevant information and asking those who saw them to repost (i.e. ‘retweet’) them to their own networks—in order to increase the potential number of people who saw these requests. Many of these tweets tagged (i.e. ‘@mentioned’) relevant professional organisations. We are not aware of previous reviewers using social media to identify grey (or peer-reviewed) literature or information.

This transition in methods from reviews 1–3 over just less than a decade reflects how information storage and sharing has changed over this time in the UK. At the same time, information storage and sharing patterns may vary internationally. Search methods need to adapt to local and international trends in information systems, and researchers should be flexible to this.

Searching efficiency

As with ‘typical’ systematic reviews [ 31 , 32 ], our searching sacrificed specificity for sensitivity. Searches yielded many results that did not meet our inclusion criteria. The resource and scientific implications of the trade-off between search specificity and sensitivity have been widely discussed in the systematic review literature [ 33 , 34 ].

Previous case studies have described very different ‘hit’ rates associated with different grey literature search strategies. In a review of interventions to promote walking and cycling, requests for help emailed to key informants added little to database searching [ 35 ]. Whereas, in a review of behaviour change interventions published only in grey literature, 70 % of items included in the final synthesis came from key informants [ 5 ]. Similarly, we found that different methods for locating information were differentially effective across our three reviews. In review 1, generic requests sent to email lists and published in the professional press were particularly useful. On a number of occasions, these requests were passed through a number of people before someone responded with relevant information—further adding to the time taken to conduct searching that is discussed below. Perhaps, similarly, in review 3, Twitter requests were particularly valuable. These were widely retweeted, vastly increasing the pool of potential viewers, but this appeared to be a much quicker process than cascading of email requests and requests in the professional press.

The efficiency of different search methods are, at least partly, dependent on the quality of the search strategy used. Simple comparisons, such as those described above, are not necessarily fair. Nor is it clear if the differences in efficiency are predictable. If the efficiency of different approaches to searching could be predicted in advance, this could help reviewers to focus their resources.

Our resources were most limited in review 2, and it became evident early in searching that we would not be able to complete a comprehensive search for all adult cooking skill interventions in England. We made a pragmatic decision to focus instead on identifying intervention types—based on delivery context and training model. As others have done [ 13 ], we borrowed the concept of ‘data saturation’ from qualitative research and stopped searching when we felt we were not identifying any new intervention types. We felt that sacrificing sensitivity in this case did not compromise our ability to meet our aims.

Using others to target searching

In reviews 2 and 3, we attempted to ask all local public health departments in England what relevant projects they were aware of. We are not aware of any peer-reviewed publications which report the efforts of other evidence synthesisers, or indeed primary researchers, who have attempted to systematically contact all local public health departments across one country in this way. That said, we recognise that the gathering of data on the activity and type of public health interventions conducted at various geographic levels is a relatively common activity. To facilitate this, we identified named individuals and contact email addresses for those with relevant roles using internet searches and telephone calls. This was a time-consuming task in itself. The requests for information we sent specifically asked recipients to pass our enquiries onto those they felt were best placed to respond. As with other email requests (see above), there were examples where messages had been passed through a number of individuals before someone responded.

Replicability of searches

Whilst in all cases, we had clear plans describing what we felt were comprehensive, systematic, and replicable approaches to information searching, it is hard to claim that these led to replicable results. Certainly, it would be possible for future investigators to replicate our search methods, but it is unlikely that these would lead to the same results on replication, as would be expected when using electronic databases. On two different occasions, different people would be likely to see calls for information on social networks or in the professional press. Even if the same people did see requests for help on different occasions, many other contextual factors may influence how likely they were to respond or pass them onto those most likely to be able to respond.

As time passes and grey literature and information becomes lost or forgotten, potential respondents’ ability to provide usable information may also decline. Whilst contacting both those currently and previously in posts as key informants may, theoretically, reduce this problem, it may not be practically possible. Others have highlighted the problem of replicability in relation to internet searching, particularly using search engines such as www.google.com which returns results based on, amongst other things, recent popularity [ 8 , 9 ].

The conclusion that searching for grey literature and information can be systematic, but not necessarily replicable, reinforces the importance of using many overlapping searching approaches. This maximises the chances that any particular piece of relevant information will be found.

Developing the ‘best’ search methods

Whilst our search methods were similar to, and built on, those described by others as well as on ‘best practice’ guidance [ 5 , 8 , 9 , 13 , 28 – 30 ], it is difficult to be sure what the ‘best’ search methods for retrieving grey literature and information are. Whilst it is possible to validate search approaches in peer-reviewed literature against a ‘gold standard’ of hand searching [ 31 , 32 ], no similar gold standard exists for grey literature and information: there is no definitive repository against which other search methods can be compared. This makes it difficult to ever be sure that all relevant information has been found or validate new search methods.

Data management

In all three reviews, we found data management to be challenging. Technology now allows fairly straightforward integration of academic databases and reference management software—both of which facilitate information organisation and record keeping. Such workflows are not well developed for grey literature and information. Developing clear filing and recording systems, using simple spreadsheets, helped us to keep track of where and how information had been identified.

However, we found it harder to capture other aspects of our searches. For example, whilst tools like NCapture allow social media content to be imported in NVivo for qualitative analysis, they do not necessarily provide a useful facility for capturing how many people (and who) ‘retweeted’ a particular tweet. It is even harder to capture when requests for information are circulated using more private methods such as email. For these reasons, we are not able to provide accurate estimates of how many people saw our requests for information.

Data extraction

In all three reviews, we developed and used data extraction forms to record information. In review 1, we adopted a similar approach to data extraction used in many ‘typical’ systematic reviews—if information was not provided in the written report we obtained, we assumed this information was missing. However, systematic review guidance encourages reviewers to attempt to minimise missing data by contacting authors of original papers [ 25 ]. We adopted an approach much more similar to this in reviews 2 and 3. In fact, many data extraction forms in these reviews were completed during telephone calls or following email conversations with key informants. To maximise accuracy, in review 3, we asked informants to check electronic versions of completed data extraction forms. As there is often little or no documentary evidence to refer back to, ensuring data extraction forms are as accurate and complete as possible is particularly important in reviews of grey literature and information. This reflects and reinforces the fact that much information on interventions in public health practice is not well documented and can be ‘temporary’: once the relevant individuals move to new posts, and interventions recede into the past, individual and institutional memories are likely to fade. This further contributes to the limited replicability of this sort of grey information searching.

Despite the efforts we made in reviews 2 and 3 to speak with those directly responsible for intervention design and delivery, we were often not able to obtain the information we intended to capture. For example, of 102 interventions identified in review 3, we were not able to obtain any information beyond a programme name in 27 cases. In most, if not all, cases, our failure to obtain information appeared to be because such information was not documented or easily obtainable. For example, many of the costs of public health interventions in everyday practice are unclear even to those responsible for them. Whilst the cost of additional staff may be explicit, costs for office space for those staff might be absorbed by organisations and so be much more implicit.

The problem of limited data availability is common to all types of evidence synthesis, but others have noted it as a particular problem when synthesising grey literature [ 7 , 13 ]. When attempting to synthesise the extent of public health practice, it may be important to be aware of the types of information that are and are not important to practitioners and easy for them to record and hence are likely to be documented. For example, service throughput appears to be much more likely to be documented than outcomes of interventions (Hillier-Brown F, Summerbell C, Moore H, Wrieden W, Abraham C, Adams J, Adamson A, Araujo-Soares V, White M, Lake A. A description of interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England: a systematic mapping and evidence synthesis. BMC public health 2015, unpublished) [ 17 ].

Risk of bias and value of information

The risk of bias of any piece of information is dependent, in part, on the question it is being used to answer. In review 2, and part of review 3, our aims were to describe the nature and range of particular classes of interventions. The risk of bias of individual pieces of grey literature and information in this case is likely to be low—there is little reason why such information would be misrecorded. In contrast, in review 1 and part of review 3, we aimed to synthesise evaluation findings. The risk of bias of grey literature and information in this case may be likely to be higher. Indeed, in reviews 1 and 3, we described some aspects of evaluation methods relating to risk of bias. In both cases, we concluded that the majority of studies were methodologically weak and at high risk of bias.

Evaluations found in grey literature and information may be at high risk of bias for a number of reasons. In public health practice, evaluations are often conducted by the same practitioners who developed and delivered an intervention. This results in an inherent conflict of interest which may increase risk of bias. In public health practice, resources for evaluation are often limited, limiting the scope of what is possible [ 36 ]. Furthermore, the interest of funders and practitioners is often on throughput rather than outcomes [ 37 ], limiting the scope of what is necessary. Whilst many evaluations included in our reviews were at high risk of bias in terms of conclusions about effects on outcomes, they may well have been fit for the purpose for which they were conducted.

Methods for assessing risk of bias in controlled trials are well established [ 38 , 39 ], and tools for other types of study are becoming available [ 40 – 42 ]. However, these approaches may be too narrow in perspective for grey literature and information. Realist synthesis takes a researcher-driven ‘value of information’ approach to assessing studies, rather than the more familiar protocol-driven risk of bias approach used in ‘typical’ systematic reviews. In the value of information paradigm, individual studies are included if the information they provide is considered relevant and rigorous enough to help contribute to answering the research question [ 6 , 43 , 44 ]. Whilst this requires researchers to make judgements about what is ‘relevant and rigorous enough’, it may result in inclusion of more potentially useful grey literature and information than stricter approaches which exclude studies based on risk of bias assessments.

Data synthesis

Many approaches to data synthesis in the context of systematic reviewing and evidence synthesis have now been described, and these are not limited to quantitative meta-analysis [ 25 , 45 ]. Although we performed a quantitative synthesis in review 1, this focused on the economic benefits of welfare rights advice to recipients (which could be summarised in £/week). We were not able to summarise health and social implications so simply and used narrative syntheses for these.

In review 3, in an attempt to capture all the data available to us, we adopted a three tiered approach to synthesis. Firstly, we listed all relevant interventions that we found ( n  = 102; tier 1). Secondly, for those interventions for which we had further information on content and delivery, we summarised this using a standard template ( n  = 75; tier 2). Finally, we summarised the results of any evaluations of included interventions ( n  = 30; tier 3). Interventions in each tier were nested within each other such that all interventions were included in the tier 1 synthesis, but only a subset of these were included in tier 2, and only a subset of those in tier 2 were included in the tier 3.

These differences in synthesis approach reflect both the contrasting aims of different reviews and how flexible and responsive researchers should be to the realities of data availability within grey literature and grey information.

Time and resources

Systematic reviews can be time and resource intensive. In ‘typical’ systematic reviews, preliminary scoping reviews can help reviewers estimate the size of a full review and resources required [ 46 ]. ‘Rapid reviews’ of peer-reviewed literature offer the hope and potential for conducting much quicker evidence syntheses that arrive at the same conclusions as full reviews [ 47 – 49 ]. Unfortunately, there is no clear equivalent of scoping or rapid reviews in relation to grey literature and information. As others have noted, searching for less formally archived information is, almost by nature, time-consuming and inefficient [ 5 , 8 , 50 ].

Encouraging public health practitioners to deposit intervention documents and information in online repositories (e.g. www.ncdlinks.org ) could enable more efficient information retrieval on current and recent practice [ 7 ]. But the utility of such databases relies heavily on their coverage, and previous attempts to ensure high coverage have been varying in their success [ 51 ]. With few obvious current incentives for busy practitioners to deposit information in these repositories, it is not necessarily clear how they could be made more useful. Further attention could be given to developing such incentives. In addition, developing better searching and retrieval methods should also facilitate syntheses of grey literature and information, particularly using more sophisticated methods for internet searching such as text analytics or data mining [ 7 , 52 ]. However, if grey information is not recorded in a searchable way (e.g. is retained only on private networks or in memory), this is also only a partial solution. Action is required to improve both information deposition and information retrieval.

Differentiating evidence synthesis from primary research

Although we approached and considered all three of our case studies as evidence syntheses, they could be considered as verging on primary research. This is particularly the case for reviews 2 and 3 where we made attempts to contact all local authorities in England and collect unpublished information via telephone or email interviews with key informants. Contacting authors is encouraged in ‘typical’ systematic reviews, particularly to collect information that may be incompletely recorded in published outputs [ 25 ]. This type of contact is not routinely considered primary research, as the contact is limited to clarifying or augmenting existing published information—rather than collecting entirely new data. However, in many cases in reviews 2 and 3, no published information was available to clarify or augment meaning that these reviews could, perhaps, be considered as collecting new data.

This grey area between evidence synthesis and primary research is particularly important in terms of research ethics. In general, research ethics committee review is not required for evidence synthesis projects because they do not involve research participants [ 53 ]. In line with this, we did not obtain research ethics committee review for any of the case studies described. It is not clear at what point ‘key informants’ become ‘research participants’ and hence when the type of evidence synthesis we conducted in reviews 2 and 3 becomes primary research that does require research ethics committee review. Further consideration, and clarification, of this issue by research ethics organisations would be helpful. In the meantime, and as has been previously proposed, it may be judicious for researchers proposing to conduct this type of work to at least discuss it informally with their local research ethics committee before proceeding [ 54 ].

We propose the term ‘grey information’ to capture a wide range of documented and undocumented information that may be excluded by common definitions of ‘grey literature’. Information on applied public health research questions relating to the nature and range of public health interventions, and many evaluations of these interventions, may be predominantly, or only, held in grey literature and grey information. Evidence syntheses on these topics need, therefore, to embrace grey literature and information. Many standard systematic review methods for searching, appraising, managing, and synthesising the evidence base can be adapted for use with grey literature and information. Evidence synthesisers should carefully consider the opportunities and problems offered by including grey literature and information. Further action to improve both information deposition and retrieval would facilitate more efficient and complete syntheses of grey literature and information.

Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev. 2007;(2):Mr000010. http://www.ncbi.nlm.nih.gov/pubmed/17443631 .

Craig P, Dieppe P, Macintyre S, Mitchie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:979–83.

Article   Google Scholar  

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O’Cathain A, Tinati T, Wight D, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

Pappas C, Williams I. Grey literature: its emerging importance. Journal of Hospital Librarianship. 2011;11(3):228–34.

Franks H, Hardiker NR, McGrath M, McQuarrie C. Public health interventions and behaviour change: reviewing the grey literature. Public Health. 2012;126(1):12–7.

Article   CAS   PubMed   Google Scholar  

Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist synthesis: an introduction. In: ESRC research methods programme. Manchester: University of Manchester; 2004.

Google Scholar  

Turner AM, Liddy ED, Bradley J, Wheatley JA. Modeling public health interventions for improved access to the gray literature. J Med Libr Assoc. 2005;93(4):487–94.

PubMed   PubMed Central   Google Scholar  

Benzies KM, Premji S, Hayden KA, Serrett K. State-of-the-evidence reviews: advantages and challenges of including grey literature. Worldviews Evid Based Nurs. 2006;3(2):55–61.

Article   PubMed   Google Scholar  

Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Research Synthesis Methods. 2014;5(3):221–34.

Schopfel J. Towards a prague definition of grey literature. In: Twelfth International Conference on Grey Literature: Transparency in Grey Literature. Prague, Czech Republic: DANS; 2010.11–26.

Chavez TA, Perrault AH, Reehling P, Crummett C. The impact of grey literature in advancing global karst research: an information needs assessment for a globally distributed interdisciplinary community. Publishing Research Quarterly. 2007;23(1):3–18.

Banks M. Blog posts and tweets: the next frontier for grey literature. In: Darace D, Schopfel J, editors. Grey literature in library and information studies. Berlin: DeGruyter Saur; 2010. p. 217–26.

Godin K, Stapleton J, Kirkpatrick SI, Hanning RM, Leatherdale ST. Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Syst Rev. 2015;4:138.

Article   PubMed   PubMed Central   Google Scholar  

Canadian Agency for Drugs and Technologies in Health. Grey Matters: A Practical Search Tool for Evidence-based Medicine. In: Health CAfDaT, editor. 2014.

Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: a systematic review. Advances in health sciences education : theory and practice. 2009;14(4):595–621.

White S, Fook J, Gardner F. Critical reflection : a review of contemporary literature and understandings. In: White S, Fook J, Gardner F, editors. Critical reflection in health and social care. Maidenhead: Open University Press; 2006. p. 3–20.

Adams J, White M, Moffatt S, Howel D, Mackintosh J. A systematic review of the health, social and financial impacts of welfare rights advice delivered in healthcare settings. BMC Public Health. 2006;6:doi: 10.1186/1471-2458-1186-1181 .

White M, Howell D, Deverill M, Moffatt S, Lock K, McColl E, Milne E, Rubin G, Aspray T. Randomised controlled trial, economic and qualitative process evaluation of domiciliary welfare rights advice for socio-economically disadvantaged older people recruited via primary health care: the Do-Well study. Newcastle upon Tyne: Newcastle University; 2013.

Haighton C, Moffatt S, Howel D, McColl E, Milne E, Deverill M, Rubin G, Aspray T, White M. The Do-Well study: protocol for a randomised controlled trial, economic and qualitative process evaluations of domiciliary welfare rights advice for socio-economically disadvantaged older people recruited via primary health care. BMC Public Health. 2012;12:382.

Adams J, Simpson E, Penn L, Adamson A, White M. Research to support the evaluation and implementation of adult cooking skills interventions in the UK: phase 1 report. Newcastle upon Tyne: Public Health Research Consortium; 2011.

Bardus M, van Beurden SB, Smith JR, Abraham C. A review and content analysis of engagement, functionality, aesthetics, information quality, and change techniques in the most popular commercial apps for weight management. Int J Behav Nutr Phys Act. 2016;13(1):1–9.

Abraham C, Michie S. A taxonomy of behaviour change techniques used in interventions. Health Psychol. 2008;27:379–87.

Rees R, Hinds K, Dickson K, O’Mara-Eves A, Thomas J. Communities that cook: a systematic review of the effectiveness and appropriateness of interventions to introduce adults to home cooking. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2012.

Hillier-Brown FC, Moore HJ, Lake AA, Adamson AJ, White M, Adams J, Araujo-Soares V, Abraham C, Summerbell CD. The effectiveness of interventions targeting specific out-of-home food outlets: protocol for a systematic review. Syst Rev. 2014;3(1):1–5.

Higgins J, Green S (eds.). Cochrane Handbook for Systematic Reviews of Interventions version 5.1. [updated March 2011]. Available from www.cochrane-handbook.org : The Cochrane Collaboration; 2011.

Systematic reviews: CRD’s guidance for undertaking reviews in health care. York: Centre for Reviews and Dissemination, University of York; 2009.

Hammerstrøm K, Wade A, Jørgensen A-MK. Searching for studies: a guide to information retrieval for Campbell systematic reviews. 2010.

Grey literature: home [ http://guides.mclibrary.duke.edu/greyliterature ]. Accessed 19 Sep 2016.

Grey literature for health sciences: getting started [ http://guides.library.ubc.ca/greylitforhealth ]. Accessed 19 Sep 2016.

How to find…: Grey literature [ http://libguides.library.curtin.edu.au/c.php?g=202368&p=1332723 ]. Accessed 19 Sep 2016.

Montori VM, Wilczynski NL, Morgan D, Haynes RB. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2005;330(7482):68.

Haynes RB, McKibbon KA, Wilczynski NL, Walter SD, Werre SR, for the Hedges Team. Optimal search strategies for retrieving scientifically strong studies of treatment from Medline: analytical survey. BMJ. 2005;330:1179.

Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62(9):944–52.

Boluyt N, Tjosvold L, Lefebvre C, Klassen TP, Offringa M. Usefulness of systematic review search strategies in finding child health systematic reviews in MEDLINE. Arch Pediatr Adolesc Med. 2008;162(2):111–6.

Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health. 2005;59(9):804–8.

Roberts K, Cavill N, Rutter H. Standard evaluation framework for weight management interventions. London: National Obesity Observatory; 2009.

Department for Health. The public health outcomes framework for England, 2013-2011. London: Department of Health; 2012.

Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savović J, Schulz KF, Weeks L, Sterne JAC. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Lundh A, Gøtzsche PC. Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies. BMC Med Res Methodol. 2008;8(1):1–9.

Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007;36(3):666–76.

Thomas B, Ciliska D, Dobbins M, Micucci S. A process for systematically reviewing the literature: providing the research evidence for public health nursing interventions. Worldviews Evid Based Nurs. 2004;1(3):176–84.

Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, Petticrew M, Altman DG. Evaluating non-randomised intervention studies. Health technology assessment (Winchester, England) . 2003;7(27):iii-x, 1-173.

Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: realist syntheses. BMC Med. 2013;11(1):21.

Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10 Suppl 1:21–34.

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Medical Research Methodology. 2009;9(59). doi: 10.1186/1471-2288-1189-1159 .

Armstrong R, Hall BJ, Doyle J, Waters E. ‘Scoping the scope’ of a cochrane review. J Public Health. 2011;33(1):147–50.

Schünemann HJ, Moja L. Reviews: rapid! rapid! rapid! …and systematic. Syst Rev. 2015;4(1):1–3.

Polisena J, Garritty C, Kamel C, Stevens A, Abou-Setta AM. Rapid review programs to support health care and policy decision making: a descriptive analysis of processes and methods. Syst Rev. 2015;4(1):1–7.

Featherstone RM, Dryden DM, Foisy M, Guise J-M, Mitchell MD, Paynter RA, Robinson KA, Umscheid CA, Hartling L. Advancing knowledge of rapid reviews: an analysis of results, conclusions and recommendations from published review articles examining rapid reviews. Syst Rev. 2015;4(1):1–8.

Cook AM, Finlay IG, Edwards AGK, Hood K, Higginson IJ, Goodwin DM, Normand CE, Douglas H-R. Efficiency of searching the grey literature in palliative care. J Pain Symptom Manage. 2001;22(3):797–801.

Kothari A, Hovanec N, Hastie R, Sibbald S. Lessons from the business sector for successful knowledge management in health care: a systematic review. BMC Health Services Research. 2011;11(173). doi: 10.1186/1472-6963-11-173 .

Lefebvre C, Glanville J, Wieland LS, Coles B, Weightman AL. Methodological developments in searching for studies for systematic reviews: past, present and future? Syst Rev. 2013;2(1):1–9.

World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA . 2013;310(20):2191-2194.

Vergnes JN, Marchal-Sixou C, Nabet C, Maret D, Hamel O. Ethics in systematic reviews. J Med Ethics. 2010;36(12):771–4.

Download references

Acknowledgements

Not applicable

This paper presents independent research funded by the National Institute for Health Research (NIHR)’s School for Public Health Research (SPHR) with support from Durham and Newcastle Universities and the NIHR Collaboration for Leadership in Applied Health Research and Care of the South West Peninsula (PenCLAHRC). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health.

The School for Public Health Research (SPHR) is funded by the National Institute for Health Research (NIHR). SPHR is a partnership between the Universities of Sheffield, Bristol, Cambridge, and UCL; The London School for Hygiene and Tropical Medicine; The Peninsula College of Medicine and Dentistry; the LiLaC collaboration between the Universities of Liverpool and Lancaster and Fuse; The Centre for Translational Research in Public Health, a collaboration between Newcastle, Durham, Northumbria, Sunderland, and Teesside Universities.

FHB, CDS, HJM, WLW, AA, VAS, and AAL are members of Fuse; JA and MW are funded by the Centre for Diet and Activity Research (CEDAR). Fuse and CEDAR are UKCRC Public Health Research Centre of Excellence. Funding for Fuse and CEDAR from the British Heart Foundation, Cancer Research UK, Economic and Social Research Council, Medical Research Council, and the National Institute for Health Research, under the auspices of the UK Clinical Research Collaboration, is gratefully acknowledged.

The funders played no role in the design of the study, analysis, and interpretation of data, writing the manuscript, or the decision to submit for publication.

Availability of data and materials

This manuscript does not refer to any new data. Of the three case studies that form the focus for the discussion, two have either been previously published and are now in the public domain [ 17 , 20 ] and one is under review (Hillier-Brown F, Summerbell C, Moore H, Wrieden W, Abraham C, Adams J, Adamson A, Araujo-Soares V, White M, Lake A. A description of interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England: a systematic mapping and evidence synthesis. BMC public health 2015, unpublished).

Authors’ contributions

JA and MW (with additional co-authors) conducted case study 1 [ 17 ]; JA and MW (with additional co-authors) conducted case study 2 [ 20 ]; FHB, CS, HJM, JA, VAS, MW, and ALA (with additional co-authors) conducted case study 3 (Hillier-Brown F, Summerbell C, Moore H, Wrieden W, Abraham C, Adams J, Adamson A, Araujo-Soares V, White M, Lake A. A description of interventions promoting healthier ready-to-eat meals (to eat in, to take away, or to be delivered) sold by specific food outlets in England: a systematic mapping and evidence synthesis. BMC public health 2015, unpublished). FHB, CS, HJM, JA, VAS, MW, and ALA contributed to the development of the ideas described in this paper. JA led the writing of the current manuscript. FHB, CS, HJM, JA, VAS, MW, and ALA provided the critical comments on previous drafts of the manuscript and agreed to submit the final version. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Ethics approval and consent to participate.

This is not a primary piece of research, no participants were recruited, and ethical approval was not required.

Author information

Authors and affiliations.

Present address: Centre for Diet and Activity Research, MRC Epidemiology Unit, University of Cambridge, CB2 0QQ, Cambridge, UK

Jean Adams & Martin White

Institute of Health and Society, Newcastle University, NE2 4AA, Newcastle, UK

Jean Adams, Vera Araujo-Soares & Martin White

Fuse, The Centre for Translational Research in Public Health, Newcastle, UK

Jean Adams, Frances C. Hillier-Brown, Helen J. Moore, Amelia A. Lake, Vera Araujo-Soares, Martin White & Carolyn Summerbell

School of Medicine, Pharmacy and Health, Durham University, TS17 3BA, Stockton-on-Tees, UK

Frances C. Hillier-Brown, Helen J. Moore, Amelia A. Lake & Carolyn Summerbell

Centre for Public Policy & Health, Durham University, TS17 6BH, Stockton-on-Tees, UK

Amelia A. Lake

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jean Adams .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Adams, J., Hillier-Brown, F.C., Moore, H.J. et al. Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: critical reflections on three case studies. Syst Rev 5 , 164 (2016). https://doi.org/10.1186/s13643-016-0337-y

Download citation

Received : 23 May 2016

Accepted : 13 September 2016

Published : 29 September 2016

DOI : https://doi.org/10.1186/s13643-016-0337-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Grey literature
• Grey information
• Systematic review
• Evidence synthesis
• Public health
• Interventions

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Grey Literature

• Introduction

About Grey Literature?

Grey literature: pros and cons, some examples of grey literature, introductory video from the university of guelph.

• Finding Grey Literature
• Searching effectively
• Getting Help

The term  grey literature  is used to describe a wide range of different information that is produced outside of traditional publishing and distribution channels, and which is often not well represented in indexing databases.

A widely accepted definition in the scholarly community for grey literature is

"information produced on all levels of government, academia, business and industry in electronic and print formats not controlled by commercial publishing" ie. where publishing is not the primary activity of the producing body." 

From: Third International Conference on Grey Literature in 1997 ( ICGL Luxembourg definition, 1997  - Expanded in New York, 2004).

Uniqueness and currency v quality and longevity

Grey Literature can be a very important research resource.

• It can record findings in niche or emerging research areas and also record research findings that produce null or negative results.  These may not be addressed by the commercial publishers - who may have a more mainstream, profit based publishing strategy.
• It may be more current than formally published research literature which can take time to pass through a sometimes lengthy peer review and editorial publishing process.
• It can connect you with content from a more diverse range of authors and institutions.  Not everyone is able to publish through commercial publication routes

You should be aware of the need to assess and capture potentially useful resources

• Grey Literature sources can vary hugely in terms of quality.
• Scholarly publications such as books and journals pass through a formal publishing process and are subject to in depth editing review. In many cases there is a peer review process where draft versions are subject to a scrutiny process by a panel of experts.  Papers may be modified and improved in light of the expert feedback before reaching final published status. 
• Not all grey literature material is subject to a similarly rigorous pre publication review process, so you should exercise caution and make your own assessment for quality, reliability and potential bias.  See the Evaluating Sources  tutorial for more guidance on this.
• Grey literature is often not formally published. For example, a report may be posted to a government website or a poster presentation may be given at a conference. In these cases you need to consider the longevity of the resource.  Something may be available on the web, or a blog for a short period only and may not be formally archived. 
• Make sure you keep a record of material you wish to use/reference - as it may not be there for discovery later down the line.

A wide, and growing, range of material can be considered as grey literature.  Not all these examples will be relevant to all researchers.    For example, clinical trial information is primarily of interest to health and medical research.  Business researchers will find company and market research information particularly useful.

You should consider which types of information you are interested in before you begin searching as this will help you target and frame your online searching.

Examples include:

• Clinical trials
• Company Information
• Conference papers/proceedings
• Discussion Forums
• Dissertations and theses
• Email discussion lists
• Government documents and reports
• Market reports
• Newsletters
• Policy statements
• Pre print articles
• Press releases
• Research reports
• Statistical Reports
• Survey results
• Working papers

If you prefer an audiovisual introduction, take a look at this short video, produced by the University of Guelph Library.

• << Previous: Introduction
• Next: Finding Grey Literature >>
• Last Updated: Mar 12, 2024 2:36 PM
• URL: https://libguides.exeter.ac.uk/greylit

Systematic Review and Evidence Synthesis

What is grey literature.

Grey (or gray) literature is literature produced by individuals or organizations outside of commercial and/or academic publishers. This can include information such as government reports, conference proceedings, graduate dissertations, unpublished clinical trials, and much more. The sources you select will be informed by your research question and field of study, but should likely include, at a minimum, theses and dissertations.

Why search grey literature?

The intent of an evidence synthesis is to synthesize all available evidence that is applicable to your research question. There is a strong bias in scientific publishing toward publishing studies that show some sort of significant effect. Meanwhile, many studies and trials that show no effect end up going unpublished. But knowing that an intervention had no effect is just as important as knowing that it did have an effect when it comes to making decisions for practice and policy-making. While not peer-reviewed, grey literature represents a valuable body of information that is critical to consider when synthesizing and evaluating all available evidence.

How do I search for grey literature?

Finding grey literature and searching it systematically is challenging. But there are a few approaches that you can take to add some structure to your search of this type of information:

Refer to grey literature sources used for related evidence syntheses . Refer to both published evidence syntheses and registered protocols.

Ask experts in the field for relevant grey literature sources . If you are an expert, include important grey literature sources, and ask colleagues for their recommendations.

Search databases that specialize in grey literature .

This Preprint Repository Syntax Table can help you to identify potential databases to search, and aid you in successfully constructing a search strategy

Search for theses and dissertations : There are a number of databases dedicated to theses and dissertations, which you can search using your search terms.

Search clinical trials : There may be clinical trials being conducted that are relevant to your research question, but that haven't been published yet or never were published.

Identify government agencies and international and non-governmental organizations that might publish technical papers and reports on your topic. Search their websites or any online libraries that they may provide . 

Search conference proceedings and newsletters : Identify professional organizations or conferences at which researchers might be presenting work related to your topic. Search those conference proceedings or newsletters on the organization's website or by contacting organizational boards for access to past proceedings that may not be online.

Contact known researchers in the field about whether there are any relevant ongoing or unpublished studies.

• Search professional and trade magazines. Professional magazines contain literature that is written by professionals in the field for other professionals in the field, but that may not be about research. Trade magazines contain advertisements and news very specific to a topic or industry.

Grey literature resources

Clinical trials.

Clinical trials may go unreported in the published literature. One useful method to identify unpublished clinical trials is to search clinical trials registries. The results may be available within the registries or you may need to contact the researchers associated with the trial for further information.

ClinicalTrials.gov

International Clinical Trials Registry Platform (WHO)

Clinical Trials Registers

Conference Proceedings

This list of conference proceedings is not comprehensive, but are ideas of where to search. We recommend that you search proceedings for the three most relevant conferences for your evidence synthesis.

OCLC PapersFirst OCLC PapersFirst is an OCLC index of papers presented at conferences worldwide.

Dissertations and Theses

Dissertations and theses are potentially rich sources of grey literature and are also usually considered grey literature themselves.

• Dissertations & Theses - How to find resources by format

Grey Literature Databases

WHO Library Database

The WHO library database includes governing documents, reports and technical documentation.

Preprint Repositories

Preprint repositories are free online archives that allow researchers to identify studies in rapidly developing fields and can be used to identify studies that have not gone through the more formalized and traditional peer review process​. Search and export functionality varies from site to site.

Preprints in Europe PMC

Includes bioRxiv, Research Square, Preprints.org, MedRxiv, ChemRxiv, PeerJ Preprints, F1000Res, SSRN, and others. Easy to export the basic citation to citation management software.

Free online archive of health sciences preprints (unpublished manuscripts) including original research articles, systematic reviews and meta-analyses, data articles, and articles describing methodological research/investigations and clinical research design protocols.

OSFPreprints: Preprint Archive Search

Aggregated search that covers preprint servers using the OSF infrastructure, e.g., AfricArXiv, AgriRxiv, Arabixiv, EcoEvoRxiv, EdArXiv, engRxiv, Frenxiv, IndiaRxiv, NutriXiv, PsyArXiv, SocArXiv and others, in addition to other preprint sources, e.g., arXiv, bioRxiv, cogPrints.

"SSRN is a platform for the dissemination of early-stage research. From its initial focus on the social sciences in 1994, SSRN has grown to become the most interdisciplinary service of its kind, representing disciplines across the full research spectrum, including the applied sciences, health sciences, humanities, life sciences, physical sciences, and social sciences."

Open access to over 1.9M e-prints in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.

bioRxiv: The Preprint Server for Biology

bioRxiv (bio-archive) is a free online archive and distribution service for unpublished preprints in the life sciences. Articles are not peer-reviewed, edited, or typeset before being posted online.

Preprint archive for engineering articles.

Research Preprints: Server List [Google Sheet]

Public Policy

​The World Bank now makes all of their publications openly available online.

WHO Institutional Repository for Information Sharing (IRIS)

Institutional WHO database of intergovernmental policy documents and technical reports. Can search by IRIS by region (Africa, Americas, Eastern Mediterranean, Europe, South-East Asia, Western Pacific).

Health Research Web

A wiki with descriptors of national health research systems, ethics review committees, local and regional policies and research priorities and more.

Other Resources

Grey Matters: A practical search tool for evidence-based medicine

This online manual provides a thorough list of sources for grey literature in medicine and a helpful checklist to help systematize your process.

Searching the grey literature: A handbook for searching reports, working papers, and other unpublished research

Searching the Grey Literature is for librarians and information professionals interested in learning more about grey literature. This book will aid with crafting a grey lit search successfully, from start to finish. Many types of librarians will find the content of this book useful, particularly those in health or social science.

Duke University Medical Center Guide to Resource for Searching the Grey Literature

Resources for trial registries, pharmacological studies, conference abstracts, government documents and more.

Gray Literature Resources for Agriculture Evidence Syntheses

This is a comprehensive list of agricultural grey literature sources, one of several evidence synthesis worksheets and tools provided to help in non-medical evidence synthesis development and implementation.

Searching for studies: A guide to information retrieval for Campbell Systematic Reviews

This document gives comprehensive guidance in searching for evidence syntheses, and offers a robust list of grey literature sources for social science disciplines.

• << Previous: Modify Search to Other Databases
• Next: Citation Management >>

• University of Michigan Library
• Research Guides

Grey Literature

• Strategies for Finding Grey Literature
• Conferences
• Clinical Trials/Regulatory Data
• Reports & Reviews
• Other Resources

Library Contact

General Information

Grey literature is the unpublished, non-commercial, hard-to-find information that organizations such as professional associations, research institutes, think tanks, and government departments produce.

It can be invaluable to your research: it is part of the overall evidence base and functions as an alternative source that may be used to overcome possible bias presented by published information.

While the bulk of the information here is related to the health sciences, there are general resources (databases for disserations and Scopus and Web of Science, which provide broad coverage of subjects), plus links to more technical (i.e., engineering) and hard science resources on the Other Resources page.

• Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions: AHRQ and the Effective Health Care Program From the Agency for Healthcare Research & Quality (AHRQ).
• Grey Literature Report NYAM's bi-monthly report identifying new publications in health sciences grey literature has ceased publication, but materials from 1999 to 2016 are still available.
• Grey Literature-Producing Organizations From the New York Academy of Medicine (NYAM).
• GreySource Index Provides categorized examples of web based grey literature.

How to Search for Grey Literature

Searching for grey literature can be time consuming because it is not usually included in bibliographic databases. Therefore, knowing how to search correctly for grey literature can help you use your time as effectively as possible.

On the Strategies for Finding Grey Literature page , you'll find a quick introduction to ways of thinking about searching for grey literature.

Remember, don't hesitate to ask your informationists if you need help.

Types of Grey Literature

• Blog posts & Tweets
• Census data (research guide)
• Clinical Trials
• Conference Abstracts & Proceedings
• Dissertations/Theses
• Registered Research
• Technical Reports & Systematic Reviews

About Grey Literature: Getting Started

Types of grey literature.

• Census Data
• Conference Papers/Proceedings
• Dissertations
• Government Documents
• Technical Reports
• White Papers

Producers of Grey Literature

• Government Agencies
• Institutional Repositories
• Online Repositories
• Research Centers/Think Tanks 
• Academic Conferences
• Non-governmental Organizations (NGOs)
• Intergovernmental Organizations (IGOs)
• Funding Agencies/Research Organizations    
• Professional Associations/Societies

Science & Engineering Librarian

Search Duke University Libraries for Grey Literature:

What is "Grey Literature"?

Grey literature stands for manifold document types produced on all levels of government, academics, business and industry in print and electronic formats that are protected by intellectual property rights, of sufficient quality to be collected and preserved by library holdings or institutional repositories, but not controlled by commercial publishers i.e., where publishing is not the primary activity of the producing body

The definition of "Grey Literature" was provided by the 12th International Conference on Grey Literature (Prague, 2010) 

Key Grey LiteratureTerms

• Charles P. Auger is credited with coining the term "Grey Literature" li> Used originally to refer to intelligence reports and notes on atomic research
• A part of new channels of scientific communication in a changing research environment
• The term “grey literature” was first used in 1978 at a conference held by British Lending

Effective Google Search Strategies for Grey Literature

• Helps you find credible sources other than peer-reviewed articles and books published from academic press (i.e., grey/gray literature)
• "Domain Limiters" (.gov, .edu, .org., .mil)
• Allows you to limit your search to within a site and/or domain

Benefits of Grey Literature

• Helps to prevent publication bias
• Quicker availability of information
• Reports neutral or negative results
• Diversifies available research literature

Citation Styles for Grey Literature

Search grey literature sources, recommended grey lit citation managers.

• Organization of sources
• Curation of full-text PDF files 
• Creation of bibliographies in your required citation style format
• Insert citations and bibliographies directly into your word document
• Last Updated: Mar 11, 2024 11:06 PM
• URL: https://guides.library.duke.edu/c.php?g=1369174

Services for...

• Faculty & Instructors
• Graduate Students
• Undergraduate Students
• International Students
• Patrons with Disabilities

• Harmful Language Statement
• Re-use & Attribution / Privacy
• Support the Libraries

Systematic Reviews and Meta-Analyses: Grey Literature

• Get Started
• Exploratory Search
• Where to Search
• How to Search

Grey Literature

• What about errata and retractions?
• Eligibility Screening
• Critical Appraisal
• Data Extraction
• Synthesis & Discussion
• Assess Certainty
• Share & Archive

This page provides more information about grey literature.

Conference Proceedings

Grey Literature is a broad term that varies across discipline. In very general terms, grey literature is anything other than peer-reviewed, empirical research. Examples of grey literature include: unpublished research, government publications, theses and dissertations, statistics and datasets, news articles, trade and popular magazines, blogs, podcasts, social media posts (e.g., tweets), slide decks, or factsheets. Including grey literature in your systematic review can help mitigate publication bias.

According to  GreyNet, 2013 :

Grey Literature...deals with the production, distribution, and access to multiple document types produced on all levels of government, academics, business, and organization in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body.

According to Paez, 2017 :

Publication bias refers to the propensity for only studies reporting positive findings to be published, and may skew the results of the meta-analysis and systematic review. As gray literature may describe neutral or negative results, including it with commercially published research may provide a more balanced understanding of the evidence and a more accurate effect sizes ...excluding unpublished studies may compromise the validity and reliability of meta-analyses and the specificity of systematic reviews.

Should you include grey literature? 

Can a type (or several types) of grey literature  help to answer your research question ? Do you have time for additional searches and processing additional material? If so, you should consider including grey literature in your review!

However, not all questions will benefit from grey literature and you cannot include  all  kinds of grey literature.

Wilson (2009) states:

"Ignoring the grey literature on the basis that it is of inferior methodological quality is empirically and logically invalid . Furthermore, the use of publication status as a proxy measure for methodological quality abdicates the important role of carefully evaluating the robustness of the evidence to others using unknown criteria (i.e., the journal peer-review process)."

Ćurković M, Košec A. Bubble effect: including internet search engines in systematic reviews introduces selection bias and impedes scientific reproducibility. BMC Med Res Methodol. 2018 Nov 13;18(1):130. doi: 10.1186/s12874-018-0599-2. PMID: 30424741; PMCID: PMC6234590.

Godin, K., Stapleton, J., Kirkpatrick, S.I.  et al.   Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada.   Syst Rev   4 , 138 (2015). https://doi.org/10.1186/s13643-015-0125-0

Gusenbauer M, Haddaway NR. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources . Res Synth Methods. 2020 Mar;11(2):181-217. doi: 10.1002/jrsm.1378. Epub 2020 Jan 28. PMID: 31614060; PMCID: PMC7079055

Mahood, Q., Van Eerd, D., & Irvin, E. (2014). Searching for grey literature for systematic reviews: Challenges and benefits: MAHOOD ET AL . Research Synthesis Methods , 5 (3), 221–234. https://doi.org/10.1002/jrsm.1106

Paez A.  Grey literature: An important resource in systematic reviews . J Evid Based Med. 2017 Dec 21. doi: 10.1111/jebm.12265. Epub ahead of print. PMID: 29266844.

Check out  Appendix I  for an annotated  list of grey lit sources!

Scherer, R.W., Saldanha, I.J.  How should systematic reviewers handle conference abstracts? A view from the trenches .   Syst Rev   8,  264 (2019). https://doi-org.ezproxy.lib.vt.edu/10.1186/s13643-019-1188-0

Wilson, D.B. Missing a critical piece of the pie: simple document search strategies inadequate for systematic reviews .  J Exp Criminol   5,  429–440 (2009). https://doi-org.ezproxy.lib.vt.edu/10.1007/s11292-009-9085-5

Conference proceedings are some of the most common types of grey literature to consider in systematic reviews. Many of the databases you'll search  will  include conference proceedings . But there are databases that contain only conference proceedings such as  ProceedingsFirst , as well as databases containing the conference proceedings for a specific conference organization (which may require affiliation to access). If you plan to include a specific type of grey literature, like conference proceedings, it is not enough to include what you happen to find through your primary search. You should also search purposefully for this type of literature by targeting focused databases, just as you did when searching for peer-reviewed empirical research.

According to  PRISMA-S, Item 4:

Including conference proceedings in a systematic review search helps  minimize bias ...specify the conference names , the dates of conferences included, and the method used to search the proceedings...

Scherer & Saldanha, 2019  offer a more pragmatic approach , suggesting reviewers consider their capacity and corpus:

If available evidence is  sparse or conflicting , it may be worthwhile to include conference abstracts. If results from conference abstracts are included, then it is necessary to make diligent attempts to  contact the authors  of the abstract and  examine study registers  and  published protocols  to obtain further and confirmatory information on methods and results.

This is important as, in a Cochrane Network Meta-Analysis,  Hackenbroich, Kranke, Meybohm, & Weibel (2022)  found:

Most conference abstracts on PONV were insufficiently reported regarding review's narrow inclusion criteria and could not be included in NMA. The resource-intensive search and evaluation of abstracts did not substantially extent the full-text evidence base of the review, given the few adequately reported abstracts. Conferences should oblige authors to adhere to CONSORT for abstracts.

• << Previous: How to Search
• Next: What about errata and retractions? >>
• Last Updated: Mar 28, 2024 2:54 PM
• URL: https://guides.lib.vt.edu/SRMA

• Grey Literature

The definitions of grey literature can be wide-ranging. The International Conference on Grey Literature (ICGL), the primary source for scholarly research disseminating the topic of grey literature, proposed this definition in 2010:

Grey literature (or gray literature) refers to information that falls outside the mainstream of published journal and monograph literature, and which is not controlled by commercial publishers. Grey literature is usually is not widely disseminated or promoted and may not be peer-reviewed. Grey literature can sometimes be more current than traditional sources, but the  quality may be variable  and all sources must be  c ritically evaluated. Searching the grey literature provides comprehensiveness and may help minimize publication bias as negative result trials or null trial results are less likely to be published in journal literature but can often be found in grey literature. 

Grey literature can include:

• Conference abstracts or papers
• Government or private sector research
• Clinical trials - ongoing or unpublished
• Communication with experts and researchers in the field
• Newsletters and annual reports
• Patent applications
• Drug information
• Technical reports and whitepapers
• Theses and Dissertations
• Blogs, listserv archives, tweets
• Podcasts and videos
• Images, maps, and spatial data
• Websites, repositories, digital libraries

Bonato, S. (2018).  Searching the Grey Literature . Rowman & Littlefield Publishers.

Evaluating Grey Literature:

• Grey Matters: a practical tool for searching health-related grey literature A free online resource with checklist for grey literature searching from CADTH (Canadian Agency for Drugs and Technologies in Health)
• AACODS Checklist A useful tool for evaluating grey literature from Flinders University
• Public Health Ontario Guide to Appraising Grey Literature

Grey Literature Sources

Some grey literature sources to consider:

• Government websites
• Non profit websites
• Professional association websites
• Institutional repositories such as Adelphi University's Scholarly Works
• Preprint servers such as  Medrxiv  or  Biorxiv
• AHRQ  - Agency for Healthcare Research and quality
• ClinicalTrials.gov  - U.S. and international federally and privately supported clinical trials registry and results database
• Cochrane Central Register of Controlled Trials (CENTRAL ) A highly concentrated source of reports of randomized and quasi-randomized controlled trials
• Dissertations and Theses Global  - Dissertation and theses database 
• ERIC  - Educational Resources Information Center -  online library of education research and information, sponsored by the Institute of Education Sciences of the U.S. Department of Education
• Google Scholar  - Search engine for scholarly literature across many disciplines and sources, including theses, books, abstracts and articles
• ISRCTN.org   A clinical trial registry recognized by the WHO and ICMJE that accepts planned, ongoing or completed studies of any design. Find and view clinical studies. Search by condition, intervention, funder, demographics, etc. 
• MedNar A free, medically-focused deep web search engine providing a search of authoritative public and deep web resources, returning the most relevant results to one easily navigable page. It is a US focused tool offering a Google like interface.
• OAIster  Union catalog of millions of records that represent open access resources.
• OpenDOAR  - Directory of academic repositories
• Open Grey  - System for information on grey literature in Europe. Open access to 700,000 references to the grey literature.
• TRIP Database  Clinical search engine for evidence-based medicine
• WorldCat  - Collection of library catalogs
• World Health Organization  (WHO) - Provides leadership on global health matters, shapes the health research agenda, sets norms and standards, articulates evidence-based policy options, provides technical support to countries and monitors and assesses health trends
• WHO International Clinical Trials Registry Platform  

Common sources of grey literature in healthcare:

Image source: C oad, J., Hardicre, J., & Devitt, P. (2006). Searching for and using grey literature . Nursing Times , 102 (50), 35-36. https://www.nursingtimes.net/careers/your-nursing-career/searching-for-and-using-grey-literature-12-12-2006/

• << Previous: Selected Databases
• Next: Handsearching >>
• Types of Questions
• Key Features and Limitations
• Is a Systematic Review Right for Your Research?
• Integrative Review
• Scoping Review
• Rapid Review
• Meta-Analysis/Meta-Synthesis
• Selecting a Review Type
• Reducing Bias
• Guidelines for Student Researchers
• Training Resources
• Register Your Protocol
• Handbooks & Manuals
• Reporting Guidelines
• PRESS 2015 Guidelines
• Search Strategies
• Selected Databases
• Handsearching
• Citation Searching
• Study Types & Terminology
• Quantitative vs. Qualitative Research
• Critical Appraisal of Studies
• Broad Functionality Programs & Tools
• Search Strategy Tools
• Deduplication Tools
• CItation Screening
• Critical Appraisal Tools
• Quality Assessment/Risk of Bias Tools
• Data Collection/Extraction
• Meta Analysis Tools
• Books on Systematic Reviews
• Finding Systematic Review Articles in the Databases
• Systematic Review Journals
• More Resources
• Evidence-Based Practice Research in Nursing
• Citation Management Programs
• Last Updated: Apr 5, 2024 2:54 PM
• URL: https://libguides.adelphi.edu/Systematic_Reviews

Tutorial: Evaluating Information: Gray Literature

• Evaluating Information
• Scholarly Literature Types
• Primary vs. Secondary Articles
• Peer Review
• Systematic Reviews & Meta-Analysis
• Gray Literature
• Evaluating Like a Boss
• Evaluating AV

Grey literature or types of non-formally published substantive literature

What is gray literature.

Gray (or grey) literature is literature produced by individuals or organizations outside of commercial and/or academic publishers. This type of non-formally published substantive information (often not formally peer-reviewed; especially important in all kinds of sciences) can include information such:

• theses and dissertations
• technical reports  
• working papers 
• government reports
• evaluation and think tank reports and resources
• conference proceedings, papers and posters
• publications from NGOs, INGOs, think tanks and policy institutes
• unpublished clinical trials
• and much more

The sources you select will be informed by your research question and field of study, but should likely include, at a minimum, theses and dissertations.

Why Search the Gray Literature?

Most of gray literature is considered less prestigious, reliable, and "official" than publication in a peer-reviewed journal. But they are still fully legitimate avenues of publication. Often they are used to publicize early findings, before a study is entirely complete. Or, in the case of theses, they are published as a condition of receiving an advanced degree. Government technical reports are issued either by agencies that do scientific research themselves or else by a lab that has received government funding. Increasingly, such labs may be required to publish technical reports as a condition of receiving such funding. Gray literature may be cited like any other paper although with the caveat mentioned before that it is considered less "official" and reliable than peer-reviewed scientific papers.

When doing evidence synthesiis, it's important because the intent is to synthesize  all available evidence  that is applicable to your research question. There is a strong bias in scientific publishing toward publishing studies that show some sort of significant effect. Meanwhile, many studies and trials that show no effect end up going unpublished. But knowing that an intervention had no effect is just as important as knowing that it did have an effect when it comes to making decisions for practice and policy-making. While not peer-reviewed, gray literature represents a valuable body of information that is critical to consider when synthesizing and evaluating all available evidence.

How to Manage the Grey Literature Search (particularly for evidence synthesis)

• Identify and record  the sources you will search. The sources you search will be informed by your research question and where you expect to find information related to your question.
• Document where you are searching and your search strategies, including document resource name, URL, search terms, and date searched.
• Collect citation information as you go.
• Adhere to your established inclusion and exclusion criteria when selecting sources.

Technical Reports

What are technical reports ? Technical reports present facts and conclusions about technological designs and projects. Typically, a technical report includes research about concepts as well as graphical depictions of designs and data. A technical report also follows a strict organization. Thus, when engineers read it, they can quickly locate the information they need.1

A technical report is a form of grey literature : Reports or documents produced by academia, government, industry or nonprofit organizations describing their work, proposals and/or the challenges they face. Grey literature is copyrighted like most other published work, but it usually is not distributed by commercial publishers the way a book or periodical would be but rather by the authoring organization itself. The organization that publishes the report is usually considered the 'author' (even if named individuals are credited within the report).

Some sources of technical reports:

• Sample technical reports on renewable energy Prof. Ahlers has provided these specific examples of technical reports produced by corporations and organizations regarding proposals for renewable energy projects. Take a look.
• OSTI SciTech Connect The Department of Energy (DOE) Office of Scientific and Technical Information (OSTI) maintain SciTech Connect, a portal to free, publicly available Dept. of Energy research and development results.
• NTRS - The NASA Technical Reports Server Collected from U.S. and international sources, NTRS is a collection of information that includes over 4 million bibliographic records and a growing number of full-text documents.

Preprints and Working Papers

Working papers are preliminary works, released to share ideas or invite discussion and feedback, often prior to the submission of a paper to a peer-reviewed journal or conference. Economists have traditionally relied on working papers as a method for the timely and informal communication of recent research findings.

• Bureau of Economic Analysis Working Papers
• Federal Reserve Bank of St. Louis Working Papers
• IDEAS (Hosted by RePEc) "IDEAS is the largest bibliographic database dedicated to Economics and available freely on the Internet. Based on RePEc, it indexes over 3,000,000 items of research, including over 2,700,000 that can be downloaded in full text."
• Economics Research Network
• NBER Working Papers NBER researchers initially report their findings in these scientific papers aimed at other professional economists. Nearly 700 NBER Working Papers are published each year, and many subsequently appear in scholarly journals.
• LawArXiv LawArXiv is a free, open access repository for legal scholarship, maintained and owned by legal scholars and law librarians for the benefit of the legal community.

How to Search the Grey Literature (particularly for evidence synthesis)

Finding gray literature and searching it systematically is challenging. But there are a few approaches that you can take to add some structure to your search of this type of information:

• Search   databases that specialize in gray literature: See the box below for more information.
• Search for theses and dissertations: There are a number of databases dedicated to theses and dissertations, which you can search using your search terms. See the box below for links to these resources. 
• Search clinical trials: There may be clinical trials being conducted that are relevant to your research question, but that haven't been published yet or never were published. See the box below for links to these resources. 
• Identify government agencies and international and non-governmental organizations that might publish technical papers and reports on your topic. Search their websites or any online libraries that they may provide. See the box below for links to some examples.
• Search conference proceedings and newsletters: Identify professional organizations that have and/or conferences at which researchers might be presenting work related to your topic. Search those conference proceedings or newsletters on the organization's website or by contacting organizational boards for access to past proceedings that may not be online. See the box below for some examples.
• Contact known researchers in the field to determine if there are any ongoing or unpublished studies that s/he may be aware of.
• Search professional and trade magazines . Professional magazines contain literature that is written by professionals in the field for other professionals in the field, but that may not be about research. Trade magazines contain advertisements and news very specific to a topic or industry.

Other Useful Guides

• Technical reports (NASA, DOE, etc.) by Jill Powell Last Updated Aug 31, 2023 289 views this year
• Patent Searching by Jill Powell Last Updated Feb 22, 2024 508 views this year
• Dissertations and Theses: A Finding Guide by Michael Engle Last Updated Jan 22, 2024 1142 views this year
• Dissertations & Theses in Engineering by Jill Powell Last Updated May 4, 2022 110 views this year
• Standards & Building Codes by Jill Powell Last Updated Jan 18, 2024 178 views this year

Grey Literature Sources

• Gray Literature Databases
• Theses and Dissertations
• Clinical Trials
• Public Policy
• Conference Proceedings
• Other Resources
• OpenGrey: System for Information on Grey Literature in Europe The System for Information on Grey Literature in Europe is an open access database to 700,000 bibliographical references of gray literature produced in Europe and allows you to export records and locate the documents in many research disciplines. OpenGrey covers Science, Technology, Biomedical Science, Economics, Social Science and Humanities topics.
• NY Academy of Medicine Grey Literature Report This report is a bimonthly publication of The New York Academy of Medicine (NYAM) alerting readers to new gray literature publications in health services research and selected public health topics. The database platform is keyword searchable and serves as an archive for the cataloged reports.
• WHO Library Database The WHO library database includes governing documents, reports and technical documentation.
• MedNar MedNar searches across more than 60 medical research sources, including commercial databases, medical societies, NIH resources, and other government resources.
• ProQuest Dissertations and Theses With more than 2 million entries, PQD&T is the single central, authoritative resource for information about doctoral dissertations and master's theses.
• Networked Digital Library of Theses and Dissertations (NDLT) Free international resource for theses and dissertations.
• Center for Research Libraries A resource for institutions outside of the U.S. and Canada.
• OCLC WorldCat Dissertations and Theses "This database provides fast and convenient access to the dissertations and theses available in OCLC member libraries. Many theses are available electronically, at no charge, directly from the publishing institution."
• Guide to Theses and Dissertations Resources This guide provides information on finding and accessing both Cornell and non-Cornell theses and dissertations, including many resources for non-US literature.
• EThOS Electronic Theses Online Service contains doctoral theses from candidates in the UK.
• ClinicalTrials.gov The US registry for clinical trials. Includes new, ongoing, and completed human clinical trials both in the US and countries around the world.
• Cochrane Central Register of Controlled Trials A highly concentrated source of reports of randomized and quasi-randomized controlled trials. The majority of CENTRAL records are taken from bibliographic databases (mainly MEDLINE and Embase), but records are also derived from other published and unpublished sources.
• EU Clinical Trials Register The European Union Clinical Trials Register allows you to search for protocol and results information on interventional clinical trials that are conducted in the European Union (EU) and the European Economic Area (EEA) and clinical trials conducted outside the EU / EEA that are linked to European paediatric-medicine development.
• Australia New Zealand Clinical Trials Registry Online register of clinical trials being undertaken in Australia, New Zealand and elsewhere. The ANZCTR includes trials from the full spectrum of therapeutic areas of pharmaceuticals, surgical procedures, preventive measures, lifestyle, devices, treatment and rehabilitation strategies, and complementary therapies.
• WHO International Clinical Trials Registry Platform ICTRP provides access to a central database containing the trial registration data sets provided by numerous international registries. It also provides links to the full original records.
• Wikipedia Clinical Trial Registries List This Wikipedia entry contains a list of links to individual clinical trial registries by country.
• World Bank ​The World Bank now makes all of their publications openly available online.
• WHO Institutional Repository for Information Sharing (IRIS) Institutional WHO database of intergovernmental policy documents and technical reports. Can search by IRIS by region (Africa, Americas, Eastern Mediterranean, Europe, South-East Asia, Western Pacific).
• Health Research Web A wiki with descriptors of national health research systems, ethics review committees, local and regional policies and research priorities and more.
• OCLC PapersFirst OCLC PapersFirst is an OCLC index of papers presented at conferences worldwide.
• BIOSIS Previews A subcollection of Web of Science, BIOSIS Previews searches across journals, meetings, patents, and books in the life sciences and biomedical fields.
• Grey Matters: A practical search tool for evidence-based medicine This online manual provides a thorough list of sources for gray literature in medicine and a helpful checklist to help systematize your process.
• Googling the Greys: Tips for Searching Beyond Health Databases and Turning Information into Insights This presentation by Sarah Bonato, a health science librarian at the Centre for Addiction and Mental Health in Toronto, ON, provides an excellent guide to searching Google effectively to find gray literature.
• Duke University Medical Center Guide to Resource for Searching the Grey Literature Resources for trial registries, pharmacological studies, conference abstracts, government document and more.

Grey Literature Resources for International Agriculture & Development

Not all websites are created equal! Major nongovernmental organizations and various scholarly associations can be rich and authoritative websites, especially for the kind of grey literature not published in mainstream scholarly journals (including white papers, evaluation reports, training materials, and the like). Below are a few useful places to start.

• Food and Agriculture Organization of the United Nations
• United Nations
• United Nations Development Programme
• International Development Research Centre
• CGIAR (Consultative Group on International Agricultural Research)
• IFPRI (International Food Policy Research Institute)
• USAID Agriculture portal
• USAID Development Experience Clearinghouse
• DFID (Department for International Development)
• Microfinance Gateway
• Cornell International Institute for Food, Agriculture, and Development (CIIFAD) This rich site also has a good section of other international resources at Cornell.
• Institute of Development Studies (UK)
• USAID Data Resources
• International Initiative for Impact Evaluation "3ie funds impact evaluations and systematic reviews that generate high quality evidence on what works in development and why. Evidence on development effectiveness can inform policy and improve the lives of poor people."
• British Library for Development Studies
• Development Gateway
• << Previous: Systematic Reviews & Meta-Analysis
• Next: Reading Science Info >>
• Last Updated: Oct 20, 2021 11:11 AM
• URL: https://guides.library.cornell.edu/evaluate