University of Tasmania, Australia
Systematic reviews for health: tools for systematic review.
- Handbooks / Guidelines for Systematic Reviews
- Standards for Reporting
- Registering a Protocol
- Tools for Systematic Review
- Online Tutorials
- Books and Articles about Systematic Reviews
- Finding Systematic Reviews
- Critical Appraisal
- Library Help
- Bibliographic Databases
- Grey Literature
- Handsearching
- Citation Tracking
- 1. Formulate the Research Question
- 2. Identify the Key Concepts
- 3. Develop Search Terms - Free-Text
- 4. Develop Search Terms - Controlled Vocabulary
- 5. Search Fields
- 6. Phrase Searching, Wildcards and Proximity Operators
- 7. Boolean Operators
- 8. Search Limits
- 9. Pilot Search Strategy & Monitor Its Development
- 10. Final Search Strategy
- 11. Adapt Search Syntax
- Documenting Search Strategies
- Handling Results & Storing Papers
Tools for Systematic Reviews
Managing the selection process can be challenging, particularly in a large-scale systematic review that involves multiple reviewers. There are various free and subscription-based tools available that support the study selection process ( Cochrane Handbook, 4.6.6.1 ).
This page describes various tools available to help conduct a systematic review. The University of Tasmania has access to EndNote, Covidence and JBI SUMARI.
Covidence is an online systematic review program developed by, and for, systematic reviewers. It can import citations from reference managers like EndNote, facilitate the screening of abstracts and full-text, populate risk of bias tables, assist with data extraction, and export to all common formats.
Covidence Demo video [3:24]
Covidence is a core component of Cochrane's review production toolkit and has also been endorsed by JBI.
Access to UTAS Covidence account
If you are the project leader, follow these steps to create a UTAS Covidence account:
- Create a UTAS Covidence account Step-by-step instructions on how to create at UTAS Covidence account
Once you have created your UTAS Covidence account, you can create a review and invite others to join the review.
If you are not the project leader, please wait for your invitation from your project leader to join the review (you don't need to create a UTAS Covidence account).
Covidence Training & Help
- Need some help getting started? Covidence offers monthly Covidence 101 training webinars . You also have the option to listen to the recording of a recent session.
- Work through the content of the Covidence Academy .
- Subscribe to the Covidence YouTube channel to find video tutorials and recorded webinars.
- Head to the Covidence knowledge base to get answers to FAQs.
- Contact [email protected] anytime to answer any of your questions.
Abstrackr is a software for semi-automated abstract screening for systematic reviews. At present, Abstrackr is a free, open-source tool for facilitating the citation screening process. Upload your abstracts, invite reviewers, and get to screening!
Rayyan is a free online tool that can be used for screening and coding of studies in a systematic review. It uses tagging and filtering to code and organise references.
The System for the Unified Management, Assessment and Review of Information ( SUMARI ) is JBI 's software for the systematic review of literature.
I t is designed to assist researchers to conduct systematic reviews and facilitates the entire review process. SUMARI supports 10 review types. It is especially useful for new review types and qualitative reviews.
University of Tasmania researchers have access to SUMARI via the JBI EBP Database under EBP Tools .
- JBI EBP Database via Ovid This link opens in a new window
SUMARI support:
- JBI SUMARI Video Tutorials
- JBI SUMARI Knowledge Base
RevMan 5 is the software used for preparing and maintaining Cochrane Reviews. RevMan facilitates preparation of protocols and full reviews, including text, characteristics of studies, comparison tables, and study data. It can perform meta-analysis of the data entered, and present the results graphically.
RevMan 5 is no longer being developed, but they continue to support Cochrane authors.
RevMan Web is the next generation of Cochrane's software for preparing and maintaining systematic reviews. This web-based version of RevMan works across all platforms, is installation-free, and automatically updated.
DistillerSR
DistillerSR is a systematic review software. It was designed from the ground up to provide a better review experience, faster project completion and transparent, audit-ready results.
What can you do in DistillerSR? Upload your references from any reference management software, create screening and data extraction forms, lay out workflow and assign reviewers, monitor study progress and review process, export results (incl PRISMA flowchart automation).
This software is more sophisticated and a bit harder to learn. DistillerSR attracts a fee .
The Systematic Review Toolbox is a community-driven, searchable, web-based catalogue of tools that support the systematic review process across multiple domains. The resource aims to help reviewers find appropriate tools based on how they provide support for the systematic review process. Users can perform a simple keyword search (i.e. Quick Search) to locate tools, a more detailed search (i.e. Advanced Search) allowing users to select various criteria to find specific types of tools and submit new tools to the database.
Need More Help? Book a consultation with a Learning and Research Librarian or contact [email protected] .
- << Previous: Registering a Protocol
- Next: Online Tutorials >>
- Last Updated: Mar 17, 2023 1:58 PM
- URL: https://utas.libguides.com/SystematicReviews
- Research Guides
- Health Science Libraries
Systematic Reviews & Other Review Types
Systematic review tools.
- Systematic Review Service
- Databases & Grey Literature
- Tools for Critical Appraisal
- What is a Systematic Review?
- What is a Rapid Review?
- What is a Scoping Review?
- What is a Mapping Review?
- What is a Meta-Synthesis?
- What is a Mixed Methods Review?
- What is an Integrative Review?
- What is an Overview of Reviews?
- What is a Review of Complex Interventions?
- What is a Diagnostic Test Accuracy Review?
- What is a Network Meta-Analysis?
- What is a Living Systematic Review?
This guide focuses on free or open source tools.
Directories
- SR Toolbox Community driven directory of systematic review tools.
- Finding the Reporting Guidelines
General Tools
Helpful for multiple stages of the review process.
- DistillerSR Free student program available & discounted faculty program.
- Covidence Only the trial version is free. Helpful for screening through data abstraction.
- Review Manager or RevMan
- PICO Portal First review is free
Specific Tools for Planning the Review
Reporting Guidelines: Not sure what to report in your review, locate a reporting guideline.
- PRISMA Checklist
- PRISMA 2020 Checklist (ShinyApp) Generate a 2020 checklist and download in PDF or Word
- PRISMA Flow Diagram
- PRISMA Flow Diagram Generator
- PRISMA-S extension for search reporting
- JBI Manual for Evidence Synthesis
- MOOSE-Reporting of Systematic Reviews of Observational studies
- Campbell Collaboration Equity Checklist for SR Authors A tool for authors who are considering equity or studying disadvantaged populations in their review.
Protocol Registration: A first step to organize the review and prevent bias is creating and registering a protocol.
- PRISMA-P extension Statement-Guidance for creating protocols
- PredicTER tool Predict how long your review will take.
- Methods Wizard A tool that helps with writing the systematic review protocol using semi-automation
- PROSPERO Designed as a registry for systematic reviews or meta-analysis and it is free to search what reviews have been registered and to register a protocol.
- OSF or Open Science Framework A free and open-source cloud based research management platform to researchers to store files and data & register preprints.
- SYREAF protocols/registration Systematic Reviews for Animals & Food
- Joanna Briggs Institute Systematic Review Register
- Campbell Collaboration-Register a protocol
Organizing & Collaborating with a Team: Teamwork is core to the systematic review.
- Conceptboard
- Google drive-Create a shared drive
- Loom Video Free for small teams, may require a subscription for larger teams. This is a team communication tool.
Tools for Literature Searching
Not interested in conducting your own search? You may use our systematic review search service instead!
- PubMed Pub ReMiner Text mining for PubMed
- Yale MeSH Analyzer or Analyze MeSH
- MeSH on Demand Identifies MeSH terms in your submitted text (abstract or manuscript) and lists similar articles.
- Medline Transpose Translate a search query between Ovid MEDLINE and PubMed syntax
- Litsearchr Shinyapps version of Litsearchr (no coding)
- SWIFT-Review Text mining tool specific to systematic reviews.
- Polyglot Search Helps to translate search strategies to other database (excluding subject headings).
- SnowGlobe Assists with literature searching, using the snowballing (searching through reference lists) technique. OSF demonstration available in OSF at https://osf.io/jth83/.
- Voyant General text mining (this is the download) for the web version go to http://voyant-tools.org
- TERMINE General text mining
- JSTOR Text Analyzer Recommends journal articles in JSTOR relevant to text.
- Oxford English Dictionary May be useful to locate additional terms or synonyms.
- Visuwords A visual dictionary and thesaurus
- Citationchaser An R package for forward and backward citation chasing in academic searching.
- Scite Easily check how a scientific article has been cited and if its findings have been supported or contrasted by others.
- RCT Tagger Helps identify human randomized controlled trials
- Connected Papers Enter a paper and it will build a graph of similar papers in the field.
Citation Managers & Deduplication Software
- Endnote Online Note: the desktop version is more robust for deduplication (subscription fees apply)
Tools for Screening
Did you know screening includes 2 or more reviewers and must be done independent and blinded from the other reviewer(s)?
- Rayyan-a Guide for Researchers (Temple University Libraries)
- Abstrackr Semi-automated screening using machine learning.
- Sysrev Helps facilitate screening, collaborative extraction of data from academic articles and abstracts, PDF documents, and uses machine learning. The free platform supports open access/public projects only. Fee for privacy settings.
- RobotAnalyst RobotAnalyst is designed for searching and screening reference collections obtained from literature database queries. It combines search engine functionality with machine learning and text mining.
- ASReview Utilizes active learning
- Colandr Machine learning, natural language processing, and text-mining functions to partially automate screening.
- DistillerSR (free with student program) Includes AI feature to help rank screening. Predict screening time with time averages.
- SyRF Screening tool for in-vivo or preclinical studies from CAMARADES for systematic reviews of animal data from experimental studies
- revtools revtools is an R package to support researchers working on evidence synthesis projects.
Screening References using EndNote (requires a subscription)
- Managing and Coding References for Systematic Reviews and Scoping Reviews in EndNote
Data Extraction
- Overview of data extraction tools-Chart
- What data to collect and tools (Cochrane Handbook 2019)
- SRDR+ Free data extraction/management tool
- SRDR+ course module
- Systematic Review Data Repository (SRDR) tool for the extraction and management of data for systematic review or meta-analysis. It is also an open and searchable archive of systematic reviews and their data
- REDCap No cost for Temple Health, build online surveys and databases quickly and securely.
- RAx (30 day free trial, discounted subscription for students) Smart assistant to explore, collate insights and organize research.
- AirTable This is a cloud-based spreadsheet/database hybrid. It is free or by subscription for more advanced features. It has many uses beyond systematic reviews.
- MSKTC's Using REDCap for Systematic Reviews guide
- DistillerSR (free with student program) Create customizable templates for data abstraction or choose from a template. Gives you full control of your data.
- Robot Reviewer It automatically extracts and synthesises data from Randomized Controlled Trials by dragging and dropping PDFs or by uploading the files.
- Cochrane Data Collection Form for RCTs and NON RCTs
- Plot Digitizer A java program that can extract data from graphs and is available as a free download.
- Grade Pro Helps create the SoF Summary of Findings Table and incorporating the GRADE Approach to help make recommendations of the level of evidence.
- Cochrane sample data collection form RCTs and non-RCTs
- Cochrane Data Collection Form for RCTs only
- Sample Data Collection Form for Non Randomized Studies only
Tools for Meta-Analysis
- DistillerSR Forest Plot Generator from Evidence Partners
- Meta Essentials The tool consists of a set of workbooks designed for Microsoft Excel.
- MetaLight Is a software application designed to support the teaching and learning of meta-analysis.
- Metafor The metafor package is a free and open-source add-on for conducting meta-analyses with the statistical software environment R.
- OpenMeta[Analyst] Completely open-source, cross-platform software for advanced meta-analysis.
- OpenMEE Open-source software for performing meta-analysis suited to the need of ecologist and evolutionary biologists.
- Statistical software discounts for Temple University
Tools for Writing
- RevManHAL An extension to RevMan-to automate the writing of systematic reviews (created by the Cochrane Collaboration Schizophrenia Group).
- GradeProof Automated high-quality spelling, grammar and rephrasing corrections using artificial intelligence (AI) to improve the flow of your writing. Free and subscription plans available.
Tools for Language Translation
- DeepL Limited free translation, may require a subscription.
Tools for Creating Risk of Bias Figures
- robvis Web app designed for visualizing risk-of-bias assessments. The tool creates: “traffic light” plots of the domain-level judgements for each individual result; and weighted bar plots of the distribution of risk-of-bias judgements within each bias domain.
- << Previous: Databases & Grey Literature
- Next: Tools for Critical Appraisal >>
- Last Updated: Mar 14, 2023 1:48 PM
- URL: https://guides.temple.edu/systematicreviews
Temple University
University libraries.
See all library locations
- Library Directory
- Locations and Directions
- Frequently Called Numbers
Need help? Email us at [email protected]
Start your free trial
Arrange a trial for your organisation and discover why FSTA is the leading database for reliable research on the sciences of food and health.
REQUEST A FREE TRIAL
- Research Skills Blog
5 software tools to support your systematic review processes
By Dr. Mina Kalantar on 19-Jan-2021 13:01:01
Systematic reviews are a reassessment of scholarly literature to facilitate decision making. This methodical approach of re-evaluating evidence was initially applied in healthcare, to set policies, create guidelines and answer medical questions.
Systematic reviews are large, complex projects and, depending on the purpose, they can be quite expensive to conduct. A team of researchers, data analysts and experts from various fields may collaborate to review and examine incredibly large numbers of research articles for evidence synthesis. Depending on the spectrum, systematic reviews often take at least 6 months, and sometimes upwards of 18 months to complete.
The main principles of transparency and reproducibility require a pragmatic approach in the organisation of the required research activities and detailed documentation of the outcomes. As a result, many software tools have been developed to help researchers with some of the tedious tasks required as part of the systematic review process.
hbspt.cta._relativeUrls=true;hbspt.cta.load(97439, 'ccc20645-09e2-4098-838f-091ed1bf1f4e', {"useNewLoader":"true","region":"na1"});
The first generation of these software tools were produced to accommodate and manage collaborations, but gradually developed to help with screening literature and reporting outcomes. Some of these software packages were initially designed for medical and healthcare studies and have specific protocols and customised steps integrated for various types of systematic reviews. However, some are designed for general processing, and by extending the application of the systematic review approach to other fields, they are being increasingly adopted and used in software engineering, health-related nutrition, agriculture, environmental science, social sciences and education.
Software tools
There are various free and subscription-based tools to help with conducting a systematic review. Many of these tools are designed to assist with the key stages of the process, including title and abstract screening, data synthesis, and critical appraisal. Some are designed to facilitate the entire process of review, including protocol development, reporting of the outcomes and help with fast project completion.
As time goes on, more functions are being integrated into such software tools. Technological advancement has allowed for more sophisticated and user-friendly features, including visual graphics for pattern recognition and linking multiple concepts. The idea is to digitalise the cumbersome parts of the process to increase efficiency, thus allowing researchers to focus their time and efforts on assessing the rigorousness and robustness of the research articles.
This article introduces commonly used systematic review tools that are relevant to food research and related disciplines, which can be used in a similar context to the process in healthcare disciplines.
These reviews are based on IFIS' internal research, thus are unbiased and not affiliated with the companies.
This online platform is a core component of the Cochrane toolkit, supporting parts of the systematic review process, including title/abstract and full-text screening, documentation, and reporting.
The Covidence platform enables collaboration of the entire systematic reviews team and is suitable for researchers and students at all levels of experience.
From a user perspective, the interface is intuitive, and the citation screening is directed step-by-step through a well-defined workflow. Imports and exports are straightforward, with easy export options to Excel and CVS.
Access is free for Cochrane authors (a single reviewer), and Cochrane provides a free trial to other researchers in healthcare. Universities can also subscribe on an institutional basis.
Rayyan is a free and open access web-based platform funded by the Qatar Foundation, a non-profit organisation supporting education and community development initiative . Rayyan is used to screen and code literature through a systematic review process.
Unlike Covidence, Rayyan does not follow a standard SR workflow and simply helps with citation screening. It is accessible through a mobile application with compatibility for offline screening. The web-based platform is known for its accessible user interface, with easy and clear export options.
Function comparison of 5 software tools to support the systematic review process
Eppi-reviewer.
EPPI-Reviewer is a web-based software programme developed by the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI) at the UCL Institute for Education, London .
It provides comprehensive functionalities for coding and screening. Users can create different levels of coding in a code set tool for clustering, screening, and administration of documents. EPPI-Reviewer allows direct search and import from PubMed. The import of search results from other databases is feasible in different formats. It stores, references, identifies and removes duplicates automatically. EPPI-Reviewer allows full-text screening, text mining, meta-analysis and the export of data into different types of reports.
There is no limit for concurrent use of the software and the number of articles being reviewed. Cochrane reviewers can access EPPI reviews using their Cochrane subscription details.
EPPI-Centre has other tools for facilitating the systematic review process, including coding guidelines and data management tools.
CADIMA is a free, online, open access review management tool, developed to facilitate research synthesis and structure documentation of the outcomes.
The Julius Institute and the Collaboration for Environmental Evidence established the software programme to support and guide users through the entire systematic review process, including protocol development, literature searching, study selection, critical appraisal, and documentation of the outcomes. The flexibility in choosing the steps also makes CADIMA suitable for conducting systematic mapping and rapid reviews.
CADIMA was initially developed for research questions in agriculture and environment but it is not limited to these, and as such, can be used for managing review processes in other disciplines. It enables users to export files and work offline.
The software allows for statistical analysis of the collated data using the R statistical software. Unlike EPPI-Reviewer, CADIMA does not have a built-in search engine to allow for searching in literature databases like PubMed.
DistillerSR
DistillerSR is an online software maintained by the Canadian company, Evidence Partners which specialises in literature review automation. DistillerSR provides a collaborative platform for every stage of literature review management. The framework is flexible and can accommodate literature reviews of different sizes. It is configurable to different data curation procedures, workflows and reporting standards. The platform integrates necessary features for screening, quality assessment, data extraction and reporting. The software uses Artificial Learning (AL)-enabled technologies in priority screening. It is to cut the screening process short by reranking the most relevant references nearer to the top. It can also use AL, as a second reviewer, in quality control checks of screened studies by human reviewers. DistillerSR is used to manage systematic reviews in various medical disciplines, surveillance, pharmacovigilance and public health reviews including food and nutrition topics. The software does not support statistical analyses. It provides configurable forms in standard formats for data extraction.
DistillerSR allows direct search and import of references from PubMed. It provides an add on feature called LitConnect which can be set to automatically import newly published references from data providers to keep reviews up to date during their progress.
The systematic review Toolbox is a web-based catalogue of various tools, including software packages which can assist with single or multiple tasks within the evidence synthesis process. Researchers can run a quick search or tailor a more sophisticated search by choosing their approach, budget, discipline, and preferred support features, to find the right tools for their research.
If you enjoyed this blog post, you may also be interested in our recently published blog post addressing the difference between a systematic review and a systematic literature review.
- FSTA - Food Science & Technology Abstracts
- IFIS Collections
- Ask an Expert
- Resources Hub
- Privacy policy
- Diversity Statement
- IFIS Sustainability Commitment
- Company news
- Frequently Asked Questions
Ground Floor, 115 Wharfedale Road, Winnersh Triangle, Wokingham, Berkshire RG41 5RB
Get in touch with IFIS
© International Food Information Service (IFIS Publishing) operating as IFIS – All Rights Reserved | Charity Reg. No. 1068176 | Limited Company No. 3507902 | Designed by Blend
- Skip to main content
- Skip to secondary menu
Software Tools for Conducting Systematic Reviews
Several programs support more than one step in the systematic review process.
Full-Featured Software Tools for Conducting Systematic Reviews
- EPPI-Reviewer 4 : EPPI-Reviewer is web-based software that supports reference management, screening, coding and synthesis. It is developed by the Evidence for Policy and Practice Information and Coordinating Centre in London. Pricing is based on a subscription model.
- Covidence : Covidence supports screening and data extraction. It is developed by a not-for-profit organization with partners in Australia and the UK. It is currently free of charge, but will be moving to a subscription system.
- DistillerSR : DistillerSR is web-based software that primarily focuses on screening and data extraction. It is developed by Evidence Partners in Canada. Pricing is based on a subscription model.
- System for the Unified Management, Assessment and Review of Information (SUMARI) : SUMARI is composed of a Comprehensive Review Management System (CReMS) that works with several modules that together support review of qualitative, quantitative, economic, and textual evidence. CReMs is available for Windows or Mac operating systems. The software is developed by the Joanna Briggs Institute in Australia. It is available through Lippincott for an annual subscription of $30.00.
- Rayyan QCRI : Rayyan is a 100% FREE web application to help systematic review authors perform their job in a quick, easy and enjoyable fashion. Authors create systematic reviews, collaborate on them, maintain them over time and get suggestions for article inclusion.
- SysRev : sysrev is a platform for collaborative extraction of data from academic articles and abstracts, PDF documents, and other entities.
Back to: Resources for Conducting Systematic Reviews
Quick Search
Advanced search.
Select stages of the review you want support with:
- Main navigation
- Footer menu
- Basic info & contact footer
Top navigation: Contact, Library account
- : PennKey Library Account
- Log in to Franklin with PennKey
- Do you need help because you have No PennKey? Click here.
- Library Home
- Libraries & hours
- Make an appointment
Penn Libraries: Guides
Systematic reviews: tools for data.
- Literature Search
- Manage Results
- Tools for Data
- Our Service
- Covidence Software
Risk of Bias Assessment Tools
JAMA Evidence Users Guides (Pennkey is needed)
- Cochrane Collaboration's tool for assessing risk of bias
- NEWCASTLE - Ottawa Quality Assessment Scale
Tools for Critical appraisal
- Joanna Briggs Institute Critical Appraisal Tools These tools are freely accessible and recommended for use when conducting different types of systematic reviews. The site provides checklists for a variety of studies.
Tools for Managing Data
Covidence
Covidence is "the primary screening and data extraction tool for Cochrane authors. Its support of key steps in the Cochrane Review process, cush as citation screening and Risk of bias assessment, and improved links with RevMan make the review writing process more efficient."
DistillerSR
- DistillerSR is an online application designed specifically for the screening and data extraction phases of a systematic review.
- The system automatically handles promotion and exclusion based on your form design
- Export data to any spreadsheet or database software
- Easy protocol changes, the system can re-evaluate all user responses in seconds
A free web application that assists systematic review authors in performing literature screening.
Software for semi-automatic citation screening. The web-based annotation tool, currently in beta version, allows project leads to import the citations that are to be screened for a review from either RefMan or Pubmed.
Export references from EndNote to Abstrackr
Systematic Review Toolbox
A resource to identify software tools for various steps in the systematic review process. It is a community-driven, searchable, web-based catalogue of tools that support the systematic review process across multiple domains.
J BI SUMARI
This tool "facilitates the entire review process, from protocol development, team management, study selection, critical appraisal, data extraction, data synthesis and writing systematic review report". It support a number of review types.
UPenn faculty, staff, and students can access freely this tool via Ovid platform, e.g., Ovid Medline, Ovid JBI.
Systematic Review Data Repository (SRDR)
From AHRQ, the Systematic Review Data Repository (SRDR) is a powerful and easy-to-use tool for the extraction and management of data for systematic review or meta-analysis. It is also an open and searchable archive of systematic reviews and their data.
"You can use RevMan for protocols and full reviews. It is most useful when you have formulated the question for the review, and allows you to prepare the text, build the tables showing the characteristics of studies and the comparisons in the review, and add study data. It can perform meta-analyses and present the results graphically."
Download RevMan
OpenMeta[Analyst ]
This is completely open-source, cross-platform software for advanced meta-analysis.
EPPI
The tool is designed to help those conducting systematic reviews to identify, extract and code information about a particular research study that is to be included in a systematic review. Other tools related are also available. Main tools include
- Guidelines tools - i) the REPOSE Guidelines for the reporting of education research; and ii) detailed sets of questions for coding at the Keywording and Data-extraction stage
- Data management tools - structures and tools for downloading, storing and managing information on studies used in the reviews
- EPPI-Reviewer - the EPPI-Centre's online tool for research synthesis
- MetaLight - which supports the teaching and learning of meta-analysis
- CCEMG - EPPI-Centre Cost Converter - a tool for assisting with currency conversion.
Google Search for “Evidence Table”
Data analysis tools for meta analysis
Xmeta is an R package and also an online platform to facilitate comprehensive meta-analysis for users with or without programming skills. It features two analytic paths: 1) for R users, you can install the “xmeta” package and directly call the main functions; and 2) for people who do not use R, you can use the web-based secure meta-analysis pipeline to personalize your own analysis. It includes a wide variety of analyses for univariate, multivariate and network meta-analysis, for continuous, binary and time to event outcomes. In addition, it also includes a rich set of model diagnosis tools and data visualizations for different types of analyses.
- << Previous: Manage Results
- Next: Our Service >>
- Last Updated: Mar 14, 2023 2:14 PM
- URL: https://guides.library.upenn.edu/SR
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
- Publications
- Account settings
- Advanced Search
- Journal List
- JMIR Med Inform
- v.10(5); 2022 May
Web-Based Software Tools for Systematic Literature Review in Medicine: Systematic Search and Feature Analysis
Kathryn cowie.
1 Nested Knowledge, Saint Paul, MN, United States
Asad Rahmatullah
Nicole hardy, kevin kallmes, associated data.
Supplementary Table 1: Screening Decisions for SR (systematic review) Tools Reviewed in Full.
Supplementary Table 2: Inter-observer Agreement across (1) Systematic Review (SR) Tools and (2) Features Assessed.
Systematic reviews (SRs) are central to evaluating therapies but have high costs in terms of both time and money. Many software tools exist to assist with SRs, but most tools do not support the full process, and transparency and replicability of SR depend on performing and presenting evidence according to established best practices.
This study aims to provide a basis for comparing and selecting between web-based software tools that support SR, by conducting a feature-by-feature comparison of SR tools.
We searched for SR tools by reviewing any such tool listed in the SR Toolbox, previous reviews of SR tools, and qualitative Google searching. We included all SR tools that were currently functional and required no coding, and excluded reference managers, desktop applications, and statistical software. The list of features to assess was populated by combining all features assessed in 4 previous reviews of SR tools; we also added 5 features (manual addition, screening automation, dual extraction, living review, and public outputs) that were independently noted as best practices or enhancements of transparency and replicability. Then, 2 reviewers assigned binary present or absent assessments to all SR tools with respect to all features, and a third reviewer adjudicated all disagreements.
Of the 53 SR tools found, 55% (29/53) were excluded, leaving 45% (24/53) for assessment. In total, 30 features were assessed across 6 classes, and the interobserver agreement was 86.46%. Giotto Compliance (27/30, 90%), DistillerSR (26/30, 87%), and Nested Knowledge (26/30, 87%) support the most features, followed by EPPI-Reviewer Web (25/30, 83%), LitStream (23/30, 77%), JBI SUMARI (21/30, 70%), and SRDB.PRO (VTS Software) (21/30, 70%). Fewer than half of all the features assessed are supported by 7 tools: RobotAnalyst (National Centre for Text Mining), SRDR (Agency for Healthcare Research and Quality), SyRF (Systematic Review Facility), Data Abstraction Assistant (Center for Evidence Synthesis in Health), SR Accelerator (Institute for Evidence-Based Healthcare), RobotReviewer (RobotReviewer), and COVID-NMA (COVID-NMA). Notably, of the 24 tools, only 10 (42%) support direct search, only 7 (29%) offer dual extraction, and only 13 (54%) offer living/updatable reviews.
Conclusions
DistillerSR, Nested Knowledge, and EPPI-Reviewer Web each offer a high density of SR-focused web-based tools. By transparent comparison and discussion regarding SR tool functionality, the medical community can both choose among existing software offerings and note the areas of growth needed, most notably in the support of living reviews.
Introduction
Systematic review costs and gaps.
According to the Centre for Evidence-Based Medicine, systematic reviews (SRs) of high-quality primary studies represent the highest level of evidence for evaluating therapeutic performance [ 1 ]. However, although vital to evidence-based medical practice, SRs are time-intensive, taking an average of 67.3 weeks to complete [ 2 ] and costing leading research institutions over US $141,000 in labor per published review [ 3 ]. Owing to the high costs in researcher time and complexity, up-to-date reviews cover only 10% to 17% of primary evidence in a representative analysis of the lung cancer literature [ 4 ]. Although many qualitative and noncomprehensive publications provide some level of summative evidence, SRs—defined as reviews of “evidence on a clearly formulated question that use systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included” [ 5 ]—are distinguished by both their structured approach to finding, filtering, and extracting from underlying articles and the resulting comprehensiveness in answering a concrete medical question.
Software Tools for Systematic Review
Software tools that assist with central SR activities—retrieval (searching or importing records), appraisal (screening of records), synthesis (content extraction from underlying studies), and documentation/output (presentation of SR outputs)—have shown promise in reducing the amount of effort needed in a given review [ 6 ]. Because of the time savings of web-based software tools, institutions and individual researchers engaged in evidence synthesis may benefit from using these tools in the review process [ 7 ].
Existing Studies of Software Tools
However, choosing among the existing software tools presents a further challenge to researchers; in the SR Toolbox [ 8 ], there are >240 tools indexed, of which 224 support health care reviews. Vitally, few of these tools can be used for each of the steps of SR, so comparing the features available through each tool can assist researchers in selecting an SR tool to use. This selection can be informed by feature analysis; for example, a previously published feature analysis compared 15 SR tools [ 9 ] across 21 subfeatures of interest and found that DistillerSR (Evidence Partners), EPPI-Reviewer (EPPI-Centre), SWIFT-Active Screener (Sciome), and Covidence (Cochrane) support the greatest number of features as of 2019. Harrison et al [ 10 ], Marshall et al [ 11 ], and Kohl et al [ 12 ] have completed similar analyses, but each feature assessment selected a different set of features and used different qualitative feature assessment methods, and none covered all SR tools currently available.
The SR tool landscape continues to evolve; as existing tools are updated, new software is made available to researchers, and new feature classes are developed. For instance, despite the growth of calls for living SRs, that is, reviews where the outputs are updated as new primary evidence becomes available, no feature analysis has yet covered this novel capability. Furthermore, the leading feature analyses [ 9 - 12 ] have focused on the screening phase of review, meaning that no comparison of data extraction capabilities has yet been published.
Feature Analysis of Systematic Review Tools
The authors, who are also the developers of the Nested Knowledge platform for SR and meta-analysis (Nested Knowledge, Inc) [ 13 ], have noted the lack of SR feature comparison among new tools and across all feature classes (retrieval, appraisal, synthesis, documentation/output, administration of reviews, and access/support features). To provide an updated feature analysis comparing SR software tools, we performed a feature analysis covering the full life cycle of SR across software tools.
Search Strategy
We searched the SR tools for assessment in 3 ways: first, we identified any SR tool that was published in existing reviews of SR tools (Table S1 in Multimedia Appendix 1 ). Second, we reviewed SR Toolbox [ 8 ], a repository of indexed software tools that support the SR process. Third, we performed a Google search for Systematic review software and identified any software tool that was among the first 5 pages of results. Furthermore, for any library resource pages that were among the search results, we included any SR tools mentioned by the library resource page that met our inclusion criteria. The search was completed between June and August 2021. Four additional tools, namely SRDR+ (Agency for Healthcare Research and Quality), Systematic Review Assistant-Deduplication Module (Institute for Evidence-Based Healthcare), Giotto Compliance, and Robotsearch (Robotsearch), were assessed in December 2021 following reviewer feedback.
Selection of Software Tools
The inclusion and exclusion criteria were determined by 3 authors (KK, KH, and KC). Among our search results, we queued up all software tools that had descriptions meeting our inclusion criteria for full examination of the software in a second round of review. We included any that were functioning web-based tools that require no coding by the user to install or operate, so long as they were used to support the SR process and can be used to review clinical or preclinical literature. The no coding requirement was established because the target audience of this review is medical researchers who are selecting a review software to use; thus, we aim to review only tools that this broad audience is likely to be able to adopt. We also excluded desktop applications, statistical packages, and tools built for reviewing software engineering and social sciences literature, as well as reference managers, to avoid unfairly casting these tools as incomplete review tools (as they would each score quite low in features that are not related to reference management). All software tools were screened by one reviewer (KC), and inclusion decisions were reviewed by a second (KK).
Selection of Features of Interest
We built on the previous comparisons of SR tools published by Van der Mierden et al [ 9 ], Harrison et al [ 10 ], Marshall et al [ 11 ], and Kohl et al [ 12 ], which assign features a level of importance and evaluate each feature in reference screening tools. As the studies by Van der Mierden et al [ 9 ] and Harrison et al [ 10 ] focus on reference screening, we supplemented the features with features identified in related reviews of SR tools (Table S1 in Multimedia Appendix 1 ). From a study by Kohl et al [ 12 ], we added database search, risk of bias assessment (critical appraisal), and data visualization. From Marshall et al [ 11 ], we added report writing.
We added 4 more features based on their importance to software-based SR: manual addition of records, automated full-text retrieval, dual extraction of studies, risk of bias (critical appraisal), living SR, and public outputs. Each addition represents either a best practice in SR [ 14 ] or a key feature for the accuracy, replicability, and transparency of SR. Thus, in total, we assessed the presence or absence of 30 features across 6 categories: retrieval, appraisal, synthesis, documentation/output, administration/project management, and access/support.
We adopted each feature unless it was outside of the SR process, it was required for inclusion in the present review, it duplicated another feature, it was not a discrete step for comparison, it was not necessary for English language reviews, it was not necessary for a web-based software, or it related to reference management (as we excluded reference managers from the present review). Table 1 shows all features not assessed, with rationale.
Features from systematic reviews not assessed in this review, with rationale.
Feature Assessment
To minimize bias concerning the subjective assessment of the necessity or desirability of features or of the relative performance of features, we used a binary assessment where each SR tool was scored 0 if a given feature was not present or 1 if a feature was present. Tools were assessed between June and August 2021. We assessed 30 features, divided into 6 feature classes. Of the 30 features, 77% (23/30) were identified in existing literature, and 23% (7/30) were added by the authors ( Table 2 ).
The criteria for each selected feature, as well as the rationale.
a API: application programming interface.
b Rationale only provided for features added in this review; all other features were drawn from existing feature analyses of Systematic Review Software Tools.
c RIS: Research Information System.
d PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
e AI: artificial intelligence.
Evaluation of Tools
For tools with free versions available, each of the researchers created an account and tested the program to determine feature presence. We also referred to user guides, publications, and training tutorials. For proprietary software, we gathered information on feature offerings from marketing webpages, training materials, and video tutorials. We also contacted all proprietary software providers to give them the opportunity to comment on feature offerings that may have been left out of those materials. Of the 8 proprietary software providers contacted, 38% (3/8) did not respond, 50% (4/8) provided feedback on feature offerings, and 13% (1/8) declined to comment. When providers provided feedback, we re-reviewed the features in question and altered the assessment as appropriate. One provider gave feedback after initial puplication, prompting issuance of a correction.
Feature assessment was completed independently by 2 reviewers (KC and AR), and all disagreements were adjudicated by a third (KK). Interobserver agreement was calculated using standard methods [ 19 ] as applied to binary assessments. First, the 2 independent assessments were compared, and the number of disagreements was counted per feature, per software. For each feature, the total number of disagreements was counted and divided by the number of software tools assessed. This provided a per-feature variability percentage; these percentages were averaged across all features to provide a cumulative interobserver agreement percentage.
Identification of SR Tools
We reviewed all 240 software tools offered on SR Toolbox and sent forward all studies that, based on the software descriptions, could meet our inclusion criteria; we then added in all software tools found on Google Scholar. This strategy yielded 53 software tools that were reviewed in full ( Figure 1 shows the PRISMA [Preferred Reporting Items for Systematic Reviews and Meta-Analyses]-based chart). Of these 53 software tools, 55% (29/53) were excluded. Of the 29 excluded tools, 17% (5/29) were built to review software engineering literature, 10% (3/29) were not functional as of August 2021, 7% (2/29) were citation managers, and 7% (2/29) were statistical packages. Other excluded tools included tools not designed for SRs (6/29, 21%), desktop applications (4/29, 14%), tools requiring users to code (3/29, 10%), a search engine (1/29, 3%), and a social science literature review tool (1/29, 3%). One tool, Research Screener [ 20 ], was excluded owing to insufficient information available on supported features. Another tool, the Health Assessment Workspace Collaborative, was excluded because it is designed to assess chemical hazards.
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)-based chart showing the sources of all tools considered for inclusion, including 2-phase screening and reasons for all exclusions made at the full software review stage. SR: systematic review.
Overview of SR Tools
We assessed the presence of features in 24 software tools, of which 71% (17/24) are designed for health care or biomedical sciences. In addition, 63% (15/24) of the analyzed tools support the full SR process, meaning they enable search, screening, extraction, and export, as these are the basic capabilities necessary to complete a review in a single software tool. Furthermore, 21% (5/34) of the tools support the screening stage ( Table 3 ).
Breakdown of software tools for systematic review by process type (full process, screening, extraction, or visualization; n=24).
Data Gathering
Interobserver agreement between the 2 reviewers gathering data features was 86.46%, meaning that across all feature assessments, the 2 reviewers disagreed on <15% of the applications. Final assessments are summarized in Table 4 , and Table S2 in Multimedia Appendix 2 shows the interobserver agreement on a per–SR tool and per-feature basis. Interobserver agreement was ≥70% for every feature assessed and for all SR tools except 3: LitStream (ICF; 53.3%), RevMan Web (Cochrane; 50%), and SR Accelerator (Institute for Evidence-Based Healthcare; 53.3%); on investigation, these low rates of agreement were found to be due to name changes and versioning (LitStream and RevMan Web) and due to the modular nature of the subsidiary offerings (SR Accelerator). An interactive, updatable visualization of the features offered by each tool is available in the Systematic Review Methodologies Qualitative Synthesis.
Feature assessment scores by feature class for each systematic review tool analyzed. The total number of features across all feature classes is presented in descending order.
Giotto Compliance (27/30, 90%), DistillerSR (26/30, 87%), and Nested Knowledge (26/30, 87%) support the most features, followed by EPPI-Reviewer Web (25/30, 83%), LitStream (23/30, 77%), JBI SUMARI (21/30, 70%), and SRDB.PRO (VTS Software) (21/30, 70%).
The top 16 software tools are ranked by percent of features from highest to lowest in Figure 2 . Fewer than half of all features are supported by 7 tools: RobotAnalyst (National Centre for Text Mining), SRDR (Agency for Healthcare Research and Quality), SyRF (Systematic Review Facility), Data Abstraction Assistant (Center for Evidence Synthesis in Health, Institute for Evidence-Based Healthcare), SR-Accelerator, RobotReviewer (RobotReviewer), and COVID-NMA (COVID-NMA; Table 3 ).
Stacked bar chart comparing the percentage of supported features, broken down by their feature class (retrieval, appraisal, extraction, output, admin, and access), among all analyzed software tools.
Feature Assessment: Breakout by Feature Class
Of all 6 feature classes, administrative features are the most supported, and output and extraction features are the least supported ( Figure 3 ). Only 3 tools, Covidence (Cochrane), EPPI-Reviewer, and Giotto Compliance, offer all 4 extraction features ( Table 4 ). DistillerSR and Giotto support all 5 retrieval features, while Nested Knowledge supports all 5 documentation/output features. Colandr, DistillerSR, EPPI-Reviewer, Giotto Compliance, and PICOPortal support all 6 appraisal features.
Heat map of features observed in 24 analyzed software tools. Dark blue indicates that a feature is present, and light blue indicates that a feature is not present.
Feature Class 1: Retrieval
The ability to search directly within the SR tool was only present for 42% (10/24) of the software tools, meaning that for all other SR tools, the user is required to search externally and import records. The only SR tool that did not enable importing of records was COVID-NMA, which supplies studies directly from the providers of the tool but does not enable the user to do so.
Feature Class 2: Appraisal
Among the 19 tools that have title/abstract screening, all tools except for RobotAnalyst and SRDR+ enable dual screening and adjudication. Reference deduplication is less widespread, with 58% (14/24) of the tools supporting it. A form of machine learning/automation during the screening stage is present in 54% (13/24) of the tools.
Feature Class 3: Extraction
Although 75% (18/24) of the tools offer data extraction, only 29% (7/24) offer dual data extraction (Giotto Compliance, DistillerSR, SRDR+, Cadima [Cadima], Covidence, EPPI-Reviewer, and PICOPortal [PICOPortal]). A total of 54% (13/24) of the tools enable risk of bias assessments.
Feature Class 4: Output
Exporting references or collected data is available in 71% (17/24) of the tools. Of the 24 tools, 54% (13/24) generate figures or tables, 42% (10/24) of tools generate PRISMA flow diagrams, 32% (8/24%) have report writing, and only 13% (3/34) have in-text citations.
Feature Class 5: Admin
Protocols, customer support, and training materials are available in 71% (17/24), 79% (19/24), and 83% (20/24) of the tools, respectively. Of all administrative features, the least well developed are progress/activity monitoring, which is offered 67% (16/24) of the tools, and comments, which are available in 58% (14/24) of the tools.
Feature Class 6: Access
Access features cover both collaboration during the review, cost, and availability of outputs. Of the 24 software tools, 83% (20/24) permit collaboration by allowing multiple users to work on a project. COVID-NMA, RobotAnalyst, RobotReviewer, and SR-Accelerator do not allow multiple users. In addition, of the 24 tools, 71% (17/24) offer a free subscription, whereas 29% (7/24) require paid subscriptions or licenses (Covidence, DistillerSR, EPPI-Reviewer Web, Giotto Compliance, JBI Sumari, SRDB.PRO, and SWIFT-Active Screener). Only 54% (13/24) of the software tools support living, updatable reviews.
Principal Findings
Our review found a wide range of options in the SR software space; however, among these tools, many lacked features that are either crucial to the completion of a review or recommended as best practices. Only 63% (15/24) of the SR tools covered the full process from search/import through to extraction and export. Among these 15 tools, only 67% (10/15) had a search functionality directly built in, and only 47% (7/15) offered dual data extraction (which is the gold standard in quality control). Notable strengths across the field include collaborative mechanisms (offered by 20/24, 83% tools) and easy, free access (17/24, 71% of tools are free). Indeed, the top 4 software tools in terms of number of features offered (Giotto Compliance, DistillerSR, Nested Knowledge, and EPPI-Reviewer all offered between 83% and 90% of the features assessed. However, major remaining gaps include a lack of automation of any step other than screening (automated screening offered by 13/24, 54% of tools) and underprovision of living, updatable outputs.
Major Gaps in the Provision of SR Tools
Marshall et al [ 11 ] have previously noted that “the user should be able to perform an automated search from within the tool which should identify duplicate papers and handle them accordingly” [ 11 ]. Less than a third of tools (7/24, 29%) support search, reference import, and manual reference addition.
Study Selection
Screening of references is the most commonly offered feature and has the strongest offerings across features. All software tools that offer screening also support dual screening (with the exception of RobotAnalyst and SRDR+). This demonstrates adherence to SR best practices during the screening stage.
Automation and Machine Learning
Automation in medical SR screening has been growing. Some form of machine learning or other automation for screening literature is present in over half (13/24, 54%) of all the tools analyzed. Machine learning/screening includes reordering references, topic modeling, and predicting inclusion rates.
Data Extraction
In contrast to screening, extraction is underdeveloped. Although extraction is offered by 75% (18/24) tools, few tools adhere to SR best practices of dual extraction. This is a deep problem in the methods of review, as the error rate for manual extraction without dual extraction is highly variable and has even reached 50% in independent tests [ 16 ].
Although single extraction continues to be the only commonly offered method, the scientific community has noted that automating extraction would have value in both time savings and improved accuracy, but the field is as of yet underdeveloped. To quote a recent review on the subject of automated extraction, “[automation] techniques have not been fully utilized to fully or even partially automate the data extraction step of systematic review” [ 21 ]. The technologies to automate extraction have not achieved partial extraction at a sufficiently high accuracy level to be adopted; therefore, dual extraction is a pressing software requirement that is unlikely to be surpassed in the near future.
Project Management
Administrative features are well supported by SR software. However, there is a need for improved monitoring of review progress. Project monitoring is offered by 67% (16/24) of the tools, which is among the lowest of all admin features and likely the feature most closely associated with the quality of the outputs. As collaborative access is common and highly prized, SR software providers should recognize the barriers to collaboration in medical research; lack of mutual awareness, inertia in communication, and time management and capacity constraints are among the leading reasons for failure in interinstitutional research [ 22 ]. Project monitoring tools could assist with each of these pain points and improve the transparency and accountability within the research team.
Living Reviews
The scientific community has made consistent demands for SR processes to be rendered updatable, with the goal of improving the quality of evidence available to clinicians, health policymakers, and the medical public [ 23 , 24 ]. Despite these ongoing calls for change, living, updatable reviews are not yet standard in SR software tools. Only 54% (13/24) of the tools support living reviews, largely because living review depends on providing updatability at each step up through to outputs. However, until greater provision of living review tools is achieved, reviews will continue to fall out of date and out of sync with clinical practice [ 24 ].
Study Limitations
In our study design, we elected to use a binary assessment, which limited the bias induced by the subjective appeal of any given tool. Therefore, these assessments did not include any comparison of quality or usability among the SR tools. This also meant that we did not use the Desmet [ 25 ] method, which ranks features by level of importance. We also excluded certain assessments that may impact user choices such as language translation features or translated training documentation, which is supported by some technologies, including DistillerSR. We completed the review in August 2021 but added several software tools following reviewer feedback; by adding expert additions without repeating the entire search strategy, we may have missed SR tools that launched between August and December 2021. Finally, the authors of this study are the designers of one of the leading SR tools, Nested Knowledge, which may have led to tacit bias toward this tool as part of the comparison.
By assessing features offered by web-based SR applications, we have identified gaps in current technologies and areas in need of development. Feature count does not equate to value or usability; it fails to capture benefits of simple platforms, such as ease of use, effective user interface, alignment with established workflows, or relative costs. The authors make no claim about superiority of software based on feature prevalence.
Future Directions
We invite and encourage independent researchers to assess the landscape of SR tools and build on this review. We expect the list of features to be assessed will evolve as research changes. For example, this review did not include features such as the ability to search included studies, reuse of extracted data, and application programming interface calls to read data, which may grow in importance. Furthermore, this review assessed the presence of automation at a high level without evaluating details. A future direction might be characterizing specific types of automation models used in screening, as well as in other stages, for software applications that support SR of biomedical research.
The highest-performing SR tools were DistillerSR, EPPI-Reviewer Web, and Nested Knowledge, each of which offer >80% of features. The most commonly offered and robust feature class was screening, whereas extraction (especially quality-controlled dual extraction) was underprovided. Living reviews, although strongly advocated for in the scientific community, were similarly underprovided by the SR tools reviewed here. This review enables the medical community to complete transparent and comprehensive comparison of SR tools and may also be used to identify gaps in technology for further development by the providers of these or novel SR tools.
This review of web-based software review software tools represents an attempt to best capture information from software providers’ websites, free trials, peer-reviewed publications, training materials, or software tutorials. The review is based primarily on publicly available information and may not accurately reflect feature offerings, as relevant information was not always available or clear to interpret. This evaluation does not represent the views or opinions of any of the software developers or service providers, except those of the authors. The review was completed in August 2021, and readers should refer to the respective software providers’ websites to obtain updated information on feature offerings.
Acknowledgments
The authors acknowledge the software development team from Nested Knowledge, Stephen Mead, Jeffrey Johnson, and Darian Lehmann-Plantenberg for their input in designing Nested Knowledge. The authors thank the independent software providers who provided feedback on our feature assessment, which increased the quality and accuracy of the results.
Abbreviations
Multimedia appendix 1, multimedia appendix 2.
Authors' Contributions: All authors participated in the conception, drafting, and editing of the manuscript.
Conflicts of Interest: KC, NH, and KH work for and hold equity in Nested Knowledge, which provides a software application included in this assessment. AR worked for Nested Knowledge. KL works for and holds equity in Nested Knowledge, Inc, and holds equity in Superior Medical Experts, Inc. KK works for and holds equity in Nested Knowledge, and holds equity in Superior Medical Experts.
An official website of the United States government
Here's how you know
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Tools & Resources
Software tools.
There are a variety of fee-based and open-source (i.e., free) tools available for conducting the various steps of your scoping or systematic review.
The NIH Library currently provides free access for NIH customers to Covidence . At least one user must be from NIH in order to request access and use Covidence. Please contact the NIH Library's Systematic Review Service to request access.
You can use Covidence to import citations from any citation management tool and then screen your citations at title and abstract and then full text levels. Covidence keeps track of who voted and manages the flow of the citations to ensure the correct number of screeners reviews each citation. It can also support single or dual screeners. In the full text screening step, you can upload PDFs into Covidence and it will keep track of your excluded citations and reasons for exclusion. Later, export this information to help you complete the PRISMA flow diagram. If you chose, you can also complete your data extraction and risk of bias assessments in Covidence by creating templates based on your needs and type of risk of bias tool. Finally, export all of your results for data management purposes or export your data into another data analysis tool for further work.
Other tools available for conducting scoping or systematic reviews are:
- DistillerSR (fee-based)
- EPPI-Reviewer (fee-based)
- JBI SUMARI (from the Joanna Briggs Institute for their reviews) (fee-based)
- LitStream (from ICF International)
- Systematic Review Data Repository (SRDR+) (from AHRQ) (free)
- Abstrackr (open source)
- Colandr (open source)
- Google Sheets and Forms
- HAWC (Health Assessment Workplace Collaborative) (free)
- Rayyan (open source)
And check out the Systematic Review Toolbox for additional software suggestions for conducting your review.
Quality Assessment Tools (i.e., risk of bias, critical appraisal)
- 2022 Repository of Quality Assessment and Risk of Bias Tools - A comprehensive resource for finding and selecting a risk of bias or quality assessment tool for evidence synthesis projects. Continually updated.
- AMSTAR 2 - AMSTAR 2 ( A MeaSurement Tool to Assess systematic Reviews). Use for critically appraising ONLY systematic reviews of healthcare interventions including randomised controlled clinical trials.
- JADAD Scale for Reporting Randomized Controlled Trials - The Jadad scale, sometimes known as Jadad scoring or the Oxford quality scoring system, is a procedure to independently assess the methodological quality of a clinical trial. Jadad et al. published a three-point questionnaire that formed the basis for a Jadad score.
- Joanna Briggs Institute Critical Appraisal Tools - includes 13 checklists to appraise a variety of different studies and publication types including qualitative studies.
- RoB 2.0: Cochrane Risk of Bias Tool for Randomized Trials Version 2 of the Cochrane RoB 2 can be used to assess the risk of bias in randomized trials.
- CASP (Critical Appraisal Skills Program ) - a number of checklists are available to appraise systematic reviews, randomised controlled trials, cohort studies, case control studies, economic evaluations, diagnostic studies, qualitative studies and clinical prediction rule.
- Newcastle-Ottawa Scale (NOS) for Assessing the Quality of Nonrandomised Studies in Meta-Analyses - Nonrandomised studies, including case-control and cohort studies, can be challenging to implement and conduct. Assessment of the quality of such studies is essential for a proper understanding of nonrandomised studies. The Newcastle-Ottawa Scale (NOS) is an ongoing collaboration between the Universities of Newcastle, Australia and Ottawa, Canada. It was developed to assess the quality of nonrandomised studies with its design, content and ease of use directed to the task of incorporating the quality assessments in the interpretation of meta-analytic results.
Background information on this important step of systematic reviews can be found at the following resources:
- Cochrane Handbook for Sysetmatic Reviews of Interventions (version 6.2) 2021 - see Chapter 7: Considering bias and conflicts of interest among the included studies , Chapter 8: Assessing risk of bias in a randomized trial , Chapter 13: Assessing risk of bias due to missing results in a synthesis and Chapter 25: Assessing risk of bias in a non-randomized study
- Aromataris E, Munn Z (Editors) . JBI Manual for Evidence Synthesis . JBI, 2020. https://doi.org/10.46658/JBIMES-20-01 - see appropriate chapter for type of review and the section on risk of bias.
- Chapter: Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Healthcare Interventions from AHRQ. 2017 December. Methods Guide for Effectiveness and Comparative Effectiveness Reviews . Rockville, MD, AHRQ.
- Chapter 3: Standards for Finding and Assessing Individual Studies in Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews . Washington, DC: The National Academies Press.
Grading Tools
The working group has developed a common, sensible and transparent approach to grading quality of evidence and strength of recommendations.
- Oxford Centre for Evidence Based Medicine - Levels of Evidence and Grades of Recommendations
Reporting Standards for Systematic Reviews
The Appraisal of Guidelines for Research and Evaluation (AGREE) Instrument evaluates the process of practice guideline development and the quality of reporting.
Collects guidance documents on reporting systematic reviews and other types of health research
Preferred Reporting Items for Systematic Reviews and Meta-Analyses. PRISMA 2020 was published in 2021 with an revised checklist , flow diagram , and a new elaboration and explanation paper .
The Methodological Expectations of Cochrane Intervention Reviews (MECIR) are methodological standards to which all Cochrane Protocols, Reviews, and Updates are expected to adhere
- RAMESES publication standards: meta-narrative reviews
Online Videos on Systematic Reviews
A collection of introductory and advanced videos on systematic reviews
This module provides an overview to Cochrane systematic reviews, and will take approximately 45 minutes to complete.
Dr. Aaron Carroll (The Incidental Economist) take on evidenced-based practice and systematic reviews
A collection of videos on evidence-based practice, common statistical methods in medicine, and systematic reviews
Dr. Michael Borenstein short introduction to meta-analysis
Systematic Reviews
- Introduction
- Guidelines and procedures
Management tools
Tools managing multiple processes, tools for automation of searching, tools for screening articles, critical appraisal tools, data extraction tools, synthesis tools, tools for writing reviews & protocols.
- Define the question
- Check the topic
- Determine inclusion/exclusion criteria
- Develop a protocol
- Identify keywords
- Databases and search strategies
- Grey literature
- Manage and organise
- Screen & Select
- Locate full text
- Extract data
- Examples of systematic reviews
- Accessing help
- Systematic Style Reviews Guide
A range of tools are available that can assist in managing the different processes involved in systematic and other types of reviews. This includes both online and downloadable tools, of which some are free and some are paid.
Some tools cover the entire process, whereas others assist with different stages.
This list covers just some of the tools available. See the link below for more useful tools.
- Systematic Review Toolbox A community-driven, searchable, web-based catalogue of tools to support the systematic review process across multiple domains. The resource aims to help reviewers find appropriate tools based on how they provide support for the systematic review process.
- SUMARI: Assessment and Review of Information (JCU authentication required) System for the Unified Management, Assessment and Review of Information is the Joanna Briggs Institutes premier software for the systematic review of literature. It is designed to assist researchers and practitioners in fields such as health, social sciences and humanities to appraise and synthesise evidence of feasibility, appropriateness, meaningfulness and effectiveness; and to conduct economic evaluations of activities and interventions. Online. Free to JCU users. Instructions to access... more... less... From the JBI landing page, as per link above, select, from the EBP Tools menu, SUMARI. If you have a personal Ovid account please log in with that. If not, you will need to create a personal account to use SUMARI. If you are creating a personal account now, you may need to click create twice (after an erroneous error message about your email address). When you are presented with the next login page, just click the green button EBP Network/Ovid . If you wish to share your project with a colleague, that colleague needs to have access to and an account for JBI via OVID. The invitee will receive an email with a link. That link needs to be copied and pasted into a browser to gain access to your project. The link has a short expiry time frame.
- MetaMap - A Tool For Recognizing UMLS Concepts in Text MetaMap is a highly configurable program developed by Dr. Alan (Lan) Aronson at the National Library of Medicine (NLM) to map biomedical text to the UMLS Metathesaurus or, equivalently, to discover Metathesaurus concepts referred to in text. MetaMap uses a knowledge-intensive approach based on symbolic, natural-language processing (NLP) and computational-linguistic techniques. Besides being applied for both IR and data-mining applications, MetaMap is one of the foundations of NLM's Medical Text Indexer (MTI) which is being used for both semiautomatic and fully automatic indexing of biomedical literature at NLM. (from website). Free but users are required to report on useage and to comply with MetaMap terms and conditions and comply with the UMLS Metathesaurus License Agreement.
- Polyglot search translator Enter a search strategy into the first box to see how the search should be entered in other databases. Note it does not identify subject terms that may be different in other databases.
- RCT Tagger There are other automated filters which are offered as standalone tools. One example is RCT Tagger. In this resource a subject search is entered into a search line and the search is run against PubMed. The results returned are presented in order of probability of the record being a report of an RCT according to the underlying algorithm. Cohen A. M., Smalheiser N. R. , McDonagh M. S., Yu C., Adams C. E., Davis J. M., & Yu P. S. (2015) Automated confidence ranked classification of randomized controlled trial articles: An aid to evidence-based medicine. Journal of the American Medical Informatics Association, 22 (3), 707-17. https://doi.org/10.1093/jamia/ocu025
- citationchaser This package contains functions to automate this process by making use of the Lens.org API. An input article list can be used to return a list of all referenced records, and/or all citing records in the Lens.org database (consisting of PubMed, PubMed Central, CrossRef, Microsoft Academic Graph and CORE)
- searchrefiner searchrefiner is an interactive interface for visualising and understanding queries used to retrieve medical literature for systematic reviews. You need to set up a free account to use this product.
- RobotSearch RobotSearch allows you to upload an RIS file, and it will filter out articles that do not describe RCTs using their (highly accurate) machine learning model.
- Automated Systematic Search Deduplication Tool (ASySD) A web application designed to de-duplicate large search results from multiple databases for systematic reviews. Upload your search to the web application and follow through each step to remove duplicate records. The code used to develop the web application is available via GitHub.
- Deduplicator (Systematic Review Accelerator) The Deduplicator is a tool to remove duplicates from libraries. This tool has two main functions 1) To deduplicate a library prior to screening 2) To compare two libraries and remove already screened articles
- Abstractr: Software for Semi-Automatic Citation Screening [From the website]: For optimizing the step of citation screening for systematic reviews, we are developing abstrackr: a free, open-source, web-based application for facilitating citation screening for systematic reviews. The program comprises two components; a web-based annotation tool that allows participants in a review to collaboratively screen citations for relevance, and machine learning technologies that semi-automate this process.
- EndNote Use EndNote to store, deduplicate and screen references.
- Screenatron The Screenatron is a tool designed to allow fast abstract screening when performing a systematic review.
- Disputatron The Disputatron is a tool designed to automatically detect screening disagreements between reviewers of a systematic review, who screened in EndNote.
- Excel Excel is the most basic tool for the management of the screening and data extraction stages of the systematic review process. Customized workbooks and spreadsheets can be designed for the review process, and lists of references can be exported from citation managers into Excel format for screening.
- Rayyan Rayyan helps screen studies for systematic style reviews by automatically identifying disagreements and allowing the use of keywords to help identify relevant studies. Rayyan also has a mobile app for iPhone, iPad and android devices.
- Systematic Review Toolbox Use the Study Selection checkbox to find more tools for screening and selecting studies.
- RobotReviewer Robot reviewer is a tool designed to automatically extract information about bias (using the Cochrane Risk of Bias tool), and additionally, can extract information about the participants, interventions, and outcomes studied in a trial.
- Systematic Review Toolbox Use the Quality Assessment checkbox to find more tools for appraising studies.
- NVivo NVivo is powerful software for qualitative data analysis.
- SPSS @ JCU SPSS is a comprehensive statistics package for the social sciences. The IBM SPSS Statistics Premium Edition helps data analysts, planners, forecasters, survey researchers, program evaluators and database marketers – among others – to easily accomplish tasks at every phase of the analytical process. It includes a broad array of fully integrated Statistics capabilities and related products for specialized analytical tasks across the enterprise. The software will improve productivity significantly and help achieve superior results for specific projects and business goals.
- Effect size calculator This is a web-based effect-size calculator. It is designed to facilitate the computation of effect-sizes for meta-analysis. Four effect-size types can be computed from various input data: the standardized mean difference, the correlation coefficient, the odds-ratio, and the risk-ratio.
- Systematic Review Toolbox Use the Data Extraction checkbox to find more tools for extracting data from studies.
- RobotReviewer RobotReviewer is a tool designed to automatically extract information about bias (using the Cochrane Risk of Bias tool), and can also extract information about the participants, interventions, and outcomes studied in a trial.
- RevMan RevMan facilitates preparation of protocols and full reviews, including text, characteristics of studies, comparison tables, and study data. It can perform meta-analysis of the data entered, and present the results graphically.
- CAMARADES: Meta-Analysis Tool A web application for meta-analysis of preclinical data. Functions include: Effect size calculation (NMD/SMD/odds ratio) ; Stratified meta-analysis ; Meta-regression ; Trim-and-fill analysis ; Egger's regression ; Graphs demonstration
- MAGICapp Develop and publish well-structured electronic guidelines, evidence summaries and decision aids. This tool is used post review to facilitate drilling down and examine the primary studies used in the review. No install or setup required.
- GRADEpro "GRADEpro GDT is an easy to use, all‐in‐one web solution to summarise and present information for healthcare decision making. It supports creating summary tables for systematic reviews and health technology assessments. It also provides a database of GRADE evidence tables and health care recommendations for adoption, adaptation and development of new guidelines." (website)
- PRISMA 2009 Flow Diagram
- PRISMA 2020 Statement The PRISMA 2020 statement consists of a 27-item checklist of items recommended for reporting in systematic reviews, an expanded checklist that details reporting recommendations for each item, the PRISMA 2020 abstract checklist , and revised flow diagrams for original and updated reviews .
- << Previous: Guidelines and procedures
- Next: Plan >>
- Last Updated: Feb 21, 2023 5:45 PM
- URL: https://libguides.jcu.edu.au/systematic-review
- Queen's University Library
- Research Guides
Systematic Reviews & Other Syntheses
- Review Software and Tools
- Scoping Reviews
- Systematic Reviews
- Training Modules
- Resources for Getting Started
- Reporting Standards
- Grey Literature
- Search Strategies
- Subject Heading Searching
- Obtaining Full Text Articles
- Critical Appraisal of Studies
Introduction
Systematic review toolbox, systematic review accelerator, systematic review data repository, bibliography.
- Covidence (Institutional Access)
- Citation Managers
- Librarian Support
In addition to the programs described below, some collaborative networks offer additional tools for reviewers. Check out Cochrane tools and software from the Joanna Briggs Institute .
Queen’s University Library provides access to an institutional subscription to Covidence. See the Covidence Institutional Access page for more information.
Covidence website
"Covidence is a not-for-profit service working in partnership with Cochrane to improve the production and use of systematic reviews for health and wellbeing. As systematic reviewers ourselves, we were frustrated at the systems available to reviewers to produce timely, high-quality evidence - so we've dedicated ourselves to making tools that make systematic reviewing faster, easier and more enjoyable!"
- Import citations : "Covidence works seamlessly with your favourite reference managers like EndNote, Zotero, Refworks, Mendeley or any tool that support RIS, CSV or PubMedXML formats."
- Screen titles and abstracts : "Breeze through screening with keyword highlighting & a lightning quick interface. Covidence keeps full records of who voted and also supports single or dual screeners. You can even do it on your mobile when you have a few spare minutes."
- Upload references : "Transfer PDFs stored in your reference manager to Covidence in a few clicks."
- Screen full text : "Decide quickly on studies in full text. Capture reasons for exclusion and any notes so you can resolve any disagreements quickly, with a click of a button."
- Create forms : "Be in control and stay focussed on your PICO question. Customisable extraction forms means you only spend time extracting what you need. It saves you time and money."
- Risk of bias : "Automatically populate your risk of bias tables by highlighting and commenting on text directly in your PDF. No more cut and paste."
- Do data extraction : "Extract data efficiently with a side-by-side view of your customised form and PDF. Then, when you’re done, easily compare your form with other reviewers."
- Export : "Covidence exports to all the common formats so you can continue your review in your preferred software."
View the JBI (Joanna Briggs Institute) and Covidence webinar .
Rayyan website
"Rayyan is a 100% FREE web application to help systematic review authors perform their job in a quick, easy and enjoyable fashion. Authors create systematic reviews, collaborate on them, maintain them over time and get suggestions for article inclusion."
Systematic Review Toolbox website
"The Systematic Review Toolbox is a community-driven, searchable, web-based catalogue of tools that support the systematic review process across multiple domains. The resource aims to help reviewers find appropriate tools based on how they provide support for the systematic review process. Users can perform a simple keyword search (i.e. Quick Search) to locate tools, a more detailed search (i.e. Advanced Search) allowing users to select various criteria to find specific types of tools and submit new tools to the database. Although the focus of the Toolbox is on identifying software tools to support systematic reviews, other tools or support mechanisms (such as checklists, guidelines and reporting standards) can also be found."
SR-Accelerator from Bond University is a suite of tools to speed up steps in the Systematic Review (SR) process. It is freely available for anyone in the world to use. The SRA is a modular design which means the tools can be incorporated into existing SR workflows and combined with other automation tools. Current tools are being continually reviewed and refined with additional tools being developed
SRDR website
"In an effort to reduce the burden of conducting systematic reviews, researchers and developers at the Brown University Evidence-based Practice Center (EPC), with support from the Agency for Healthcare Research and Quality (AHRQ), have developed a collaborative, Web-based repository of systematic review data. This resource serves as both an archive and data extraction tool and is shared among organizations and individuals producing systematic reviews worldwide, enabling the creation of a central database of systematic review data which may be critiqued, updated, and augmented on an ongoing basis. This database is freely accessible to facilitate evidence reviews and thus improve and speed up policy-making with regards to healthcare."
Gates, A., Johnson, C., & Hartling, L. (2018). Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool . Systematic Reviews, 7(1), 45. Kellermeyer, L., Harnke, B., & Knight, S. (2018). Covidence and Rayyan . Journal of the Medical Library Association: JMLA, 106(4), 580. Olofsson, H., Brolund, A., Hellberg, C., Silverstein, R., Stenström, K., Österberg, M., & Dagerhamn, J. (2017). Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan . Research Synthesis Methods, 8 (3). https://doi.org/10.1002/jrsm.1237 Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016). Rayyan - A web and mobile app for systematic reviews . Systematic Reviews, 5 (1), 1–10. https://doi.org/10.1186/s13643-016-0384-4 Rathbone, J., Hoffmann, T., & Glasziou, P. (2015). Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers . Systematic Reviews, 4 (80). https://doi.org/10.1186/s13643-015-0067-6 Swab, M. (2016). Product review: Mendeley Data . Journal of the Canadian Health Libraries Association, 37 (3), 121–123. https://doi.org/10.5596/c16-022
- << Previous: Critical Appraisal of Studies
- Next: Covidence (Institutional Access) >>
- Last Updated: Mar 13, 2023 10:29 AM
- Subjects: Bachelor of Health Sciences , Health Quality , Life Sciences and Biochemistry , Medicine , Nursing , Rehabilitation Therapy
- Tags: knowledge synthesis , systematic reviews
IMAGES
VIDEO
COMMENTS
Systematic Reviews for Health: Tools for Systematic Review · Tools for Systematic Reviews · Covidence · Abstrackr · Rayyan · JBI SUMARI · RevMan.
Tools for Literature Searching · PubMed Pub ReMiner. Text mining for PubMed · Yale MeSH Analyzer or Analyze MeSH · MeSH on Demand. Identifies MeSH terms in your
Systematic reviews are a reassessment of scholarly literature to facilitate decision making. This methodical approach of re-evaluating
Full-Featured Software Tools for Conducting Systematic Reviews · EPPI-Reviewer 4: EPPI-Reviewer is web-based software that supports reference
*tool added* http://pitts.ai - a machine learning and semi-automated web application for conducting systematic reviews and living systematic reviews http://
From AHRQ, the Systematic Review Data Repository (SRDR) is a powerful and easy-to-use tool for the extraction and management of data for
The highest-performing SR tools were DistillerSR, EPPI-Reviewer Web, and Nested Knowledge, each of which offer >80% of features. The most
Software Tools · DistillerSR (fee-based) · EPPI-Reviewer (fee-based) · JBI SUMARI (from the Joanna Briggs Institute for their reviews) (fee-based) · LitStream (from
Tools for screening articles · Automated Systematic Search Deduplication Tool (ASySD) · Deduplicator (Systematic Review Accelerator) · Abstractr:
"The Systematic Review Toolbox is a community-driven, searchable, web-based catalogue of tools that support the systematic review process across