critical appraisal of quantitative research checklist

CASP Checklists

How to use our CASP Checklists

Referencing and Creative Commons

• Online Training Courses
• CASP Workshops
• What is Critical Appraisal
• Study Designs
• Useful Links
• Bibliography
• View all Tools and Resources
• Testimonials

Critical Appraisal Checklists

We offer a number of free downloadable checklists to help you more easily and accurately perform critical appraisal across a number of different study types.

The CASP checklists are easy to understand but in case you need any further guidance on how they are structured, take a look at our guide on how to use our CASP checklists .

CASP Randomised Controlled Trial Checklist

• Print & Fill

CASP Systematic Review Checklist

CASP Qualitative Studies Checklist

CASP Cohort Study Checklist

CASP Diagnostic Study Checklist

CASP Case Control Study Checklist

CASP Economic Evaluation Checklist

CASP Clinical Prediction Rule Checklist

Checklist Archive

• CASP Randomised Controlled Trial Checklist 2018 fillable form
• CASP Randomised Controlled Trial Checklist 2018

CASP Checklist

Need more information?

• Online Learning
• Privacy Policy

Critical Appraisal Skills Programme

Critical Appraisal Skills Programme (CASP) will use the information you provide on this form to be in touch with you and to provide updates and marketing. Please let us know all the ways you would like to hear from us:

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.

Copyright 2024 CASP UK - OAP Ltd. All rights reserved Website by Beyond Your Brand

• Teesside University Student & Library Services
• Learning Hub Group

Critical Appraisal for Health Students

• Critical Appraisal of a quantitative paper
• Critical Appraisal: Help
• Critical Appraisal of a qualitative paper
• Useful resources

Appraisal of a Quantitative paper: Top tips

• Introduction

Critical appraisal of a quantitative paper (RCT)

This guide, aimed at health students, provides basic level support for appraising quantitative research papers. It's designed for students who have already attended lectures on critical appraisal. One framework for appraising quantitative research (based on reliability, internal and external validity) is provided and there is an opportunity to practise the technique on a sample article.

Please note this framework is for appraising one particular type of quantitative research a Randomised Controlled Trial (RCT) which is defined as 

a trial in which participants are randomly assigned to one of two or more groups: the experimental group or groups receive the intervention or interventions being tested; the comparison group (control group) receive usual care or no treatment or a placebo.  The groups are then followed up to see if there are any differences between the results.  This helps in assessing the effectiveness of the intervention.(CASP, 2020)

Support materials

• Framework for reading quantitative papers (RCTs)
• Critical appraisal of a quantitative paper PowerPoint

To practise following this framework for critically appraising a quantitative article, please look at the following article:

Marrero, D.G.  et al  (2016) 'Comparison of commercial and self-initiated weight loss programs in people with prediabetes: a randomized control trial',  AJPH Research , 106(5), pp. 949-956.

Critical Appraisal of a quantitative paper (RCT): practical example

• Internal Validity
• External Validity
• Reliability Measurement Tool

How to use this practical example 

Using the framework, you can have a go at appraising a quantitative paper - we are going to look at the following article:

Marrero, d.g.  et al  (2016) 'comparison of commercial and self-initiated weight loss programs in people with prediabetes: a randomized control trial',  ajph research , 106(5), pp. 949-956.,            step 1.  take a quick look at the article, step 2.  click on the internal validity tab above - there are questions to help you appraise the article, read the questions and look for the answers in the article. , step 3.   click on each question and our answers will appear., step 4.    repeat with the other aspects of external validity and reliability. , questioning the internal validity:, randomisation : how were participants allocated to each group did a randomisation process taken place, comparability of groups: how similar were the groups eg age, sex, ethnicity – is this made clear, blinding (none, single, double or triple): who was not aware of which group a patient was in (eg nobody, only patient, patient and clinician, patient, clinician and researcher) was it feasible for more blinding to have taken place , equal treatment of groups: were both groups treated in the same way , attrition : what percentage of participants dropped out did this adversely affect one group has this been evaluated, overall internal validity: does the research measure what it is supposed to be measuring, questioning the external validity:, attrition: was everyone accounted for at the end of the study was any attempt made to contact drop-outs, sampling approach: how was the sample selected was it based on probability or non-probability what was the approach (eg simple random, convenience) was this an appropriate approach, sample size (power calculation): how many participants was a sample size calculation performed did the study pass, exclusion/ inclusion criteria: were the criteria set out clearly were they based on recognised diagnostic criteria, what is the overall external validity can the results be applied to the wider population, questioning the reliability (measurement tool) internal validity:, internal consistency reliability (cronbach’s alpha). has a cronbach’s alpha score of 0.7 or above been included, test re-test reliability correlation. was the test repeated more than once were the same results received has a correlation coefficient been reported is it above 0.7 , validity of measurement tool. is it an established tool if not what has been done to check if it is reliable pilot study expert panel literature review criterion validity (test against other tools): has a criterion validity comparison been carried out was the score above 0.7, what is the overall reliability how consistent are the measurements , overall validity and reliability:, overall how valid and reliable is the paper.

• << Previous: Critical Appraisal of a qualitative paper
• Next: Useful resources >>
• Last Updated: Aug 25, 2023 2:48 PM
• URL: https://libguides.tees.ac.uk/critical_appraisal

Methods for the development of NICE public health guidance (third edition)

NICE process and methods [PMG4] Published: 26 September 2012

• 1 Introduction
• 2 Topic selection and scoping the guidance
• 3 Determining the evidence for review and consideration
• 4 Identifying the evidence
• 5 Reviewing the scientific evidence
• 6 Incorporating health economics
• 7 Developing recommendations
• Appendix A Conceptual framework for the work of the Centre for Public Health Excellence (CPHE)
• Appendix B Electronic resources
• Appendix C Example of audit information to accompany search strategies
• Appendix D Glossary of study designs
• Appendix E Algorithm for classifying quantitative (experimental and observational) study designs

Appendix F Quality appraisal checklist – quantitative intervention studies

• Appendix G Quality appraisal checklist – quantitative studies reporting correlations and associations
• Appendix H Quality appraisal checklist – qualitative studies
• Appendix I Quality appraisal checklist – economic evaluations
• Appendix J Process for using review-level material in exceptional circumstances
• Appendix K Examples of evidence tables
• Appendix L NICE review format
• Appendix M Fieldwork
• Appendix N Expert testimony summary template
• Update information
• About this document

NICE process and methods

Notes on the use of the quantitative studies checklist, references and further reading, appendix f quality appraisal checklist – quantitative intervention studies.

Public health interventions comprise a vast range of approaches, from the relatively simple through to complex national policy interventions. As a consequence, research questions about the effectiveness and efficacy of public health interventions will typically rely on quantitative evidence from a range of sources (see section 3.2 ). This will include evidence from small (experimental) randomised controlled trials through to large-scale observational studies (see appendix E for an algorithm outlining the range of experimental and observational quantitative study designs).

Rather than include an exhaustive list of critical appraisal tools for each individual study design, this checklist [ 12 ] is designed to be used for randomised controlled trials, case–control studies, cohort studies, controlled before-and-after studies and interrupted time series. It is based on the 'Graphical appraisal tool for epidemiological studies (GATE)', developed by Jackson et al. (2006), revised and tailored to be more suitable for public health interventions. It is anticipated that the majority of study designs used to determine the effect of an intervention on a (quantitative) outcome will be amenable to critical appraisal with this revised tool.

It enables a reviewer to appraise a study's internal and external validity after addressing the following key aspects of study design:

characteristics of study participants

definition of, and allocation to, intervention and control conditions

outcomes assessed over different time periods

methods of analyses.

GATE is intended to be used in an electronic (Excel) format that will facilitate both the sharing and storage of data, and through linkage with other documents, the compilation of research reports. Much of the guidance to support the completion of the critical appraisal form that is reproduced below also appears in 'pop-up' windows in the electronic version [ 13 ] .

There are 5 sections of the revised GATE. Section 1 seeks to assess the key population criteria for determining the study's external validity – that is, the extent to which the findings of a study are generalisable beyond the confines of the study to the study's source population.

Sections 2 to 4 assess the key criteria for determining the study's internal validity – that is, making sure that the study has been carried out carefully, and that the outcomes are likely to be attributable to the intervention being assessed, rather than some other (often unidentified) factor. In an internally valid study, any differences observed between groups of patients allocated to receive different interventions may (apart from the possibility of random error) be attributed to the intervention under investigation. Biases are characteristics that are likely to make estimates of effect differ systematically from the truth. Each of the critical appraisal checklist questions covers an aspect of methodology that research has shown makes a significant difference to the conclusions of a study.

Checklist items are worded so that 1 of 5 responses is possible:

In addition, the reviewer is requested to complete in detail the comments section of the quality appraisal form so that the grade awarded for each study aspect is as transparent as possible.

Each study is then awarded an overall study quality grading for internal validity (IV) and a separate one for external validity (EV):

The following sections outline the checklist questions, the prompts provided as pop-up boxes in the electronic version (highlighted in boxes) and additional guidance notes to aid the reviewer in assessing the study's internal and external validity.

This section seeks to assess the key population criteria for determining the study's external validity .

Although there are checklists for assessing external validity of RCTs (with a particular focus on clinical interventions) (see for example [Rothwell 2005]), there don't appear to be any checklists specific for public health interventions.

The questions asked in this section ask the reviewer to identify and describe the source population of the study (that is, those the study aims to represent), the eligible population (those that meet the study eligibility criteria), and the study participants (those that agreed to participate in the study). Where a study assesses an intervention delivered to a particular geographical setting or area (rather than delivered to individuals), the questions in this section relate to describing the source area or setting, and how the study areas or settings were chosen. For example, a study might assess the effect on health outcomes of neighbourhood renewal schemes and this section seeks to identify and describe how those neighbourhoods were chosen and whether they are representative of the neighbourhoods the study seeks to represent.

External validity is defined as the extent to which the findings of a study are generalisable beyond the confines of the study itself to the source population. So, for example, findings from a study conducted in a school setting in the USA might be generalisable to other schools in the USA (the source population of the study). An assessment of external validity will consider how representative of the source population the study population is and whether or not there are any specific population, demographic or geographic features of the selected population that might limit or support generalisability. Also important are considerations of the setting, intervention and outcomes assessed. These factors will be considered in sections 2 and 3 of the checklist.

1.1 Is the source population or source area well described?

This question seeks to determine the study's source population or area (that is, to whom or what the study aims to represent). The source population is usually best identified by referring to the study's original research question.

It is important to consider those population demographic characteristics such as age, sex, sexual orientation, disability, ethnicity, religion, place of residence, occupation, education, socioeconomic position and social capital [ 14 ] that can help to assess the impact of interventions on health inequalities and may help guide recommendations for specific population subgroups.

1.2 Is the eligible population or area representative of the source population or area?

To determine if the eligible population or area (for example, smokers responding to a media advertisement, areas of high density housing in a particular catchment area) are representative of the source population (for example, smokers or areas of high density housing), consider the means by which the eligible population was defined or identified and the implicit or explicit inclusion and exclusion criteria used. Were important groups likely to have been missed or under-represented? For example, were recruitment strategies geared toward more affluent or motivated groups? (For example, recruitment from more affluent areas or local fitness centres.) Were significant numbers of potentially eligible participants likely to have been inadvertently excluded? (For example, through referral to practitioners not involved in the research study.)

1.3 Do the selected participants or areas represent the eligible population or area?

Consider whether the method of selection of participants or areas from the eligible population or area was well described (for example, consecutive cases or random sampling). Were any significant sources of biases likely to have been introduced? Consider what proportion of selected individuals or clusters agreed to participate. Was there a bias toward more healthier or motivated individuals or wealthier areas?

Also consider whether the inclusion and exclusion criteria were well described and whether they were appropriate given the study objectives and the source population. Strict eligibility criteria can limit the external validity of intervention studies if the selected participants are not representative of the eligible population. This has been well-documented for RCTs where recruited participants have been found to differ from those who are eligible but not recruited, in terms of age, sex, race, severity of disease, educational status, social class and place of residence (Rothwell 2005).

Finally, consider whether sufficient detail of the demographic (for example, age, education, socioeconomic status, employment) or personal health-related (for example, smoking, physical activity levels) characteristics of the selected participants were presented. Are selected participants representative of the eligible population?

Section 2: method of allocation to intervention (or comparison)

This section aims to assess the likelihood of selection bias and confounding being introduced into a study.

Selection bias exists when there are systematic differences between the participants in the different intervention groups. As a result, the differences in the outcome observed may be explained by pre-existing differences between the groups, rather than because of the intervention itself. For example, if the people in 1 group are generally in poorer health compared with the second group, then they are more likely to have a worse outcome, regardless of the effect of the intervention. The intervention groups should be similar at the start of the study so that the only difference between the groups should be the intervention received.

2.1 Allocation to intervention or comparison. How was confounding minimised?

Consider the method by which individuals were allocated to either intervention or control conditions. Random allocation of individuals (as in RCTs) to receive 1 or other of the interventions under investigation, is considered the most reliable means of minimising the risk of selection bias and confounding.

If an appropriate method of randomisation has been used, each participant should have an equal chance of ending up in each of the intervention groups. Examples of random allocation sequences include random numbers generated by computer, tables of random numbers and drawing of lots or envelopes. However, if the description of randomisation is poor, or the process used is not truly random (for example, if the allocation sequence is predictable, such as date of birth or alternating between 1 group and another) or can otherwise be seen as flawed, this component should be given a lower quality rating.

2.2 Were the interventions (and comparisons) well-described and appropriate?

2.3 was the allocation concealed.

If investigators are aware of the allocation group for the next individual to be enrolled in the study, there is potential for people to be enrolled in an order that results in imbalances in important characteristics. For example, a practitioner might feel that people with mild rather than severe mental health problems would be more likely to do better on a new, behavioural intervention and be tempted to only enrol such individuals when they know they will be allocated to that group. This would result in the intervention group being, on average, less severe at baseline than control group. Concealment of treatment group may not always be feasible but concealment of allocation up until the point of enrolment in the study should always be possible.

Information should be presented in the paper that provides some assurance that allocations were not known until at least the point of allocation. Centralised allocation, computerised allocation systems and the use of coded identical containers would all be regarded as adequate methods of concealment. Sealed envelopes can be considered as adequate concealment if the envelopes are serially numbered, sealed and opaque, and allocation is performed by a third party. Poor methods of allocation concealment include alternation, or the use of case record numbers, date of birth or day of the week.

If the method of allocation concealment used is regarded as poor, or relatively easy to subvert, the study should be given a lower quality rating. If a study does not report any concealment approach, this should be scored as 'not reported'.

2.4 Were participants and investigators blind to exposure and comparison?

Blinding refers to the process of withholding information about treatment allocation or exposure status from those involved in the study who could potentially be influenced by this information. This can include participants, investigators, those administering care and those involved in data collection and analysis.

Unblinded individuals can bias the results of studies, either intentionally or unintentionally, through the use of other effective co-interventions, decisions about withdrawal, differential reporting of symptoms, or influencing concordance with treatment.

The terms 'single blind', 'double blind' and even 'triple blind' are sometimes used in studies. Unfortunately, they are not always used consistently. Commonly, when a study is described as 'single blind', only the participants are blind to their group allocation. When both participants and investigators are blind to group allocation the study is often described as 'double blind'. It is preferable to record exactly who was blinded, if reported, to avoid misunderstanding.

It is important to note that blinding of participants and researchers is not always possible, and it is important to think about the likely size and direction of bias caused by failure to blind in making an assessment of this component.

2.5 Is the exposure to the intervention and comparison adequate?

2.6 is contamination acceptably low, 2.7 were other interventions similar in both groups.

This question seeks to establish if there were any important differences between the intervention groups aside from the intervention received. If some patients received additional intervention (known as 'co-intervention'), this additional intervention is a potential confounding factor in the presence of which can make it difficult to attribute any observed effect to the intervention rather than to the other factors.

2.8 Were there other confounding factors?

2.9 were all participants accounted for at study conclusion.

Section 2 also aims to assess the likelihood of attrition bias being introduced into a study.

Attrition bias occurs when there are systematic differences between the comparison groups with respect to participants lost, or differences between participants lost to the study and those who remain. Attrition can occur at any point after participants have been allocated to their intervention groups. As such, it includes participants who are excluded post-allocation (and may indicate a violation of eligibility criteria), those who fail to complete the intervention and those who fail to complete outcome measurement (regardless of whether or not the intervention was completed).

It is a concern if the number of participants who were lost to follow-up (that is, dropped out) is high – typically >20%, although it is not unreasonable to expect a higher drop-out rate in studies conducted over a longer period of time.

Consideration should also be given to the reasons why participants dropped out. Participants who dropped out of a study may differ in some significant way from those who remained in the study. Drop-out rates and reasons for dropping out should be similar across all treatment groups. In good quality studies, the proportion of participants lost after allocation is reported and the possibility of attrition bias considered in the analysis.

2.10 Did the setting reflect usual UK practice?

2.11 did the intervention or control comparison reflect usual uk practice, section 3: outcomes.

Some of the items on this checklist may need to be filled in separately for each of the different outcomes reported by the study. For example, a study may report only 1 outcome of interest, measured by 1 tool, at 1 point in time, in which case each of the components (for example, reliability of outcome measure, relevance, withdrawals and drop-outs) can be assessed based on that 1 tool. However, if a study reports multiple outcomes of interest, scored by multiple tools (for example, self-report AND biochemically validated measures), at multiple points in time (for example, 6-month follow-up AND 1-year follow-up) individual components will need to be assessed for each outcome of interest.

It is important, therefore, that the reviewer has a clear idea of what the important outcomes are and over what timeframe, before appraising a study. The important outcomes for a piece of guidance will be identified through consultation with the NICE project team, the public health advisory committee and stakeholders.

3.1 Were the outcome measures reliable?

This question seeks to determine how reliable (that is, how consistently the method measures a particular outcome) and valid (that is, the method measures what it claims to measure) the outcome measures were. For example, a study assessing effectiveness of a smoking cessation intervention may report on a number of outcomes using a number of different tools, including self-reported smoking rates (a subjective outcome measure that is often unreliable) and biochemically validated smoking rates (an objective outcome measure that is likely to be more reliable).

If the outcome measures were subjective, it is also important to consider if the participant or researcher was blinded to the intervention or exposure (see question 2.4) as blinding may rescue the reliability of some subjective outcome measures.

3.2 Were the outcome measurements complete?

3.3 were all important outcomes assessed, 3.4 were outcomes relevant, 3.5 were there similar follow-up times in exposure and comparison groups.

It is possible to overcome differences in the length of follow-up between groups in the analyses, for example, by adjusting the denominator to take the time into account (by using person-years).

3.6 Was follow-up time meaningful?

The duration of post-intervention follow-up of participants should be of an adequate length to identify the outcome of interest.

Section 4: analyses

4.1 were the exposure and comparison groups similar at baseline if not, were these adjusted.

Studies may report the distributions or important differences in potential confounding factors between intervention groups. However, formal tests comparing the groups are problematic – failure to detect a difference does not mean a difference does not exist, and multiple comparisons of factors may falsely detect some differences that are not real.

It is important to assess whether all likely confounders have been considered. Confounding factors may differ by outcome, so potential confounding factors for all of the outcomes that are of interest will need to be considered.

4.2 Intention to treat analysis?

4.3 was the study sufficiently powered to detect an intervention effect (if one exists).

For cluster RCTs in particular, it is important to consider whether the cluster design has been appropriately taken into account in calculating required sample size for adequate power.

4.4 Were estimates of effect size given or calculable?

4.5 were the analytical methods appropriate.

There are a large number of considerations in deciding whether analytical methods were appropriate. For example, it is important to review the appropriateness of any subgroup analyses (and whether pre-specified or exploratory) that are presented. Although subgroup analyses can often provide valuable information on which to base further research (that is, are often exploratory), it is important that findings of subgroup analyses are not over (or under) emphasised. Meaningful results from subgroup analyses are beset by the problems of multiplicity of testing (in which the risk of a false positive result increases with the number of tests performed) and low statistical power (that is, studies generally only enrol sufficient participants to ensure that testing the primary study hypothesis is adequately powered) (Assmann et al. 2000). In a good quality paper, subgroup analyses are restricted to pre-specified subgroups and are often confined to primary outcome measures. Data are analysed using formal statistical tests of interaction (that assess whether intervention effect differs between subgroups) rather than comparison of subgroup p values. A correction for multiple testing is performed where appropriate (for example, 'Bonferroni correction' where a stricter significance level is used to define statistical significance). The results are delineated carefully, and full details of how analyses were performed are provided (Assmann et al. 2000; Guillemin 2007).

The appropriateness of some analytical methods will also depend on the study design under investigation. For example, with cluster RCTs, because participants are randomised at the group level and are not independent 'units' (as is the case with RCTs based on individuals without clustering), and outcomes are often assessed at the individual level, statistical adjustments are necessary before pooled intervention and control group outcomes can be compared.

Likewise, it is also important to consider whether the degree of similarity or difference in clusters has been considered in analyses of cluster RCTs. Good quality cluster-RCTs will determine the intra-class correlation coefficient of their study (a statistical measure of the interdependence in each cluster that is calculated by taking the ratio of the variance between groups compared with variance in groups).

Studies may also report other forms of statistical analysis such as regression, time series, factor analysis and discriminant analysis, as well as epidemiological or economic modelling. Economic modelling is covered in more detail in chapter 6 . The other topics are specialised and advice should be sought from the NICE project team before attempting to assess such studies.

4.6 Was the precision of intervention effects given or calculable? Were they meaningful?

Section 5: summary, 5.1 are the study results internally valid (that is, unbiased), 5.2 are the findings generalisable to the source population (that is, externally valid).

Assmann SF, Pocock SJ, Enos LE et al. (2000) Subgroup analysis and other (mis)uses of baseline data in clinical trials. The Lancet 355:1064–9

Effective Public Health Practice Project Quality assessment tool for quantitative studies

Guillemin F (2007) Primer: the fallacy of subgroup analysis. Nature Clinical Practice Rheumatology 3: 407–13

Heller RF, Verma A, Gemmell I et al. (2008) Critical appraisal for public health: a new checklist. Public Health122: 92–8

Jackson R, Ameratunga S, Broad J et al. (2006) The GATE frame: critical appraisal with pictures. Evidence based Medicine11: 35–8

Gough D, Oliver S, Thomas J (2012) (eds) An Introduction to Systematic Reviews, London: Sage.

Rothwell PM (2005) External validity of randomised controlled trials: to whom do the results of this trial apply? Lancet 365: 82–93

[ 12 ] Appraisal form derived from: Jackson R, Ameratunga S, Broad J et al. (2006) The GATE frame: critical appraisal with pictures. Evidence Based Medicine 11: 35–8.

[ 13 ] Available from CPHE on request.

[ 14 ] Demographic criteria as outlined by the PROGRESS-Plus categorisation (Kavanagh et al. 2008).

Critical Appraisal : Critical appraisal full list of checklists and tools

• What is critical appraisal?
• Where to start
• Education and childhood studies
• Occupational Therapy
• Physiotherapy
• Interpreting statistics
• Further reading and resources

Which checklist or tool should I use?

There are hundreds of critical appraisal checklists and tools you can choose from, which can be very overwhelming. There are so many because there are many kinds of research, knowledge can be communicated in a wide range of ways, and whether something is appropriate to meet your information needs depends on your specific context. 

We have asked for recommendations from lecturers in different academic departments, to give you an idea about which checklists and tools may be the most relevant for you. Please hover over the drop-down menu at the top of the page, underneath 'Critical appraisal checklists and tools' to view the individual subject pages.

Below are lists of as many critical appraisal tools and checklists as we have been able to find. These are split into health sciences and social sciences because the two areas tend to take different approaches to evaluation, for various reasons!

To see a selection of checklists more suitable for your subject, hover over the top tab of this page.  

Critical appraisal checklists and tools for Health Sciences

• AACODS  Checklist for appraising grey literature
• AMSTAR 2  critical appraisal tool for systematic reviews that include randomised and non-randomised studies of healthcare interventions or both
• AOTA Critically Appraised Papers  American Occupational Therapy Association 
• Bandolier - "Evidence based thinking about healthcare"
• BestBETS critical appraisal worksheet
• BMJ critical appraisal checklists
• CASP  Critical Appraisal Skills Programme includes checklists for case control studies, clinical prediction rule, cohort studies, diagnostic studies, economic evaluation, qualitative studies, RCTs and systematic reviews
• Centre for Evidence Based Medicine (Oxford) Critical Appraisal Tools  CEBM's worksheets to assess systematic reviews, diagnostic, prognosis, and RCTs
• Centre for Evidence Based Medicine (Oxford) CATmaker and EBM calculator  CEBM's computer assisted critical appraisal tool CATmaker 
• CEMB critical appraisal sheets  (Centre for Evidence Based Medicine)
• Cochrane Assessing Risk of Bias in a Randomized Trial
• Critical appraisal: a checklist from Students for Best Evidence S4BE (student network with simple explanations of difficult concepts)
• Critical appraisal and statistical skills (Knowledge for Healthcare)
• Critical appraisal of clinical trials  from Testing Treatments International
• Critical appraisal of clinical trials (Medicines Learning Portal)
• Critical appraisal of quantitative research  
• Critical appraisal of a quantitative paper  from Teeside University
• Critical appraisal of a qualitative paper  from Teeside University
• Critical appraisal tools  from the Centre for Evidence-Based Medicine
• Critical Evaluation of Research Papers – Qualitative Studies from Teeside University
• Critical Evaluation of Research Papers – RCTs/Experimental Studies from Teeside University
• Evaluation tool for mixed methods study designs 
• GRADE - The Grading of Recommendations Assessment, Development and Evaluation working group  guidelines and publications for grading the quality of evidence in healthcare research and policy
• HCPRDU Evaluation Tool for Mixed Methods Studies  - University of Salford Health Care Practice R&D Unit 
• HCPRDU Evaluation Tool for Qualitative Studies  - University of Salford Health Care Practice R&D Unit 
• HCPRDU Evaluation Tool for Quantitative Studies  - University of Salford Health Care Practice R&D Unit 
• JBI Joanna Briggs Institute critical appraisal tools  checklists for Analytical cross sectional studies, case control studies, case reports, case series, cohort studies, diagnostic test accuracy, economic evaluations, prevalence studies, qualitative research, quasi-experimental (non-randomised) studies, RCTs, systematic reviews and for text and opinion  
• Knowledge Translation Program  - Toronto based KTP critical appraisal worksheets for systematic reviews, prognosis, diagnosis, harm and therapy
• MATT Mixed Methods Appraisal Tool 
• McMaster University Evidence Based Practice Research Group quantitative and qualitative review forms
• NHLBI (National Heart, Blood and lung Institute) study quality assessment tools for case control studies, case series, controlled intervention, observational cohort and cross sectional studies, before-after (pre-post) studies with no control group, systematic reviews and meta analyses 
• NICE Guidelines, The Manual Appendix H. pp9-24
• QUADAS-2  tool for evaluating risk of bias in systematic reviews from the University of Bristol
• PEDro  PEDro (Physiotherapy Evidence Database) Scale - appraisal resources including a tutorial and appraisal tool
• RoB 2   A revised Cochrane risk-of-bias tool for randomized trials
• ROBINS-I Risk Of Bias In Non-Randomized Studies of Interventions 
• ROBIS  Risk of Bias in Systematic Reviews
• ROB-ME   A tool for assessing Risk Of Bias due to Missing Evidence in a synthesis
• SIGN  - Critical appraisal notes and checklists for case control studies, cohort studies, diagnostic studies, economic studies, RCTs, meta-analyses and systematic reviews
• Strength of Recommendation Taxonomy  - the SORT scale for quality, quantity and consistency of evidence in individual studies or bodies of evidence
• STROBE (Strengthening the Reporting of Observational studies in Epidemiology)  for cohort, case-control, and cross-sectional studies (combined),  cohort, case-control, cross-sectional studies and conference abstracts
• SURE Case Controlled Studies Critical Appraisal checklist
• SURE Case Series Studies Critical Appraisal checklist
• SURE Cohort Studies Critical Appraisal checklist
• SURE Cross-sectional Studies Critical Appraisal checklist
• SURE Experimental Studies Critical Appraisal checklist
• SURE Qualitative Studies Critical Appraisal checklist
• SURE Systematic Review Critical Appraisal checklist

Critical appraisal checklists and tools for Social Sciences

• AACODS   Checklist for appraising grey literature
• CRAAP test to evaluate sources of information 
• Critical Appraisal of an Article on an Educational Intervention  (variable study design) from the University of Glasgow
• Educational Interventions Critical Appraisal worksheet  from BestBETs
• PROMPT  from Open University
• PROVEN  - tool to evaluate any source of information 

SIFT (The Four Moves)  to help students distinguish between truth and fake news 

Some Guidelines for the Critical Reviewing of Conceptual Papers

• << Previous: Where to start
• Next: Business >>
• Last Updated: Apr 1, 2024 11:00 AM
• URL: https://libguides.qmu.ac.uk/critical-appraisal

• Critical appraisal checklists

There are numerous checklists available to act as memory aids for evaluating studies and they can be very useful.

The SR toolbox is a website providing regularly updated lists of the available guidance and software for each stage of the systematic review process, including screening and quality assessment.

Critical appraisal is much more than a 'tick box' exercise. It is needed to evaluate the weight we can put on the findings, and how far we can generalise the results from trials into routine practice to inform clinical care.

Since these pragmatic issues always need to be borne in mind when critically appraising study data, here we present some examples of checklists for different study types and some information for tricky critical appraisal scenarios.

Appraising two-armed RCTs Appraising multiple-armed RCTs Appraising diagnostic test studies Appraising systematic reviews Assessing multiple systematic reviews on the same question

Content created by BMJ Knowledge Centre

• Statistics calculators
• A glossary of EBM terms
• A glossary of health economics terms
• Bibliography of key resources
• Links to other EBM resources

EBM Toolkit home

Learn, Practise, Discuss, Tools

Library Services

UCL LIBRARY SERVICES

• Guides and databases
• Library skills

LibrarySkills@UCL for NHS staff

• Critical Appraisal of a quantitative study (RCT)
• Library skills for NHS staff
• Accessing resources
• Evidence-based resources
• Bibliographic databases
• Developing your search
• Reviewing and refining your search
• Health management information
• Critical appraisal of a qualitative study
• Critical appraisal of a systematic review
• Referencing basics
• Referencing software
• Referencing styles
• Publishing and sharing research outputs
• Publishing a protocol for your systematic or scoping review
• Communicating and disseminating research
• Help and training

Critical appraisal of a quantitative study (RCT)

The following video (5 mins, 36 secs.) helps to clarify the process of critical appraisal, how to systematically examine research, e.g. using checklists; the variety of tools /checklists available, and guidance on identifying the type of research you are faced with (so you can select the most appropriate appraisal tool).

Critical appraisal of an RCT: introduction to use of CASP checklists

The following video (4 min. 58 sec.) introduces the use of CASP checklists, specifically for critical appraisal of a randomised controlled trial (RCT) study paper; how the checklist is structured, and how to effectively use it.

Webinar recording of critical appraisal of an RCT

The following video is a recording of a webinar, with facilitator and participants using a CASP checklist, to critically appraise a randomised controlled trial paper, and determine whether it constitutes good practice.

'Focus on' videos

The following videos (all approx. 2-7 mins.) focus on a particular aspects of critical appraisal methodology for quantitative studies. 

• << Previous: Evaluating information & critical appraisal
• Next: Critical appraisal of a qualitative study >>
• Last Updated: Apr 11, 2024 9:57 AM
• URL: https://library-guides.ucl.ac.uk/nhs

• Mayo Clinic Libraries
• Systematic Reviews
• Critical Appraisal by Study Design

Systematic Reviews: Critical Appraisal by Study Design

• Knowledge Synthesis Comparison
• Knowledge Synthesis Decision Tree
• Standards & Reporting Results
• Materials in the Mayo Clinic Libraries
• Training Resources
• Review Teams
• Develop & Refine Your Research Question
• Develop a Timeline
• Project Management
• Communication
• PRISMA-P Checklist
• Eligibility Criteria
• Register your Protocol
• Other Resources
• Other Screening Tools
• Grey Literature Searching
• Citation Searching
• Data Extraction Tools
• Minimize Bias
• Synthesis & Meta-Analysis
• Publishing your Systematic Review

Tools for Critical Appraisal of Studies

“The purpose of critical appraisal is to determine the scientific merit of a research report and its applicability to clinical decision making.” 1 Conducting a critical appraisal of a study is imperative to any well executed evidence review, but the process can be time consuming and difficult. 2 The critical appraisal process requires “a methodological approach coupled with the right tools and skills to match these methods is essential for finding meaningful results.” 3 In short, it is a method of differentiating good research from bad research.

Critical Appraisal by Study Design (featured tools)

• Non-RCTs or Observational Studies
• Diagnostic Accuracy
• Animal Studies
• Qualitative Research
• Tool Repository
• AMSTAR 2 The original AMSTAR was developed to assess the risk of bias in systematic reviews that included only randomized controlled trials. AMSTAR 2 was published in 2017 and allows researchers to “identify high quality systematic reviews, including those based on non-randomised studies of healthcare interventions.” 4 more... less... AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)
• ROBIS ROBIS is a tool designed specifically to assess the risk of bias in systematic reviews. “The tool is completed in three phases: (1) assess relevance(optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help assess specific concerns about potential biases with the review.” 5 more... less... ROBIS (Risk of Bias in Systematic Reviews)
• BMJ Framework for Assessing Systematic Reviews This framework provides a checklist that is used to evaluate the quality of a systematic review.
• CASP Checklist for Systematic Reviews This CASP checklist is not a scoring system, but rather a method of appraising systematic reviews by considering: 1. Are the results of the study valid? 2. What are the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Systematic Reviews Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• JBI Critical Appraisal Tools, Checklist for Systematic Reviews JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• NHLBI Study Quality Assessment of Systematic Reviews and Meta-Analyses The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• RoB 2 RoB 2 “provides a framework for assessing the risk of bias in a single estimate of an intervention effect reported from a randomized trial,” rather than the entire trial. 6 more... less... RoB 2 (revised tool to assess Risk of Bias in randomized trials)
• CASP Randomised Controlled Trials Checklist This CASP checklist considers various aspects of an RCT that require critical appraisal: 1. Is the basic study design valid for a randomized controlled trial? 2. Was the study methodologically sound? 3. What are the results? 4. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CONSORT Statement The CONSORT checklist includes 25 items to determine the quality of randomized controlled trials. “Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report.” 7 more... less... CONSORT (Consolidated Standards of Reporting Trials)
• NHLBI Study Quality Assessment of Controlled Intervention Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• JBI Critical Appraisal Tools Checklist for Randomized Controlled Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• ROBINS-I ROBINS-I is a “tool for evaluating risk of bias in estimates of the comparative effectiveness… of interventions from studies that did not use randomization to allocate units… to comparison groups.” 8 more... less... ROBINS-I (Risk Of Bias in Non-randomized Studies – of Interventions)
• NOS This tool is used primarily to evaluate and appraise case-control or cohort studies. more... less... NOS (Newcastle-Ottawa Scale)
• AXIS Cross-sectional studies are frequently used as an evidence base for diagnostic testing, risk factors for disease, and prevalence studies. “The AXIS tool focuses mainly on the presented [study] methods and results.” 9 more... less... AXIS (Appraisal tool for Cross-Sectional Studies)
• NHLBI Study Quality Assessment Tools for Non-Randomized Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. • Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies • Quality Assessment of Case-Control Studies • Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group • Quality Assessment Tool for Case Series Studies more... less... NHLBI (National Heart, Lung, and Blood Institute)
• Case Series Studies Quality Appraisal Checklist Developed by the Institute of Health Economics (Canada), the checklist is comprised of 20 questions to assess “the robustness of the evidence of uncontrolled, [case series] studies.” 10
• Methodological Quality and Synthesis of Case Series and Case Reports In this paper, Dr. Murad and colleagues “present a framework for appraisal, synthesis and application of evidence derived from case reports and case series.” 11
• MINORS The MINORS instrument contains 12 items and was developed for evaluating the quality of observational or non-randomized studies. 12 This tool may be of particular interest to researchers who would like to critically appraise surgical studies. more... less... MINORS (Methodological Index for Non-Randomized Studies)
• JBI Critical Appraisal Tools for Non-Randomized Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis. • Checklist for Analytical Cross Sectional Studies • Checklist for Case Control Studies • Checklist for Case Reports • Checklist for Case Series • Checklist for Cohort Studies
• QUADAS-2 The QUADAS-2 tool “is designed to assess the quality of primary diagnostic accuracy studies… [it] consists of 4 key domains that discuss patient selection, index test, reference standard, and flow of patients through the study and timing of the index tests and reference standard.” 13 more... less... QUADAS-2 (a revised tool for the Quality Assessment of Diagnostic Accuracy Studies)
• JBI Critical Appraisal Tools Checklist for Diagnostic Test Accuracy Studies JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• STARD 2015 The authors of the standards note that “[e]ssential elements of [diagnostic accuracy] study methods are often poorly described and sometimes completely omitted, making both critical appraisal and replication difficult, if not impossible.”10 The Standards for the Reporting of Diagnostic Accuracy Studies was developed “to help… improve completeness and transparency in reporting of diagnostic accuracy studies.” 14 more... less... STARD 2015 (Standards for the Reporting of Diagnostic Accuracy Studies)
• CASP Diagnostic Study Checklist This CASP checklist considers various aspects of diagnostic test studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Diagnostic Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• SYRCLE’s RoB “[I]mplementation of [SYRCLE’s RoB tool] will facilitate and improve critical appraisal of evidence from animal studies. This may… enhance the efficiency of translating animal research into clinical practice and increase awareness of the necessity of improving the methodological quality of animal studies.” 15 more... less... SYRCLE’s RoB (SYstematic Review Center for Laboratory animal Experimentation’s Risk of Bias)
• ARRIVE 2.0 “The [ARRIVE 2.0] guidelines are a checklist of information to include in a manuscript to ensure that publications [on in vivo animal studies] contain enough information to add to the knowledge base.” 16 more... less... ARRIVE 2.0 (Animal Research: Reporting of In Vivo Experiments)
• Critical Appraisal of Studies Using Laboratory Animal Models This article provides “an approach to critically appraising papers based on the results of laboratory animal experiments,” and discusses various “bias domains” in the literature that critical appraisal can identify. 17
• CEBM Critical Appraisal of Qualitative Studies Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• CASP Qualitative Studies Checklist This CASP checklist considers various aspects of qualitative research studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• Quality Assessment and Risk of Bias Tool Repository Created by librarians at Duke University, this extensive listing contains over 100 commonly used risk of bias tools that may be sorted by study type.
• Latitudes Network A library of risk of bias tools for use in evidence syntheses that provides selection help and training videos.

References & Recommended Reading

1.     Kolaski, K., Logan, L. R., & Ioannidis, J. P. (2024). Guidance to best tools and practices for systematic reviews .  British Journal of Pharmacology ,  181 (1), 180-210

2.    Portney LG.  Foundations of clinical research : applications to evidence-based practice.  Fourth edition. ed. Philadelphia: F A Davis; 2020.

3.     Fowkes FG, Fulton PM.  Critical appraisal of published research: introductory guidelines.   BMJ (Clinical research ed).  1991;302(6785):1136-1140.

4.     Singh S.  Critical appraisal skills programme.   Journal of Pharmacology and Pharmacotherapeutics.  2013;4(1):76-77.

5.     Shea BJ, Reeves BC, Wells G, et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.   BMJ (Clinical research ed).  2017;358:j4008.

6.     Whiting P, Savovic J, Higgins JPT, et al.  ROBIS: A new tool to assess risk of bias in systematic reviews was developed.   Journal of clinical epidemiology.  2016;69:225-234.

7.     Sterne JAC, Savovic J, Page MJ, et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials.  BMJ (Clinical research ed).  2019;366:l4898.

8.     Moher D, Hopewell S, Schulz KF, et al.  CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials.  Journal of clinical epidemiology.  2010;63(8):e1-37.

9.     Sterne JA, Hernan MA, Reeves BC, et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.  BMJ (Clinical research ed).  2016;355:i4919.

10.     Downes MJ, Brennan ML, Williams HC, Dean RS.  Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS).   BMJ open.  2016;6(12):e011458.

11.   Guo B, Moga C, Harstall C, Schopflocher D.  A principal component analysis is conducted for a case series quality appraisal checklist.   Journal of clinical epidemiology.  2016;69:199-207.e192.

12.   Murad MH, Sultan S, Haffar S, Bazerbachi F.  Methodological quality and synthesis of case series and case reports.  BMJ evidence-based medicine.  2018;23(2):60-63.

13.   Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J.  Methodological index for non-randomized studies (MINORS): development and validation of a new instrument.   ANZ journal of surgery.  2003;73(9):712-716.

14.   Whiting PF, Rutjes AWS, Westwood ME, et al.  QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.   Annals of internal medicine.  2011;155(8):529-536.

15.   Bossuyt PM, Reitsma JB, Bruns DE, et al.  STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies.   BMJ (Clinical research ed).  2015;351:h5527.

16.   Hooijmans CR, Rovers MM, de Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW.  SYRCLE's risk of bias tool for animal studies.   BMC medical research methodology.  2014;14:43.

17.   Percie du Sert N, Ahluwalia A, Alam S, et al.  Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0.  PLoS biology.  2020;18(7):e3000411.

18.   O'Connor AM, Sargeant JM.  Critical appraisal of studies using laboratory animal models.   ILAR journal.  2014;55(3):405-417.

• << Previous: Minimize Bias
• Next: GRADE >>
• Last Updated: Apr 12, 2024 9:46 AM
• URL: https://libraryguides.mayo.edu/systematicreviewprocess

Occupational Therapy and Rehabilitation Sciences

• Defining the Research Question(s)
• Reference Resources
• Evidence Summaries & Clinical Guidelines
• Health Data & Statistics
• Patient & Consumer Facing Materials
• Images/Streaming Video
• Database Tutorials
• Crafting a Search
• Narrowing / Filtering a Search
• Expanding a Search
• Cited Reference Searching
• Find Grey Literature
• Save Your Searches
• Cite Sources
• Critical Appraisal
• Different Types of Literature Reviews
• Conducting & Reporting Systematic Reviews
• Finding Systematic Reviews
• Tutorials & Tools for Literature Reviews
• Mobile Apps for Health

PRISMA  or Preferred Reporting Items for Systematic Reviews and Meta-Analyses is an evidence-based protocol for reporting information in systematic reviews and meta-analyses.

• The PRISMA STATEMENT , a 27-item checklist and a four-phase flow diagram to help authors improve the reporting of systematic reviews and meta-analyses.
• PRISMA also offers editable templates for the flow diagram as PDF and Word documents 

Appraising the Evidence: Getting Started

To appraise the quality of evidence, it is essential understand the nature of the evidence source. Begin the appraisal process by considering these general characteristics:

• Is the source primary, secondary or tertiary? (See University of Minnesota Library -  Primary, Secondary, and Tertiary Sources in the Health Sciences )
• If the source is a journal article, what kind of article is it? (A report of original research? A review article? An opinion or commentary?)
• If the source is reporting original research, what was the purpose of the research?
• What is the date of publication?
• Would the evidence presented in the source still be applicable today? (Consider: has technology changed? Have recommended best clinical practices changed? Has consensus understanding of a disease, condition, or treatment changed?)

Authority/Accuracy

• Who is the author? What are the author's credentials and qualifications and to write on the topic?
• Was the source published by a credible entity? (a scholarly journal? a popular periodical, e.g, newspaper or magazine?  an association? an organization?)
• Did the source go through a peer review or editorial process before being published? (See this section of the guide for more information about locating peer reviewed articles)

Determining Study Methodology

Understanding how a study was conducted (the methodology) is fundamental for determining the level of evidence that was generated by the study, as well as assessing the quality of the evidence it generated.  While some papers state explicitly in the title what kind of method was used, it is often not so straightforward.  When looking at report of a study, there are a few techniques you can use to help classify the study design.

1. Notice Metadata in Database Records

In some bibliographic databases, there is information found in the Subject field, or the Publication Type field of the record that can provide information about a study's methodology.  Try to locate the record for the article of interest in CINAHL, PubMed or PsycINFO and look for information describing the study (e.g., is it tagged as a "randomized controlled trial,"  a "case report," and "observational study", a "review" article, etc).

• A word of caution : A  "review" article is not necessarily a "systematic review."  Even if the title or abstract says "systematic review," carefully evaluate what type of review it is (a systematic review of interventions? a mixed methods SR? a scoping review? a narrative review?).

2. Read the Methods Section

While there may be some information in the abstract that indicates a study's design, it is often necessary to read the full methods section in order to truly understand how the study was conducted.  For help understanding the major types of research methodologies within the health sciences, see:

• Understanding Research Study Designs  (University of Minnesota Library)
• Study Designs  (Centre for Evidence Based Medicine)
• Jeremey Howick's  Introduction to Study Designs  (Flow Chart) [PDF]
• Quantitative Study Designs  (Deakin University Library)
• Grimes, D. A., & Schulz, K. F. (2002). An overview of clinical research: the lay of the land .  Lancet (London, England) ,  359 (9300), 57–61. https://doi.org/10.1016/S0140-6736(02)07283-5
• Deconstructing the Research Article (May/Jun2022; 42(3): 138-140)
• Background, Significance, and Literature Review (Jul-Aug2022; 42(4): 203-205)
• Purpose Statement, Research Questions, and Hypotheses (Sep/Oct2022; 42(5): 249-257)
• Quantitative Research Designs (Nov/Dec2022; 42(6): 303-311)
• Qualitative Research Designs (Jan/Feb2023; 43(1): 41-45)
• Non-Experimental Research Designs (Mar/Apr2023; 43(2): 99-102)

Once the study methodology is understood, a tool or checklist can be selected to appraise the quality of the evidence that was generated by that study.  

Critical Appraisal Resources

In order to select a tool for critical appraisal (also known as quality assessment or "risk of bias" assessment), it is necessary to understand what methodology was used in the study.  (For help understanding study design, see this section of the guide .)

The list below sets of contains critical appraisal tools and checklists, with information about what types of studies those tools are meant for.  Additionally, there are links to reporting guidelines for different types of students, which can also be useful for quality assessment.  

If you're new to critical appraisal, check out this helpful video overview of some of the common tools:

Checklists & Tools

The AGREE II an instrument is valid and reliable tool that can be applied to any practice guideline in any disease area and can be used by health care providers, guideline developers, researchers, decision/policy makers, and educators.

For help using the AGREE II instrument, see the AGREE II Training Tools

• AMSTAR 2 AMSTAR 2 is the revised version of the popular AMSTAR tool (a tool for critically appraising systematic reviews of RCTs). AMSTAR 2 can be used to critically appraise systematic reviews that include randomized or non-randomized studies of healthcare interventions, or both.

A collection of checklists for a number of purposes related to EBM, including finding, interpreting, and evaluating research evidence.

Found in Appendix 1 of Greenhalgh, Trisha. (2010). How to Read a Paper : The Basics of Evidence Based Medicine, 4th edition .

Systematic reviews Randomised controlled trials Qualitative research studies Economic evaluation studies Cohort studies Case control studies Diagnostic test studies

CEBM offers Critical Appraisal Sheets for:

• GRADE The GRADE working group has developed a common, sensible and transparent approach to grading quality of a body of evidence and strength of recommendations that can be drawn from randomized and non-randomized trials . GRADE is meant for use in systematic reviews and other evidence syntheses (e.g., clinical guidelines) where a recommendation impacting practice will be made.

JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers. There are checklists available for:

• The Patient Education Materials Assessment Tool (PEMAT) and User’s Guide The Patient Education Materials Assessment Tool (PEMAT) is a systematic method to evaluate and compare the understandability and actionability of patient education materials . It is designed as a guide to help determine whether patients will be able to understand and act on information. Separate tools are available for use with print and audiovisual materials.
• MMAT (Mixed Methods Appraisal Tool) 2018 "The MMAT is a critical appraisal tool that is designed for the appraisal stage of systematic mixed studies reviews, i.e., reviews that include qualitative, quantitative and mixed methods studies. It permits to appraise the methodological quality of five categories to studies: qualitative research, randomized controlled trials, non randomized studies, quantitative descriptive studies, and mixed methods studies."
• PEDro Scale (Physiotherapy Evidence Database) The PEDro scale was developed to help users rapidly identify trials that are likely to be internally valid and have sufficient statistical information to guide clinical decision-making.
• Risk of Bias (RoB) Tools The RoB 2 tool is designed for assessing risk of bias in randomized trials , while the ROBINS-I tool is meant for assessing non-randomized studies of interventions .
• CanChild / McMaster EBP Research Group - Evidence Review Forms Evidence review forms from the McMaster University Occupational Therapy Evidence-Based Practice for appraising quantitative and qualitative evidence.

Reporting Guidelines

• CONSORT (CONsolidated Standards Of Reporting Trials) The CONSORT Statement is an evidence-based, minimum set of standards for reporting of randomized trials . It offers a standard way for authors to prepare reports of trial findings, facilitating their complete and transparent reporting, and aiding their critical appraisal and interpretation.
• TREND (Transparent Reporting of Evaluations with Nonrandomized Designs) The TREND statement has a 22-item checklist specifically developed to guide standardized reporting of non-randomized controlled trials .

RISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses . PRISMA primarily focuses on the reporting of reviews evaluating the effects of interventions, but can also be used as a basis for reporting systematic reviews with objectives other than evaluating interventions.

There are also extensions available for scoping reviews , as well as other aspects or types of systematic reviews.

• SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence) The SQUIRE guidelines provide a framework for reporting new knowledge about how to improve healthcare (i.e., quality improvement ). These guidelines are intended for reports that describe system level work to improve the quality, safety, and value of healthcare, and used methods to establish that observed outcomes were due to the intervention(s).

Searchable Registries of Appraisal Tools & Reporting Guidelines

• Equator Network: Enhancing the QUAlity and Transparency Of health Research Comprehensive searchable database of reporting guidelines for main study types and also links to other resources relevant to research reporting.
• The Registry of Methods and Tools for Evidence-Informed Decision Making The Registry of Methods and Tools for Evidence-Informed Decision Making ("the Registry") is a collection of resources to support evidence-informed decision making in practice, programs and policy. This curated, searchable resource offers a selection of methods and tools for each step in the evidence-informed decision-making process. Includes tools related to implementation science , assessing the applicability and transferability of evidence.

For a list of additional tools, as well as some commentary on their use, see:

Ma, L.-L., Wang, Y.-Y., Yang, Z.-H., Huang, D., Weng, H., & Zeng, X.-T. (2020). Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: What are they and which is better ? Military Medical Research, 7 (1), 7. https://doi.org/10.1186/s40779-020-00238-8

Determining Level of Evidence

Determining the level of evidence for a particular study or information source depends on understanding, the nature of the research question that is being investigated and the  methodology  that was used to collect the evidence.  See these these resources for help understanding study methodologies .  

There are a number of evidence hierarchies that could be used to 'rank' evidence. Which hierarchy is applied often depends on disciplinary norms - students should refer to materials and guidance from their professors about which hierarchy is appropriate to use.

• Oxford Centre for Evidence Based Medicine - Levels of Evidence The CEBM has put together a suite of documents to enable ranking of evidence into levels. Where a study falls in the ranking depends on the methodology of the study, and what kind of question (e.g., therapy, prognosis, diagnosis) is being addressed.
• Joanna Briggs Levels of Evidence [PDF] The JBI Levels of Evidence and Grades of Recommendation are meant to be used alongside the supporting document (PDF) outlining their use.
• << Previous: Evidence-Based Practice
• Next: Literature Reviews >>
• Last Updated: Apr 13, 2024 9:13 PM
• URL: https://guides.nyu.edu/ot

• Submission Guidelines

ebp appraisal 2021 header

Learning Objectives

1) Explain steps in conducting a rapid critical appraisal

2) Evaluate validity, reliability, and applicability 

3) Determine relevance to patient or project

Kelly’s literature search produced a considerable number of articles from the database she selected. But how can she narrow things down and determine which items are most relevant to her project? She needs to do a quick overview of them to choose which ones have the most accurate and meaningful results and can best provide information to answer her PICOT question. Is there a process she can follow that is relatively quick yet detailed?

fter you’ve collected the relevant literature for your clinical question , it’s time to evaluate the relevance of the literature to your PICOT (or to your evidence-based practice project) by performing a rapid critical appraisal.  

What is a rapid critical appraisal?

A rapid critical appraisal (RCA) is simply evaluating a study for its worth, based on its validity, reliability and applicability to clinical practice. Generally, an RCA involves some kind of a checklist to help the reviewer quickly determine the appropriateness of the study and whether it can apply to their patients. 

Although several RCA tools exist , most answer 3 very important questions to evaluate a study’s worth, namely:

• Are the results of the study valid?
• What were the results and are they important?
• Do the results provide an answer to my PICOT question that will help me care for my patient in this setting?

Rapid critical appraisal checklist

We can use a checklist to guide us through answering these three questions. First, let's break down the checklist by the broader question and review the specific approach in each section. 

Question 1: Are the results of the study valid?

Validity refers to the accuracy of the results. In other words, this question asks, “How accurate are the results of the study and can the results be trusted?” 

To answer this question, we need to find some answers. 

First, we need to consider how study participants were chosen. Were they selected at random or by some other method? Were measures taken to decrease biased results based on how participants were selected?  

Next, we need to consider the accuracy and completeness of the data. Did the authors of the study use a measure or instrument (such as a survey tool) that was validated and reliable? 

Finally, we need to determine what data was included and what data was excluded. Does it appear that the researchers took reasonable steps to assure accurate and complete data? Reliability refers to dependability and consistency—do we know the results are an accurate representation of what is being measured?

Question 2: What were the results and are they important?

Essentially, this question asks, “What did the study find and is it meaningful?” 

For the answers to some of these questions, you’ll need to understand statistics so we’ll run through statistical terminology. 

One thing you’ll often see in the literature is a reference to the “treatment effect.” We need to understand effect size and level of significance to understand treatment effect.

Effect Size 

Effect size refers to the strength of the relationship between the variables. The greater the effect size, the stronger the relationship between the two. Generally, effect size is designated as small (.2), medium (.5), and large (.8). The higher the effect size, the greater the relationship between variables. 

For example, the relationship between smoking and lung cancer would have a large effect size because they have a very strong relationship. The effect size between eating dark chocolate and headaches would be relatively small, if a relationship exists at all. 

Level of Significance

The level of significance deals with how likely something is to happen or not happen. In studies, it is often depicted by the p-value, or probability. The smaller the p-value, the less likely it is that the reported results happened because of a fluke or chance. Therefore, you can feel more confident in the significance of the data you are reviewing. Most studies use a p-value of .05 as “clinically significant.”    For example, a p-value of .05 means that there is less than a 5% chance, or 5 times out of 100, that the reported occurrence happened by chance or “fluke.” A p-value of .01 means that there is less than a 1% chance, or 1 times out of 100, that the reported occurrence happened by chance. However, even with extremely small p-values, such as .001 (1 times out of 1000), we still cannot say that our results prove anything.

It’s also important to consider how plausible and believable the results were. Do the results of the study seem likely? Do they appear that they could be reproduced? If claims seem far-fetched, it may warrant searching for similar studies to determine if similar results were achieved—otherwise, the intervention may not be beneficial and may have the potential to harm your patient.

Question 3: Do the results provide an answer to my PICOT question that will help me care for my patient in this setting?

This question is also known as clinical significance. Just because something has statistical significance doesn’t mean that it will be clinically significant to your patient population. In determining if it is clinically significant to your patient, you need to consider the following questions, many of which rely on your clinical expertise and understanding of your patient population.

Do the results apply to my patient? 

First, you need to assess whether your patient or patient population is similar to those in the study. There are likely to be differences, but you must determine what the important differences are and if it will impact the effectiveness on your patient population. 

For example, say you reviewed a study that demonstrated weekly group education classes were highly effective in helping control blood sugar fluctuations in patients newly diagnosed with type I diabetes. When asking how the results apply to your patient population, you first determine that your clients are also newly diagnosed with type I diabetes. If they aren’t newly diagnosed, the intervention may not produce the same results as those experienced by study participants. When you look at how the populations differ, you note that your patient population is located in a rural setting with limited access to care and that the study population in the published study was located in a major metropolitan city. The results of the original study may be important and meaningful, but due to some differences, it may not help you care for the patients in your setting.

What are the risks and benefits of treatment?

In some studies, researchers are able to carefully monitor and control the risks. In clinical settings, we may not be able to provide monitoring or control for side effects in the same manner as the original study, and the risks may outweigh the benefits for your patient population.

Is the treatment feasible in my clinical setting?

Using our type 1 diabetes example, the study may have found that weekly educational sessions helped to control blood sugar levels, but given the financial costs and available resources at your facility, you may not be able to offer weekly group education sessions.

What are my patient’s values and expectations for the outcome?

Using our same example, if you have a patient who is independent and introverted, group education sessions may not be consistent with these values, even though the intervention was shown to be effective. If your patient is unlikely to attend, it won’t be a helpful intervention for them. 

Returning to our case study: While Kelly’s literature search produced a considerable number of articles, she is able to narrow those articles down and determine the most relevant ones by asking 1) whether the results are valid, 2) whether they are important, and 3) whether they will help answer her PICOT question and help her care for her patients.

After you’ve appraised the evidence, you’ll plan how to implement your project and monitor the outcomes, which we’ll address in the next module.

True or false? Take the quiz

Originally published 2021

Barbara Wilson

Mary-jean (gigi) austria, tallie casucci.

Quantitative and qualitative methods are the engine behind evidence-based knowledge. Tallie Casucci, Gigi Austria, and Barbara Wilson provide a basic overview of how to differentiate between the two.

Relationship building isn’t typically the focus of medical training but is a necessary skill for truly excellent clinicians. Deirdre, Joni, Jared and colleagues developed a model to integrate relationship management skills into medical training, helping create a more well-rounded, complete clinician.

Medical students Rachel Tsolinas and Sam Wilkinson, along with SOM professor Kathryn Moore, share a practical tool all health care professionals can use to broaden our understanding of how culture influences decisions and events.

Subscribe to our newsletter

Receive the latest insights in health care equity, improvement, leadership, resilience, and more..

Contact the Accelerate Team

50 North Medical Drive   |   Salt Lake City, Utah 84132   |   801-587-2157

• Critical Appraisal Tools
• Introduction
• Related Guides
• Getting Help

Critical Appraisal of Studies

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value/relevance in a particular context by providing a framework to evaluate the research. During the critical appraisal process, researchers can:

• Decide whether studies have been undertaken in a way that makes their findings reliable as well as valid and unbiased
• Make sense of the results
• Know what these results mean in the context of the decision they are making
• Determine if the results are relevant to their patients/schoolwork/research

Burls, A. (2009). What is critical appraisal? In What Is This Series: Evidence-based medicine. Available online at  What is Critical Appraisal?

Critical appraisal is included in the process of writing high quality reviews, like systematic and integrative reviews and for evaluating evidence from RCTs and other study designs. For more information on systematic reviews, check out our  Systematic Review  guide.

• Next: Critical Appraisal Tools >>
• Last Updated: Nov 16, 2023 1:27 PM
• URL: https://guides.library.duq.edu/critappraise

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 21, Issue 4
• How to appraise quantitative research
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

This article has a correction. Please see:

• Correction: How to appraise quantitative research - April 01, 2019

• Xabi Cathala 1 ,
• Calvin Moorley 2
• 1 Institute of Vocational Learning , School of Health and Social Care, London South Bank University , London , UK
• 2 Nursing Research and Diversity in Care , School of Health and Social Care, London South Bank University , London , UK
• Correspondence to Mr Xabi Cathala, Institute of Vocational Learning, School of Health and Social Care, London South Bank University London UK ; cathalax{at}lsbu.ac.uk and Dr Calvin Moorley, Nursing Research and Diversity in Care, School of Health and Social Care, London South Bank University, London SE1 0AA, UK; Moorleyc{at}lsbu.ac.uk

https://doi.org/10.1136/eb-2018-102996

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

Some nurses feel that they lack the necessary skills to read a research paper and to then decide if they should implement the findings into their practice. This is particularly the case when considering the results of quantitative research, which often contains the results of statistical testing. However, nurses have a professional responsibility to critique research to improve their practice, care and patient safety. 1  This article provides a step by step guide on how to critically appraise a quantitative paper.

Title, keywords and the authors

The authors’ names may not mean much, but knowing the following will be helpful:

Their position, for example, academic, researcher or healthcare practitioner.

Their qualification, both professional, for example, a nurse or physiotherapist and academic (eg, degree, masters, doctorate).

This can indicate how the research has been conducted and the authors’ competence on the subject. Basically, do you want to read a paper on quantum physics written by a plumber?

The abstract is a resume of the article and should contain:

Introduction.

Research question/hypothesis.

Methods including sample design, tests used and the statistical analysis (of course! Remember we love numbers).

Main findings.

Conclusion.

The subheadings in the abstract will vary depending on the journal. An abstract should not usually be more than 300 words but this varies depending on specific journal requirements. If the above information is contained in the abstract, it can give you an idea about whether the study is relevant to your area of practice. However, before deciding if the results of a research paper are relevant to your practice, it is important to review the overall quality of the article. This can only be done by reading and critically appraising the entire article.

The introduction

Example: the effect of paracetamol on levels of pain.

My hypothesis is that A has an effect on B, for example, paracetamol has an effect on levels of pain.

My null hypothesis is that A has no effect on B, for example, paracetamol has no effect on pain.

My study will test the null hypothesis and if the null hypothesis is validated then the hypothesis is false (A has no effect on B). This means paracetamol has no effect on the level of pain. If the null hypothesis is rejected then the hypothesis is true (A has an effect on B). This means that paracetamol has an effect on the level of pain.

Background/literature review

The literature review should include reference to recent and relevant research in the area. It should summarise what is already known about the topic and why the research study is needed and state what the study will contribute to new knowledge. 5 The literature review should be up to date, usually 5–8 years, but it will depend on the topic and sometimes it is acceptable to include older (seminal) studies.

Methodology

In quantitative studies, the data analysis varies between studies depending on the type of design used. For example, descriptive, correlative or experimental studies all vary. A descriptive study will describe the pattern of a topic related to one or more variable. 6 A correlational study examines the link (correlation) between two variables 7  and focuses on how a variable will react to a change of another variable. In experimental studies, the researchers manipulate variables looking at outcomes 8  and the sample is commonly assigned into different groups (known as randomisation) to determine the effect (causal) of a condition (independent variable) on a certain outcome. This is a common method used in clinical trials.

There should be sufficient detail provided in the methods section for you to replicate the study (should you want to). To enable you to do this, the following sections are normally included:

Overview and rationale for the methodology.

Participants or sample.

Data collection tools.

Methods of data analysis.

Ethical issues.

Data collection should be clearly explained and the article should discuss how this process was undertaken. Data collection should be systematic, objective, precise, repeatable, valid and reliable. Any tool (eg, a questionnaire) used for data collection should have been piloted (or pretested and/or adjusted) to ensure the quality, validity and reliability of the tool. 9 The participants (the sample) and any randomisation technique used should be identified. The sample size is central in quantitative research, as the findings should be able to be generalised for the wider population. 10 The data analysis can be done manually or more complex analyses performed using computer software sometimes with advice of a statistician. From this analysis, results like mode, mean, median, p value, CI and so on are always presented in a numerical format.

The author(s) should present the results clearly. These may be presented in graphs, charts or tables alongside some text. You should perform your own critique of the data analysis process; just because a paper has been published, it does not mean it is perfect. Your findings may be different from the author’s. Through critical analysis the reader may find an error in the study process that authors have not seen or highlighted. These errors can change the study result or change a study you thought was strong to weak. To help you critique a quantitative research paper, some guidance on understanding statistical terminology is provided in  table 1 .

• View inline

Some basic guidance for understanding statistics

Quantitative studies examine the relationship between variables, and the p value illustrates this objectively.  11  If the p value is less than 0.05, the null hypothesis is rejected and the hypothesis is accepted and the study will say there is a significant difference. If the p value is more than 0.05, the null hypothesis is accepted then the hypothesis is rejected. The study will say there is no significant difference. As a general rule, a p value of less than 0.05 means, the hypothesis is accepted and if it is more than 0.05 the hypothesis is rejected.

The CI is a number between 0 and 1 or is written as a per cent, demonstrating the level of confidence the reader can have in the result. 12  The CI is calculated by subtracting the p value to 1 (1–p). If there is a p value of 0.05, the CI will be 1–0.05=0.95=95%. A CI over 95% means, we can be confident the result is statistically significant. A CI below 95% means, the result is not statistically significant. The p values and CI highlight the confidence and robustness of a result.

Discussion, recommendations and conclusion

The final section of the paper is where the authors discuss their results and link them to other literature in the area (some of which may have been included in the literature review at the start of the paper). This reminds the reader of what is already known, what the study has found and what new information it adds. The discussion should demonstrate how the authors interpreted their results and how they contribute to new knowledge in the area. Implications for practice and future research should also be highlighted in this section of the paper.

A few other areas you may find helpful are:

Limitations of the study.

Conflicts of interest.

Table 2 provides a useful tool to help you apply the learning in this paper to the critiquing of quantitative research papers.

Quantitative paper appraisal checklist

• 1. ↵ Nursing and Midwifery Council , 2015 . The code: standard of conduct, performance and ethics for nurses and midwives https://www.nmc.org.uk/globalassets/sitedocuments/nmc-publications/nmc-code.pdf ( accessed 21.8.18 ).
• Gerrish K ,
• Moorley C ,
• Tunariu A , et al
• Shorten A ,

Competing interests None declared.

Patient consent Not required.

Provenance and peer review Commissioned; internally peer reviewed.

Correction notice This article has been updated since its original publication to update p values from 0.5 to 0.05 throughout.

Linked Articles

• Miscellaneous Correction: How to appraise quantitative research BMJ Publishing Group Ltd and RCN Publishing Company Ltd Evidence-Based Nursing 2019; 22 62-62 Published Online First: 31 Jan 2019. doi: 10.1136/eb-2018-102996corr1

Read the full text or download the PDF:

Critical Appraisal of Quantitative Research

• Living reference work entry
• Later version available View entry history
• First Online: 27 February 2018
• Cite this living reference work entry

• Rocco Cavaleri 2 ,
• Sameer Bhole 3 &
• Amit Arora 2 , 4 , 5  

235 Accesses

Critical appraisal skills are important for anyone wishing to make informed decisions or improve the quality of healthcare delivery. A good critical appraisal provides information regarding the believability and usefulness of a particular study. However, the appraisal process is often overlooked, and critically appraising quantitative research can be daunting for both researchers and clinicians. This chapter introduces the concept of critical appraisal and highlights its importance in evidence-based practice. Readers are then introduced to the most common quantitative study designs and key questions to ask when appraising each type of study. These studies include systematic reviews, experimental studies (randomized controlled trials and non-randomized controlled trials), and observational studies (cohort, case-control, and cross-sectional studies). This chapter also provides the tools most commonly used to appraise the methodological and reporting quality of quantitative studies. Overall, this chapter serves as a step-by-step guide to appraising quantitative research in healthcare settings.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Altman DG, Bland JM. Treatment allocation in controlled trials: why randomise? BMJ. 1999;318(7192):1209.

Article   Google Scholar  

Arora A, Scott JA, Bhole S, Do L, Schwarz E, Blinkhorn AS. Early childhood feeding practices and dental caries in preschool children: a multi-centre birth cohort study. BMC Public Health. 2011;11(1):28.

Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, … Lijmer JG. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med. 2003;138(1):W1–12.

Google Scholar  

Cavaleri R, Schabrun S, Te M, Chipchase L. Hand therapy versus corticosteroid injections in the treatment of de quervain’s disease: a systematic review and meta-analysis. J Hand Ther. 2016;29(1):3–11. https://doi.org/10.1016/j.jht.2015.10.004 .

Centre for Evidence-based Management. Critical appraisal tools. 2017. Retrieved 20 Dec 2017, from https://www.cebma.org/resources-and-tools/what-is-critical-appraisal/ .

Centre for Evidence-based Medicine. Critical appraisal worksheets. 2017. Retrieved 3 Dec 2017, from http://www.cebm.net/blog/2014/06/10/critical-appraisal/ .

Clark HD, Wells GA, Huët C, McAlister FA, Salmi LR, Fergusson D, Laupacis A. Assessing the quality of randomized trials: reliability of the jadad scale. Control Clin Trials. 1999;20(5):448–52. https://doi.org/10.1016/S0197-2456(99)00026-4 .

Critical Appraisal Skills Program. Casp checklists. 2017. Retrieved 5 Dec 2017, from http://www.casp-uk.net/casp-tools-checklists .

Dawes M, Davies P, Gray A, Mant J, Seers K, Snowball R. Evidence-based practice: a primer for health care professionals. London: Elsevier; 2005.

Dumville JC, Torgerson DJ, Hewitt CE. Research methods: reporting attrition in randomised controlled trials. BMJ. 2006;332(7547):969.

Greenhalgh T, Donald A. Evidence-based health care workbook: understanding research for individual and group learning. London: BMJ Publishing Group; 2000.

Guyatt GH, Sackett DL, Cook DJ, Guyatt G, Bass E, Brill-Edwards P, … Gerstein H. Users’ guides to the medical literature: II. How to use an article about therapy or prevention. JAMA. 1993;270(21):2598–601.

Guyatt GH, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, … Jaeschke R. GRADE guidelines: 1. Introduction – GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4), 383–94.

Herbert R, Jamtvedt G, Mead J, Birger Hagen K. Practical evidence-based physiotherapy. London: Elsevier Health Sciences; 2005.

Hewitt CE, Torgerson DJ. Is restricted randomisation necessary? BMJ. 2006;332(7556):1506–8.

Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.0.2. The cochrane collaboration. 2009. Retrieved 3 Dec 2017, from http://www.cochrane-handbook.org .

Hoffmann T, Bennett S, Del Mar C. Evidence-based practice across the health professions. Chatswood: Elsevier Health Sciences; 2013.

Hoffmann T, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, … Johnston M. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ, 2014;348: g1687.

Joanna Briggs Institute. Critical appraisal tools. 2017. Retrieved 4 Dec 2017, from http:// joannabriggs.org/research/critical-appraisal-tools.html .

Mhaskar R, Emmanuel P, Mishra S, Patel S, Naik E, Kumar A. Critical appraisal skills are essential to informed decision-making. Indian J Sex Transm Dis. 2009;30(2):112–9. https://doi.org/10.4103/0253-7184.62770 .

Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Med Res Methodol. 2001;1(1):2. https://doi.org/10.1186/1471-2288-1-2 .

Moher D, Liberati A, Tetzlaff J, Altman DG, Prisma Group. Preferred reporting items for systematic reviews and meta-analyses: the prisma statement. PLoS Med. 2009;6(7):e1000097.

National Health and Medical Research Council. NHMRC additional levels of evidence and grades for recommendations for developers of guidelines. Canberra: NHMRC; 2009. Retrieved from https://www.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_120423.pdf .

National Heart Lung and Blood Institute. Study quality assessment tools. 2017. Retrieved 17 Dec 2017, from https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools .

Physiotherapy Evidence Database. PEDro scale. 2017. Retrieved 10 Dec 2017, from https://www.pedro.org.au/english/downloads/pedro-scale/ .

Portney L, Watkins M. Foundations of clinical research: application to practice. 2nd ed. Upper Saddle River: F.A. Davis Company/Publishers; 2009.

Roberts C, Torgerson DJ. Understanding controlled trials: baseline imbalance in randomised controlled trials. BMJ. 1999;319(7203):185.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, … Kristjansson E. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. https://doi.org/10.1136/bmj.j4008 .

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, … Boutron I. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.

Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, … Thacker SB. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283(15):2008–12.

Von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, Initiative S. The strengthening the reporting of observational studies in epidemiology (strobe) statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495–9.

Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, … Bossuyt PM. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011;155(8):529–36.

Download references

Author information

Authors and affiliations.

School of Science and Health, Western Sydney University, Sydney, NSW, Australia

Rocco Cavaleri & Amit Arora

Faculty of Dentistry, The University of Sydney, Surry Hills, NSW, Australia

Sameer Bhole

Discipline of Child and Adolescent Health, Sydney Medical School, Sydney, NSW, Australia

Oral Health Services, Sydney Local Health District and Sydney Dental Hospital, NSW Health, Sydney, NSW, Australia

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rocco Cavaleri .

Editor information

Editors and affiliations.

Health, Locked Bag 1797, CA.02.35, Western Sydney Univ, School of Science & Health, Locked Bag 1797, CA.02.35, Penrith, New South Wales, Australia

Pranee Liamputtong

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this entry

Cite this entry.

Cavaleri, R., Bhole, S., Arora, A. (2018). Critical Appraisal of Quantitative Research. In: Liamputtong, P. (eds) Handbook of Research Methods in Health Social Sciences . Springer, Singapore. https://doi.org/10.1007/978-981-10-2779-6_120-1

Download citation

DOI : https://doi.org/10.1007/978-981-10-2779-6_120-1

Received : 20 January 2018

Accepted : 12 February 2018

Published : 27 February 2018

Publisher Name : Springer, Singapore

Print ISBN : 978-981-10-2779-6

Online ISBN : 978-981-10-2779-6

eBook Packages : Springer Reference Social Sciences Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

• Publish with us

Policies and ethics

Chapter history

DOI: https://doi.org/10.1007/978-981-10-2779-6_120-2

DOI: https://doi.org/10.1007/978-981-10-2779-6_120-1

• Find a journal
• Track your research

• En español – ExME
• Em português – EME

Critical Appraisal: A Checklist

Posted on 6th September 2016 by Robert Will

Critical appraisal of scientific literature is a necessary skill for healthcare students. Students can be overwhelmed by the vastness of search results. Database searching is a skill in itself, but will not be covered in this blog. This blog assumes that you have found a relevant journal article to answer a clinical question. After selecting an article, you must be able to sit with the article and critically appraise it. Critical appraisal of a journal article is a literary and scientific systematic dissection in an attempt to assign merit to the conclusions of an article. Ideally, an article will be able to undergo scrutiny and retain its findings as valid.

The specific questions used to assess validity change slightly with different study designs and article types. However, in an attempt to provide a generalized checklist, no specific subtype of article has been chosen. Rather, the 20 questions below should be used as a quick reference to appraise any journal article. The first four checklist questions should be answered “Yes.” If any of the four questions are answered “no,” then you should return to your search and attempt to find an article that will meet these criteria.

Critical appraisal of…the Introduction

• Does the article attempt to answer the same question as your clinical question?
• Is the article recently published (within 5 years) or is it seminal (i.e. an earlier article but which has strongly influenced later developments)?
• Is the journal peer-reviewed?
• Do the authors present a hypothesis?

Critical appraisal of…the Methods

• Is the study design valid for your question?
• Are both inclusion and exclusion criteria described?
• Is there an attempt to limit bias in the selection of participant groups?
• Are there methodological protocols (i.e. blinding) used to limit other possible bias?
• Do the research methods limit the influence of confounding variables?
• Are the outcome measures valid for the health condition you are researching?

Critical appraisal of…the Results

• Is there a table that describes the subjects’ demographics?
• Are the baseline demographics between groups similar?
• Are the subjects generalizable to your patient?
• Are the statistical tests appropriate for the study design and clinical question?
• Are the results presented within the paper?
• Are the results statistically significant and how large is the difference between groups?
• Is there evidence of significance fishing (i.e. changing statistical tests to ensure significance)?

Critical appraisal of…the Discussion/Conclusion

• Do the authors attempt to contextualise non-significant data in an attempt to portray significance? (e.g. talking about findings which had a  trend  towards significance as if they were significant).
• Do the authors acknowledge limitations in the article?
• Are there any conflicts of interests noted?

This is by no means a comprehensive checklist of how to critically appraise a scientific journal article. However, by answering the previous 20 questions based on a detailed reading of an article, you can appraise most articles for their merit, and thus determine whether the results are valid. I have attempted to list the questions based on the sections most commonly present in a journal article, starting at the introduction and progressing to the conclusion. I believe some of these items are weighted heavier than others (i.e. methodological questions vs journal reputation). However, without taking this list through rigorous testing, I cannot assign a weight to them. Maybe one day, you will be able to critically appraise my future paper:  How Online Checklists Influence Healthcare Students’ Ability to Critically Appraise Journal Articles.

Feature Image by Arek Socha from Pixabay

Robert Will

Leave a reply cancel reply.

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

No Comments on Critical Appraisal: A Checklist

Hi Ella, I have found a checklist here for before and after study design: https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools and you may also find a checklist from this blog, which has a huge number of tools listed: https://s4be.cochrane.org/blog/2018/01/12/appraising-the-appraisal/

What kind of critical appraisal tool can be used for before and after study design article? Thanks

Hello, I am currently writing a book chapter on critical appraisal skills. This chapter is limited to 1000 words so your simple 20 questions framework would be the perfect format to cite within this text. May I please have your permission to use your checklist with full acknowledgement given to you as author? Many thanks

Thank you Robert, I came across your checklist via the Royal College of Surgeons of England website; https://www.rcseng.ac.uk/library-and-publications/library/blog/dissecting-the-literature-the-importance-of-critical-appraisal/ . I really liked it and I have made reference to it for our students. I really appreciate your checklist and it is still current, thank you.

Hi Kirsten. Thank you so much for letting us know that Robert’s checklist has been used in that article – that’s so good to see. If any of your students have any comments about the blog, then do let us know. If you also note any topics that you would like to see on the website, then we can add this to the list of suggested blogs for students to write about. Thank you again. Emma.

i am really happy with it. thank you very much

A really useful guide for helping you ask questions about the studies you are reviewing BRAVO

Dr.Suryanujella,

Thank you for the comment. I’m glad you find it helpful.

Feel free to use the checklist. S4BE asks that you cite the page when you use it.

I have read your article and found it very useful , crisp with all relevant information.I would like to use it in my presentation with your permission

That’s great thank you very much. I will definitely give that a go.

I find the MEAL writing approach very versatile. You can use it to plan the entire paper and each paragraph within the paper. There are a lot of helpful MEAL resources online. But understanding the acronym can get you started.

M-Main Idea (What are you arguing?) E-Evidence (What does the literature say?) A-Analysis (Why does the literature matter to your argument?) L-Link (Transition to next paragraph or section)

I hope that is somewhat helpful. -Robert

Hi, I am a university student at Portsmouth University, UK. I understand the premise of a critical appraisal however I am unsure how to structure an essay critically appraising a paper. Do you have any pointers to help me get started?

Thank you. I’m glad that you find this helpful.

Very informative & to the point for all medical students

How can I know what is the name of this checklist or tool?

This is a checklist that the author, Robert Will, has designed himself.

Thank you for asking. I am glad you found it helpful. As Emma said, please cite the source when you use it.

Greetings Robert, I am a postgraduate student at QMUL in the UK and I have just read this comprehensive critical appraisal checklist of your. I really appreciate you. if I may ask, can I have it downloaded?

Please feel free to use the information from this blog – if you could please cite the source then that would be much appreciated.

Robert Thank you for your comptrehensive account of critical appraisal. I have just completed a teaching module on critical appraisal as part of a four module Evidence Based Medicine programme for undergraduate Meducal students at RCSI Perdana medical school in Malaysia. If you are agreeable I would like to cite it as a reference in our module.

Anthony, Please feel free to cite my checklist. Thank you for asking. I hope that your students find it helpful. They should also browse around S4BE. There are numerous other helpful articles on this site.

Subscribe to our newsletter

You will receive our monthly newsletter and free access to Trip Premium.

Related Articles

Risk Communication in Public Health

Learn why effective risk communication in public health matters and where you can get started in learning how to better communicate research evidence.

Why was the CONSORT Statement introduced?

The CONSORT statement aims at comprehensive and complete reporting of randomized controlled trials. This blog introduces you to the statement and why it is an important tool in the research world.

Measures of central tendency in clinical research papers: what we should know whilst analysing them

Learn more about the measures of central tendency (mean, mode, median) and how these need to be critically appraised when reading a paper.

Evaluating the effects of triage education on triage accuracy within the emergency department: An integrative review

Affiliations.

• 1 School of Nursing, University of Auckland/Waipapa Taumata Rau, Auckland, New Zealand; Emergency Department, Whakatane Hospital, Te Whatu Ora Hauora a Toi, Bay of Plenty, New Zealand. Electronic address: [email protected].
• 2 School of Nursing, University of Auckland/Waipapa Taumata Rau, Auckland, New Zealand; Auckland Emergency Department, Auckland City Hospital, Te Whatu Ora Te Toka Tumai, Auckland, New Zealand.
• 3 School of Nursing, University of Auckland/Waipapa Taumata Rau, Auckland, New Zealand; National Institute for Health Innovation, University of Auckland/Waipapa Taumata Rau, Auckland, New Zealand.
• PMID: 37597277
• DOI: 10.1016/j.ienj.2023.101322

Introduction: Triage accuracy can affect patient outcomes. Education to ensure nurses provide the most accurate triage scores is paramount for patient safety.The objective was to investigate whether ongoing triage education increases triage accuracy, knowledge or behaviour.

Method: An integrative review was conducted by searching five databases to identify studies that included triage-based education. A systematic search strategy was completed followed by analysis with critical appraisal using the Critical Appraisal Skills Programme, a TIDieR Checklist and thematic analysis.

Findings: Four thousand five hundred seventy-six studies were retrieved, with 34 studies selected for inclusion. Thirty-one studies were quantitative, and three were mixed methods. 18 out of 34 studies showed improvement in triage accuracy. Seven showed increased knowledge. Six studies showed no improvement in triage accuracy. Sixteen studies assessed triage behaviour and showed improvement post-intervention, with five showing no changes. Only three studies compared interventions. Fifty-three opportunities for changes to triage accuracy, knowledge or behaviour were found, 41 showed improvements.

Conclusion: Triage education interventions can improve accuracy, knowledge and behaviour, but whether improvements are sustained needs further research.

Keywords: Education; Educational Intervention; Emergency Department; Emergency Nursing; Integrative review; Nursing; Training; Triage.

Copyright © 2023 Elsevier Ltd. All rights reserved.

Publication types