medical research bias types

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base
Research bias

Types of Bias in Research | Definition & Examples

Research bias results from any deviation from the truth, causing distorted results and wrong conclusions. Bias can occur at any phase of your research, including during data collection , data analysis , interpretation, or publication. Research bias can occur in both qualitative and quantitative research .

Understanding research bias is important for several reasons.

Bias exists in all research, across research designs , and is difficult to eliminate.
Bias can occur at any stage of the research process .
Bias impacts the validity and reliability of your findings, leading to misinterpretation of data.

It is almost impossible to conduct a study without some degree of research bias. It’s crucial for you to be aware of the potential types of bias, so you can minimize them.

For example, the success rate of the program will likely be affected if participants start to drop out ( attrition ). Participants who become disillusioned due to not losing weight may drop out, while those who succeed in losing weight are more likely to continue. This in turn may bias the findings towards more favorable results.

Information bias, interviewer bias.

Publication bias

Researcher bias

Response bias.

Selection bias

Cognitive bias

How to avoid bias in research

Other types of research bias, frequently asked questions about research bias.

Information bias , also called measurement bias, arises when key study variables are inaccurately measured or classified. Information bias occurs during the data collection step and is common in research studies that involve self-reporting and retrospective data collection. It can also result from poor interviewing techniques or differing levels of recall from participants.

The main types of information bias are:

Recall bias
Observer bias

Performance bias

Regression to the mean (rtm).

Over a period of four weeks, you ask students to keep a journal, noting how much time they spent on their smartphones along with any symptoms like muscle twitches, aches, or fatigue.

Recall bias is a type of information bias. It occurs when respondents are asked to recall events in the past and is common in studies that involve self-reporting.

As a rule of thumb, infrequent events (e.g., buying a house or a car) will be memorable for longer periods of time than routine events (e.g., daily use of public transportation). You can reduce recall bias by running a pilot survey and carefully testing recall periods. If possible, test both shorter and longer periods, checking for differences in recall.

A group of children who have been diagnosed, called the case group
A group of children who have not been diagnosed, called the control group

Since the parents are being asked to recall what their children generally ate over a period of several years, there is high potential for recall bias in the case group.

The best way to reduce recall bias is by ensuring your control group will have similar levels of recall bias to your case group. Parents of children who have childhood cancer, which is a serious health problem, are likely to be quite concerned about what may have contributed to the cancer.

Thus, if asked by researchers, these parents are likely to think very hard about what their child ate or did not eat in their first years of life. Parents of children with other serious health problems (aside from cancer) are also likely to be quite concerned about any diet-related question that researchers ask about.

Observer bias is the tendency of research participants to see what they expect or want to see, rather than what is actually occurring. Observer bias can affect the results in observationa l and experimental studies, where subjective judgment (such as assessing a medical image) or measurement (such as rounding blood pressure readings up or down) is part of the d ata collection process.

Observer bias leads to over- or underestimation of true values, which in turn compromise the validity of your findings. You can reduce observer bias by using double-blinded and single-blinded research methods.

Based on discussions you had with other researchers before starting your observations , you are inclined to think that medical staff tend to simply call each other when they need specific patient details or have questions about treatments.

At the end of the observation period, you compare notes with your colleague. Your conclusion was that medical staff tend to favor phone calls when seeking information, while your colleague noted down that medical staff mostly rely on face-to-face discussions. Seeing that your expectations may have influenced your observations, you and your colleague decide to conduct semi-structured interviews with medical staff to clarify the observed events. Note: Observer bias and actor–observer bias are not the same thing.

Performance bias is unequal care between study groups. Performance bias occurs mainly in medical research experiments, if participants have knowledge of the planned intervention, therapy, or drug trial before it begins.

Studies about nutrition, exercise outcomes, or surgical interventions are very susceptible to this type of bias. It can be minimized by using blinding , which prevents participants and/or researchers from knowing who is in the control or treatment groups. If blinding is not possible, then using objective outcomes (such as hospital admission data) is the best approach.

When the subjects of an experimental study change or improve their behavior because they are aware they are being studied, this is called the Hawthorne effect (or observer effect). Similarly, the John Henry effect occurs when members of a control group are aware they are being compared to the experimental group. This causes them to alter their behavior in an effort to compensate for their perceived disadvantage.

Regression to the mean (RTM) is a statistical phenomenon that refers to the fact that a variable that shows an extreme value on its first measurement will tend to be closer to the center of its distribution on a second measurement.

Medical research is particularly sensitive to RTM. Here, interventions aimed at a group or a characteristic that is very different from the average (e.g., people with high blood pressure) will appear to be successful because of the regression to the mean. This can lead researchers to misinterpret results, describing a specific intervention as causal when the change in the extreme groups would have happened anyway.

In general, among people with depression, certain physical and mental characteristics have been observed to deviate from the population mean .

This could lead you to think that the intervention was effective when those treated showed improvement on measured post-treatment indicators, such as reduced severity of depressive episodes.

However, given that such characteristics deviate more from the population mean in people with depression than in people without depression, this improvement could be attributed to RTM.

Interviewer bias stems from the person conducting the research study. It can result from the way they ask questions or react to responses, but also from any aspect of their identity, such as their sex, ethnicity, social class, or perceived attractiveness.

Interviewer bias distorts responses, especially when the characteristics relate in some way to the research topic. Interviewer bias can also affect the interviewer’s ability to establish rapport with the interviewees, causing them to feel less comfortable giving their honest opinions about sensitive or personal topics.

Participant: “I like to solve puzzles, or sometimes do some gardening.”

You: “I love gardening, too!”

In this case, seeing your enthusiastic reaction could lead the participant to talk more about gardening.

Establishing trust between you and your interviewees is crucial in order to ensure that they feel comfortable opening up and revealing their true thoughts and feelings. At the same time, being overly empathetic can influence the responses of your interviewees, as seen above.

Publication bias occurs when the decision to publish research findings is based on their nature or the direction of their results. Studies reporting results that are perceived as positive, statistically significant , or favoring the study hypotheses are more likely to be published due to publication bias.

Publication bias is related to data dredging (also called p -hacking ), where statistical tests on a set of data are run until something statistically significant happens. As academic journals tend to prefer publishing statistically significant results, this can pressure researchers to only submit statistically significant results. P -hacking can also involve excluding participants or stopping data collection once a p value of 0.05 is reached. However, this leads to false positive results and an overrepresentation of positive results in published academic literature.

Researcher bias occurs when the researcher’s beliefs or expectations influence the research design or data collection process. Researcher bias can be deliberate (such as claiming that an intervention worked even if it didn’t) or unconscious (such as letting personal feelings, stereotypes, or assumptions influence research questions ).

The unconscious form of researcher bias is associated with the Pygmalion effect (or Rosenthal effect ), where the researcher’s high expectations (e.g., that patients assigned to a treatment group will succeed) lead to better performance and better outcomes.

Researcher bias is also sometimes called experimenter bias, but it applies to all types of investigative projects, rather than only to experimental designs .

Good question: What are your views on alcohol consumption among your peers?
Bad question: Do you think it’s okay for young people to drink so much?

Response bias is a general term used to describe a number of different situations where respondents tend to provide inaccurate or false answers to self-report questions, such as those asked on surveys or in structured interviews .

This happens because when people are asked a question (e.g., during an interview ), they integrate multiple sources of information to generate their responses. Because of that, any aspect of a research study may potentially bias a respondent. Examples include the phrasing of questions in surveys, how participants perceive the researcher, or the desire of the participant to please the researcher and to provide socially desirable responses.

Response bias also occurs in experimental medical research. When outcomes are based on patients’ reports, a placebo effect can occur. Here, patients report an improvement despite having received a placebo, not an active medical treatment.

While interviewing a student, you ask them:

“Do you think it’s okay to cheat on an exam?”

Common types of response bias are:

Acquiescence bias

Demand characteristics.

Social desirability bias

Courtesy bias

Question-order bias

Extreme responding

Acquiescence bias is the tendency of respondents to agree with a statement when faced with binary response options like “agree/disagree,” “yes/no,” or “true/false.” Acquiescence is sometimes referred to as “yea-saying.”

This type of bias occurs either due to the participant’s personality (i.e., some people are more likely to agree with statements than disagree, regardless of their content) or because participants perceive the researcher as an expert and are more inclined to agree with the statements presented to them.

Q: Are you a social person?

People who are inclined to agree with statements presented to them are at risk of selecting the first option, even if it isn’t fully supported by their lived experiences.

In order to control for acquiescence, consider tweaking your phrasing to encourage respondents to make a choice truly based on their preferences. Here’s an example:

Q: What would you prefer?

A quiet night in
A night out with friends

Demand characteristics are cues that could reveal the research agenda to participants, risking a change in their behaviors or views. Ensuring that participants are not aware of the research objectives is the best way to avoid this type of bias.

On each occasion, patients reported their pain as being less than prior to the operation. While at face value this seems to suggest that the operation does indeed lead to less pain, there is a demand characteristic at play. During the interviews, the researcher would unconsciously frown whenever patients reported more post-op pain. This increased the risk of patients figuring out that the researcher was hoping that the operation would have an advantageous effect.

Social desirability bias is the tendency of participants to give responses that they believe will be viewed favorably by the researcher or other participants. It often affects studies that focus on sensitive topics, such as alcohol consumption or sexual behavior.

You are conducting face-to-face semi-structured interviews with a number of employees from different departments. When asked whether they would be interested in a smoking cessation program, there was widespread enthusiasm for the idea.

Note that while social desirability and demand characteristics may sound similar, there is a key difference between them. Social desirability is about conforming to social norms, while demand characteristics revolve around the purpose of the research.

Courtesy bias stems from a reluctance to give negative feedback, so as to be polite to the person asking the question. Small-group interviewing where participants relate in some way to each other (e.g., a student, a teacher, and a dean) is especially prone to this type of bias.

Question order bias

Question order bias occurs when the order in which interview questions are asked influences the way the respondent interprets and evaluates them. This occurs especially when previous questions provide context for subsequent questions.

When answering subsequent questions, respondents may orient their answers to previous questions (called a halo effect ), which can lead to systematic distortion of the responses.

Extreme responding is the tendency of a respondent to answer in the extreme, choosing the lowest or highest response available, even if that is not their true opinion. Extreme responding is common in surveys using Likert scales , and it distorts people’s true attitudes and opinions.

Disposition towards the survey can be a source of extreme responding, as well as cultural components. For example, people coming from collectivist cultures tend to exhibit extreme responses in terms of agreement, while respondents indifferent to the questions asked may exhibit extreme responses in terms of disagreement.

Selection bias is a general term describing situations where bias is introduced into the research from factors affecting the study population.

Common types of selection bias are:

Sampling or ascertainment bias

Attrition bias
Self-selection (or volunteer) bias
Survivorship bias
Nonresponse bias
Undercoverage bias

Sampling bias occurs when your sample (the individuals, groups, or data you obtain for your research) is selected in a way that is not representative of the population you are analyzing. Sampling bias threatens the external validity of your findings and influences the generalizability of your results.

The easiest way to prevent sampling bias is to use a probability sampling method . This way, each member of the population you are studying has an equal chance of being included in your sample.

Sampling bias is often referred to as ascertainment bias in the medical field.

Attrition bias occurs when participants who drop out of a study systematically differ from those who remain in the study. Attrition bias is especially problematic in randomized controlled trials for medical research because participants who do not like the experience or have unwanted side effects can drop out and affect your results.

You can minimize attrition bias by offering incentives for participants to complete the study (e.g., a gift card if they successfully attend every session). It’s also a good practice to recruit more participants than you need, or minimize the number of follow-up sessions or questions.

You provide a treatment group with weekly one-hour sessions over a two-month period, while a control group attends sessions on an unrelated topic. You complete five waves of data collection to compare outcomes: a pretest survey, three surveys during the program, and a posttest survey.

Self-selection or volunteer bias

Self-selection bias (also called volunteer bias ) occurs when individuals who volunteer for a study have particular characteristics that matter for the purposes of the study.

Volunteer bias leads to biased data, as the respondents who choose to participate will not represent your entire target population. You can avoid this type of bias by using random assignment —i.e., placing participants in a control group or a treatment group after they have volunteered to participate in the study.

Closely related to volunteer bias is nonresponse bias , which occurs when a research subject declines to participate in a particular study or drops out before the study’s completion.

Considering that the hospital is located in an affluent part of the city, volunteers are more likely to have a higher socioeconomic standing, higher education, and better nutrition than the general population.

Survivorship bias occurs when you do not evaluate your data set in its entirety: for example, by only analyzing the patients who survived a clinical trial.

This strongly increases the likelihood that you draw (incorrect) conclusions based upon those who have passed some sort of selection process—focusing on “survivors” and forgetting those who went through a similar process and did not survive.

Note that “survival” does not always mean that participants died! Rather, it signifies that participants did not successfully complete the intervention.

However, most college dropouts do not become billionaires. In fact, there are many more aspiring entrepreneurs who dropped out of college to start companies and failed than succeeded.

Nonresponse bias occurs when those who do not respond to a survey or research project are different from those who do in ways that are critical to the goals of the research. This is very common in survey research, when participants are unable or unwilling to participate due to factors like lack of the necessary skills, lack of time, or guilt or shame related to the topic.

You can mitigate nonresponse bias by offering the survey in different formats (e.g., an online survey, but also a paper version sent via post), ensuring confidentiality , and sending them reminders to complete the survey.

You notice that your surveys were conducted during business hours, when the working-age residents were less likely to be home.

Undercoverage bias occurs when you only sample from a subset of the population you are interested in. Online surveys can be particularly susceptible to undercoverage bias. Despite being more cost-effective than other methods, they can introduce undercoverage bias as a result of excluding people who do not use the internet.

Cognitive bias refers to a set of predictable (i.e., nonrandom) errors in thinking that arise from our limited ability to process information objectively. Rather, our judgment is influenced by our values, memories, and other personal traits. These create “ mental shortcuts” that help us process information intuitively and decide faster. However, cognitive bias can also cause us to misunderstand or misinterpret situations, information, or other people.

Because of cognitive bias, people often perceive events to be more predictable after they happen.

Although there is no general agreement on how many types of cognitive bias exist, some common types are:

Anchoring bias
Framing effect
Actor-observer bias
Availability heuristic (or availability bias)
Confirmation bias
Halo effect
The Baader-Meinhof phenomenon

Anchoring bias

Anchoring bias is people’s tendency to fixate on the first piece of information they receive, especially when it concerns numbers. This piece of information becomes a reference point or anchor. Because of that, people base all subsequent decisions on this anchor. For example, initial offers have a stronger influence on the outcome of negotiations than subsequent ones.

Framing effect

Framing effect refers to our tendency to decide based on how the information about the decision is presented to us. In other words, our response depends on whether the option is presented in a negative or positive light, e.g., gain or loss, reward or punishment, etc. This means that the same information can be more or less attractive depending on the wording or what features are highlighted.

Actor–observer bias

Actor–observer bias occurs when you attribute the behavior of others to internal factors, like skill or personality, but attribute your own behavior to external or situational factors.

In other words, when you are the actor in a situation, you are more likely to link events to external factors, such as your surroundings or environment. However, when you are observing the behavior of others, you are more likely to associate behavior with their personality, nature, or temperament.

One interviewee recalls a morning when it was raining heavily. They were rushing to drop off their kids at school in order to get to work on time. As they were driving down the highway, another car cut them off as they were trying to merge. They tell you how frustrated they felt and exclaim that the other driver must have been a very rude person.

At another point, the same interviewee recalls that they did something similar: accidentally cutting off another driver while trying to take the correct exit. However, this time, the interviewee claimed that they always drive very carefully, blaming their mistake on poor visibility due to the rain.

Availability heuristic

Availability heuristic (or availability bias) describes the tendency to evaluate a topic using the information we can quickly recall to our mind, i.e., that is available to us. However, this is not necessarily the best information, rather it’s the most vivid or recent. Even so, due to this mental shortcut, we tend to think that what we can recall must be right and ignore any other information.

Confirmation bias

Confirmation bias is the tendency to seek out information in a way that supports our existing beliefs while also rejecting any information that contradicts those beliefs. Confirmation bias is often unintentional but still results in skewed results and poor decision-making.

Let’s say you grew up with a parent in the military. Chances are that you have a lot of complex emotions around overseas deployments. This can lead you to over-emphasize findings that “prove” that your lived experience is the case for most families, neglecting other explanations and experiences.

The halo effect refers to situations whereby our general impression about a person, a brand, or a product is shaped by a single trait. It happens, for instance, when we automatically make positive assumptions about people based on something positive we notice, while in reality, we know little about them.

The Baader-Meinhof phenomenon

The Baader-Meinhof phenomenon (or frequency illusion) occurs when something that you recently learned seems to appear “everywhere” soon after it was first brought to your attention. However, this is not the case. What has increased is your awareness of something, such as a new word or an old song you never knew existed, not their frequency.

While very difficult to eliminate entirely, research bias can be mitigated through proper study design and implementation. Here are some tips to keep in mind as you get started.

Clearly explain in your methodology section how your research design will help you meet the research objectives and why this is the most appropriate research design.
In quantitative studies , make sure that you use probability sampling to select the participants. If you’re running an experiment, make sure you use random assignment to assign your control and treatment groups.
Account for participants who withdraw or are lost to follow-up during the study. If they are withdrawing for a particular reason, it could bias your results. This applies especially to longer-term or longitudinal studies .
Use triangulation to enhance the validity and credibility of your findings.
Phrase your survey or interview questions in a neutral, non-judgmental tone. Be very careful that your questions do not steer your participants in any particular direction.
Consider using a reflexive journal. Here, you can log the details of each interview , paying special attention to any influence you may have had on participants. You can include these in your final analysis.
Baader–Meinhof phenomenon
Sampling bias
Ascertainment bias
Self-selection bias
Hawthorne effect
Omitted variable bias
Pygmalion effect
Placebo effect

Research bias affects the validity and reliability of your research findings , leading to false conclusions and a misinterpretation of the truth. This can have serious implications in areas like medical research where, for example, a new form of treatment may be evaluated.

Observer bias occurs when the researcher’s assumptions, views, or preconceptions influence what they see and record in a study, while actor–observer bias refers to situations where respondents attribute internal factors (e.g., bad character) to justify other’s behavior and external factors (difficult circumstances) to justify the same behavior in themselves.

Response bias is a general term used to describe a number of different conditions or factors that cue respondents to provide inaccurate or false answers during surveys or interviews. These factors range from the interviewer’s perceived social position or appearance to the the phrasing of questions in surveys.

Nonresponse bias occurs when the people who complete a survey are different from those who did not, in ways that are relevant to the research topic. Nonresponse can happen because people are either not willing or not able to participate.

Is this article helpful?

Other students also liked.

Attrition Bias | Examples, Explanation, Prevention
Observer Bias | Definition, Examples, Prevention
What Is Social Desirability Bias? | Definition & Examples

8.4 Introduction to sources of bias in clinical trials

The reliability of the results of a randomized trial depends on the extent to which potential sources of bias have been avoided. A key part of a review is to consider the risk of bias in the results of each of the eligible studies. A useful classification of biases is into selection bias, performance bias, attrition bias, detection bias and reporting bias. In this section we describe each of these biases and introduce seven corresponding domains that are assessed in the Collaboration’s ‘Risk of bias’ tool. These are summarized in Table 8.4.a . We describe the tool for assessing the seven domains in Section 8.5 . We provide more detailed consideration of each issue in Sections 8.9 to 8.15 .

8.4.1 Selection bias

Selection bias refers to systematic differences between baseline characteristics of the groups that are compared. The unique strength of randomization is that, if successfully accomplished, it prevents selection bias in allocating interventions to participants. Its success in this respect depends on fulfilling several interrelated processes. A rule for allocating interventions to participants must be specified, based on some chance (random) process. We call this sequence generation . Furthermore, steps must be taken to secure strict implementation of that schedule of random assignments by preventing foreknowledge of the forthcoming allocations. This process if often termed allocation concealment , although could more accurately be described as allocation sequence concealment. Thus, one suitable method for assigning interventions would be to use a simple random (and therefore unpredictable) sequence, and to conceal the upcoming allocations from those involved in enrolment into the trial.

For all potential sources of bias, it is important to consider the likely magnitude and the likely direction of the bias. For example, if all methodological limitations of studies were expected to bias the results towards a lack of effect, and the evidence indicates that the intervention is effective, then it may be concluded that the intervention is effective even in the presence of these potential biases.

8.4.2 Performance bias

Performance bias refers to systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest. . After enrolment into the study, blinding (or masking) of study participants and personnel may reduce the risk that knowledge of which intervention was received, rather than the intervention itself, affects outcomes. Effective blinding can also ensure that the compared groups receive a similar amount of attention, ancillary treatment and diagnostic investigations. Blinding is not always possible, however. For example, it is usually impossible to blind people to whether or not major surgery has been undertaken.

8.4.3 Detection bias

Detection bias refers to systematic differences between groups in how outcomes are determined. Blinding (or masking) of outcome assessors may reduce the risk that knowledge of which intervention was received, rather than the intervention itself, affects outcome measurement. Blinding of outcome assessors can be especially important for assessment of subjective outcomes, such as degree of postoperative pain.

8.4.4 Attrition bias

Attrition bias refers to systematic differences between groups in withdrawals from a study. Withdrawals from the study lead to incomplete outcome data. There are two reasons for withdrawals or incomplete outcome data in clinical trials. Exclusions refer to situations in which some participants are omitted from reports of analyses, despite outcome data being available to the trialists. Attrition refers to situations in which outcome data are not available.

8.4.5 Reporting bias

Reporting bias refers to systematic differences between reported and unreported findings. Within a published report those analyses with statistically significant differences between intervention groups are more likely to be reported than non-significant differences. This sort of ‘within-study publication bias’ is usually known as outcome reporting bias or selective reporting bias, and may be one of the most substantial biases affecting results from individual studies (Chan 2005).

8.4.6 Other biases

In addition there are other sources of bias that are relevant only in certain circumstances. These relate mainly to particular trial designs (e.g. carry-over in cross-over trials and recruitment bias in cluster-randomized trials); some can be found across a broad spectrum of trials, but only for specific circumstances (e.g. contamination, whereby the experimental and control interventions get ‘mixed’, for example if participants pool their drugs); and there may be sources of bias that are only found in a particular clinical setting.

Open access
Published: 13 April 2010

Reporting bias in medical research - a narrative review

Natalie McGauran 1 ,
Beate Wieseler 1 ,
Julia Kreis 1 ,
Yvonne-Beatrice Schüler 1 ,
Heike Kölsch 1 &
Thomas Kaiser 1

Trials volume 11 , Article number: 37 ( 2010 ) Cite this article

99k Accesses

292 Citations

147 Altmetric

Metrics details

Reporting bias represents a major problem in the assessment of health care interventions. Several prominent cases have been described in the literature, for example, in the reporting of trials of antidepressants, Class I anti-arrhythmic drugs, and selective COX-2 inhibitors. The aim of this narrative review is to gain an overview of reporting bias in the medical literature, focussing on publication bias and selective outcome reporting. We explore whether these types of bias have been shown in areas beyond the well-known cases noted above, in order to gain an impression of how widespread the problem is. For this purpose, we screened relevant articles on reporting bias that had previously been obtained by the German Institute for Quality and Efficiency in Health Care in the context of its health technology assessment reports and other research work, together with the reference lists of these articles.

We identified reporting bias in 40 indications comprising around 50 different pharmacological, surgical (e.g. vacuum-assisted closure therapy), diagnostic (e.g. ultrasound), and preventive (e.g. cancer vaccines) interventions. Regarding pharmacological interventions, cases of reporting bias were, for example, identified in the treatment of the following conditions: depression, bipolar disorder, schizophrenia, anxiety disorder, attention-deficit hyperactivity disorder, Alzheimer's disease, pain, migraine, cardiovascular disease, gastric ulcers, irritable bowel syndrome, urinary incontinence, atopic dermatitis, diabetes mellitus type 2, hypercholesterolaemia, thyroid disorders, menopausal symptoms, various types of cancer (e.g. ovarian cancer and melanoma), various types of infections (e.g. HIV, influenza and Hepatitis B), and acute trauma. Many cases involved the withholding of study data by manufacturers and regulatory agencies or the active attempt by manufacturers to suppress publication. The ascertained effects of reporting bias included the overestimation of efficacy and the underestimation of safety risks of interventions.

In conclusion, reporting bias is a widespread phenomenon in the medical literature. Mandatory prospective registration of trials and public access to study data via results databases need to be introduced on a worldwide scale. This will allow for an independent review of research data, help fulfil ethical obligations towards patients, and ensure a basis for fully-informed decision making in the health care system.

Peer Review reports

The reporting of research findings may depend on the nature and direction of results, which is referred to as "reporting bias" [ 1 , 2 ]. For example, studies in which interventions are shown to be ineffective are sometimes not published, meaning that only a subset of the relevant evidence on a topic may be available [ 1 , 2 ]. Various types of reporting bias exist (Table 1 ), including publication bias and outcome reporting bias, which concern bias from missing outcome data on 2 levels: the study level, i.e. "non-publication due to lack of submission or rejection of study reports", and the outcome level, i.e. "the selective non-reporting of outcomes within published studies" [ 3 ].

Reporting bias on a study level

Results of clinical research are largely underreported or reported with delay. Various analyses of research protocols submitted to institutional review boards and research ethics committees in Europe, the United States, and Australia found that on average, only about half of the protocols had been published, with higher publication rates in Anglo-Saxon countries [ 4 – 10 ].

Similar analyses have been performed of trials submitted to regulatory authorities: a cohort study of trials supporting new drugs approved by the Food and Drug Administration (FDA) identified over 900 trials of 90 new drugs in FDA reviews; only 43% of the trials were published [ 11 ]. Wide variations in publication rates have been shown for specific indications [ 12 – 16 ]. The selective submission of clinical trials with positive outcomes to regulatory authorities has also been described [ 17 ]. Even if trials are published, the time lapse until publication may be substantial [ 8 , 18 , 19 ].

There is no simple classification of a clinical trial into "published" or "unpublished", as varying degrees of publication exist. These range from full-text publications in peer-reviewed journals that are easily identifiable through a search in bibliographic databases, to study information entered in trial registries, so-called grey literature (e.g. abstracts and working papers), and data on file in drug companies and regulatory agencies, which may or may not be provided to health technology assessment (HTA) agencies or other researchers after being requested. If such data are transmitted, they may then be fully published or not (e.g. the German Institute for Quality and Efficiency in Health Care [Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, IQWiG] publishes all data used in its assessment reports [ 20 ], whereas the UK National Institute for Clinical Excellence [NICE] may accept "commercial in confidence" data [ 21 ]).

Even if studies are presented at meetings, this does not necessarily mean subsequent full publication: an analysis of nearly 30,000 meeting abstracts from various disciplines found a publication rate of 63% for randomized or controlled clinical trials [ 22 ].

Reporting bias on an outcome level

Selective reporting within a study may involve (a) selective reporting of analyses or (b) selective reporting of outcomes. This may include, for example, the reporting of (a) per-protocol (PP) versus intention-to-treat (ITT) analyses or adjusted versus unadjusted analyses, and (b) outcomes from different time points or statistically significant versus non-significant outcomes [ 3 , 23 ].

Various reviews have found extensive selective reporting in study publications [ 3 , 14 , 24 – 28 ]. For example, comparisons of publications with study protocols have shown that primary outcomes had been newly introduced, omitted, or changed in about 40% to 60% of cases [ 3 , 24 ]. Selective reporting particularly concerns the underreporting of adverse events [ 12 , 29 – 32 ]. For example, an analysis of 192 randomized drug trials in various indications showed that only 46% of publications stated the frequency of specific reasons for treatment discontinuation due to toxicity [ 29 ]. Outcomes are not only selectively reported, but negative results are reported in a positive manner and conclusions are often not supported by results data [ 16 , 26 , 33 – 35 ]. For instance, a comparison of study characteristics reported in FDA reviews of New Drug Applications (NDAs) with those reported in publications found that 9 of 99 conclusions had been changed in the publications, all in favour of the new drug [ 26 ].

Factors associated with reporting bias

Characteristics of published studies.

The fact that studies with positive or favourable results are more likely to be published than those with negative or unfavourable results was already addressed in the 1950s [ 36 ], and has since been widely confirmed [ 3 , 6 – 8 , 14 , 37 – 40 ]. Studies with positive or favourable results have been associated with various other factors such as faster publication [ 8 , 18 , 19 , 37 ], publication in higher impact factor journals [ 7 , 41 ], a greater number of publications [ 7 ] (including covert duplicate publications [ 42 ]), more frequent citation [ 43 – 45 ], and more likely publication in English [ 46 ].

Several other factors have been linked to successful publication, for example, methodological quality [ 47 ], study type [ 47 ], sample size [ 5 , 7 , 48 ], multicentre status [ 5 , 6 , 41 ], and non-commercial funding [ 5 , 6 , 49 , 50 ]. However, for some factors, these associations are inconsistent [ 6 , 37 ].

Submission and rejection of studies

One of the main reasons for the non-publication of negative studies seems to be the non-submission of manuscripts by investigators, not the rejection of manuscripts by medical journals. A follow-up of studies approved by US institutional review boards showed that only 6 of 124 unpublished studies had actually been rejected for publication [ 6 ]. A prospective cohort study of 745 manuscripts submitted to JAMA showed no statistically significant difference in publication rates between studies with positive and those with negative results [ 51 ], which has been confirmed by further analyses of other journals [ 47 , 52 ]. Author surveys have shown that the most common reasons for not submitting papers were negative results and a lack of interest, time, or other resources [ 39 – 41 , 53 ].

The role of the pharmaceutical industry

An association has been shown between industry sponsorship or industry affiliation of authors and positive research outcomes and conclusions, both in publications of primary studies and in systematic reviews [ 49 , 54 – 63 ]. For example, in a systematic review of the scope and impact of financial conflicts of interest in biomedical research, an aggregation of the results of 8 analyses of the relation between industry sponsorship and outcomes showed a statistically significant association between industry sponsorship and pro-industry conclusions [ 55 ]. A comparison of the methodological quality and conclusions in Cochrane reviews with those in industry-supported meta-analyses found that the latter were less transparent, less critical of methodological limitations of the included trials, and drew more favourable conclusions [ 57 ]. In addition, publication constraints and active attempts to prevent publication have been identified in industry-sponsored research [ 55 , 64 – 68 ]. Other aspects of industry involvement, such as design bias, are beyond the scope of this paper.

Rationale, aim and procedure

IQWiG produces HTA reports of drug and non-drug interventions for the decision-making body of the statutory health care funds, the Federal Joint Committee. The process of report production includes requesting information on published and unpublished studies from manufacturers; unfortunately, compliance by manufacturers is inconsistent, as recently shown in the attempted concealment of studies on antidepressants [ 69 ]. Reporting bias in antidepressant research has been shown before [ 16 , 70 ]; other well-known cases include Class I anti-arrhythmic drugs [ 71 , 72 ] and selective COX-2 inhibitors [ 73 , 74 ].

The aim of this narrative review was to gain an overview of reporting bias in the medical literature, focussing on publication bias and selective outcome reporting. We wished to explore whether this type of bias has been shown in areas beyond the well-known cases noted above, in order to obtain an impression of how widespread this problem is. The review was based on the screening of full-text publications on reporting bias that had either been obtained by the Institute in the context of its HTA reports and other research work or were identified by the screening of the reference lists of the on-site publications. The retrieved examples were organized according to indications and interventions. We also discuss the effects of reporting bias, as well as the measures that have been implemented to solve this problem.

The term "reporting bias" traditionally refers to the reporting of clinical trials and other types of studies; if one extends this term beyond experimental settings, for example, to the withholding of information on any beneficial medical innovation, then an early example of reporting bias was noted by Rosenberg in his article "Secrecy in medical research", which describes the invention of the obstetrical forceps. This device was developed by the Chamberlen brothers in Europe in the 17th century; however, it was kept secret for commercial reasons for 3 generations and as a result, many women and neonates died during childbirth [ 75 ]. In the context of our paper, we also considered this extended definition of reporting bias.

We identified reporting bias in 40 indications comprising around 50 different interventions. Examples were found in various sources, e.g. journal articles of published versus unpublished data, reviews of reporting bias, editorials, letters to the editor, newspaper reports, expert and government reports, books, and online sources. The following text summarizes the information presented in these examples. More details and references to the background literature are included in Additional file 1 : Table S2.

Mental and behavioural disorders

Reporting bias is common in psychiatric research (see below). This also includes industry-sponsorship bias [ 76 – 82 ].

Turner et al compared FDA reviews of antidepressant trials including over 12,000 patients with the matching publications and found that 37 out of 38 trials viewed as positive by the FDA were published [ 16 ]. Of the 36 trials having negative or questionable results according to the FDA, 22 were unpublished and 11 of the 14 published studies conveyed a positive outcome. According to the publications, 94% of the trials had positive results, which was in contrast to the proportion reported by the FDA (51%). The overall increase in effect size in the published trials was 32%. In a meta-analysis of data from antidepressant trials submitted to the FDA, Kirsch et al requested data on 6 antidepressants from the FDA under the Freedom of Information Act. However, the FDA did not disclose relevant data from 9 of 47 trials, all of which failed to show a statistically significant benefit over placebo. Data from 4 of these trials were available on the GlaxoSmithKline (GSK) website. In total, the missing data represented 38% of patients in sertraline trials and 23% of patients in citalopram trials. The analysis of trials investigating the 4 remaining antidepressants showed that drug-placebo differences in antidepressant efficacy were relatively small, even for severely depressed patients [ 83 ].

Selective serotonin reuptake inhibitors (SSRIs)

One of the biggest controversies surrounding unpublished data was the withholding of efficacy and safety data from SSRI trials. In a lawsuit launched by the Attorney General of the State of New York it was alleged that GSK had published positive information about the paediatric use of paroxetine in major depressive disorder (MDD), but had concealed negative safety and efficacy data [ 84 ]. The company had conducted at least 5 trials on the off-label use of paroxetine in children and adolescents but published only one, which showed mixed results for efficacy. The results of the other trials, which did not demonstrate efficacy and suggested a possible increased risk of suicidality, were suppressed [ 84 ]. As part of a legal settlement, GSK agreed to establish an online clinical trials registry containing results summaries for all GSK-sponsored studies conducted after a set date [ 85 , 86 ].

Whittington et al performed a systematic review of published versus unpublished data on SSRIs in childhood depression. While published data indicated a favourable risk-benefit profile for some SSRIs, the inclusion of unpublished data indicated a potentially unfavourable risk-benefit profile for all SSRIs investigated except fluoxetine [ 70 ].

Newer antidepressants

IQWiG published the preliminary results of an HTA report on reboxetine, a selective norepinephrine reuptake inhibitor, and other antidepressants. At least 4600 patients had participated in 16 reboxetine trials, but the majority of data were unpublished. Despite a request for information the manufacturer Pfizer refused to provide these data. Only data on about 1600 patients were analysable and IQWiG concluded that due to the risk of publication bias, no statement on the benefit or harm of reboxetine could be made [ 69 , 87 ]. The preliminary HTA report mentioned above also included an assessment of mirtazapine, a noradrenergic and specific serotonergic antidepressant. Four potentially relevant trials were identified in addition to 27 trials included in the assessment, but the manufacturer Essex Pharma did not provide the study reports. Regarding the other trials, the manufacturer did not send the complete study reports, so the full analyses were not available. IQWiG concluded that the results of the assessment of mirtazapine may have been biased by unpublished data [ 69 , 87 ]. After the behaviour of Pfizer and Essex Pharma had been widely publicized, the companies provided the majority of study reports for the final HTA report. The preliminary report's conclusion on the effects of mirtazapine was not affected by the additional data. For reboxetine, the analysis of the published and unpublished data changed the conclusion from "no statement possible" to "no benefit proven" [ 88 ].

Bipolar disorder

Lamotrigine.

A review by Nassir Ghaemi et al of data on lamotrigine in bipolar disorder provided on the GSK website showed that data from negative trials were available on the website but that the studies had not been published in detail or publications emphasized positive secondary outcomes instead of negative primary ones. Outside of the primary area of efficacy (prophylaxis of mood episodes), the drug showed very limited efficacy in indications such as acute bipolar depression, for which clinicians were supporting its use [ 35 ].

Gabapentin, a GABA analogue, was approved by the FDA in 1993 for a certain type of epilepsy, and in 2002 for postherpetic neuralgia. As of February 1996, 83% of gabapentin use was for epilepsy, and 17% for off-label indications (see the expert report by Abramson [ 89 ]). As the result of a comprehensive marketing campaign by Pfizer, the number of patients in the US taking gabapentin rose from about 430,000 to nearly 6 million between 1996 and 2001; this increase was solely due to off-label use for indications, including bipolar disorder. As of September 2001, 93.5% of gabapentin use was for off-label indications [ 89 ]. In a further expert report, Dickersin noted "extensive evidence of reporting bias" [ 34 ], which she further analysed in a recent publication with Vedula et al [ 90 ]. Concerning the trials of gabapentin for bipolar disorders, 2 of the 3 trials (all published) were negative for the primary outcome. However, these publications showed "extensive spin and misrepresentation of data" [ 34 ].

Schizophrenia

The Washington Post reported that a trial on quetiapine, an atypical antipsychotic, was "silenced" in 1997, the same year it was approved by the FDA to treat schizophrenia. The study ("Study 15") was not published. Patients taking quetiapine had shown high rates of treatment discontinuations and had experienced significant weight increases. However, data presented by the manufacturer AstraZeneca in 1999 at European and US meetings actually indicated that the drug helped psychotic patients lose weight [ 91 ].

Panic disorder

Turner described an example of reporting bias in the treatment of panic disorder: according to a review article, 3 "well designed studies" had apparently shown that the controlled-release formulation of paroxetine had been effective in patients with this condition. However, according to the corresponding FDA statistical review, only one study was strongly positive, the second study was non-significant regarding the primary outcome (and marginally significant for a secondary outcome), and the third study was clearly negative [ 92 ].

Further examples of reporting bias in research on mental and behavioural disorders are included in Additional file 1 : Table S2.

Disorders of the nervous system

Alzheimer's disease.

Internal company analyses and information provided by the manufacturer Merck & Co to the FDA on rofecoxib, a selective COX-2 inhibitor, were released during litigation procedures. The documents referred to trials investigating the effects of rofecoxib on the occurrence or progression of Alzheimer's disease. Psaty and Kronmal performed a review of these documents and 2 trial publications and showed that, although presenting mortality data, the publications had not included analyses or statistical tests of these data and both had concluded that regarding safety, rofecoxib was "well tolerated". In contrast, in April 2001, Merck's internal ITT analyses of pooled data from these 2 trials showed a significant increase in total mortality. However, this information was neither disclosed to the FDA nor published in a timely fashion [ 74 ]. Rofecoxib was taken off the market by Merck in 2004 [ 93 ], among allegations that the company had been aware of the safety risks since 2000 [ 73 ].

In their article "An untold story?", Lenzer and Brownlee reported the case of valdecoxib, another selective COX-2 inhibitor withdrawn from the market due to cardiovascular concerns [ 94 , 95 ]. In 2001, the manufacturer Pfizer had applied for approval in 4 indications, including acute pain. The application for acute pain was rejected and some of the information about the corresponding trials removed from the FDA website for confidentiality reasons. Further examples of reporting bias in research on pain are presented in Additional file 1 : Table S2.

According to the expert report by Dickersin, all 3 trials on gabapentin for migraine showed negative results for the primary outcome. Substantial reporting bias was present. One trial was fully published (seemingly with a redefined primary outcome showing positive results in a subgroup of patients), one was unpublished, and preliminary (positive) results were presented for the third trial [ 34 ].

Disorders of the circulatory system

Coronary heart disease (bleeding prophylaxis during bypass surgery).

In his article on observational studies on drug safety, Hiatt reported the case of aprotinin, an antifibrinolytic drug formerly marketed to reduce bleeding during heart bypass graft surgery. In 2006, data from 2 published observational studies indicated serious concerns about the drug's safety [ 96 ]. The FDA subsequently convened an expert meeting in which the safety data presented by the manufacturer Bayer did not reveal any increased risk of fatal or nonfatal cardiovascular events. However, it turned out that Bayer had not presented additional observational data, which, according to an FDA review, indicated that aprotinin may be associated with an increased risk of death and other serious adverse events. In November 2007 Bayer suspended the worldwide marketing of aprotinin, after requests and advice from various drug regulating authorities [ 97 ].

Prevention of arrhythmia

Class i anti-arrhythmic drugs.

In a clinical trial conducted in 1980, 9 out of 49 patients with suspected acute myocardial infarction treated with a class Ic anti-arrhythmic drug (lorcainide) died, versus only one patient in the placebo group; the investigators interpreted this finding as an "effect of chance" [ 71 ]. The development of lorcainide was discontinued for commercial reasons, and the results of the trial were not published until 1993. The investigators then stated that if the trial had been published earlier, it "might have provided an early warning of trouble ahead" [ 71 ]. Instead, during the 1980s, class I drugs were widely used, even though concerns as to their lack of effect were published as early as 1983 [ 98 , 99 ]. Further reviews and trials confirmed this suspicion, as well as an increase in mortality [ 100 – 102 ]. In his book "Deadly Medicine", Moore described the consequences as "America's worst drug disaster", which had "produced a death toll larger than the United States' combat losses in wars such as Korea and Vietnam" [ 72 ]. Further examples of reporting bias in research on disorders of the circulatory system are presented in Additional file 1 : Table S2.

Disorders of the digestive system

Irritable bowel syndrome.

Barbehenn et al compared a published trial on alosetron, a 5-HT 3 antagonist, in women with irritable bowel syndrome with data obtained from the FDA [ 103 ]. She noted that according to the graphics in the publication, which presented relative differences in pain and discomfort scores, the drug seemed effective. However, when plotting the absolute data from the FDA review, the data points were almost superimposable. After discussions with the FDA about potential serious side effects, the drug was withdrawn from the market by the manufacturer in 2000, but reapproved with restrictions in 2002 [ 104 ]. A further example of reporting bias in research on disorders of the digestive system is presented in Additional file 1 : Table S2.

Disorders of the genitourinary system/Perinatal medicine

Urinary incontinence.

Lenzer and Brownlee also reported cases of suicide in a trial investigating the selective serotonin and noradrenalin reuptake inhibitor duloxetine for a new indication, urinary incontinence in women. However, the FDA refused to release data on these cases, citing trade secrecy laws. These laws "permit companies to withhold all information, even deaths, about drugs that do not win approval for a new indication, even when the drug is already on the market for other indications" [ 94 ]. Two examples of reporting bias in perinatal research are presented in Additional file 1 : Table S2.

Disorders of the musculoskeletal system

Osteoarthritis.

In 2000, a trial comparing upper gastrointestinal toxicity of rofecoxib, a selective COX-2 inhibitor, and naproxen in over 8000 patients with rheumatoid arthritis reported that rofecoxib was associated with significantly fewer clinically important upper gastrointestinal events. The significantly lower myocardial infarction rate in the naproxen group was attributed to a cardioprotective effect of naproxen (VIGOR trial, [ 105 ]). Concerns about the risk of selective COX-2-inhibitor-related cardiovascular events were raised as early as 2001 [ 106 ], and in 2002, an analysis including previously unpublished data from FDA reports of the VIGOR trial showed a statistically significant increase of serious cardiovascular thrombotic events in patients using rofecoxib [ 107 ].

In their article on access to pharmaceutical data at the FDA, Lurie and Zieve presented the example of the selective COX-2 inhibitor celecoxib: in a journal publication of a trial investigating the gastrointestinal toxicity with celecoxib versus other pain medications, the study authors concluded that the drug was associated with a lower incidence of gastrointestinal ulcers after 6 months of therapy [ 108 , 109 ]. However, they failed to disclose that at the time of publication they had already received data for the full study period (12 months), which showed no advantage over the comparator drugs for the above outcome [ 109 ].

Disorders of the skin

Atopic dermatitis, evening primrose oil.

In his editorial "Evening primrose oil for atopic dermatitis - Time to say goodnight", Williams reported that he and his colleague, who had performed an individual patient meta-analysis of evening primrose oil for atopic dermatitis commissioned by the UK Department of Health, were not given permission to publish their report, which included 10 previously unpublished studies. After submission of the report to the Department of Health, Searle, the company then responsible for product marketing, required the authors and referees to sign a written statement that the contents of the report had not been leaked. Other research had not shown convincing evidence of a benefit, and in 2002 the UK Medicines Control Agency withdrew marketing authorisation [ 66 ].

Endocrine and metabolic disorders

Diabetes mellitus type 2.

Rosiglitazone

The US cardiologist Steven Nissen commented on safety issues surrounding rosiglitazone, a thiazolidinedione used to treat type 2 diabetes. After the drug's approval, the FDA was informed in August 2005 by the manufacturer GSK that it had performed a meta-analysis of 42 randomized clinical trials of rosiglitazone, which suggested a 31% increase in the risk of ischaemic cardiovascular complications. GSK posted this finding on its website. However, neither GSK nor the FDA disseminated their findings in a broad way to the scientific community and the public [ 110 ]. The safety concerns were supported by a controversially discussed meta-analysis performed by Nissen and Wolski, who found that treatment with rosiglitazone was associated with a significantly increased risk of myocardial infarction and an increase in the risk of death from cardiovascular causes that had borderline significance [ 111 ]. More examples of reporting bias in diabetes research are presented in Additional file 1 : Table S2.

Hypercholesterolaemia

Ezetimibe and simvastatin.

In his article "Controversies surround heart drug study" Mitka described a trial that compared the 2 anticholesterol drugs ezetimibe and simvastatin versus simvastatin alone in patients with heterozygous familial hypercholesterolaemia [ 112 ]. No statistically significant difference between treatment groups was found for the primary outcome (mean change in the carotid intima-media thickness) after 2 years [ 113 ]. The trial, which was sponsored by Merck & Co. and Schering-Plough, was concluded in April 2006. A delay of almost 2 years in the reporting of results followed amidst allegations that the manufacturers had attempted to change the study endpoints prior to the publication of results [ 112 ]. A further case of reporting bias in research on ezetimibe is included in Additional file 1 : Table S2.

Cerivastatin

Psaty et al conducted a review of the published literature on the statin cerivastatin and also analysed internal company documents that became available during litigation procedures [ 114 ]. In the published literature, cerivastatin was associated with a substantially higher risk of rhabdomyolysis than other statins; this particularly referred to cerivastatin-gemfibrozil combination therapy. Cerivastatin was launched in the US in 1998 by Bayer, and within 3 to 4 months, internal documents indicated there had been multiple cases of cerivastatin-gemfibrozil interactions. However, it took more than 18 months until a contraindication about the concomitant use of the 2 drugs was added to the package insert. The unpublished data available in 1999 also suggested an association between high-dose cerivastatin monotherapy and rhabdomyolysis. In 1999/2000, the company analysed FDA adverse event reporting system data, which suggested that compared with atorvastatin, cerivastatin monotherapy substantially increased the risk of rhabdomyolysis. However, these findings were not disseminated or published. Cerivastatin was removed from the market in August 2001 [ 114 ]. In the same month, the German Ministry of Health accused Bayer of withholding vital information from its federal drug agency [ 115 ].

Thyroid disorders

Levothyroxine.

The Wall Street Journal reported the suppression of the results of a trial comparing the bioavailability of generic and brand-name levothyroxine products in the treatment of hypothyroidism; the investigators had concluded that the products were bioequivalent and in most cases interchangeable [ 116 , 117 ]. The trial was completed in 1990; over the next 7 years, the manufacturer of the brand-name product Synthroid, Boots pharmaceuticals, successfully delayed publication [ 65 ]. The manuscript was finally published in 1997.

Menopausal symptoms

A study investigating tibolone, a synthetic steroid, in breast-cancer patients with climacteric complaints was terminated prematurely after it was shown that this drug significantly increased the risk of cancer recurrence [ 118 ]. According to the German TV programme Frontal 21, the manufacturer (Schering-Plough, formerly NV Organon) informed regulatory authorities and ethics committees, as well as study centres and participants. However, the study results were not published until 1.5 years later [ 119 ].

Oncology is another area in which reporting bias is common [ 40 , 50 , 54 , 120 – 127 ]. A review of over 2000 oncology trials registered in ClinicalTrials.gov showed that less than 20% were available in PubMed, with substantial differences between trials sponsored by clinical trial networks and those sponsored by industry regarding both publication rates (59% vs. 6%) and the proportion of trials with positive results (50% vs. 75%) [ 50 ].

Ovarian cancer

Combination chemotherapy.

In one of the earliest publications measuring the effects of reporting bias, Simes compared published oncology trials and trials identified in cancer registries that investigated the survival impact of initial alkylating agent (AA) therapy versus combination chemotherapy (CC) in advanced ovarian cancer. A meta-analysis of the published trials showed a significant survival advantage for CC; however, no such advantage was shown in the meta-analysis of registered trials [ 121 ].

Multiple myeloma

The above study also investigated the survival impact of AA/prednisone versus CC in multiple myeloma. The meta-analysis of published trials demonstrated a significant survival advantage for CC. A survival benefit was also shown in the registered trials; however, the estimated magnitude of the benefit was reduced [ 121 ]. A further example of reporting bias in cancer research is presented in Additional file 1 : Table S2.

Disorders of the blood

Thalassaemia major, iron-chelation agents.

In his editorial "Thyroid storm", Rennie, among other things, discussed events surrounding a US researcher who had been involved in a trial investigating the effects of an oral iron-chelation agent in patients with thalassaemia major. She had initially published an optimistic article on the effects of this agent. However, further research showed a lack of effectiveness and a potential safety risk. She had signed a confidentiality agreement but, because of her concerns, decided to break confidentiality and report her results at a meeting; the manufacturer unsuccessfully attempted to block her presentation [ 128 ].

Bacterial, fungal, and viral infections

Oseltamivir.

The BMJ and Channel 4 News reported on the difficulties in obtaining data for an updated Cochrane review on neuraminidase inhibitors in influenza [ 129 ]. A previous analysis of oseltamivir, which was used in the prior Cochrane review [ 130 ], was based on 10 industry-sponsored trials of which only 2 had been published in peer-reviewed journals [ 131 ]. The manufacturer Roche initially declined to provide the necessary data to reproduce the analysis and then only provided a selection of files [ 129 ]. The Cochrane authors (Jefferson et al) subsequently concluded that "Evidence on the effects of oseltamivir in complications from lower respiratory tract infections, reported in our 2006 Cochrane review, may be unreliable" [ 132 ]. Roche has since agreed to provide public access to study summaries and password-protected access to the full study reports [ 129 ].

Anti-HIV agents

Ioannidis et al identified several examples of publication bias in trials investigating medications against HIV. At least 13 trials of 6 antiviral agents including at least 3779 patients had remained unpublished for more than 3 years from the time of their meeting presentation or completion. At least 9 of these trials had negative preliminary or final results. For example, 2 large negative isoprinosine trials were unpublished, whilst a positive trial had been published in a high impact journal [ 33 ]. Further examples of reporting bias in research on infections are presented in Additional file 1 : Table S2.

Acute trauma

Acute spinal cord injury, high-dose steroids.

Lenzer and Brownlee described the concerns of neurosurgeons regarding the use of high-dose steroids in patients with acute spinal cord injury. They noted that one neurosurgeon believed that several thousand patients had died as a result of this intervention; 2 surveys showed that many other neurosurgeons shared his concerns. The single available study, which had been funded by the NIH, was potentially flawed and several researchers had unsuccessfully lobbied for the release of the underlying data [ 94 ].

Human albumin infusion

In the UK Health Committee's 2004-2005 report on the influence of the pharmaceutical industry, Chalmers mentioned a systematic review of human albumin solution, which is used in the treatment of shock, e.g. in patients with burns. The results showed no evidence that albumin was helpful and suggested that this intervention may actually be harmful. Although the UK Medicines Control Agency subsequently slightly modified the labelling, it kept confidential the evidence upon which the drug had been re-licensed in 1993 [ 133 , 134 ].

Vaccinations

Hiv-1 vaccine.

McCarthy reported the case of an HIV-1 vaccine study that was terminated early when no difference in efficacy between the vaccine and placebo was found. After the lead investigators refused to include a post-hoc analysis arguing that it had not been part of the study protocol and that invalid statistical methods had been used, the manufacturer, Immune Response, filed an (unsuccessful) claim seeking to prevent publication. After publication, the manufacturer filed a claim against the study's lead investigators and their universities asking for US $7-10 million in damages [ 135 ].

Cancer vaccines

Rosenberg provided various examples of how researchers and companies withheld information on cancer vaccines for competitive reasons; for example, researchers were asked to keep information confidential that might have prevented cancer patients from receiving ineffective or even harmful doses of a new agent [ 75 ].

Other indications

Nocturnal leg cramps.

Man-Song-Hing et al performed a meta-analysis including unpublished individual patient data (IPD) obtained from the FDA on trials investigating quinine for the treatment of nocturnal leg cramps. They showed that the pooling only of published studies overestimated efficacy by more than 100% [ 136 ]. Further examples of reporting bias in other indications are presented in Additional file 1 : Table S2.

Further research areas

Reporting bias has also been shown in other research areas, such as genetics [ 137 , 138 ], effects of passive smoking [ 139 , 140 ] and nicotine [ 141 , 142 ], and effects of air pollution [ 143 ].

The numerous examples identified show that reporting bias concerns not only previously highlighted therapies such as antidepressants, pain medication, or cancer drugs, but affects a wide range of indications and interventions. Many cases involved the withholding of study data by manufacturers and regulatory agencies or the active attempt to suppress publication by manufacturers, which either resulted in substantial delays in publication (time-lag bias) or no publication at all.

Limitations of the review

The review does not provide a complete overview of reporting bias in clinical research. Although our efforts to identify relevant literature went beyond the usual efforts applied in narrative reviews, the review is non-systematic and we emphasized this feature in the title. A substantial amount of relevant literature was available in-house and further relevant literature was obtained by screening reference lists. We dispensed with our initial plan to conduct a systematic review to identify cases of reporting bias, as we noticed that many cases were not identifiable by screening titles and abstracts of citations from bibliographic databases, but were "hidden" in the discussion sections of journal articles or mentioned in other sources such as newspapers, books, government reports or websites. As a search of bibliographic databases and the Internet using keywords related to reporting bias produces thousands of potentially relevant hits, we would therefore have had to obtain and read an excessive amount of full texts in order to ensure that we had not missed any examples. This was not feasible due to resource limitations. However, within the framework of a previous publication [ 144 ] we had conducted a literature search in PubMed, and some of the citations retrieved formed the basis of our literature pool for the current review. In spite of this non-systematic approach, we were able to identify dozens of cases of reporting bias in numerous indications.

Another potential limitation of the review is the validity of the sources describing cases of reporting bias. Although the majority of examples were identified in peer-reviewed journals, several cases were based on information from other sources such as newspaper articles and websites. However, we also regard these sources to be valuable as they provide a broader overview of reporting bias beyond well-known examples and also offer a starting point for more systematic research on the additional examples identified.

Effects of reporting bias

Published evidence tends to overestimate efficacy and underestimate safety risks. The extent of misestimation is often unknown. The few identified comparisons that quantified overestimates of treatment effects in fully published versus unpublished or not fully published data showed wide variations in their results. Comparisons of pooled published versus pooled published and unpublished FDA data showed a greater treatment effect of 11% to 69% for individual antidepressants, 32% for the class of antidepressants [ 16 ], and over 100% for an agent to treat nocturnal leg cramps [ 136 ]. In addition, published studies have shown a 9% to 15% greater treatment effect than grey literature studies [ 145 , 146 ]. Thus, the conclusions of systematic reviews and meta-analyses based on published evidence alone may be misleading [ 5 , 7 , 38 ]. This is a serious concern as these documents are being used increasingly to support decision making in the health care system. Reporting bias may consequently result in inappropriate health care decisions by policy makers and clinicians, which harm patients, waste resources, and misguide future research [ 4 , 5 , 34 ].

Trial registration and public access to study data

There is an ethical obligation to publish research findings [ 120 , 147 – 150 ]. For example, patients who participate in clinical trials do so in the belief that they are contributing to medical progress, and this will only be the case if these trials are published. Deliberate non- or selective reporting represents unethical behaviour and scientific misconduct [ 34 , 147 ]. Public access to study data may also help identify safety problems at an earlier stage, which in the past have in some cases not always been detected by regulatory authorities [ 151 – 153 ]. Two concepts can help solve the issue of reporting bias: firstly, the mandatory and prospective registration of clinical trials, and secondly, the mandatory publication of full study results in results databases after study completion.

Non-industry initiatives

One of the first searchable computerized international registries of clinical trials was introduced in the United States in 1967; since then, several national and international trial registries have been created [ 154 ], such as the US government's trial registry and results database ClinicalTrials.gov (see Tse et al for an update on this registry [ 155 , 156 ]). The various controversies surrounding reporting bias, particularly the non-reporting of safety data, accelerated the movement both for trial registration and the establishment of results databases. Numerous researchers, organizations, regulatory and governmental authorities started various initiatives to achieve these goals [ 148 , 157 – 165 ].

In 2004, the International Committee of Medical Journal Editors (ICMJE) announced that it would make registration of clinical trials in a public registry a condition of consideration for publication [ 158 ]; this statement has since been updated [ 166 , 167 ].

In 2006, the WHO established the International Clinical Trials Registry Platform (ICTRP) in an initiative to bring national trial registries together in a global network providing a single point of access to registered trials [ 157 ]. However, to date no consensus has been found between the parties involved concerning which characteristics must be made publicly available at registration [ 168 ].

Section 801 of the US FDA Amendments Act 2007 (FDAAA, [ 169 ]) requires the registration at inception of all clinical trials involving a drug, biological product, or device regulated by the FDA. Trials must be registered on ClinicalTrials.gov and a defined set of results must be posted in the same registry within 12 months of study completion. Exceptions are phase I drug trials and early feasibility device trials. Non-compliance is sanctioned with monetary fines [ 163 , 170 ].

In 2004, the European Agency for the Evaluation of Medicinal Products (now European Medicines Agency) launched the European clinical trials database EudraCT (eudract.emea.europa.eu) to provide national authorities with a common set of information on clinical trials conducted in the EU. The database was initially supposed to be available only to the responsible authorities of the member states, as well as to the European Commission and the European Medicines Agency [ 171 ]. In 2006, the regulation on medicinal products for paediatric use was published, which required that information about European paediatric clinical trials of investigational medicinal products was to be made publicly available on EudraCT [ 172 , 173 ], and in February 2009, the European Commission published a guideline including the list of data fields to be made public [ 174 ]. On the same date, a similar list was published for all trials [ 175 ]. However, the legal obligation to publish information on trials in adults is not fully clear, and it is also unclear when all relevant information from EudraCT will be made publicly accessible.

With the introduction of the above-mentioned legislation, regulatory agencies are on the one hand helping to solve the problem of reporting bias, but on the other hand, they are also part of the problem: several of the examples identified refer to the non-publication or active withholding of study data by regulatory agencies [ 83 , 94 , 109 , 133 ]. This is partly due to existing confidentiality regulations such as Exemption 4 of the US Freedom of Information Act [ 176 ]. To solve the problems resulting from this situation, current legislation has to be changed to allow for the publication of comprehensive information on study methods and results by regulatory agencies. In his essay "A taxpayer-funded clinical trials registry and results database", Turner called for increased access to the FDA information sources, which would at least enable the assessment of drugs marketed in the USA [ 92 ]. Although the FDA posts selected reviews of NDAs on its website after the approval process following the Electronic Freedom of Information Act [ 177 ], the availability of these reviews is limited [ 92 ]. Moreover, according to the FDAAA, the results of older trials of approved drugs or of drugs that were never approved need not be disclosed [ 170 ], which is why a retrospective registry and results database is needed [ 178 ].

Industry initiatives

In 2002, the US Pharmaceutical Research and Manufacturers Association (PhRMA) member companies committed themselves to the registration of all hypothesis-testing clinical trials at initiation and to the timely disclosure of summary results, regardless of outcome [ 179 , 180 ]. PhRMA also launched the clinical study results database ClinicalStudyResults.org in 2004. In 2005, a similar commitment was made by several pharmaceutical industry associations [ 181 ], which has since been updated [ 182 ]. Following the legal settlement in the paroxetine case, GSK established a trial registry on its website gsk-clinicalstudyregister.com and other large companies have followed. In 2008, the German Association of Research-Based Pharmaceutical Companies (VFA) published a position paper on the issue of publication bias and claimed that, because of the voluntary self-commitment of the pharmaceutical industry and the introduction of legislation for the reporting of study data, publication bias had become a "historical" topic [ 183 ]. However, even after the update of the position paper in January 2009 [ 184 ], in Germany alone further attempts by drug companies to withhold study data have occurred [ 69 ], which shows that voluntary self-commitment is insufficient.

Conclusions

Reporting bias is widespread in the medical literature and has harmed patients in the past. Mandatory prospective registration of trials and public access to study data via results databases need to be introduced on a worldwide level. This would help fulfil ethical obligations towards patients by enabling proactive publication and independent reviews of clinical trial data, and ensure a basis for fully informed decision making in the health care system. Otherwise, clinical decision making based on the "best evidence" will remain an illusion.

Green S, Higgins S, editors: Glossary. Cochrane Handbook for Systematic Reviews of Interventions 4.2.5. Last update May 2005 [accessed 22 Feb 2010], http://www.cochrane.org/resources/handbook/

Sterne J, Egger M, Moher D: Addressing reporting biases. Cochrane handbook for systematic reviews of interventions. Edited by: Higgins JPT, Green S. 2008, Chichester: Wiley, 297-334. full_text.

Google Scholar

Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, Ghersi D, Ioannidis JP, Simes J, Williamson PR: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE. 2008, 3: e3081-10.1371/journal.pone.0003081.

PubMed PubMed Central Google Scholar

Blumle A, Antes G, Schumacher M, Just H, Von Elm E: Clinical research projects at a German medical faculty: follow-up from ethical approval to publication and citation by others. J Med Ethics. 2008, 34: e20-10.1136/jme.2008.024521.

CAS PubMed Google Scholar

Von Elm E, Rollin A, Blumle A, Huwiler K, Witschi M, Egger M: Publication and non-publication of clinical trials: longitudinal study of applications submitted to a research ethics committee. Swiss Med Wkly. 2008, 138: 197-203.

PubMed Google Scholar

Dickersin K, Min YI, Meinert CL: Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. JAMA. 1992, 267: 374-378. 10.1001/jama.267.3.374.

Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR: Publication bias in clinical research. Lancet. 1991, 337: 867-872. 10.1016/0140-6736(91)90201-Y.

Stern JM, Simes RJ: Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997, 315: 640-645.

CAS PubMed PubMed Central Google Scholar

Pich J, Carne X, Arnaiz JA, Gomez B, Trilla A, Rodes J: Role of a research ethics committee in follow-up and publication of results. Lancet. 2003, 361: 1015-1016. 10.1016/S0140-6736(03)12799-7.

Decullier E, Lheritier V, Chapuis F: Fate of biomedical research protocols and publication bias in France: retrospective cohort study. BMJ. 2005, 331: 19-24. 10.1136/bmj.38488.385995.8F.

Lee K, Bacchetti P, Sim I: Publication of clinical trials supporting successful new drug applications: a literature analysis. PLoS Med. 2008, 5: e191-10.1371/journal.pmed.0050191.

Hemminki E: Study of information submitted by drug companies to licensing authorities. Br Med J. 1980, 280: 833-836. 10.1136/bmj.280.6217.833.

MacLean CH, Morton SC, Ofman JJ, Roth EA, Shekelle PG: How useful are unpublished data from the Food and Drug Administration in meta-analysis?. J Clin Epidemiol. 2003, 56: 44-51. 10.1016/S0895-4356(02)00520-6.

Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B: Evidence b(i)ased medicine: selective reporting from studies sponsored by pharmaceutical industry; review of studies in new drug applications. BMJ. 2003, 326: 1171-1173. 10.1136/bmj.326.7400.1171.

Benjamin DK, Smith PB, Murphy MD, Roberts R, Mathis L, Avant D, Califf RM, Li JS: Peer-reviewed publication of clinical trials completed for pediatric exclusivity. JAMA. 2006, 296: 1266-1273. 10.1001/jama.296.10.1266.

Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R: Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008, 358: 252-260. 10.1056/NEJMsa065779.

Bardy AH: Bias in reporting clinical trials. Br J Clin Pharmacol. 1998, 46: 147-150. 10.1046/j.1365-2125.1998.00759.x.

Ioannidis JP: Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA. 1998, 279: 281-286. 10.1001/jama.279.4.281.

Hopewell S, Clarke M, Stewart L, Tierney J: Time to publication for results of clinical trials. Cochrane Database Syst Rev. 2007, MR000011-2

Institute for Quality and Efficiency in Health Care: General methods: version 3.0. Last update 27 May 2008 [accessed 22 Feb 2010], http://www.iqwig.de/download/IQWiG_General_methods_V-3-0.pdf

National Institute for Health and Clinical Excellence: Guide to the methods of technology appraisal. London. 2008

Scherer RW, Langenberg P, Von Elm E: Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2007, MR000005-2

Altman D: Outcome reporting bias in meta-analyses. Last update 2007 [accessed 24 Feb 2010], http://www.chalmersresearch.com/bmg/docs/t2p1.pdf

Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004, 291: 2457-2465. 10.1001/jama.291.20.2457.

Chan AW, Altman DG: Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005, 330: 753-10.1136/bmj.38356.424606.8F.

Rising K, Bacchetti P, Bero L: Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Med. 2008, 5: e217-10.1371/journal.pmed.0050217.

Chan AW, Krleza-Jeric K, Schmid I, Altman DG: Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004, 171: 735-740.

Al-Marzouki S, Roberts I, Evans S, Marshall T: Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet. Lancet. 2008, 372: 201-10.1016/S0140-6736(08)61060-0.

Ioannidis JP, Lau J: Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. JAMA. 2001, 285: 437-443. 10.1001/jama.285.4.437.

Hazell L, Shakir SA: Under-reporting of adverse drug reactions: a systematic review. Drug Saf. 2006, 29: 385-396. 10.2165/00002018-200629050-00003.

Bonhoeffer J, Zumbrunn B, Heininger U: Reporting of vaccine safety data in publications: systematic review. Pharmacoepidemiol Drug Saf. 2005, 14: 101-106. 10.1002/pds.979.

Loke YK, Derry S: Reporting of adverse drug reactions in randomised controlled trials: a systematic survey. BMC Clin Pharmacol. 2001, 1: 3-10.1186/1472-6904-1-3.

Ioannidis JP, Cappelleri JC, Sacks HS, Lau J: The relationship between study design, results, and reporting of randomized clinical trials of HIV infection. Control Clin Trials. 1997, 18: 431-444. 10.1016/S0197-2456(97)00097-4.

Dickersin K: Reporting and other biases in studies of Neurontin for migraine, psychiatric/bipolar disorders, nociceptive pain, and neuropathic pain. Last update 10 Aug 2008 [accessed 26 Feb 2010], http://dida.library.ucsf.edu/pdf/oxx18r10

Nassir Ghaemi S, Shirzadi AA, Filkowski M: Publication bias and the pharmaceutical industry: the case of lamotrigine in bipolar disorder. Medscape J Med. 2008, 10: 211-

Sterling T: Publication decisions and their possible effects on inferences drawn from tests of significances. J Am Stat Assoc. 1959, 54: 30-34. 10.2307/2282137.

Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K: Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009, MR000006-1

Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ: Publication and related biases. Health Technol Assess. 2000, 4: 1-115.

Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H: Publication bias and clinical trials. Control Clin Trials. 1987, 8: 343-353. 10.1016/0197-2456(87)90155-3.

Krzyzanowska MK, Pintilie M, Tannock IF: Factors associated with failure to publish large randomized trials presented at an oncology meeting. JAMA. 2003, 290: 495-501. 10.1001/jama.290.4.495.

Timmer A, Hilsden RJ, Cole J, Hailey D, Sutherland LR: Publication bias in gastroenterological research: a retrospective cohort study based on abstracts submitted to a scientific meeting. BMC Med Res Methodol. 2002, 2: 7-10.1186/1471-2288-2-7.

Tramer MR, Reynolds DJ, Moore RA, McQuay HJ: Impact of covert duplicate publication on meta-analysis: a case study. BMJ. 1997, 315: 635-640.

Gotzsche PC: Reference bias in reports of drug trials. Br Med J (Clin Res Ed). 1987, 295: 654-656. 10.1136/bmj.295.6599.654.

CAS Google Scholar

Kjaergard LL, Gluud C: Citation bias of hepato-biliary randomized clinical trials. J Clin Epidemiol. 2002, 55: 407-410. 10.1016/S0895-4356(01)00513-3.

Ravnskov U: Quotation bias in reviews of the diet-heart idea. J Clin Epidemiol. 1995, 48: 713-719. 10.1016/0895-4356(94)00222-C.

Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G: Language bias in randomised controlled trials published in English and German. Lancet. 1997, 350: 326-329. 10.1016/S0140-6736(97)02419-7.

Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, Bero LA: Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust. 2006, 184: 621-626.

Callaham ML, Wears RL, Weber EJ, Barton C, Young G: Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA. 1998, 280: 254-257. 10.1001/jama.280.3.254.

Lexchin J, Bero LA, Djulbegovic B, Clark O: Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003, 326: 1167-1170. 10.1136/bmj.326.7400.1167.

Ramsey S, Scoggins J: Commentary: practicing on the tip of an information iceberg? Evidence of underpublication of registered clinical trials in oncology. Oncologist. 2008, 13: 925-929. 10.1634/theoncologist.2008-0133.

Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, Zhu Q, Reiling J, Pace B: Publication bias in editorial decision making. JAMA. 2002, 287: 2825-2828. 10.1001/jama.287.21.2825.

Okike K, Kocher MS, Mehlman CT, Heckman JD, Bhandari M: Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the Journal of Bone and Joint Surgery (American Volume). J Bone Joint Surg Am. 2008, 90: 595-601. 10.2106/JBJS.G.00279.

Dickersin K, Min YI: NIH clinical trials and publication bias. Online J Curr Clin Trials. 1993, Doc No 50:[4967 words; 4953 paragraphs].

Hartmann M, Knoth H, Schulz D, Knoth S: Industry-sponsored economic studies in oncology vs studies sponsored by nonprofit organisations. Br J Cancer. 2003, 89: 1405-1408. 10.1038/sj.bjc.6601308.

Bekelman JE, Li Y, Gross CP: Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA. 2003, 289: 454-465. 10.1001/jama.289.4.454.

Sismondo S: Pharmaceutical company funding and its consequences: a qualitative systematic review. Contemp Clin Trials. 2008, 29: 109-113. 10.1016/j.cct.2007.08.001.

Jorgensen AW, Hilden J, Gotzsche PC: Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ. 2006, 333: 782-10.1136/bmj.38973.444699.0B.

Liss H: Publication bias in the pulmonary/allergy literature: effect of pharmaceutical company sponsorship. Isr Med Assoc J. 2006, 8: 451-454.

Ridker PM, Torres J: Reported outcomes in major cardiovascular clinical trials funded by for-profit and not-for-profit organizations: 2000-2005. JAMA. 2006, 295: 2270-2274. 10.1001/jama.295.19.2270.

Als-Nielsen B, Chen W, Gluud C, Kjaergard LL: Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events?. JAMA. 2003, 290: 921-928. 10.1001/jama.290.7.921.

Perlis CS, Harwood M, Perlis RH: Extent and impact of industry sponsorship conflicts of interest in dermatology research. J Am Acad Dermatol. 2005, 52: 967-971. 10.1016/j.jaad.2005.01.020.

Bhandari M, Busse JW, Jackowski D, Montori VM, Schünemann H, Sprague S, Mears D, Schemitsch EH, Heels-Ansdell D, Devereaux PJ: Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. CMAJ. 2004, 170: 477-480.

Kjaergard LL, Als-Nielsen B: Association between competing interests and authors' conclusions: epidemiological study of randomised clinical trials published in the BMJ. BMJ. 2002, 325: 249-10.1136/bmj.325.7358.249.

Lauritsen K, Havelund T, Laursen LS, Rask-Madsen J: Withholding unfavourable results in drug company sponsored clinical trials. Lancet. 1987, 1: 1091-10.1016/S0140-6736(87)90515-0.

Wise J: Research suppressed for seven years by drug company. BMJ. 1997, 314: 1145-

Williams HC: Evening primrose oil for atopic dermatitis. BMJ. 2003, 327: 1358-1359. 10.1136/bmj.327.7428.1358.

Henry DA, Kerridge IH, Hill SR, McNeill PM, Doran E, Newby DA, Henderson KM, Maguire J, Stokes BJ, Macdonald GJ, Day RO: Medical specialists and pharmaceutical industry-sponsored research: a survey of the Australian experience. Med J Aust. 2005, 182: 557-560.

Gotzsche PC, Hrobjartsson A, Johansen HK, Haahr MT, Altman DG, Chan AW: Constraints on publication rights in industry-initiated clinical trials. JAMA. 2006, 295: 1645-1646. 10.1001/jama.295.14.1645.

Stafford N: German agency refuses to rule on drug's benefits until Pfizer discloses all trial results. BMJ. 2009, 338: b2521-10.1136/bmj.b2521.

Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E: Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet. 2004, 363: 1341-1345. 10.1016/S0140-6736(04)16043-1.

Cowley AJ, Skene A, Stainer K, Hampton JR: The effect of lorcainide on arrhythmias and survival in patients with acute myocardial infarction: an example of publication bias. Int J Cardiol. 1993, 40: 161-166. 10.1016/0167-5273(93)90279-P.

Moore TJ: Deadly medicine: why tens of thousands of heart patients died in America's worst drug disaster. 1995, New York: Simon & Schuster

Mathews A, Martinez B: E-mails suggest Merck knew Vioxx's dangers at early stage. Wall Street Journal. 2004, A1-

Psaty BM, Kronmal RA: Reporting mortality findings in trials of rofecoxib for Alzheimer disease or cognitive impairment: a case study based on documents from rofecoxib litigation. JAMA. 2008, 299: 1813-1817. 10.1001/jama.299.15.1813.

Rosenberg SA: Secrecy in medical research. N Engl J Med. 1996, 334: 392-394. 10.1056/NEJM199602083340610.

Baker CB, Johnsrud MT, Crismon ML, Rosenheck RA, Woods SW: Quantitative analysis of sponsorship bias in economic studies of antidepressants. Br J Psychiatry. 2003, 183: 498-506. 10.1192/bjp.183.6.498.

Moncrieff J: Clozapine v. conventional antipsychotic drugs for treatment-resistant schizophrenia: a re-examination. Br J Psychiatry. 2003, 183: 161-166. 10.1192/bjp.183.2.161.

Montgomery JH, Byerly M, Carmody T, Li B, Miller DR, Varghese F, Holland R: An analysis of the effect of funding source in randomized clinical trials of second generation antipsychotics for the treatment of schizophrenia. Control Clin Trials. 2004, 25: 598-612. 10.1016/j.cct.2004.09.002.

Procyshyn RM, Chau A, Fortin P, Jenkins W: Prevalence and outcomes of pharmaceutical industry-sponsored clinical trials involving clozapine, risperidone, or olanzapine. Can J Psychiatry. 2004, 49: 601-606.

Perlis RH, Perlis CS, Wu Y, Hwang C, Joseph M, Nierenberg AA: Industry sponsorship and financial conflict of interest in the reporting of clinical trials in psychiatry. Am J Psychiatry. 2005, 162: 1957-1960. 10.1176/appi.ajp.162.10.1957.

Heres S, Davis J, Maino K, Jetzinger E, Kissling W, Leucht S: Why olanzapine beats risperidone, risperidone beats quetiapine, and quetiapine beats olanzapine: an exploratory analysis of head-to-head comparison studies of second-generation antipsychotics. Am J Psychiatry. 2006, 163: 185-194. 10.1176/appi.ajp.163.2.185.

Kelly RE, Cohen LJ, Semple RJ, Bialer P, Lau A, Bodenheimer A, Neustadter E: Relationship between drug company funding and outcomes of clinical psychiatric research. Psychol Med. 2006, 36: 1647-1656. 10.1017/S0033291706008567.

Kirsch I, Deacon BJ, Huedo-Medina TB, Scoboria A, Moore TJ, Johnson BT: Initial severity and antidepressant benefits: a meta-analysis of data submitted to the Food and Drug Administration. PLoS Med. 2008, 5: e45-10.1371/journal.pmed.0050045.

Office of the Attorney General: Major pharmaceutical firm concealed drug information. Last update 02 Jun 2004 [accessed 24 Feb 2010], http://www.oag.state.ny.us/media_center/2004/jun/jun2b_04.html

Office of the Attorney General: Settlement sets new standard for release of drug information. Last update 26 Aug 2004 [accessed 26 Feb 2010], http://www.oag.state.ny.us/media_center/2004/aug/aug26a_04.html

Gibson L: GlaxoSmithKline to publish clinical trials after US lawsuit. BMJ. 2004, 328: 1513-10.1136/bmj.328.7455.1513-a.

Institute for Quality and Efficiency in Health Care: Bupropion, mirtazapine and reboxetine in the treatment of depression: executive summary of preliminary report; commission no A05-20C. Last update 29 May 2009 [accessed 26 Feb 2010], http://www.iqwig.de/download/A05-20C_Executive_summary_Bupropion_mirtazapine_and_reboxetine_in_the_treatment_of_depression.pdf

Institute for Quality and Efficiency in Health Care: Antidepressants: benefit of reboxetine not proven. Last update 24 Nov 2009 [accessed 26 Feb 2010], http://www.iqwig.de/antidepressants-benefit-of-reboxetine-not-proven.981.en.html

Abramson J: Expert report. Last update 11 Aug 2008 [accessed 26 Feb 2010], http://dida.library.ucsf.edu/pdf/oxx18v10

Vedula SS, Bero L, Scherer RW, Dickersin K: Outcome reporting in industry-sponsored trials of gabapentin for off-label use. N Engl J Med. 2009, 361: 1963-1971. 10.1056/NEJMsa0906126.

Vedantam S: A silenced drug study creates an uproar. Washington Post. 2009, A01-

Turner EH: A taxpayer-funded clinical trials registry and results database. PLoS Med. 2004, 1: e60-10.1371/journal.pmed.0010060.

Singh D: Merck withdraws arthritis drug worldwide. BMJ. 2004, 329: 816-10.1136/bmj.329.7470.816-a.

Lenzer J, Brownlee S: An untold story?. BMJ. 2008, 336: 532-534. 10.1136/bmj.39504.662685.0F.

Waknine Y: Bextra withdrawn from market. Medscape Today [Online]. 2005, http://www.medscape.com/viewarticle/502642

Hiatt WR: Observational studies of drug safety--aprotinin and the absence of transparency. N Engl J Med. 2006, 355: 2171-2173. 10.1056/NEJMp068252.

Tuffs A: Bayer withdraws heart surgery drug. BMJ. 2007, 335: 1015-10.1136/bmj.39395.644826.DB.

Furberg CD: Effect of antiarrhythmic drugs on mortality after myocardial infarction. Am J Cardiol. 1983, 52: 32C-36C. 10.1016/0002-9149(83)90629-X.

Antes G: Tödliche Medizin. Unpublizierte Studien - harmlos? [Fatal medicine. Unpublished studies - harmless?]. MMW Fortschr Med. 2006, 148: 8-

Hine LK, Laird N, Hewitt P, Chalmers TC: Meta-analytic evidence against prophylactic use of lidocaine in acute myocardial infarction. Arch Intern Med. 1989, 149: 2694-2698. 10.1001/archinte.149.12.2694.

MacMahon S, Collins R, Peto R, Koster RW, Yusuf S: Effects of prophylactic lidocaine in suspected acute myocardial infarction: an overview of results from the randomized, controlled trials. JAMA. 1988, 260: 1910-1916. 10.1001/jama.260.13.1910.

Preliminary report: effect of encainide and flecainide on mortality in a randomized trial of arrhythmia suppression after myocardial infarction. The Cardiac Arrhythmia Suppression Trial (CAST) Investigators. N Engl J Med. 1989, 321: 406-412.

Barbehenn E, Lurie P, Wolfe SM: Alosetron for irritable bowel syndrome. Lancet. 2000, 356: 2009-2010. 10.1016/S0140-6736(05)72978-0.

Moynihan R: Alosetron: a case study in regulatory capture, or a victory for patients' rights?. BMJ. 2002, 325: 592-595. 10.1136/bmj.325.7364.592.

Bombardier C, Laine L, Reicin A, Shapiro D, Burgos-Vargas R, Davis B, Day R, Ferraz MB, Hawkey CJ, Hochberg MC, Kvien TK, Schnitzer TJ, VIGOR Study Group: Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis. N Engl J Med. 2000, 343: 1520-1528. 10.1056/NEJM200011233432103.

Mukherjee D, Nissen SE, Topol EJ: Risk of cardiovascular events associated with selective COX-2 inhibitors. JAMA. 2001, 286: 954-959. 10.1001/jama.286.8.954.

McCormack JP, Rangno R: Digging for data from the COX-2 trials. CMAJ. 2002, 166: 1649-1650.

Silverstein FE, Faich G, Goldstein JL, Simon LS, Pincus T, Whelton A, Makuch R, Eisen G, Agrawal NM, Stenson WF, Burr AM: Gastrointestinal toxicity with celecoxib vs nonsteroidal anti-inflammatory drugs for osteoarthritis and rheumatoid arthritis: the CLASS study: a randomized controlled trial. JAMA. 2000, 284: 1247-1255. 10.1001/jama.284.10.1247.

Lurie P, Zieve A: Sometimes the silence can be like the thunder: access to pharmaceutical data at the FDA. Law Contemp Probl. 2008, 69: 85-97.

Nissen S, Califf R: A conversation about rosiglitazone. Medscape Diabetes & Endocrinology [Online]. 2007, http://www.medscape.com/viewarticle/561666

Nissen SE, Wolski K: Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. N Engl J Med. 2007, 356: 2457-2471. 10.1056/NEJMoa072761.

Mitka M: Controversies surround heart drug study: questions about Vytorin and trial sponsors' conduct. JAMA. 2008, 299: 885-887. 10.1001/jama.299.8.885.

Kastelein JJ, Akdim F, Stroes ES, Zwinderman AH, Bots ML, Stalenhoef AF, Visseren FL, Sijbrands EJ, Trip MD, Stein EA, Duivenvoorden R, Veltri EP, Marais AD, de Groot E, ENHANCE Investigators: Simvastatin with or without ezetimibe in familial hypercholesterolemia. N Engl J Med. 2008, 358: 1431-1443. 10.1056/NEJMoa0800742.

Psaty BM, Furberg CD, Ray WA, Weiss NS: Potential for conflict of interest in the evaluation of suspected adverse drug reactions: use of cerivastatin and risk of rhabdomyolysis. JAMA. 2004, 292: 2622-2631. 10.1001/jama.292.21.2622.

Tuffs A: Bayer faces potential fine over cholesterol lowering drug. BMJ. 2001, 323: 415-10.1136/bmj.323.7310.415.

King RT: Bitter pill: how a drug firm paid for university study, then undermined it. Wall Street Journal. 1996, 1: A13-

Dong BJ, Hauck WW, Gambertoglio JG, Gee L, White JR, Bubp JL, Greenspan FS: Bioequivalence of generic and brand-name levothyroxine products in the treatment of hypothyroidism. JAMA. 1997, 277: 1205-1213. 10.1001/jama.277.15.1205.

Kenemans P, Bundred NJ, Foidart JM, Kubista E, von Schoultz B, Sismondi P, Vassilopoulou-Sellin R, Yip CH, Egberts J, Mol-Arts M, Mulder R, van Os S, Beckmann MW, LIBERATE Study Group: Safety and efficacy of tibolone in breast-cancer patients with vasomotor symptoms: a double-blind, randomised, non-inferiority trial. Lancet Oncol. 2009, 10: 135-146. 10.1016/S1470-2045(08)70341-3.

Lippegaus O, Prokscha S, Thimme C: Verharmloste Gefahren. Krebs durch Hormonbehandlung [Trivialised dangers. Cancer caused by hormone therapy]. Last update 2009 [accessed 26 Feb 2010], http://frontal21.zdf.de/ZDFde/inhalt/11/0,1872,7593675,00.html

Doroshow JH: Commentary: publishing cancer clinical trial results: a scientific and ethical imperative. Oncologist. 2008, 13: 930-932. 10.1634/theoncologist.2008-0168.

Simes RJ: Publication bias: the case for an international registry of clinical trials. J Clin Oncol. 1986, 4: 1529-1541.

Takeda A, Loveman E, Harris P, Hartwell D, Welch K: Time to full publication of studies of anti-cancer medicines for breast cancer and the potential for publication bias: a short systematic review. Health Technol Assess. 2008, 12: iii-x. 1-46

Peppercorn J, Blood E, Winer E, Partridge A: Association between pharmaceutical involvement and outcomes in breast cancer clinical trials. Cancer. 2007, 109: 1239-1246. 10.1002/cncr.22528.

Kyzas PA, Loizou KT, Ioannidis JP: Selective reporting biases in cancer prognostic factor studies. J Natl Cancer Inst. 2005, 97: 1043-1055.

Begg CB, Pocock SJ, Freedman L, Zelen M: State of the art in comparative cancer clinical trials. Cancer. 1987, 60: 2811-2815. 10.1002/1097-0142(19871201)60:11<2811::AID-CNCR2820601136>3.0.CO;2-P.

Kyzas PA, Denaxa-Kyza D, Ioannidis JP: Almost all articles on cancer prognostic markers report statistically significant results. Eur J Cancer. 2007, 43: 2559-2579.

Manheimer E, Anderson D: Survey of public information about ongoing clinical trials funded by industry: evaluation of completeness and accessibility. BMJ. 2002, 325: 528-531. 10.1136/bmj.325.7363.528.

Rennie D: Thyroid storm. JAMA. 1997, 277: 1238-1243. 10.1001/jama.277.15.1238.

Godlee F, Clarke M: Why don't we have all the evidence on oseltamivir?. BMJ. 2009, 339: b5351-10.1136/bmj.b5351.

Jefferson TO, Demicheli V, Di Pietrantonj C, Jones M, Rivetti D: Neuraminidase inhibitors for preventing and treating influenza in healthy adults. Cochrane Database Syst Rev. 2006, 3: CD001265-

Kaiser L, Wat C, Mills T, Mahoney P, Ward P, Hayden F: Impact of oseltamivir treatment on influenza-related lower respiratory tract complications and hospitalizations. Arch Intern Med. 2003, 163: 1667-1672. 10.1001/archinte.163.14.1667.

Jefferson T, Jones M, Doshi P, Del Mar C: Neuraminidase inhibitors for preventing and treating influenza in healthy adults: systematic review and meta-analysis. BMJ. 2009, 339: b5106-10.1136/bmj.b5106.

The influence of the pharmaceutical industry; formal minutes, oral and written evidence. 2005, London: Stationery Office, 2: [House of Commons, Health Committee (Series Editor): Report of session 2004-05; vol 4]

Cochrane Injuries Group Albumin Reviewers: Human albumin administration in critically ill patients: systematic review of randomised controlled trials. BMJ. 1998, 317: 235-240.

McCarthy M: Company sought to block paper's publication. Lancet. 2000, 356: 1659-10.1016/S0140-6736(00)03166-4.

Man-Son-Hing M, Wells G, Lau A: Quinine for nocturnal leg cramps: a meta-analysis including unpublished data. J Gen Intern Med. 1998, 13: 600-606. 10.1046/j.1525-1497.1998.00182.x.

Marshall E: Is data-hoarding slowing the assault on pathogens?. Science. 1997, 275: 777-780. 10.1126/science.275.5301.777.

Campbell EG, Clarridge BR, Gokhale M, Birenbaum L, Hilgartner S, Holtzman NA, Blumenthal D: Data withholding in academic genetics: evidence from a national survey. JAMA. 2002, 287: 473-480. 10.1001/jama.287.4.473.

Misakian AL, Bero LA: Publication bias and research on passive smoking: comparison of published and unpublished studies. JAMA. 1998, 280: 250-253. 10.1001/jama.280.3.250.

Barnes DE, Bero LA: Why review articles on the health effects of passive smoking reach different conclusions. JAMA. 1998, 279: 1566-1570. 10.1001/jama.279.19.1566.

Hilts PJ: Philip Morris blocked paper showing addiction, panel finds. New York Times. 1994, A7-

Hilts PJ: Scientists say Philip Morris withheld nicotine findings. New York Times. 1994, A1-A7.

Anderson HR, Atkinson RW, Peacock JL, Sweeting MJ, Marston L: Ambient particulate matter and health effects: publication bias in studies of short-term associations. Epidemiology. 2005, 16: 155-163. 10.1097/01.ede.0000152528.22746.0f.

Peinemann F, McGauran N, Sauerland S, Lange S: Negative pressure wound therapy: potential publication bias caused by lack of access to unpublished study results data. BMC Med Res Methodol. 2008, 8: 4-10.1186/1471-2288-8-4.

McAuley L, Pham B, Tugwell P, Moher D: Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?. Lancet. 2000, 356: 1228-1231. 10.1016/S0140-6736(00)02786-0.

Hopewell S, McDonald S, Clarke M, Egger M: Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev. 2007, MR000010-2

Chalmers I: Underreporting research is scientific misconduct. JAMA. 1990, 263: 1405-1408. 10.1001/jama.263.10.1405.

World Medical Association: Declaration of Helsinki: ethical principles for medical research involving human subjects. Last update Oct 2008 [accessed 26 Feb 2010], http://www.wma.net/en/30publications/10policies/b3/index.html

Pearn J: Publication: an ethical imperative. BMJ. 1995, 310: 1313-1315.

The Nuremberg code. Trials of war criminals before the Nuremberg Military Tribunals under Control Council Law no10. 1949, Washington, D.C.: US Government Printing Office, 2: 181-182.

Healy D: Did regulators fail over selective serotonin reuptake inhibitors?. BMJ. 2006, 333: 92-95. 10.1136/bmj.333.7558.92.

Topol EJ: Failing the public health: rofecoxib, Merck, and the FDA. N Engl J Med. 2004, 351: 1707-1709. 10.1056/NEJMp048286.

Rennie D: When evidence isn't: trials, drug companies and the FDA. J Law Policy. 2007, 15: 991-1012.

Dickersin K, Rennie D: Registering clinical trials. JAMA. 2003, 290: 516-523. 10.1001/jama.290.4.516.

Tse T, Williams RJ, Zarin DA: Update on Registration of Clinical Trials in ClinicalTrials.gov. Chest. 2009, 136: 304-305. 10.1378/chest.09-1219.

Tse T, Williams RJ, Zarin DA: Reporting "basic results" in ClinicalTrials.gov. Chest. 2009, 136: 295-303. 10.1378/chest.08-3022.

WHO clinical trials initiative to protect the public. Bull World Health Organ. 2006, 84: 10-11.

De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJ, Schroeder TV, Sox HC, Weyden Van Der MB, International Committee of Medical Journal Editors: Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004, 351: 1250-1251. 10.1056/NEJMe048225.

Deutsches Cochrane Zentrum, Deutsches Netzwerk Evidenzbasierte Medizin: Stellungnahme [Comment]. Last update 22 Sep 2004 [accessed 26 Feb 2010], http://www.ebm-netzwerk.de/netzwerkarbeit/images/stellungnahme_anhoerung_probandenschutz.pdf

Krleza-Jeric K: International dialogue on the Public Reporting Of Clinical Trial Outcome and Results: PROCTOR meeting. Croat Med J. 2008, 49: 267-268. 10.3325/cmj.2008.2.267.

Krleza-Jeric K, Chan AW, Dickersin K, Sim I, Grimshaw J, Gluud C: Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1). BMJ. 2005, 330: 956-958. 10.1136/bmj.330.7497.956.

European Research Council: ERC Scientific Council guidelines on open access. Last update 17 Dec 2007 [accessed 25 Feb 2010], http://erc.europa.eu/pdf/ScC_Guidelines_Open_Access_revised_Dec07_FINAL.pdf

Groves T: Mandatory disclosure of trial results for drugs and devices. BMJ. 2008, 336: 170-10.1136/bmj.39469.465139.80.

Steinbrook R: Public access to NIH-funded research. N Engl J Med. 2005, 352: 1739-1741. 10.1056/NEJMp058088.

Dickersin K: Report from the Panel on the Case for Registers of Clinical Trials at the Eighth Annual Meeting of the Society for Clinical Trials. Control Clin Trials. 1988, 9: 76-81.

De Angelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C: Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. N Engl J Med. 2005, 352: 2436-2438. 10.1056/NEJMe058127.

Laine C, Horton R, DeAngelis CD, Drazen JM, Frizelle FA, Godlee F, Haug C, Hebert PC, Kotzin S, Marusic A, Sahni P, Schroeder TV, Sox HC, Weyden Van der MB, Verheugt FW: Clinical trial registration--looking back and moving ahead. N Engl J Med. 2007, 356: 2734-2736. 10.1056/NEJMe078110.

Krleza-Jeric K: Clinical trial registration: the differing views of industry, the WHO, and the Ottawa Group. PLoS Med. 2005, 2: e378-10.1371/journal.pmed.0020378.

Food and Drug Administration: FDA Amendments Act (FDAAA) of 2007, public law no. 110-85 §801. Last update 2007 [accessed 26 Feb 2010], http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=110_cong_public_laws%26docid=f:publ085.110.pdf

Wood AJJ: Progress and deficiencies in the registration of clinical trials. N Engl J Med. 2009, 360: 824-830. 10.1056/NEJMsr0806582.

European Medicines Agency: EMEA launches EudraCT database. Last update 06 May 2004 [accessed 25 Feb 2010], http://www.emea.europa.eu/pdfs/general/direct/pr/1258904en.pdf

Smyth RL: Making information about clinical trials publicly available. BMJ. 2009, 338: b2473-10.1136/bmj.b2473.

Regulation (EC) No 1901/2006 of the European Parliament and of the Council of 12 December 2006 on medicinal products for paediatric use and amending regulation (EEC) no 1768/92, directive 2001/20/EC, directive 2001/83/EC and regulation (EC) No 726/2004. Official J Eur Commun. 2006, 49: L378/1-L378/19.

European Commission: List of fields to be made public from EudraCT for paediatric clinical trials in accordance with article 41 of regulation (EC) no 1901/2006 and its implementing guideline 2009/C28/01. Last update 26 Mar 2009 [accessed 26 Feb 2010], http://ec.europa.eu/enterprise/pharmaceuticals/eudralex/vol-10/2009_02_04_guidelines_paed.pdf

European Commission: List of fields contained in the 'EudraCT' clinical trials database to be made public, in accordance with Article 57(2) of Regulation (EC) No 726/2004 and its implementing guideline 2008/c168/021. Last update 04 Feb 2009 [accessed 25 Feb 2010], http://ec.europa.eu/enterprise/pharmaceuticals/eudralex/vol-10/2009_02_04_guideline.pdf

Committee on Government Reform: A citizen's guide on using the Freedom of Information Act and the Privacy Act of 1974 to request government records. Last update 20 Sep 2005 [accessed 26 Feb 2010], http://www.fas.org/sgp/foia/citizen.pdf

Food and Drug Administration: Executive summary of the Food and Drug Administration's consumer roundtable on consumer protection priorities. Last update 2000 [accessed 26 Feb 2010], http://www.fda.gov/ohrms/dockets/dockets/00n_1665/cr00001.pdf

Turner EH: Closing a loophole in the FDA Amendments Act. Science. 2008, 322: 44-46. 10.1126/science.322.5898.44c.

Pharmaceutical Research and Manufacturers of America: PhRMA clinical trial registry proposal. Last update 2010 [accessed 26 Feb 2010], http://www.phrma.org/node/446

Principles on the conduct of clinical trials and communication of clinical trial results. 2002, Washington DC: Pharmaceutical Research and Manufacturers of America

International Federation of Pharmaceutical Manufacturers & Associations: Joint position on the disclosure of clinical trial information via clinical trial registries and databases. Last update 2005 [accessed 24 Feb 2010], http://www.phrma.org/files/attachments/2005-01-06.1113.PDF

International Federation of Pharmaceutical Manufacturers & Associations: Joint position on the disclosure of clinical trial information via clinical trial registries and databases. Last update Nov 2008 [accessed 11 Mar 2010], http://www.ifpma.org/pdf/Revised_Joint_Industry_Position_26Nov08.pdf

Verband Forschender Arzneimittelhersteller: VFA-Positionspapier zum Thema "publication bias" [VFA position paper on the subject of "publication bias"]. 2008, Berlin: VFA

Verband Forschender Arzneimittelhersteller: VFA-Positionspapier zum Thema "publication bias" [VFA position paper on the subject of "publication bias"]. Last update Jan 2009 [accessed 26 Feb 2010], http://www.vfa.de/download/SAVE/de/presse/positionen/pos-publication-bias.html/pos-publication-bias.pdf

Mathew SJ, Charney DS: Publication bias and the efficacy of antidepressants. Am J Psychiatry. 2009, 166: 140-145. 10.1176/appi.ajp.2008.08071102.

Abbott A: British panel bans use of antidepressant to treat children. Nature. 2003, 423: 792-

Mitka M: FDA alert on antidepressants for youth. JAMA. 2003, 290: 2534-10.1001/jama.290.19.2534.

Garland EJ: Facing the evidence: antidepressant treatment in children and adolescents. CMAJ. 2004, 170: 489-491.

Herxheimer A, Mintzes B: Antidepressants and adverse effects in young patients: uncovering the evidence. CMAJ. 2004, 170: 487-489.

Dyer O: GlaxoSmithKline faces US lawsuit over concealment of trial results. BMJ. 2004, 328: 1395-10.1136/bmj.328.7453.1395.

Jureidini JN, McHenry LB, Mansfield PR: Clinical trials and drug promotion: selective reporting of study 329. Int J Risk Safety Med. 2008, 73-81.

Institute for Quality and Efficiency in Health Care: Preliminary report on antidepressants published. Last update 10 Jun 2009 [accessed 26 Feb 2010], http://www.iqwig.de/index.867.en.html

Steinman MA, Bero LA, Chren MM, Landefeld CS: Narrative review: the promotion of gabapentin: an analysis of internal industry documents. Ann Intern Med. 2006, 145: 284-293.

Steinman MA, Harper GM, Chren MM, Landefeld CS, Bero LA: Characteristics and impact of drug detailing for gabapentin. PLoS Med. 2007, 4: e134-10.1371/journal.pmed.0040134.

Landefeld CS, Steinman MA: The Neurontin legacy: marketing through misinformation and manipulation. N Engl J Med. 2009, 360: 103-106. 10.1056/NEJMp0808659.

Mack A: Examination of the evidence for off-label use of gabapentin. J Manag Care Pharm. 2003, 9: 559-568.

Petersen M: Memos cast shadow on drug's promotion. New York Times. 2002, C2-

U.S. Department of Justice: Warner-Lambert to pay $430 million to resolve criminal & civil health care liability relating to off-label promotion. Last update 13 May 2004 [accessed 13 Mar 2010], http://www.usdoj.gov/opa/pr/2004/May/04_civ_322.htm

Feeley J, Cronin Fisk M: AstraZeneca Seroquel studies 'buried,' papers show (update 3). Last update 27 Feb 2009 [accessed 19 Mar 2010], http://www.bloomberg.com/apps/news?pid=20601087%26sid=aS_.NqzMArG8#

Milford P: AstraZeneca may link Seroquel, diabetes, doctor says (update 1). Last update 11 Mar 2009 [accessed 19 Mar 2010], http://www.bloomberg.com/apps/news?pid=newsarchive%26sid=ayzJsK2HlF6s

Whalen J: AstraZeneca chalks up Seroquel dismissal in State Court. Wall Street Journal Health Blog [Online]. 2009, http://blogs.wsj.com/health/2009/06/10/astrazeneca-chalks-up-seroquel-dismissal-in-state-court

Kapczinski F, Lima MS, Souza JS, Schmitt R: Antidepressants for generalized anxiety disorder. Cochrane Database Syst Rev. 2003, CD003592-2

Bang LM, Keating GM: Paroxetine controlled release. CNS Drugs. 2004, 18: 355-364. 10.2165/00023210-200418060-00003.

Lenzer J: NIH secrets: study break. Last update 19.10.2006 [accessed 13 Mar 2010], http://www.ahrp.org/cms/index2.php?option=com_content%26do_pdf=1%26id=398

Ray WA, Stein CM, Daugherty JR, Hall K, Arbogast PG, Griffin MR: COX-2 selective non-steroidal anti-inflammatory drugs and risk of serious coronary heart disease. Lancet. 2002, 360: 1071-1073. 10.1016/S0140-6736(02)11131-7.

Juni P, Nartey L, Reichenbach S, Sterchi R, Dieppe PA, Egger M: Risk of cardiovascular events and rofecoxib: cumulative meta-analysis. Lancet. 2004, 364: 2021-2029. 10.1016/S0140-6736(04)17514-4.

Curfman GD, Morrissey S, Drazen JM: Expression of concern: Bombardier et al., "Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis," N Engl J Med 2000;343:1520-8. N Engl J Med. 2005, 353: 2813-2814. 10.1056/NEJMe058314.

Waxman HA: The lessons of Vioxx: drug safety and sales. N Engl J Med. 2005, 352: 2576-2578. 10.1056/NEJMp058136.

Waxman HA: The Marketing of Vioxx to Physicians (Memorandum to Democratic members of the Government Reform Committee). 2005

Krumholz HM, Ross JS, Presler AH, Egilman DS: What have we learnt from Vioxx?. BMJ. 2007, 334: 120-123. 10.1136/bmj.39024.487720.68.

Charatan F: Merck to pay $58 m in settlement over rofecoxib advertising. BMJ. 2008, 336: 1208-1209. 10.1136/bmj.39591.705231.DB.

DeAngelis CD, Fontanarosa PB: Impugning the integrity of medical science: the adverse effects of industry influence. JAMA. 2008, 299: 1833-1835. 10.1001/jama.299.15.1833.

Hill KP, Ross JS, Egilman DS, Krumholz HM: The ADVANTAGE seeding trial: a review of internal documents. Ann Intern Med. 2008, 149: 251-258.

Ross JS, Hill KP, Egilman DS, Krumholz HM: Guest authorship and ghostwriting in publications related to rofecoxib: a case study of industry documents from rofecoxib litigation. JAMA. 2008, 299: 1800-1812. 10.1001/jama.299.15.1800.

Moynihan R: Merck defends Vioxx in court, as publisher apologises for fake journal. BMJ. 2009, 338: b1914-10.1136/bmj.b1914.

West RR, Jones DA: Publication bias in statistical overview of trials: example of psychological rehabilitation following myocardial infarction [Abstract]. Proceedings of the 2nd International Conference on the Scientific Basis of Health Services and 5th Annual Cochrane Colloquium; 1997 Oct 8-12; Amsterdam. Amsterdam. 1999, 17-

Mangano DT, Tudor IC, Dietzel C: The risk associated with aprotinin in cardiac surgery. N Engl J Med. 2006, 354: 353-365. 10.1056/NEJMoa051379.

Karkouti K, Beattie WS, Dattilo KM, McCluskey SA, Ghannam M, Hamdy A, Wijeysundera DN, Fedorko L, Yau TM: A propensity score case-control comparison of aprotinin and tranexamic acid in high-transfusion-risk cardiac surgery. Transfusion (Paris). 2006, 46: 327-338.

Hauser RG, Maron BJ: Lessons from the failure and recall of an implantable cardioverter-defibrillator. Circulation. 2005, 112: 2040-2042. 10.1161/CIRCULATIONAHA.105.580381.

Kesselheim AS, Mello MM: Confidentiality laws and secrecy in medical research: improving public access to data on drug safety. Health Aff (Millwood). 2007, 26: 483-491. 10.1377/hlthaff.26.2.483.

Sackner-Bernstein JD, Kowalski M, Fox M, Aaronson K: Short-term risk of death after treatment with nesiritide for decompensated heart failure: a pooled analysis of randomized controlled trials. JAMA. 2005, 293: 1900-1905. 10.1001/jama.293.15.1900.

Camilleri M, Northcutt AR, Kong S, Dukes GE, McSorley D, Mangel AW: Efficacy and safety of alosetron in women with irritable bowel syndrome: a randomised, placebo-controlled trial. Lancet. 2000, 355: 1035-1040. 10.1016/S0140-6736(00)02033-X.

Horton R: Lotronex and the FDA: a fatal erosion of integrity. Lancet. 2001, 357: 1544-1545. 10.1016/S0140-6736(00)04776-0.

Lenzer J: FDA warns that antidepressants may increase suicidality in adults. BMJ. 2005, 331: 70-10.1136/bmj.331.7508.70-b.

Lenzer J: Drug secrets: what the FDA isn't telling. Slate Magazine. 2005, http://www.slate.com/id/2126918

Saunders MC, Dick JS, Brown IM, McPherson K, Chalmers I: The effects of hospital admission for bed rest on the duration of twin pregnancy: a randomised trial. Lancet. 1985, 2: 793-795. 10.1016/S0140-6736(85)90792-5.

Nissen SE: The DREAM trial. Lancet. 2006, 368: 2049-10.1016/S0140-6736(06)69825-5.

Drazen JM, Morrissey S, Curfman GD: Rosiglitazone: continued uncertainty about safety. N Engl J Med. 2007, 357: 63-64. 10.1056/NEJMe078118.

Home PD, Pocock SJ, Beck-Nielsen H, Gomis R, Hanefeld M, Jones NP, Komajda M, McMurray JJ: Rosiglitazone evaluated for cardiovascular outcomes: an interim analysis. N Engl J Med. 2007, 357: 28-38. 10.1056/NEJMoa073394.

Nathan DM: Rosiglitazone and cardiotoxicity: weighing the evidence. N Engl J Med. 2007, 357: 64-66. 10.1056/NEJMe078117.

Psaty BM, Furberg CD: The record on rosiglitazone and the risk of myocardial infarction. N Engl J Med. 2007, 357: 67-69. 10.1056/NEJMe078116.

Psaty BM, Furberg CD: Rosiglitazone and cardiovascular risk. N Engl J Med. 2007, 356: 2522-2524. 10.1056/NEJMe078099.

Rosen CJ: The rosiglitazone story: lessons from an FDA Advisory Committee meeting. N Engl J Med. 2007, 357: 844-846. 10.1056/NEJMp078167.

Singh S, Loke YK, Furberg CD: Long-term risk of cardiovascular events with rosiglitazone: a meta-analysis. JAMA. 2007, 298: 1189-1195. 10.1001/jama.298.10.1189.

Shuster JJ, Schatz DA: The rosiglitazone meta-analysis: lessons for the future. Diabetes Care. 2008, 31: e10-10.2337/dc07-2147.

Friedrich JO, Beyene J, Adhikari NK: Rosiglitazone: can meta-analysis accurately estimate excess cardiovascular risk given the available data? Re-analysis of randomized trials using various methodologic approaches. BMC Res Notes. 2009, 2: 5-10.1186/1756-0500-2-5.

Home PD, Pocock SJ, Beck-Nielsen H, Curtis PS, Gomis R, Hanefeld M, Jones NP, Komajda M, McMurray JJ: Rosiglitazone evaluated for cardiovascular outcomes in oral agent combination therapy for type 2 diabetes (RECORD): a multicentre, randomised, open-label trial. Lancet. 2009, 373: 2125-2135. 10.1016/S0140-6736(09)60953-3.

Merck/Schering-Plough Pharmaceuticals: Merck/Schering-Plough Pharmaceuticals provides results of the ENHANCE trial. Last update 14 Jan 2008 [accessed 13 Mar 2010], http://www.msppharma.com/msppharma/documents/press_release/ENHANCE_news_release_1-14-08.pdf

Greenland P, Lloyd-Jones D: Critical lessons from the ENHANCE trial. JAMA. 2008, 299: 953-955. 10.1001/jama.299.8.953.

Lenzer J: Unreported cholesterol drug data released by company. BMJ. 2008, 336: 180-181. 10.1136/bmj.39468.610775.DB.

Berenson A: Data about Zetia risks was not fully revealed. New York Times. 2007

Furberg CD, Pitt B: Withdrawal of cerivastatin from the world market. Curr Control Trials Cardiovasc Med. 2001, 2: 205-207. 10.1186/CVM-2-5-205.

Wooltorton E: Bayer pulls cerivastatin (Baycol) from market. CMAJ. 2001, 165: 632-

Marwick C: Bayer is forced to release documents over withdrawal of cerivastatin. BMJ. 2003, 326: 518-10.1136/bmj.326.7388.518/a.

Piorkowski JD: Bayer's response to "potential for conflict of interest in the evaluation of suspected adverse drug reactions: use of cerivastatin and risk of rhabdomyolysis". JAMA. 2004, 292: 2655-2657. 10.1001/jama.292.21.2655.

Zinberg DS: A cautionary tale. Science. 1996, 273: 411-10.1126/science.273.5274.411.

Begg CB, Berlin JA: Publication bias and dissemination of clinical research. J Natl Cancer Inst. 1989, 81: 107-115. 10.1093/jnci/81.2.107.

Nathan DG, Weatherall DJ: Academia and industry: lessons from the unfortunate events in Toronto. Lancet. 1999, 353: 771-772. 10.1016/S0140-6736(99)00072-0.

Harris G: Approval of antibiotic worried safety officials. New York Times. 2006

Ross DB: The FDA and the case of Ketek. N Engl J Med. 2007, 356: 1601-1604. 10.1056/NEJMp078032.

Johansen HK, Gotzsche PC: Problems in the design and reporting of trials of antifungal agents encountered during meta-analysis. JAMA. 1999, 282: 1752-1759. 10.1001/jama.282.18.1752.

McKenzie R, Fried MW, Sallie R, Conjeevaram H, Di Bisceglie AM, Park Y, Savarese B, Kleiner D, Tsokos M, Luciano C: Hepatic failure and lactic acidosis due to fialuridine (FIAU), an investigational nucleoside analogue for chronic hepatitis B. N Engl J Med. 1995, 333: 1099-1105. 10.1056/NEJM199510263331702.

Blumsohn A: Authorship, ghostscience, access to data and control of the pharmaceutical scientific literature: who stands behind the word?. Prof Ethics Rep. 2006, 19: 1-4.

Bracken MB, Shepard MJ, Holford TR, Leo-Summers L, Aldrich EF, Fazl M, Fehlings M, Herr DL, Hitchon PW, Marshall LF, Nockels RP, Pascale V, Perot PL, Piepmeier J, Sonntag VK, Wagner F, Wilberger JE, Winn HR, Young W: Administration of methylprednisolone for 24 or 48 hours or tirilazad mesylate for 48 hours in the treatment of acute spinal cord injury: results of the Third National Acute Spinal Cord Injury Randomized Controlled Trial. JAMA. 1997, 277: 1597-1604. 10.1001/jama.277.20.1597.

Download references

Acknowledgements

The authors thank Dirk Eyding, Daniel Fleer, Elke Hausner, Regine Potthast, Andrea Steinzen, and Siw Waffenschmidt for helping to screen reference lists and Verena Wekemann for formatting citations.

Funding source

This work was supported by the German Institute for Quality and Efficiency in Health Care. All authors are employees of the Institute.

Author information

Authors and affiliations.

Institute for Quality and Efficiency in Health Care, Dillenburger Str 27, 51105, Cologne, Germany

Natalie McGauran, Beate Wieseler, Julia Kreis, Yvonne-Beatrice Schüler, Heike Kölsch & Thomas Kaiser

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Natalie McGauran .

Additional information

Competing interests.

Non-financial competing interests: All authors are employees of the German Institute for Quality and Efficiency in Health Care. In order to produce unbiased HTA reports, the Institute depends on access to all of the relevant data on the topic under investigation. We therefore support the mandatory worldwide establishment of trial registries and study results databases.

Authors' contributions

NM and BW had the idea for the manuscript. NM, HK, YBS, and JK screened reference lists. JK and YBS reviewed titles and abstracts of potentially relevant citations identified in the screening process. NM extracted relevant examples from the full-text publications. BW and TK checked the extracted examples. NM drafted the first version of the manuscript. The remaining authors contributed important intellectual content to the final version. All authors approved the final version.

Electronic supplementary material

13063_2009_448_moesm1_esm.doc.

Additional file 1: Table S2: Examples of reporting bias in the medical literature. Extracts from 50 publications presenting examples of reporting bias. (DOC 634 KB)

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

McGauran, N., Wieseler, B., Kreis, J. et al. Reporting bias in medical research - a narrative review. Trials 11 , 37 (2010). https://doi.org/10.1186/1745-6215-11-37

Download citation

Received : 07 November 2009

Accepted : 13 April 2010

Published : 13 April 2010

DOI : https://doi.org/10.1186/1745-6215-11-37

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Irritable Bowel Syndrome
Atopic Dermatitis

ISSN: 1745-6215

Submission enquiries: Access here and click Contact Us
General enquiries: [email protected]

Research article
Open access
Published: 01 June 2020

Publication and related biases in health services research: a systematic review of empirical evidence

Abimbola A. Ayorinde 1 ,
Iestyn Williams 2 ,
Russell Mannion 2 ,
Fujian Song 3 ,
Magdalena Skrybant 4 ,
Richard J. Lilford 4 &
Yen-Fu Chen ORCID: orcid.org/0000-0002-9446-2761 1

BMC Medical Research Methodology volume 20 , Article number: 137 ( 2020 ) Cite this article

7355 Accesses

13 Citations

13 Altmetric

Metrics details

Publication and related biases (including publication bias, time-lag bias, outcome reporting bias and p-hacking) have been well documented in clinical research, but relatively little is known about their presence and extent in health services research (HSR). This paper aims to systematically review evidence concerning publication and related bias in quantitative HSR.

Databases including MEDLINE, EMBASE, HMIC, CINAHL, Web of Science, Health Systems Evidence, Cochrane EPOC Review Group and several websites were searched to July 2018. Information was obtained from: (1) Methodological studies that set out to investigate publication and related biases in HSR; (2) Systematic reviews of HSR topics which examined such biases as part of the review process. Relevant information was extracted from included studies by one reviewer and checked by another. Studies were appraised according to commonly accepted scientific principles due to lack of suitable checklists. Data were synthesised narratively.

After screening 6155 citations, four methodological studies investigating publication bias in HSR and 184 systematic reviews of HSR topics (including three comparing published with unpublished evidence) were examined. Evidence suggestive of publication bias was reported in some of the methodological studies, but evidence presented was very weak, limited in both quality and scope. Reliable data on outcome reporting bias and p-hacking were scant. HSR systematic reviews in which published literature was compared with unpublished evidence found significant differences in the estimated intervention effects or association in some but not all cases.

Conclusions

Methodological research on publication and related biases in HSR is sparse. Evidence from available literature suggests that such biases may exist in HSR but their scale and impact are difficult to estimate for various reasons discussed in this paper.

Systematic review registration

PROSPERO 2016 CRD42016052333.

Peer Review reports

Publication bias occurs when the publication, non-publication or late publication of research findings is influenced by the direction or strength of the results, and consequently the findings that are published or published early may differ systematically from those that remain unpublished or for which publication is delayed [ 1 , 2 ]. Other related biases, however, may occur between the generation of research evidence and its eventual publication. These include: p-hacking, which involves repeated analyses using different methods or subsets of data until statistically significant results are obtained [ 3 ]; and outcome reporting bias, whereby among those examined, only favourable outcomes are reported [ 4 ]. For brevity, we use the term “publication and related bias” in this paper to encompass these various types of biases (Fig. 1 ).

Publication related biases and other biases at various stages of research

Publication bias is a major concern in health care as biased evidence available to decision makers may lead to suboptimal decisions that a) negatively impact on the care and the health of patients and b) lead to an inefficient and inequitable allocation of scarce resources. This problem has been documented extensively in the clinical research literature [ 2 , 4 , 5 ], and several high-profile cases of non-publication of studies showing unfavourable results have led to the introduction of mandatory prospective registration of clinical trials [ 6 ]. By comparison, publication bias appears to have received scant attention in health services research (HSR). A recent methodological study of Cochrane reviews of HSR topics found that less than one in 10 of the reviews explicitly assessed publication bias [ 7 ].

However, it is unlikely that HSR is immune from publication and related biases, and these problems may be anticipated on theoretical grounds. In contrast with clinical research, where mandatory registration of all studies involving human subjects has long been advocated through the declaration of Helsinki [ 8 ] and publication of results of commercial trials are increasingly enforced by regulatory bodies, the registration and regulation of HSR studies are much more variable. In addition, studies in HSR often examine a large number of factors (independent variables, mediating variables, contextual variables and outcome variables) along a long service delivery causal chain [ 9 ]. The scope for ‘data dredging’ associated with use of multiple subsets of data and analytical techniques is substantial [ 10 ]. Furthermore, there is a grey area between research and non-research, particularly in the evaluation of quality improvement projects [ 11 ], which are usually initiated under a service imperative rather than to produce generalizable knowledge. In these settings there are fewer checks against the motivation that may arise post hoc to selectively publish “newsworthy” findings from evaluations showing promising results.

The first step towards improving our understanding of publication and related biases in HSR, which is the main aim of this review, is to systematically examine the existing literature. We anticipated that we might find two broad types of literature: (1) methodological research that set out with the prime purpose of investigating publication and related bias in HSR; (2) systematic reviews of substantive HSR topics but in which the authors had investigated the possibility of publication and related biases as part of the methodology used to explore the validity of their findings.

We adopted the definition of HSR used by the United Kingdom’s National Institute for Health Research Health Services & Delivery Research (NIHR HS & DR) Programme: “research to produce evidence on the quality, accessibility and organisation of health services”, including evaluation of how healthcare organizations might improve the delivery of services. The definition is deliberately broad in recognition of the many associated disciplines and methodologies, and is compatible with other definitions of HSR such as those offered by the Agency for Healthcare Research and Quality (AHRQ). We were aware that publication bias may arise in qualitative research [ 12 ], but as the mechanisms and manifestations are likely to be very different, we focused on publication bias related to quantitative research in this review. The protocol for this systematic review was pre-registered in the PROSPERO International prospective register of systematic reviews (2016:CRD42016052333). We followed the PRISMA statement [ 13 ] for undertaking and reporting this review where applicable (see Additional file 1 for the PRISMA checklist).

Inclusion criteria

Included studies needed to be concerned with HSR related topics based on the NIHR HS & DR Programme’s definition described above. The types of study included were either:

(1) methodological studies that set out to investigate data dredging/p-hacking, outcome reporting bias or publication bias by one or more of: a) tracking a cohort of studies from inception or from a pre-publication stage such as conference presentation to publication (or not); b) surveying researchers about their experiences related to research publication; c) investigating statistical techniques to prevent, detect or mitigate the above biases;

(2) systematic reviews of substantive HSR topics that provided empirical evidence concerning publication and related biases. Such evidence could take various forms such as comparing findings in published vs. grey literature; statistical analyses (e.g. funnel plots and Egger’s test); and assessment of selective outcome reporting within individual studies included in the reviews.

Exclusion criteria

Articles were excluded if they assessed publication and related biases in subject areas other than HSR (e.g. basic sciences; clinical and public health research) or publication bias purely in relation to qualitative research. Biases in the dissemination of evidence following research publication, such as citation bias and media attention bias, were not included since they can be alleviated by systematic search [ 2 ]. Studies of bias relating to study design (such as recall bias) were also excluded. No language restriction was applied.

Search strategy

We used a judicious combination of information sources and searching methods to ensure that our coverage of the relevant HSR literature was as comprehensive as possible. MEDLINE (1946 to 16 March 2017), EMBASE (1947 to 16 March 2017), Health Management Information Consortium (HMIC, 1979 to January 2017), CINAHL (1981 to 17 March 2017), and Web of Science (all years) were searched using indexed terms and text words related to HSR [ 14 ], combined with search terms relating to publication bias. In April 2017 we searched HSR-specific databases including Health Systems Evidence (HSE) and the Cochrane Effective Practice and Organisation of Care (EPOC) Review Group using publication bias related terms. The search strategy for MEDLINE is provided in Appendix 1 (see Additional file 2 ).

For the included studies, we used forward and backward citation searches (using Google Scholar/PubMed and manual check of reference lists) to identify additional studies that had not been captured in the electronic database searches. We searched the webpages of major organizations related to HSR, including the Institute for Healthcare Improvement (USA), The AHRQ (USA), and the Research and Development (RAND) Corporation (USA), Health Foundation (UK), King’s Fund (UK) (last searched on 20th September 2017). We also searched the UK NIHR HSDR Programme website and the US HSRProj (Health Services Research Projects in Progress) database for previously commissioned and ongoing studies (last searched on 20th February 2018). All the searches were updated between 30th July and 2nd August 2018 in order to identify any new relevant methodological studies. Members of the project steering and management committees were consulted to identify any additional studies.

Citations retrieved were imported and de-duplicated in the EndNote software, and were screened for relevance based on titles and abstracts. Full-text publications were retrieved for potentially relevant records and articles were included/excluded based on the selection criteria described above. The screening and study selection were carried out by two reviewers independently, with any disagreement resolved by discussion with the wider research team.

Data extraction

Methodological studies.

For the included methodological studies set out to examine publication and related biases, a data extraction form was designed to collect the following information: citation details; methods of selecting study sample; characteristics of study sample; methods of investigating publication and related biases; key findings; limitations; and conclusions. Data extraction was conducted by one reviewer and checked by another reviewer.

Systematic reviews of substantive topics of HSDR

For systematic reviews that directly compared published literature with grey literature/unpublished studies, the following data were collected by one reviewer and checked by another: the topic being examined; methods used to identify grey literature and unpublished studies; findings of comparisons between published and grey/unpublished literature; limitations and conclusions. A separate data extraction form was used to collect data from the remaining HSR systematic reviews. Information concerning techniques used to investigate publication bias and outcome reporting bias was extracted along with findings of these investigations. Due to the large number of identified HSR systematic reviews falling into this category, the data extraction was carried out only by a single reviewer.

Risk of bias assessment

No single risk of bias assessment tool could capture the dimensions of quality for the types of methodological studies included [ 2 ]. We therefore critically appraised individual methodological studies and systematic reviews directly comparing published vs unpublished evidence on the basis of adherence to commonly accepted scientific principles, including: representativeness of published/unpublished HSR studies being examined or health services researchers being surveyed; rigour in data collection and analysis; and whether attention was paid to factors that could confound the association between study findings and publication status. Each study was read by at least two reviewers and any methodological issues identified are presented as commentary alongside study findings in the results section. No quality assessment was carried out for the remaining HSR systematic reviews, as we were only interested in their findings in relation to publication and related biases rather than the effects or associations examined in these reviews per se. We anticipated that it would not be feasible to use quantitative methods (such as funnel plots) for evaluating potential publication bias across studies due to heterogeneous methods and measures adopted to assess publication bias in the methodological studies included in this review.

Data synthesis and presentation

As included studies used diverse approaches and measures to investigate publication and related biases, meta-analyses could not be performed. Findings were therefore presented narratively [ 15 ].

Literature search and selection

The initial searches of the electronic databases yielded 6155 references, which were screened on the basis of titles/abstracts. The full-text for 422 of them and six additional articles identified from other sources were then retrieved and assessed (Fig. 2 ). Two hundred and forty articles did not meet the inclusion criteria primarily because no empirical evidence on publication and related biases was reported or the subject areas lay outside the domain of HSR as described above. An updated search yielded 1328 new records but no relevant methodological studies were identified.

Flow diagram showing study selection process

We found four methodological studies that set out with the primary purpose of investigating publication and related biases in HSR [ 16 , 17 , 18 , 19 ]. We identified 184 systematic reviews of HSR topics where the authors of reviews looked for evidence of publication and related biases. Three of these 184 systematic reviews provided direct evidence on publication bias by comparing findings of published articles with those of grey literature and unpublished studies [ 20 , 21 , 22 ]. The remaining 181 review provided only indirect evidence on publication and related biases (Fig. 2 ).

Methodological studies setting out to investigate publication and related biases

The characteristics of the four included methodological studies are presented in Table 1 . Three studies [ 16 , 17 , 19 ] explored the presence or absence of publication bias in health informatics research. The remaining study [ 18 ] focused on p-hacking or reporting bias that may arise when authors of research papers compete by reporting ‘more extreme and spectacular results’ in order to optimize chances of journal publication. A brief summary of each of the studies is provided below.

Only one study was an inception cohort study, which tracked individual research projects from their start. Such a study provides direct evidence of publication bias [ 19 ]. This study assessed publication bias in clinical trials of electronic health records registered with ClinicalTrials.gov during 2000–8 and reported that results from 76% (47/62) of completed trials were subsequently published. Of the published studies, 74% (35/47) reported predominantly positive results, 21% (10/47) reported neutral results (no effect) and 4% (2/47) reported negative/harmful results. Data were available from investigators for seven of the 15 unpublished trials: four reported neutral results and three reported positive results. Based on these data, the authors concluded that trials with positive results are more likely to be published than those with null results, although we noticed that this finding was not statistically significant (see Table 1 ). The authors cautioned that few trials were registered in the early years of ClinicalTrials.gov and those registered may be more likely to publish their findings and thus systematically different from those not registered. They further noted that the registered data were often unreliable during that period.

The second study reported a pilot survey of academics in order to assess rates of non-publication in IT evaluation studies and reasons for any non-publication [ 16 ]. The survey asked what information systems the respondents had evaluated in the past 3 years, whether the results of the evaluation(s) were published, and if not published, the reasons behind the non-publication. The findings show that approximately 50% of the identified evaluation studies were published in peer reviewed journals, proceedings or books. Of the remaining studies, some were published in internal reports and/or local publications (such as masters’ theses and local conferences) and approximately one third were unpublished at the time of the survey. The reasons cited for non-publication included: “results not of interest for others”; “publication in preparation”; “no time for publication”; “limited scientific quality of study”; “political or legal reasons”, and “study only conducted for internal use”. The main limitation of this study is a low response rate with only 118 of 722 (18.8%) targeted participants providing valid responses.

The third methodological study used three different approaches to assess publication bias in health informatics [ 17 ]. However, for one of the approaches (statistical analyses of publication bias/small study effects) the authors were unable to find enough studies which reported findings using the same outcome measures; while the remaining two approaches adopted in this study (i.e. examining percentage of HSR evaluation studies reporting positive results and percentage of HSR reviews reaching positive conclusion) provided little information on publication bias since there is no estimate of what the “unbiased” proportion of positive findings should be for HSR evaluation studies and reviews (Table 1 ).

The fourth methodological study included in this review examined quantitative estimates of income elasticity of health care and price elasticity of prescription drugs reported in the published literature [ 18 ]. Using funnel plots and meta-regressions the authors identified a positive correlation between effect sizes and the standard errors of income/price elasticity estimates, which suggested potential publication bias [ 18 ]. In addition, they found an independent association between effect size and journal impact factor, indicating that given similar standard errors (which reflect sample sizes), studies reporting larger effect sizes (i.e. more striking findings) were more likely to be published in ‘high-impact’ journals. As other confounding factors could not be ruled out for these observed associations and no unpublished studies were examined, the evidence is suggestive rather than conclusive.

Systematic reviews of HSR topics providing evidence on publication and related bias

We identified 184 systematic reviews of HSR topics in which empirical evidence on publication and related bias was reported. Three of these reviews provided direct evidence on publication bias by comparing evidence from studies published in academic journals with those from grey literature or unpublished studies [ 20 , 21 , 22 ]. These reviews are described in detail in the next sub-section. The remaining 181 reviews only provided indirect evidence and are summarised briefly in the subsequent sub-section and in Appendix 2 (see Additional file 2 ).

HSR systematic reviews comparing published and grey/unpublished evidence

Three HSR systematic reviews made such comparisons [ 20 , 21 , 22 ]. The topics of these reviews and their findings are summarised in Table 2 . The first review evaluated the effectiveness of mass mailings for increasing the utilization of influenza vaccine [ 22 ], focusing on evidence from controlled trials. The authors found one published study reporting statistically significant intervention effects, but additionally identified five unpublished studies through a Medicare quality improvement project database. All the unpublished studies reported clinically trivial intervention effects (no effect or an increase of less than two percentage point in uptake). This case illustrated the practical implications of publication bias: the authors highlighted that further mass mailing interventions were being considered by service planners on the basis of results from the first published study when they presented the review findings.

The second review compared the grey literature [ 20 ] with published literature [ 23 ] on the effectiveness and cost-effectiveness of strategies to improve immunization coverage in developing countries, and found that the quality and nature of evidence differed between these two sources of evidence, and that the recommendations about the most cost-effective interventions would differ between the two reviews (Table 2 ).

The third review assessed nine associations between various measures of organisational culture, organisational climate and nurse’s job satisfaction [ 21 ]. The author included both published literature and doctoral dissertations in the review, and statistically significant differences in the pooled estimates between these two types of literature were found in three of the nine associations (Table 2 ).

Findings from other systematic reviews of HSR topics

Of the 181 remaining systematic reviews, 100 examined potential publication bias across studies included in the reviews using funnel plots and related techniques, and 108 attempted to assess outcome reporting bias within individual included studies, generally as part of the risk of bias assessment. The methods used in these reviews and key findings in relation to publication bias and outcome reporting bias are summarised in Appendix 2 (see Additional file 2 ). Fifty-one of the 100 reviews which attempted to assess publication bias showed some evidence of its existence (through the assumption that observed small study effects were caused by publication bias).

For the assessment of outcome reporting bias, reviewers frequently reported difficulties in judging outcome reporting bias due to the absence of a published protocol for the included studies. For instance, a Cochrane review of the effectiveness of interventions to enhance medication adherence included 182 RCTs and judged eight and 32 RCTs to be of high and low risk for outcome reporting bias respectively, but the remaining 142 RCTs were judged to be of unclear risk, primarily due to unavailability of protocols [ 24 ]. In the absence of a protocol, some reviewers assessed outcome reporting bias by comparing outcomes specified in the methods to those presented in the results section, or made subjective judgements on the extent to which all important outcomes were reported. However, the validity of such approaches remains unclear. All but one of the reviews that assessed outcome reporting bias used either the Cochrane risk of bias tool (the checklist developed by the Cochrane Collaboration for assessing internal validity of individual RCTs) or bespoke tools derived from this. The remaining review - of the effectiveness of interventions for hypertension care in the community - undertook a sensitivity analysis to explore the influence of studies that otherwise met the inclusion criteria except for not providing sufficient data on relevant outcomes [ 25 ]. This was achieved by imputing zero effects (with average standard deviations) for the studies with missing outcomes (40 to 49% of potentially eligible studies), including them in the meta-analysis and recalculating the pooled effect. They found that the pooled effect was considerably reduced although still statistically significant [ 25 ]. These reviews illustrate the challenges of assessing outcome reporting bias in HSR and in identifying its potential consequences.

Delay in publication arising from the direction or strength of the study findings, referred to as time lag bias, was assessed in one of the reviews which evaluated the effectiveness of interventions for increasing the uptake of mammography in low and middle income countries [ 26 ]. The authors classified the time lag from end of intervention to the publication date into ≤4 years and > 4 years and reported that studies published within 4 years showed stronger association between intervention and mammography uptake (risk differences: 0.10, 95% CI 0.08, 0.12) when compared to studies published more than 4 years after completion (0.08, 95% CI 0.04, 0.11). However, the difference between the two subgroups was very small and not statistically significant (F ratio = 2.94, p = 0.10), and it was not clear whether this analysis and the cut-off time lag for defining the subgroups were specified a priori.

This systematic review examined current empirical evidence on publication and related biases in HSR. Very few methodological studies that directly investigated these issues were found. Nonetheless, a small number of available studies focusing on publication bias suggested its existence: findings of studies were not always reported/published; those published were often with positive results, and were sometimes of different nature, which could impact upon their applicability and relevance for different users of the evidence. There was also evidence suggesting that studies reporting larger effect sizes were more likely to be published in high impact journals. However, there are methodological weaknesses behind these pieces of evidence, which does not allow a firm conclusion to be drawn.

Reasons for non-publication of HSR findings described in the only survey we found appear to be similar to those of clinical research [ 27 ]. Lack of time and interest from the part of the researcher appears to be a major factor, which could exacerbate when the study findings are uninteresting. Also of note are comments such as “not of interest for others” and “only meant for internal use”. These not only illustrate context-sensitive nature of evidence for HSR, but also highlight issues arising from the hazy boundary between research and non-research for many evaluations undertaken in healthcare organizations, such as quality improvement projects and service audits. As promising findings are likely to motivate publication of these quality improvement projects, caution is required in interpreting and particularly in generalizing their findings. Another reason given for non-publication in HSR is “political and legal reasons”. Publication bias and restriction of access to data arising from conflict of interest is well documented in clinical research [ 2 ] and one might expect similar issues in HSR. We did not identify methodological research specifically related to the impact of conflict of interest on publication of findings in HSR, although anecdotal evidence of financial arrangement influencing editorial process exists [ 28 ], and there are debates concerning public’s accessibility of information related to health services and policy [ 29 ].

It is currently difficult to gauge the true scale and impact of publication and related biases given the sparse high quality evidence. Among the four methodological studies identified in this review, only one was an inception cohort study that provided direct evidence. This paucity of evidence is in stark contrast with a methodological review assessing publication bias and outcome reporting bias in clinical research, in which 20 inception cohort studies of RCTs were found [ 4 ]. The difference between these two fields is likely to be in part attributable to the less frequent use of RCTs in HSR and lack of requirement for study registration. The lesser reliance on RCTs and lack of study registration present a major methodological challenge in studying publication bias in HSR as there is no reliable way to identify studies that have been conducted but not subsequently published.

The lack of prospective study registration poses further challenges in assessing outcome reporting bias, which could be a greater concern for HSR than clinical research given the more exploratory approaches to examining a larger number of variables and associations in HSR. Empirical evidence on selective outcome reporting has primarily been obtained from RCTs as study protocols are made available in the trial registration process [ 4 ]. Calls for prospective registration of study protocols of observational studies have been made [ 30 ] and repositories of quality improvement projects are emerging [ 31 ]. HSR and quality improvement communities will need to consider and evaluate the feasibility and values of adopting these practices.

Statistical techniques such as funnel plots and regression methods are commonly used in HSR systematic reviews to identify potential publication bias, as in clinical research. Assumptions (e.g. any observed small study effects are caused by publication bias) and conditions (e.g. at least 10 studies measuring the same effect) related to the appropriate use of these techniques hold true for HSR, but heterogeneity commonly found among HSR studies resulting from the inherent complexity and variability of service delivery interventions and their interaction with contextual factors [ 32 , 33 ] may further influence the validity of funnel plots and related methods [ 34 ], and findings from these methods should be treated with caution [ 35 ].

In addition to the conventional methods discussed above, new methods such as p-curves for detecting p-hacking have emerged in recent years [ 36 , 37 ]. P-curves have been tested in various scientific disciplines [ 3 , 38 , 39 ], although no studies that we examined in the field of HSR have used this technique. The validity and usefulness of p-curves are subject to debate and accumulation of further empirical evidence [ 40 , 41 , 42 , 43 ].

Given the limitations of statistical methods, search of grey literature and contacting stakeholders to unearth unpublished studies remain an important means of mitigating publication bias, although this is often resource intensive and does not completely eliminate the risk. The finding from Batt et al. (2004) described above highlighted that published and grey literature could differ in their geographical coverage and nature of evidence [ 20 ]. This has important implications given the context-sensitive nature of HSR.

The limited evidence that we found does not allow us to estimate precisely the scale and impact of publication and related biases in HSR. It may be argued that publication bias may not be as prevalent in HSR as in clinical research because of the complexity of health systems which makes it often necessary to investigate the associations between a large number of variables along the service delivery causal pathway. As a result, HSR studies may be less likely to have completely null results or to depend for their contribution on single outcomes. Conversely, this heterogeneity and complexity may increase the scope for p-hacking and outcome reporting bias in HSR, which are even more difficult to prevent and detect.

A major challenge for this review was to delineate a boundary between HSR and other health/medical research. We used a broad range of search terms and identified a large number of studies, many of which were subsequently excluded after screening. We have used the definition of HSR provided by the UK NIHR and therefore our review may not have covered some areas of HSR if defined more broadly. We combined publication bias related terms with HSR related terms in our searches. As a result, we might not have captured some HSR related studies which have investigated publication and related bias but which did not mention them in their titles, abstracts or indexed terms. This is most likely to occur for systematic reviews of substantive HSR topics, in which funnel plot and related methods might have been deployed as a routine procedure to examine potential publication bias. Nevertheless, it is well known that statistical techniques such as funnel plot and related tests have low statistical power, and publication bias is just one of the many potential reasons behind ‘small study effects’ which these methods actually detect [ 34 ]. Findings from these systematic reviews are therefore of limited value in terms of confirming or refuting the existence of publication bias. Despite the limitation related to the search strategy, we identified and briefly examined more than 180 systematic reviews as shown in Appendix 2 in the supplementary file , but except for the small number of systematic reviews highlighted in the Results section, very little conclusion in relation to publication bias could be drawn from these reviews.

A further limitation of this study is that we have focused on publication and related biases related to quantitative studies and have not covered qualitative research, which plays an important role in HSR. It is also worth noting that three of the four included studies relate to the specific sub-field of health informatics which places limits on the extent to which our conclusions can be generalised to other subfields of HSR. Lastly, although we attempted to search several databases as well as grey literature, the possibility that evidence included in this review is subject to publication and related bias cannot be ruled out.

There is a paucity of empirical evidence and methodological literature addressing the issue of publication and related biases in HSR. While the available evidence suggests the presence of publication bias in this field, its magnitude and impact is yet to be fully explored and understood. Further research evaluating the existence of publication and related biases in HSR, what factors contribute towards their occurrence, their impact and the range of potential strategies to mitigate them, is therefore warranted.

Availability of data and materials

All data generated and/or analysed during this review are included within this article and its additional files. This systematic review was part of a large project investigating publication and related bias in HSR. The full technical report for the project will be published in the UK National Institute for Health Research (NIHR) Journals Library: https://www.journalslibrary.nihr.ac.uk/programmes/hsdr/157106/#/

Abbreviations

Agency for Healthcare Research and Quality

Effective Practice and Organisation of Care

Health Systems Evidence

Health Services Research

National Institute for Health Research Health Services & Delivery Research Programme

Randomised controlled trials

Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database Syst Rev. 2007;2:MR000011.

Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing C, Kwok CS, Pang C, Harvey I. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010;14(8):1–193.

Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. The extent and consequences of p-hacking in science. PLoS Biol. 2015;13(3):e1002106.

Article Google Scholar

Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One. 2013;8(7):e66844.

Kicinski M, Springate DA, Kontopantelis E. Publication bias in meta-analyses from the Cochrane database of systematic reviews. Stat Med. 2015;34(20):2781–93.

Gulmezoglu AM, Pang T, Horton R, Dickersin K. WHO facilitates international collaboration in setting standards for clinical trial registration. Lancet. 2005;365(9474):1829–31.

Li X, Zheng Y, Chen T-L, Yang K-H, Zhang Z-J. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy. 2015;119(4):503–10.

Article CAS Google Scholar

The World Medical Association. WMA declaration of Helsinki - ethical principles for medical research involving human subjects. In: Current policies. The World Medical Association; 2013. https://www.wma.net/policies-post/wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/ . Accessed 26 Apr 2020.

Lilford RJ, Chilton PJ, Hemming K, Girling AJ, Taylor CA, Barach P. Evaluating policy and service interventions: framework to guide selection and interpretation of study end points. BMJ. 2010;341:c4413.

Gelman A, Loken E. The garden of forking paths: why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time (2013). http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf . Accessed 25 July 2018.

Google Scholar

Smith R. Quality improvement reports: a new kind of article. They should allow authors to describe improvement projects so others can learn. BMJ. 2000;321(7274):1428.

Toews I, Glenton C, Lewin S, Berg RC, Noyes J, Booth A, Marusic A, Malicki M, Munthe-Kaas HM, Meerpohl JJ. Extent, awareness and perception of dissemination bias in qualitative research: an explorative survey. PLoS One. 2016;11(8):e0159290.

Liberati A, Altman DG, Tetzlaff J. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700.

Wilczynski NL, Haynes RB, Lavis JN, Ramkissoonsingh R, Arnold-Oatley AE, The HSRHT. Optimal search strategies for detecting health services research studies in MEDLINE. CMAJ. 2004;171(10):1179–85.

Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy. 2005;10(Suppl 1):6–20.

Ammenwerth E, de Keizer N. A viewpoint on evidence-based health informatics, based on a pilot survey on evaluation studies in health care informatics. JAMIA. 2007;14(3):368–71.

PubMed Google Scholar

Machan C, Ammenwerth E, Bodner T. Publication bias in medical informatics evaluation research: is it an issue or not? Stud Health Technol Inform. 2006;124:957–62.

Costa-Font J, McGuire A, Stanley T. Publication selection in health policy research: the winner's curse hypothesis. Health Policy. 2013;109(1):78–87.

Vawdrey DK, Hripcsak G. Publication bias in clinical trials of electronic health records. J Biomed Inform. 2013;46(1):139–41.

Batt K, Fox-Rushby JA, Castillo-Riquelme M. The costs, effects and cost-effectiveness of strategies to increase coverage of routine immunizations in low- and middle-income countries: systematic review of the grey literature. Bull World Health Organ. 2004;82(9):689–96.

PubMed PubMed Central Google Scholar

Fang Y. A meta-analysis of relationships between organizational culture, organizational climate, and nurse work outcomes (PhD thesis). Baltimore: University of Maryland; 2007.

Maglione MA, Stone EG, Shekelle PG. Mass mailings have little effect on utilization of influenza vaccine among Medicare beneficiaries. Am J Prev Med. 2002;23(1):43–6.

Pegurri E, Fox-Rushby JA, Damian W. The effects and costs of expanding the coverage of immunisation services in developing countries: a systematic literature review. Vaccine. 2005;23(13):1624–35.

Nieuwlaat R, Wilczynski N, Navarro T, Hobson N, Jeffery R, Keepanasseril A, Agoritsas T, Mistry N, Iorio A, Jack S, et al. Interventions for enhancing medication adherence. Cochrane Database Syst Rev. 2014;11:CD000011.

Lu Z, Cao S, Chai Y, Liang Y, Bachmann M, Suhrcke M, Song F. Effectiveness of interventions for hypertension care in the community--a meta-analysis of controlled studies in China. BMC Health Serv Res. 2012;12:216.

Gardner MP, Adams A, Jeffreys M. Interventions to increase the uptake of mammography amongst low income women: a systematic review and meta-analysis. PLoS One. 2013;8(2):e55574.

Song F, Loke Y, Hooper L. Why are medical and health-related studies not being published? A systematic review of reasons given by investigators. PLoS One. 2014;9(10):e110418.

Homedes N, Ugalde A. Are private interests clouding the peer-review process of the WHO bulletin? A case study. Account Res. 2016;23(5):309–17.

Dyer C. Information commissioner condemns health secretary for failing to publish risk register. BMJ. 2012;344:e3480.

Swaen GMH, Urlings MJE, Zeegers MP. Outcome reporting bias in observational epidemiology studies on phthalates. Ann Epidemiol. 2016;26(8):597–599.e594.

Bytautas JP, Gheihman G, Dobrow MJ. A scoping review of online repositories of quality improvement projects, interventions and initiatives in healthcare. BMJ Qual Safety. 2017;26(4):296–303.

Long KM, McDermott F, Meadows GN. Being pragmatic about healthcare complexity: our experiences applying complexity theory and pragmatism to health services research. BMC Med. 2018;16(1):94.

Greenhalgh T, Papoutsi C. Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med. 2018;16(1):95.

Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, Carpenter J, Rücker G, Harbord RM, Schmid CH, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002.

Lau J, Ioannidis JPA, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ. 2006;333(7568):597–600.

Simonsohn U, Nelson LD, Simmons JP. P-curve: a key to the file-drawer. J Exp Psychol Gen. 2014;143(2):534–47.

Simonsohn U, Nelson LD, Simmons JP. P-curve and effect size: correcting for publication Bias using only significant results. Perspect Psychol Sci. 2014;9(6):666–81.

Carbine KA, Larson MJ. Quantifying the presence of evidential value and selective reporting in food-related inhibitory control training: a p-curve analysis. Health Psychol Rev. 2019;13(3):318–43.

Carbine KA, Lindsey HM, Rodeback RE, Larson MJ. Quantifying evidential value and selective reporting in recent and 10-year past psychophysiological literature: a pre-registered P-curve analysis. Int J Psychophysiol. 2019;142:33–49.

Bishop DV, Thompson PA. Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value. PeerJ. 2016;4:e1715.

Bruns SB, Ioannidis JPA. P-curve and p-hacking in observational research. PLoS One. 2016;11(2):e0149144.

Simonsohn U, Simmons JP, Nelson LD. Better P-curves: making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a reply to Ulrich and Miller (2015). J Exp Psychol Gen. 2015;144(6):1146–52.

Ulrich R, Miller J. Some properties of p-curves, with an application to gradual publication bias. Psychol Methods. 2018;23(3):546–60.

Download references

Acknowledgements

We are grateful for the advice and guidance provided by members of the Study Steering Committee for the project.

This project is funded by the UK NIHR Health Services and Delivery Research Programme (project number 15/71/06). The authors are required to notify the funder prior to the publication of study findings, but the funder does not otherwise have any roles in the preparation of the manuscript and the decision to submit and publish it. MS and RJL are also supported by the NIHR Applied Research Collaboration (ARC) West Midlands. The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the HS&DR Programme, NIHR, National Health Services or the Department of Health.

Author information

Authors and affiliations.

Warwick Centre for Applied Health Research & Delivery, Division of Health Sciences, Warwick Medical School, University of Warwick, Coventry, UK

Abimbola A. Ayorinde & Yen-Fu Chen

Health Services Management Centre, School of Social Policy, University of Birmingham, Birmingham, UK

Iestyn Williams & Russell Mannion

Norwich Medical School, University of East Anglia, Norwich, UK

Fujian Song

Institute of Applied Health Research, University of Birmingham, Birmingham, UK

Magdalena Skrybant & Richard J. Lilford

You can also search for this author in PubMed Google Scholar

Contributions

YFC and RJL conceptualised the study. AAA and YFC contributed to all stages of the review and drafted the paper. IW, RM, FS, MS, RJL were involved in planning the study, advised on the conduct of the review and interpretation of the findings. All authors reviewed and helped revising drafts of this paper and approved its submission.

Corresponding author

Correspondence to Yen-Fu Chen .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1..

PRISMA checklist.

Additional file 2.

Appendices.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Ayorinde, A.A., Williams, I., Mannion, R. et al. Publication and related biases in health services research: a systematic review of empirical evidence. BMC Med Res Methodol 20 , 137 (2020). https://doi.org/10.1186/s12874-020-01010-1

Download citation

Received : 28 January 2019

Accepted : 07 May 2020

Published : 01 June 2020

DOI : https://doi.org/10.1186/s12874-020-01010-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Publication bias
Outcome reporting bias
Dissemination bias
Grey literature
Research publication
Research registration
Health services research
Systematic review
Research methodology
Funnel plots

BMC Medical Research Methodology

ISSN: 1471-2288

General enquiries: [email protected]

Bias in clinical research

Affiliation.

1 CNR-IBIM, Clinical Epidemiology and Physiopathology of Renal Diseases and Hypertension, Reggio Calabria, Italy. [email protected]
PMID: 17978812
DOI: 10.1038/sj.ki.5002648

The quality of a clinical study depends on internal and external factors. Studies have internal validity when, random error apart, reported differences between exposed and unexposed individuals can be attributed only to the exposure under investigation. Internal validity may be affected by bias, that is, by any systematic error that occurs in the design or in the conduction of a clinical research. Here we focus on two major categories of bias: selection bias and information bias. We describe three types of selection biases (incidence-prevalence bias, loss-to-follow-up bias, and publication bias) and a series of information biases (i.e. misclassification bias--recall bias, interviewer bias, observer bias, and regression dilution bias--and lead-time bias).

Biomedical Research*
Data Collection
Follow-Up Studies
Publication Bias

University Library, University of Illinois at Urbana-Champaign

Evidence-Based Medicine (EBM)

EBM as Process
ASK - PICO Quesitons
Tools & Databases for Evidence
Appraisal Tools
Appraisal by Study Type
Calculators
Users' Guides to the Medical Literature
APPLY - Using Evidence
Organizations & Agencies

Tools for assessing risk of bias

OHAT risk of Bias Rating Tool for Human and Animal Studies This document is written to outline a tool for evaluating individual study risk of bias or internal validity – the assessment of whether the design and conduct of a study compromised the credibility of the link between exposure and outcome (Higgins and Green 2011, IOM 2011, Viswanathan et al. 2012). The risk-of-bias rating tool presents a parallel approach to evaluating risk of bias in human and non-human animal studies to facilitate consideration of risk of bias across elements and across evidence streams with common terms and categories.
ROBIS ROBIS is a new tool for assessing the risk of bias in systematic reviews (rather than in primary studies). ROBIS has been developed using rigorous methodology and is currently aimed at four broad categories of reviews mainly within healthcare settings: interventions, diagnosis, prognosis and aetiology. The target audience of ROBIS is primarily guideline developers, authors of overviews of systematic reviews (“reviews of reviews”) and review authors who might want to assess or avoid risk of bias in their reviews.

Types of Bias

Publication bias - How similar are results from published versus unpublished studies?

Incomplete or selective reporting of outcomes
Arbitrary limits such as language or choice of resources
Truncation bias - study is published in a briefer form with less details
Time-lag bias - delayed publication of findings
Language bias - more likely to be published in English
Citation Bias - citation/non-citation of research findings
Selective outcome reporting bias - selective reporting of some outcomes but not others
Location bias - journals with different ease of access/levels of indexing in standard databases
Multiple (duplicate) publications
Database bias - some databases are more likely to index certain languages/journals

Source: Rothstein, D. H. R., Sutton, D. A. J., & Borenstein, D. M. (2006). Publication Bias in Meta-Analysis. Publication Bias in Meta-Analysis (pp. 1-7) doi:10.1002/0470870168.ch1

Cochrane Collaboration - Introduction to sources of bias in clinical trials

7: Considering bias and conflicts of interest among the included studies Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook.

8.4 Introduction to sources of bias in clinical trials

8.4.1 Selection bias

8.4.2 Performance bias

8.4.3 Detection bias

8.4.4 Attrition bias

8.4.5 Reporting bias

8.4.6 Other biases

Source: https://handbook-5-1.cochrane.org/chapter_8/8_4_introduction_to_sources_of_bias_in_clinical_trials.htm

Understanding Bias Reources

Catalogue of Bias To better understand the persistent presence, diversity, and impact of biases, we are compiling a Catalogue of Biases, stemming from original work by David Sackett. The entries are a work in progress and describe a wide range of biases – outlining their potential impact in research studies.
Medical Biostatistics & Research "Medical Biostatistics comprises statistical methods that are used to manage uncertainties in the field of medicine and health." Indrayan A. Medical Biostatistics, Third Edition. Chapman & Hall/CRC Press, 2012:2. Redefining Biostatistics
COMPARE - Tracking switched outcomes in clinical trials Outcome switching in clinical trials is a serious problem. Between October 2015 and January 2016, the COMPare team systematically checked every trial published in the top five medical journals, to see if they misreported their findings. We are now submitting the first set of findings from the project as an academic paper, summarising the quantitative results, and the themes of responses from journal editors and trialists in collaboration with a qualitative researcher. Prior to publication, cite our data and methods as per the reference at the bottom of this page.
Statistics in clinical trials: Bias From European Patients' Academy EUPATI - Statistical methods provide formal accounting for sources of variability in patients’ responses to treatment. The use of statistics allows the clinical researcher to form reasonable and accurate inferences from collected information, and sound decisions in the presence of uncertainty. Statistics are key to preventing errors and biases in medical research. This article covers the concept of bias in clinical trials.
Understanding Health Research Common Sources of Bias One of the main problems with scientific studies is that bias (the conscious or unconscious influencing of the study and its results) can make them less dependable. bias can occur in a number of different ways and it is important for researchers to be aware of these and find ways to minimize bias. There are a great number of ways that bias can occur, these are a few common examples:

Equity and Bias

Campbell and Cochrane Equity Methods Group

The aim is to encourage authors of Campbell and Cochrane reviews to include explicit descriptions of the effect of the interventions not only on the whole population but to describe their effect upon the disadvantaged and/or their ability to reduce socioeconomic inequalities in health and to promote their use to the wider community.

University of Minnesota Libraries - Conducting research through an anti-racism lens

This guide is for students, staff, and faculty who are incorporating an anti-racist lens at all stages of the research life cycle.

Impact of Bias

Beware evdience "spin": an important source of bias in the reporting of clinical research For many researchers, the number of publications, and the impact of those publications, is the usual currency for measuring professional worth. Furthermore, we are increasingly seeing researchers discuss their work in public through mainstream and social media, as more of these opportunities arise. With this in mind it probably won’t come as such a shock to imagine that researchers might be tempted to report their results in a more favorable (again, even glowing) way than they deserve i.e. to add some “spin”.
Outcome reporting bias: is it ok to be a little selective? A large part of being a scientist is venturing into the unknown. You come up with hypotheses and test them through experiments. The problem is that more often than not, the experiments infrequently give you the BIG outcome you were perhaps hoping for, the one that might revolutionise clinical practice tomorrow, and instead often give marginal or equivocal (not significant) results.
<< Previous: Appraisal by Study Type
Next: Calculators >>
Last Updated: Dec 20, 2023 10:49 AM
URL: https://guides.library.illinois.edu/ebm

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

View all journals
My Account Login
Explore content
About the journal
Publish with us
Sign up for alerts
Open access
Published: 23 August 2021

Mitigating bias in machine learning for medicine

Kerstin N. Vokinger ORCID: orcid.org/0000-0002-6997-7384 1 , 2 ,
Stefan Feuerriegel 3 , 4 &
Aaron S. Kesselheim 2

Communications Medicine volume 1 , Article number: 25 ( 2021 ) Cite this article

16k Accesses

103 Citations

60 Altmetric

Metrics details

Health care
Medical research

Several sources of bias can affect the performance of machine learning systems used in medicine and potentially impact clinical care. Here, we discuss solutions to mitigate bias across the different development steps of machine learning-based systems for medical applications.

Machine learning (ML) is an artificial intelligence technique that can be used to train algorithms to learn from and act on data 1 . ML in medicine aims to improve patient care by deriving new and relevant insights from the vast amount of data generated by individual patients and the collective experience of many patients 1 , 2 . The number of authorized ML-based systems has increased over the past years, with many being authorized by agencies such as the US Food and Drug Administration (FDA) for use in radiology, for example, to support tumor detection in diagnostic imaging 3 .

The creation of ML-based systems in medicine involves a number of steps. Relevant clinical data must first be collected and prepared for use in model development. Development of the model involves selection and training of suitable mathematical algorithms to perform the intended task. Model performance must subsequently be evaluated in independent cohorts of patients before potential authorization by agencies and deployment in clinical practice.

The FDA has recognized challenges due to bias in ML and released an Action Plan in January 2021, highlighting the importance of identifying and mitigating bias in ML-based systems for medicine.

It has been demonstrated that the outcomes of ML-based systems can be subject to systematic errors in their ability to classify subgroups of patients, estimate risk levels, or make predictions. These errors can be introduced across the various stages of development. Such errors are commonly referred to as bias 4 , 5 , 6 . For example, previous research has found that the application of a commercial prediction algorithm resulted in significant racial bias in predicting outcomes. Black patients assigned the same level of risk by the algorithm were sicker than white patients. This bias occurred because the algorithm used health costs as a proxy for health needs. Since less money is spent on black patients who have the same level of need, the algorithm falsely concluded that black patients were healthier than equally sick white patients 5 .

The FDA has recognized challenges due to bias in ML and released an Action Plan in January 2021, highlighting the importance of identifying and mitigating bias in ML-based systems for medicine 6 . Underrepresented groups in medical research are particularly susceptible to the impact of bias.

Previous studies have focused on the detection of bias 4 , but more discussion of possible solutions is needed. Here, we outline proposed solutions on how to mitigate bias across the different development steps of ML-based systems for medical applications (Fig. 1 ): data collection and data preparation, model development, model evaluation, and deployment in clinical practice (post-authorization).

Diagram outlining proposed solutions on how to mitigate bias across the different development steps of ML-based systems for medical applications: (1) Data collection and data preparation, (2) Model development, (3) Model evaluation, and (4) Deployment.

Data collection and data preparation

The first steps in which bias can be introduced are data collection and data preparation. In many ML-based applications in medicine, data of predictors, such as risk factors or other clinical parameters, serve as input, and, based on them, an outcome is predicted. For example, in the Framingham score, a tool to assess the cardiovascular risk of a patient, sociodemographics and other risk factors represent the predictors as input, while the risk level is the outcome that is to be predicted.

If the training data used to develop the ML-based system is subject to sampling bias, meaning that when the patient cohort in the data for training the ML model is not representative of the population for which the ML system is intended to be used, the same bias may be replicated when the system is applied in the clinical setting. For example, if a ML-based system is trained to recognize skin disease, such as melanoma, based on images from people with white skin, it might misinterpret images from patients with a darker skin tone, and might fail to diagnose melanoma 7 . This can lead to potentially serious consequences, since melanoma is responsible for the majority of skin cancer-associated deaths, and early diagnosis is critical for it to be curable 7 . To mitigate such bias, the developers of ML-based systems for medical applications should be transparent about the selected training data with regard to patient demographic and baseline characteristics, such as number of patients, distribution of patients’ age, representation of race and ethnicity, as well as gender. To address this type of bias, one should strive to compile datasets that are as diverse and large as possible to have a better representation of all patient groups. Developers can also carefully monitor error rates of ML software applied to different patient cohorts, and identify when the performance level deteriorates for a subset of patients. The performance level should then be disclosed by the developer in the authorization process.

In order to assess the risk of this type of bias, reporting checklists, such as PROBAST (prediction model risk of bias assessment tool), have been developed 8 . Such reporting checklists should be completed by developers and could guide the FDA in the authorization process to better understand the risk of potential bias of an ML-based system. Furthermore, it will also allow the end-users, such as physicians, to better understand whether the ML-based system is suitable in a specific setting for a specific group of patients.

Model development

The modeling step uses mathematical algorithms to perform predictions, estimations or classifications in clinical parameters based on training data. The modeling step can perpetuate existing bias in the data. A naive application of a ML-based system without accounting for bias learns good predictions for the average population but does not necessarily incentivize the model to learn good predictions for those that are underrepresented in the data due to sampling bias (i.e., the underrepresented groups). In consequence, a model might perform overall better, yet it trades in a better performance for groups that are well represented at the cost of a lower performance (i.e., systematic errors) for the underrepresented groups.

Beyond creating diverse datasets for model development, there are mathematical approaches for de-biasing that mitigate the risk of bias at this step, such as adversarial de-biasing 9 or oversampling 10 . Such approaches force the model to account for underrepresented groups and achieve a better performance when applied to them. However, techniques for de-biasing have only recently emerged in computer science and more research is still needed to demonstrate proof-of-principle and show that de-biasing reliably achieves its intended purpose. Systematic errors might also be reduced through continual learning 11 , whereby an ML-based system is continuously updated through new data while retaining previously learned knowledge 12 .

Model evaluation

The model evaluation step, which is performed prior to authorization, is concerned with assessing how well the model makes predictions in independent groups of patients and in independent clinical studies and/or trials. This validates how the model generalizes to data from different patients and thus provides insights on how and where errors occur. Hence, it allows developers to identify bias that is introduced during the modeling step and also pinpoints to predictors that might be biased (e.g., due to a measurement error). Developers should carefully evaluate model performance, for example across certain subgroups of the patients, and inspect whether the model predicts incorrect outcomes.

Strategies to inspect how a ML model reaches an outcome can be grouped into techniques for interpretability or explainability 13 . Interpretability refers to models that are transparent in how outcomes are generated and where a user can understand the internal decision logic of the model (i.e., because the model has only a few parameters). By contrast, explainability means that a second model is created to explain the actual ML-based system, but where the actually ML-based system is not necessarily transparent (i.e., because it has millions of parameters). Explainability in ML is supported via various software tools, such as SHAP 14 or LIME 15 . By understanding how a ML model reaches outcomes, developers can then validate the insights against prior knowledge from clinical research to ensure that a mathematical model considers known risk factors. In particular, developers may identify systematic errors in a ML-based system and then revise the model development accordingly, for instance by removing the responsible predictor or choosing a different model. However, caution would be recommended since ML explainability can be inaccurate or non-meaningful due to the underlying mathematical assumptions 13 . Hence, for high-risk ML-based systems in medicine, it might be better that models are limited to those that are interpretable.

In the deployment step, when the ML-based system has passed regulatory authorization and is implemented in clinical practice, bias can occur in situations where the patient cohort in clinical practice differs from the patient cohort in the training data, which is known as domain shift. This can lead to a deterioration in the performance with potentially negative outcomes for patients. Such a domain shift can, for example, occur if a ML-based system was developed with data from a US population, but is implemented in other geographies. To identify such unwanted bias, it is crucial that the ML-based systems are carefully monitored after authorization. This monitoring should include the following dimensions: the sociodemographics characteristics of patients to assess whether these are representative of the patients included in the training data; risk factors to check whether the patients have the same overall risk level because since, if the risk level differs, the ML model might no longer be precise; and the prediction performance of the ML-based system overall and across patients subgroups to identify other sources of error that were not known during model development. Monitoring ML-based systems after authorization is important to ensure that the performance does not degrade in clinical practice. If this occurs, the ML-based system needs to be updated with new post-authorization data.

Unwanted bias in clinical practice can also result from feedback loops. Feedback loops occur to when outcomes influence clinical practice so that a new bias is created. Post-authorization monitoring would also help identify such feedback loops so that steps can be taken to address its impact in clinical practice.

Bias in ML-based systems for medical applications can occur across the different development steps, data collection and data preparation, model development, model evaluation, and post-authorization deployment in clinical practice. However, there are various strategies that reduce the risk of bias, including transparency about the selected training datasets, mathematical approaches to de-biasing, ML interpretability or explainability, and post-authorization monitoring. These strategies should become best practice for any ML-based system that is developed with medical application in mind. It is crucial that bias is mitigated when developing and deploying ML-based systems in medicine to prevent health care inequality for particular patient groups and to ensure a functionality that is safe for all patients.

FDA. Artificial intelligence and machine learning in software as a medical device 2021. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device (2021).

Hwang, T. J., Kesselheim, A. S. & Vokinger, K. N. Lifecycle regulation of artificial intelligence- and machine learning-based software devices in medicine. J. Am. Med. Assoc. 322 , 2285 (2019).

Article Google Scholar

Muehlematter, U. J., Daniore, P. & Vokinger, K. N. Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015-20): a comparative analysis. Lancet Digit. Health 3 , E195-E203 (2021).

Parikh, R. B., Teeple, S. & Navathe, A. S. Addressing bias in artificial intelligence in health care. J. Am. Med. Assoc. 322 , 2377 (2019).

Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366 , 447–453 (2019).

Article CAS Google Scholar

FDA. Artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD) action plan. https://www.fda.gov/media/145022/download (2021).

Adamson, A. S. & Smith, A. Machine learning and health care disparities in dermatology. JAMA Dermatol. 154 , 1247 (2018).

Wolff, R. F. et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann. Intern. Med. 170 , 51 (2019).

Zhang, B. H., Lemoine, B. & Mitchell, M. Mitigating unwanted biases with adversarial learning. in Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society . (ed. Furman, Jason, Marchant, Gary, Price, Huw, Rossi, Francesca) 335–340 (ACM, 2018).

Kamiran, F. & Calders, T. Data preprocessing techniques for classification without discrimination. Knowl. Inf. Syst. 33 , 1–33 (2012).

Vokinger, K. N., Feuerriegel, S. & Kesselheim, A. S. Continual learning in medical devices: FDA’s action plan and beyond. Lancet Digit. Health 3 , e337–e338 (2021).

Lee, C. S. & Lee, A. Y. Clinical applications of continual learning machine learning. Lancet Digital Health 2 , e279–e281 (2020).

Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1 , 206–215 (2019).

Lundberg, S. & Lee, S.-I. A unified approach to interpreting model predictions. in Proceedings of the International Conference on Neural Information Processing Systems 4768–4777 (2017).

Ribeiro, M. T., Singh, S. & Guestrin, C. “Why should i trust you?”: explaining the predictions of any classifier. in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (ed. Krishnapuram, Balaji, Shah, Mohak, Smola, Alex, Aggarwal, Charu, Shen, Dou, Rastogi, Rajeev) 1135–1144 (ACM, 2016).

Download references

Acknowledgements

K.N.V. is supported by the Swiss Cancer Research Foundation (Krebsforschung Schweiz) and the Swiss National Science Foundation (SNSF). S.F. is supported by the Swiss National Science Foundation (SNSF). A.S.K. is supported by Arnold Ventures.

Author information

Authors and affiliations.

Institute of Law, University of Zurich, Zurich, Switzerland

Kerstin N. Vokinger

Program on Regulation, Therapeutics, and Law (PORTAL), Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, MA, USA

Kerstin N. Vokinger & Aaron S. Kesselheim

ETH Zurich, Zurich, Switzerland

Stefan Feuerriegel

LMU Munich, Munich, Germany

You can also search for this author in PubMed Google Scholar

Contributions

K.N.V., S.F., and A.S.K. contributed to writing.

Corresponding author

Correspondence to Kerstin N. Vokinger .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Vokinger, K.N., Feuerriegel, S. & Kesselheim, A.S. Mitigating bias in machine learning for medicine. Commun Med 1 , 25 (2021). https://doi.org/10.1038/s43856-021-00028-w

Download citation

Received : 27 May 2021

Accepted : 30 July 2021

Published : 23 August 2021

DOI : https://doi.org/10.1038/s43856-021-00028-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

An introduction to machine learning and generative artificial intelligence for otolaryngologists—head and neck surgeons: a narrative review.

Isaac L. Alter
Anaïs Rameau

European Archives of Oto-Rhino-Laryngology (2024)

Mining Data Wrangling Workflows for Design Patterns Discovery and Specification

Abdullah AlMasaud
Sandra Sampaio
Pedro Sampaio

Information Systems Frontiers (2024)

Behind the mask: a critical perspective on the ethical, moral, and legal implications of AI in ophthalmology

Daniele Veritti
Leopoldo Rubinato
Paolo Lanzetta

Graefe's Archive for Clinical and Experimental Ophthalmology (2024)

Systematic Review of Machine Learning Applied to the Secondary Prevention of Ischemic Stroke

Dongbao Qian
Chunying Pang

Journal of Medical Systems (2024)

Ethical challenges of using artificial intelligence in healthcare delivery: a thematic analysis of a systematic review of reviews

Mohsen Khosravi
Reyhane Izadi

Journal of Public Health (2024)

Quick links

Explore articles by subject
Guide to authors
Editorial policies

Search Search …

Bias in medical research

A bias in evidence based medicine is any factor that leads to conclusions that are systematically different from the truth.

Bias is universal. Some study designs are better than others, but there is no perfect study. Although in general parlance “bias” has moral or ethical implications, research bias does not refer to the researcher’s character, just the validity of the study. (I do not include fraud as a type of research bias, but it is important to be aware that the medical literature is full of examples of both major and minor research fraud.)

Bias is not something that can be accounted for with statistics. Larger sample sizes will can create more precision, but that doesn’t help if the numbers aren’t accurate. Ideally, we want to see research that is both precise and accurate, but I would take accurate over precise any day.

I really believe that evidence based medicine is easy . Most types of research bias are actually quite easy to understand. Unfortunately, critical appraisal novices are frequently scared off by the sheer number of biases and the technical jargon that is often used to describe them. For that reason, I have started this glossary of research biases.

Bias Glossary

Allocation bias

Ascertainment bias

Attention bias

Attrition bias

Chronological bias

Co-intervention bias

Contamination bias

Diagnostic access bias

Diagnostic suspicion bias

Detection bias

Detection-signal bias

Expectation bias

Exposure bias

Hawthorne effect

Incorporation bias

Inflation bias

Insensitive measurement bias

Intervention bias

Lead time bias

Measurement bias

Membership bias

Neyman bias

Non-respondent bias

Observer bias

Prevalence-incidence bias

Proficiency bias

Publication bias

Recall bias

Referral bias

Reporting bias

Response bias

Sampling bias

Selection bias

Spectrum bias

Survival bias

Time lag bias

Timing bias

Unmasking bias

Verification bias (or partial verification bias)

Volunteer bias

There is no official list of research biases to refer to. As a consequence, a number of these terms are used differently by different people. I have included competing definitions wherever I found them. If there are other research biases that you think should be included, or have suggestions to improve this resource, please let me know .

More evidence based medicine resources can be found here .

Key Point: The existence of research bias does not indicate wrongdoing on behalf of the researchers. For example, a researcher could run a methodologically perfect weight loss trial, but because people that volunteer for such trials are systematically different from the general population, the results could be impacted by selection bias, limiting the generalizability of the results.

Other Resources

(Lots of other great videos on http://www.sketchyebm.com/ )

http://www.jameslindlibrary.org/topics/

https://catalogofbias.org/

Sackett DL. Bias in analytic research. Journal of chronic diseases. 1979; 32(1-2):51-63. PMID: 447779

Justin Morgenstern

What’s the word on Word catheters?

A brief review of the evidence in the management of Bartholin’s abscesses

DOSE VF double sequential defibrillation

Dose VF: A double sequential defibrillation game changer?

I can almost guarantee that everyone has heard about this paper already. It is an exciting topic. It is resuscitation and gadgets. […]

Idiopathic Intracranial Hypertension: a rapid review

Suzie, an otherwise healthy 32-year-old female, presents to your Emergency Department complaining of her typical migraine-type headache. Slam dunk. She’ll have a […]

A general approach to resuscitation

An expanded ABCs approach to be used as a framework to guide resuscitation.

Discover more from First10EM

Subscribe now to keep reading and get access to the full archive.

Type your email…

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

Advanced Search
Journal List
Biochem Med (Zagreb)
v.23(1); 2013 Feb

Bias in research

By writing scientific articles we communicate science among colleagues and peers. By doing this, it is our responsibility to adhere to some basic principles like transparency and accuracy. Authors, journal editors and reviewers need to be concerned about the quality of the work submitted for publication and ensure that only studies which have been designed, conducted and reported in a transparent way, honestly and without any deviation from the truth get to be published. Any such trend or deviation from the truth in data collection, analysis, interpretation and publication is called bias. Bias in research can occur either intentionally or unintentionally. Bias causes false conclusions and is potentially misleading. Therefore, it is immoral and unethical to conduct biased research. Every scientist should thus be aware of all potential sources of bias and undertake all possible actions to reduce or minimize the deviation from the truth. This article describes some basic issues related to bias in research.

Introduction

Scientific papers are tools for communicating science between colleagues and peers. Every research needs to be designed, conducted and reported in a transparent way, honestly and without any deviation from the truth. Research which is not compliant with those basic principles is misleading. Such studies create distorted impressions and false conclusions and thus can cause wrong medical decisions, harm to the patient as well as substantial financial losses. This article provides the insight into the ways of recognizing sources of bias and avoiding bias in research.

Definition of bias

Bias is any trend or deviation from the truth in data collection, data analysis, interpretation and publication which can cause false conclusions. Bias can occur either intentionally or unintentionally ( 1 ). Intention to introduce bias into someone’s research is immoral. Nevertheless, considering the possible consequences of a biased research, it is almost equally irresponsible to conduct and publish a biased research unintentionally.

It is worth pointing out that every study has its confounding variables and limitations. Confounding effect cannot be completely avoided. Every scientist should therefore be aware of all potential sources of bias and undertake all possible actions to reduce and minimize the deviation from the truth. If deviation is still present, authors should confess it in their articles by declaring the known limitations of their work.

It is also the responsibility of editors and reviewers to detect any potential bias. If such bias exists, it is up to the editor to decide whether the bias has an important effect on the study conclusions. If that is the case, such articles need to be rejected for publication, because its conclusions are not valid.

Bias in data collection

Population consists of all individuals with a characteristic of interest. Since, studying a population is quite often impossible due to the limited time and money; we usually study a phenomenon of interest in a representative sample. By doing this, we hope that what we have learned from a sample can be generalized to the entire population ( 2 ). To be able to do so, a sample needs to be representative of the population. If this is not the case, conclusions will not be generalizable, i.e. the study will not have the external validity.

So, sampling is a crucial step for every research. While collecting data for research, there are numerous ways by which researchers can introduce bias in the study. If, for example, during patient recruitment, some patients are less or more likely to enter the study than others, such sample would not be representative of the population in which this research is done. In that case, these subjects who are less likely to enter the study will be under-represented and those who are more likely to enter the study will be over-represented relative to others in the general population, to which conclusions of the study are to be applied to. This is what we call a selection bias . To ensure that a sample is representative of a population, sampling should be random, i.e. every subject needs to have equal probability to be included in the study. It should be noted that sampling bias can also occur if sample is too small to represent the target population ( 3 ).

For example, if the aim of the study is to assess the average hsCRP (high sensitive C-reactive protein) concentration in healthy population in Croatia, the way to go would be to recruit healthy individuals from a general population during their regular annual health check up. On the other hand, a biased study would be one which recruits only volunteer blood donors because healthy blood donors are usually individuals who feel themselves healthy and who are not suffering from any condition or illness which might cause changes in hsCRP concentration. By recruiting only healthy blood donors we might conclude that hsCRP is much lower that it really is. This is a kind of sampling bias, which we call a volunteer bias .

Another example for volunteer bias occurs by inviting colleagues from a laboratory or clinical department to participate in the study on some new marker for anemia. It is very likely that such study would preferentially include those participants who might suspect to be anemic and are curious to learn it from this new test. This way, anemic individuals might be over-represented. A research would then be biased and it would not allow generalization of conclusions to the rest of the population.

Generally speaking, whenever cross-sectional or case control studies are done exclusively in hospital settings, there is a good chance that such study will be biased. This is called admission bias . Bias exists because the population studied does not reflect the general population.

Another example of sampling bias is the so called survivor bias which usually occurs in cross-sectional studies. If a study is aimed to assess the association of altered KLK6 (human Kallikrein-6) expression with a 10 year incidence of Alzheimer’s disease, subjects who died before the study end point might be missed from the study.

Misclassification bias is a kind of sampling bias which occurs when a disease of interest is poorly defined, when there is no gold standard for diagnosis of the disease or when a disease might not be easy detectable. This way some subjects are falsely classified as cases or controls whereas they should have been in another group. Let us say that a researcher wants to study the accuracy of a new test for an early detection of the prostate cancer in asymptomatic men. Due to absence of a reliable test for the early prostate cancer detection, there is a chance that some early prostate cancer cases would go misclassified as disease-free causing the under- or over-estimation of the accuracy of this new marker.

As a general rule, a research question needs to be considered with much attention and all efforts should be made to ensure that a sample is as closely matched to the population, as possible.

Bias in data analysis

A researcher can introduce bias in data analysis by analyzing data in a way which gives preference to the conclusions in favor of research hypothesis. There are various opportunities by which bias can be introduced during data analysis, such as by fabricating, abusing or manipulating the data. Some examples are:

reporting non-existing data from experiments which were never done (data fabrication);
eliminating data which do not support your hypothesis (outliers, or even whole subgroups);
using inappropriate statistical tests to test your data;
performing multiple testing (“fishing for P”) by pair-wise comparisons ( 4 ), testing multiple endpoints and performing secondary or subgroup analyses, which were not part of the original plan in order “to find” statistically significant difference regardless to hypothesis.

For example, if the study aim is to show that one biomarker is associated with another in a group of patients, and this association does not prove significant in a total cohort, researchers may start “torturing the data” by trying to divide their data into various subgroups until this association becomes statistically significant. If this sub-classification of a study population was not part of the original research hypothesis, such behavior is considered data manipulation and is neither acceptable nor ethical. Such studies quite often provide meaningless conclusions such as:

CRP was statistically significant in a subgroup of women under 37 years with cholesterol concentration > 6.2 mmol/L;
lactate concentration was negatively associated with albumin concentration in a subgroup of male patients with a body mass index in the lowest quartile and total leukocyte count below 4.00 × 10 9 /L.

Besides being biased, invalid and illogical, those conclusions are also useless, since they cannot be generalized to the entire population.

There is a very often quoted saying (attributed to Ronald Coase, but unpublished to the best of my knowledge), which says: “If you torture the data long enough, it will confess to anything”. This actually means that there is a good chance that statistical significance will be reached only by increasing the number of hypotheses tested in the work. The question is then: is this significant difference real or did it occur by pure chance?

Actually, it is well known that if 20 tests are performed on the same data set, at least one Type 1 error (α) is to be expected. Therefore, the number of hypotheses to be tested in a certain study needs to determined in advance. If multiple hypotheses are tested, correction for multiple testing should be applied or study should be declared as exploratory.

Bias in data interpretation

By interpreting the results, one needs to make sure that proper statistical tests were used, that results were presented correctly and that data are interpreted only if there was a statistical significance of the observed relationship ( 5 ). Otherwise, there may be some bias in a research.

However, wishful thinking is not rare in scientific research. Some researchers tend to believe so much in their original hypotheses that they tend to neglect the original findings and interpret them in favor of their beliefs. Examples are:

discussing observed differences and associations even if they are not statistically significant (the often used expression is “borderline significance”);
discussing differences which are statistically significant but are not clinically meaningful;
drawing conclusions about the causality, even if the study was not designed as an experiment;
drawing conclusions about the values outside the range of observed data (extrapolation);
overgeneralization of the study conclusions to the entire general population, even if a study was confined to the population subset;
Type I (the expected effect is found significant, when actually there is none) and type II (the expected effect is not found significant, when it is actually present) errors ( 6 ).

Even if this is done as an honest error or due to the negligence, it is still considered a serious misconduct.

Publication bias

Unfortunately, scientific journals are much more likely to accept for publication a study which reports some positive than a study with negative findings. Such behavior creates false impression in the literature and may cause long-term consequences to the entire scientific community. Also, if negative results would not have so many difficulties to get published, other scientists would not unnecessarily waste their time and financial resources by re-running the same experiments.

Journal editors are the most responsible for this phenomenon. Ideally, a study should have equal opportunity to be published regardless of the nature of its findings, if designed in a proper way, with valid scientific assumptions, well conducted experiments and adequate data analysis, presentation and conclusions. However, in reality, this is not the case. To enable publication of studies reporting negative findings, several journals have already been launched, such as Journal of Pharmaceutical Negative Results, Journal of Negative Results in Biomedicine, Journal of Interesting Negative Results and some other. The aim of such journals is to counterbalance the ever-increasing pressure in the scientific literature to publish only positive results.

It is our policy at Biochemia Medica to give equal consideration to submitted articles, regardless to the nature of its findings.

One sort of publication bias is the so called funding bias which occurs due to the prevailing number of studies funded by the same company, related to the same scientific question and supporting the interests of the sponsoring company. It is absolutely acceptable to receive funding from a company to perform a research, as long as the study is run independently and not being influenced in any way by the sponsoring company and as long as the funding source is declared as a potential conflict of interest to the journal editors, reviewers and readers.

It is the policy of our Journal to demand such declaration from the authors during submission and to publish this declaration in the published article ( 7 ). By this we believe that scientific community is given an opportunity to judge on the presence of any potential bias in the published work.

There are many potential sources of bias in research. Bias in research can cause distorted results and wrong conclusions. Such studies can lead to unnecessary costs, wrong clinical practice and they can eventually cause some kind of harm to the patient. It is therefore the responsibility of all involved stakeholders in the scientific publishing to ensure that only valid and unbiased research conducted in a highly professional and competent manner is published ( 8 ).

Potential conflict of interest

None declared.

IMAGES

Research bias: What it is, Types & Examples
6 Types Of Bias
Types of Bias in Research: Definition, Examples, and Prevention
Types of Bias in Epidemiology
PPT
Table 1 from The problem of bias in medical research and its

VIDEO

Sampling Bias in Research
TYPES OF INFORMATION BIAS
T14-2 Bias Types
Types of Research Bias (with MCQ’s)
Replicability vs Reproducibility in Research || 2023 Exam New Topics Research Aptitude ||
Explanation of Bias

COMMENTS

Types of Bias in Research
Information bias occurs during the data collection step and is common in research studies that involve self-reporting and retrospective data collection. It can also result from poor interviewing techniques or differing levels of recall from participants. The main types of information bias are: Recall bias. Observer bias.
Study Bias
Channeling and procedure bias are other forms of selection bias that can be encountered and addressed during the planning stage of a study. Channeling bias is a type of selection bias noted in observational studies. It occurs most frequently when patient characteristics, such as age or severity of illness, affect cohort assignment.
Bias in Medicine
Type of Bias Description; Anchoring bias: Implicit reference point of first data: Attribution bias: Attempts to discover reason for observations: ... we are unlikely to consider its influence in medical decision making or research. Bias has been extensively studied in the social sciences but has often been ignored in medicine . There has been ...
Identifying and Avoiding Bias in Research
Abstract. This narrative review provides an overview on the topic of bias as part of Plastic and Reconstructive Surgery 's series of articles on evidence-based medicine. Bias can occur in the planning, data collection, analysis, and publication phases of research. Understanding research bias allows readers to critically and independently review ...
8.4 Introduction to sources of bias in clinical trials
This sort of 'within-study publication bias' is usually known as outcome reporting bias or selective reporting bias, and may be one of the most substantial biases affecting results from individual studies (Chan 2005). 8.4.6 Other biases. In addition there are other sources of bias that are relevant only in certain circumstances.
Reducing bias and improving transparency in medical research: a
'Reporting bias' encompasses several sub-biases caused by selective disclosure or withholding of information, either intentionally or unintentionally, related to study design, methods and/or findings. 25 While several types of reporting biases have been described, we will focus on two of the most widely studied: publication bias and spin.
Risk of bias: why measure it, and how?
What types of bias exist, and how can we assess them There are five main forms of bias that are important to consider for clinical trials: Selection Bias , Performance Bias, Detection Bias ...
Best Available Evidence or Truth for the Moment: Bias in Research
The major types of bias in quantitative research occur with the study design, participant selection, data collection, or analysis or during publication. ... data are used to confirm the exposure or outcome with confirmatory findings such as physical exams or medical reports. Selection bias can occur during the process of recruiting individuals ...
Bias in Medicine: Lessons Learned and Mitigation Strategies
In this paper article, we review the types of bias that can lead to flawed clinical decisions, as well as the delays in publishing of scientific articles that do not fit into current accepted scientific norms. ... we are unlikely to consider its influence in medical decision making or research. Bias has been extensively studied in the social ...
Reporting bias in medical research
The reporting of research findings may depend on the nature and direction of results, which is referred to as "reporting bias" [1, 2].For example, studies in which interventions are shown to be ineffective are sometimes not published, meaning that only a subset of the relevant evidence on a topic may be available [1, 2].Various types of reporting bias exist (Table 1), including publication ...
Tackling Implicit Bias in Health Care
DOI: 10.1056/NEJMp2201180. Implicit and explicit biases are among many factors that contribute to disparities in health and health care. 1 Explicit biases, the attitudes and assumptions that we ...
Information bias in health research: definition, pitfalls, and
Specifying the types of bias can be essential to limit its effects and, the use of adjustment methods might serve to improve clinical evaluation and health care practice. ... Self-reporting is a common approach for gathering data in epidemiologic and medical research. This method requires participants to respond to the researcher's questions ...
Publication and related biases in health services research: a
Publication and related biases (including publication bias, time-lag bias, outcome reporting bias and p-hacking) have been well documented in clinical research, but relatively little is known about their presence and extent in health services research (HSR). This paper aims to systematically review evidence concerning publication and related bias in quantitative HSR.
PDF Bias in research
Bias in research Joanna Smith,1 Helen Noble2 The aim of this article is to outline types of 'bias' across research designs, and consider strategies to minimise bias. Evidence-based nursing, deﬁned as the "process by which evidence, nursing theory, and clinical expertise are critically evaluated and considered, in conjunction
Bias in clinical research
Bias in clinical research Kidney Int. 2008 Jan;73(2):148-53. doi: 10.1038/sj.ki.5002648. Epub 2007 Oct 31. ... selection bias and information bias. We describe three types of selection biases (incidence-prevalence bias, loss-to-follow-up bias, and publication bias) and a series of information biases (i.e. misclassification bias--recall bias ...
Information bias in health research: definition, pitfalls, and
Specifying the types of bias can be essential to limit its effects and, the use of adjustment methods might serve to improve clinical evaluation and health care practice. Keywords: ... Self-reporting is a common approach for gathering data in epidemiologic and medical research. This method requires participants to respond to the researcher's ...
Bias
Attrition bias refers to systematic differences between groups in withdrawals from a study. Withdrawals from the study lead to incomplete outcome data. There are two reasons for withdrawals or incomplete outcome data in clinical trials. Exclusions refer to situations in which some participants are omitted from reports of analyses, despite ...
Reporting bias in medical research
Reporting bias in antidepressant research has been shown before [16,70]; other well-known cases include Class I anti-arrhythmic drugs [71,72] and selective COX-2 inhibitors [73,74]. The aim of this narrative review was to gain an overview of reporting bias in the medical literature, focussing on publication bias and selective outcome reporting.
Mitigating bias in machine learning for medicine
Here, we discuss solutions to mitigate bias across the different development steps of machine learning-based systems for medical applications. Vokinger et al. discuss potential sources of bias in ...
Types of bias in medical research
It would be hard to say that the college love this, but it has certainly showed up in the exams of late: Question 26 from the first paper of 2014 and Question 5 from the second paper of 2013 asked the candidates to define bias and discuss strategies to minimise it. Bias in medical research. There is a good article on bias in research from the journal Radiology.
Bias in medical research
Most types of research bias are actually quite easy to understand. Unfortunately, critical appraisal novices are frequently scared off by the sheer number of biases and the technical jargon that is often used to describe them. ... - 112 Types of Bias in Medical Research - MRA; 3 […] are also many different types of selection bias, which Dr ...
Bias in research
Definition of bias. Bias is any trend or deviation from the truth in data collection, data analysis, interpretation and publication which can cause false conclusions. Bias can occur either intentionally or unintentionally ( 1 ). Intention to introduce bias into someone's research is immoral.

Have a language expert improve your writing

Types of Bias in Research | Definition & Examples

Table of contents

Researcher bias

How to avoid bias in research

Performance bias

Acquiescence bias

Courtesy bias

Extreme responding

Question order bias

Sampling or ascertainment bias

Self-selection or volunteer bias

Anchoring bias

Actor–observer bias

The Baader-Meinhof phenomenon

Is this article helpful?

More interesting articles

8.4 Introduction to sources of bias in clinical trials

8.4.1 Selection bias

8.4.2 Performance bias

8.4.3 Detection bias

8.4.4 Attrition bias

8.4.5 Reporting bias

8.4.6 Other biases

Reporting bias in medical research - a narrative review

Reporting bias on a study level

Reporting bias on an outcome level

Factors associated with reporting bias

Submission and rejection of studies

The role of the pharmaceutical industry

Rationale, aim and procedure

Mental and behavioural disorders

Selective serotonin reuptake inhibitors (SSRIs)

Newer antidepressants

Bipolar disorder

Schizophrenia

Panic disorder

Disorders of the nervous system

Disorders of the circulatory system

Prevention of arrhythmia

Disorders of the digestive system

Disorders of the genitourinary system/Perinatal medicine

Disorders of the musculoskeletal system

Disorders of the skin

Endocrine and metabolic disorders

Hypercholesterolaemia

Cerivastatin

Thyroid disorders

Menopausal symptoms

Ovarian cancer

Multiple myeloma

Disorders of the blood

Bacterial, fungal, and viral infections

Anti-HIV agents

Acute trauma

Human albumin infusion

Vaccinations

Cancer vaccines

Other indications

Further research areas

Limitations of the review

Effects of reporting bias

Trial registration and public access to study data

Non-industry initiatives

Industry initiatives

Conclusions

Acknowledgements

Author information

Corresponding author

Additional information

Authors' contributions

Electronic supplementary material

Rights and permissions

About this article

Share this article

Publication and related biases in health services research: a systematic review of empirical evidence

Conclusions

Systematic review registration

Inclusion criteria

Exclusion criteria