How to rate publication bias

See this topic in the GRADE handbook: Publication bias

The text below is taken from the GRADE workinggroup official JCE series, article number 5:
GRADE guidelines: 5. Rating the quality of evidence—publication bias

Publication bias vs. selective reporting bias 

In some classification systems, reporting bias has two subcategories: selective outcome reporting, with which we have dealt in the previous article in the series, and publication bias. However, all the sources of bias that we have considered under study limitations, including selective outcome reporting, can be addressed in single studies. In contrast, when an entire study remains unreported and reporting is related to the size of the effect—publication bias—one can assess the likelihood of publication bias only by looking at a group of studies [2], [3], [4], [5], [6], [7]. Currently, we follow the Cochrane approach and consider selective reporting bias as an issue in risk of bias (study limitations). This issue is currently under review by the Cochrane Collaboration, and both Cochrane and GRADE may revise this in future.

Using study results to estimate the likelihood of publication bias 
Another criterion for publication bias is the pattern of study results. Suspicion may increase if visual inspection demonstrates an asymmetrical (Fig. 1b) rather than a symmetrical (Fig. 1a) funnel plot or if statistical tests of asymmetry are positive [29],[30]. Although funnel plots may be helpful, review authors and guideline developers should bear in mind that visual assessment of funnel plots is distressingly prone to error [31], [32]. Enhancements of funnel plots may (or may not) help to improve reproducibility and validity associated with their use [33].

  • View full-size image.
  • Fig. 1 

    (a) Funnel plot. The circles represent the point estimates of the trials. The pattern of distribution resembles an inverted funnel. Larger studies tend to be closer to the pooled estimate (the dashed line). In this case, the effect sizes of the smaller studies are more or less symmetrically distributed around the pooled estimate. (b) Publication bias. This funnel plot shows that the smaller studies are not symmetrically distributed around either the point estimate (dominated by the larger trials) or the results of the larger trials themselves. The trials expected in the bottom right quadrant are missing. One possible explanation for this set of results is publication bias—an overestimate of the treatment effect relative to the underlying truth.


Go to the orginal article for full text, or go to a specific chapter in the article:

See the Assessing Publication Bias Training video from McMaster CE&B GRADE site:

Feedback and Knowledge Base