How to rate imprecision

Modified on Mon, 13 Jan at 11:29 AM

See this topic in the GRADE handbook: Imprecision

The text below is taken from the GRADE workinggroup official JCE series, article number 6:
GRADE guidelines 6. Rating the quality of evidence—imprecision

Guidelines: are results of a binary outcome sufficiently precise to support a recommendation?

The following example illustrates how guideline developers must consider the context of their particular recommendations in making judgments about precision. A hypothetical systematic review of randomized conrol trials (RCTs) of an intervention to prevent major strokes yields a pooled estimate of the absolute reduction in strokes of 1.3%, with a 95% CI of 0.6% to 2.0% (Fig. 1). Thus, we must treat 77 (100/1.3) patients for a year to prevent a single major stroke. The 95% CI around the number needed to treat (NNT)—50 to 167—tells us that while 77 is our best estimate, we may need to treat as few as 50 or as many as 167 people to prevent a single stroke.

Further, assume that the intervention is a drug with no serious adverse effects, minimal inconvenience, and modest cost. Under these circumstances, even a small effect would warrant a strong recommendation. For instance, we may strongly recommend the intervention were it to reduce strokes by as little as 0.5% (vertical middle line in Fig. 1)—an NNT of 200. The entire CI (0.6% to 2.0%) around the effect on stroke reduction lies to the left of the clinical decision threshold of 0.5% and therefore excludes a benefit smaller than the threshold. We can therefore conclude that the precision of the evidence is sufficient to support a strong recommendation.

What if, however, treatment is associated with serious toxicity? Were this true, we may be reluctant to recommend treatment unless the absolute stroke reduction is at least 1% (NNT of 100—left verticle line in Fig. 1). Under these circumstances, the precision is insufficient to support a strong recommendation as the CI encompasses treatment effects smaller than this threshold and therefore does not exclude an absolute benefit appreciably less than 1%. Because the point estimate of 1.3% meets the threshold criterion, a recommendation in favor of treatment would still be appropriate, although the imprecision-generated uncertainty regarding the true effect would mandate a weak recommendation (Box 1).

The impact of undesirable consequences on precision

The hypothetical example presented in Fig. 1 and the accompanying text shows that greater levels of precision are required to support a recommendation in favor of a treatment when a large effect is required to make treatment worth the appreciable undesirable consequences. When appreciable undesirable consequences exist, CIs are more likely to span not only regions of effect that would mandate treating but also regions that would mandate not treating. Thus, the existence of appreciable undesirable consequences makes it more likely that guideline developers will rate down the evidence regarding an apparently beneficial intervention for imprecision.

Go to the orginal article for full text, or go to a specific chapter in the article:

Criteria for imprecision differ for guidelines and systematic reviews
Confidence intervals capture the extent of imprecision—mostly

Guidelines: are results of a binary outcome sufficiently precise to support a recommendation?

Real world examples of the clinical decision threshold approach to precision

Confidence intervals can be misleading because of fragility

The danger of initial trials with impressive positive results

Addressing the vulnerability of CIs: the optimal information size

Low event rates with large sample size: an exception to the need for OIS

Rating precision for binary variables in guidelines: summary and conclusions

Standards for adequate precision of binary variables in systematic reviews: application of the OIS

Systematic reviews of binary variables: meeting threshold OIS may not ensure precision

Rating down two levels for imprecision