How to rate indirectness

← GRADE and methodology

See this topic in the GRADE handbook: Indirectness of evidence

The text below is taken from the GRADE workinggroup official JCE series, article number 8:
GRADE guidelines: 8. Rating the quality of evidence—indirectness

Four types of indirectness
We are more confident in the results when we have direct evidence. By direct evidence, we mean research that directly compares the interventions in which we are interested delivered to the populations in which we are interested and measures the outcomes important to patients. Thus, we can have concerns about indirectness when the population, intervention, or outcomes differ from those in which we are interested. A fourth, different type of indirectness, occurs when there are no head-to-head comparisons between the alternative management strategies under comparison. Indirectness of outcomes and indirect comparisons are equally relevant to systematic reviews and practice guidelines; indirectness related to populations and interventions (sometimes referred to as applicability) is more relevant to guidelines.

Evidence is lower quality if comparisons are indirect

Question of interest: Oseltamivir for prophylaxis of avian flu caused by influenza A virus

Source of indirectness: Differences in population: Randomized trials of oseltamivir are available for seasonal influenza, but not for avian flu

Question of interest: Colonoscopic screening for prevention of colon cancer mortality

Source of indirectness: Differences in intervention: Randomized trials of fecal occult blood screening provide indirect evidence bearing on the potential effectiveness of colonoscopy

Question of interest: Sevelamer- vs. calcium-based phosphate binders in chronic renal failure

Source of indirectness: Differences in outcome: Reducing the calcium-phosphate load is hypothesized to reduce vascular calcification, which is hypothesized to reduce vascular events

Question of interest: Choice of antidepressant
Source of indirectness: Indirect comparison: Some antidepressants have been compared directly with others, but many have not

Indirectness: differences in outcome measures (surrogate outcomes)

GRADE specifies that both those conducting systematic reviews and those developing practice guidelines should begin by specifying every important outcome of interest. The available studies may have measured the impact of the intervention of interest on outcomes related to, but different from, those of primary importance to patients.

The difference between desired and measured outcomes may relate to time frame. For example, a systematic review of behavioral and cognitive-behavioral interventions for outwardly directed aggressive behavior in people with learning disabilities showed that a program of 3-week relaxation training significantly reduced disruptive behaviors at 3 months [3]. Unfortunately, no eligible trial assessed the review authors' predefined outcome of interest, the long-term impact defined as effect at 9 months or greater. The argument for rating down becomes even stronger when one considers that other types of behavioral interventions have shown an early beneficial effect that was not sustained at 6 months follow-up [3]. When there is a discrepancy between the time frame of measurement and that of interest, whether to rate down by one or two levels will depend on the magnitude of the discrepancy. In this case, one could argue for either option.

Another source of indirectness related to measurement of outcomes is the use of substitute or surrogate endpoints in place of the patient-important outcome of interest. Table 2 lists a number of such surrogate measures that are common in current clinical investigation"

Table 2. Examples of surrogate outcomes

Condition	Patient-important outcome(s)	Surrogate outcome(s)
Diabetes mellitus	Diabetic symptoms, hospital admission, complications (cardiovascular, eye, renal, neuropathic)	Blood glucose, A1C
Hypertension	Cardiovascular death, myocardial infarction, stroke	Blood pressure
Dementia	Patient function, behavior, caregiver burden	Cognitive function
Osteoporosis	Fractures	Bone density
Adult Respiratory Distress Syndrome	Mortality	Oxygenation
End-stage renal disease	Quality of life, morbidity (such as shunt thrombosis or heart failure), mortality	Hemoglobin
Venous thrombosis	Symptomatic venous thrombosis	Asymptomatic venous thrombosis
Chronic respiratory disease	Quality of life, exacerbations, mortality	Pulmonary function, exercise capacity
Cardiovascular disease/risk	Vascular events, mortality