How to rate indirectness

See this topic in the GRADE handbook: Indirectness of evidence

The text below is taken from the GRADE workinggroup official JCE series, article number 8:
GRADE guidelines: 8. Rating the quality of evidence—indirectness

Four types of indirectness
We are more confident in the results when we have direct evidence. By direct evidence, we mean research that directly compares the interventions in which we are interested delivered to the populations in which we are interested and measures the outcomes important to patients. Thus, we can have concerns about indirectness when the population, intervention, or outcomes differ from those in which we are interested. A fourth, different type of indirectness, occurs when there are no head-to-head comparisons between the alternative management strategies under comparison. Indirectness of outcomes and indirect comparisons are equally relevant to systematic reviews and practice guidelines; indirectness related to populations and interventions (sometimes referred to as applicability) is more relevant to guidelines.

Evidence is lower quality if comparisons are indirect
Question of interest: Oseltamivir for prophylaxis of avian flu caused by influenza A virus
Source of indirectness: Differences in population: Randomized trials of oseltamivir are available for seasonal influenza, but not for avian flu

Question of interest: Colonoscopic screening for prevention of colon cancer mortality
Source of indirectnessDifferences in intervention: Randomized trials of fecal occult blood screening provide indirect evidence bearing on the potential effectiveness of colonoscopy

Question of interest: Sevelamer- vs. calcium-based phosphate binders in chronic renal failure
Source of indirectnessDifferences in outcome: Reducing the calcium-phosphate load is hypothesized to reduce vascular calcification, which is hypothesized to reduce vascular events

Question of interest: Choice of antidepressant

Source of indirectness: Indirect comparison: Some antidepressants have been compared directly with others, but many have not

Indirectness: differences in outcome measures (surrogate outcomes) 

GRADE specifies that both those conducting systematic reviews and those developing practice guidelines should begin by specifying every important outcome of interest. The available studies may have measured the impact of the intervention of interest on outcomes related to, but different from, those of primary importance to patients.

The difference between desired and measured outcomes may relate to time frame. For example, a systematic review of behavioral and cognitive-behavioral interventions for outwardly directed aggressive behavior in people with learning disabilities showed that a program of 3-week relaxation training significantly reduced disruptive behaviors at 3 months [3]. Unfortunately, no eligible trial assessed the review authors' predefined outcome of interest, the long-term impact defined as effect at 9 months or greater. The argument for rating down becomes even stronger when one considers that other types of behavioral interventions have shown an early beneficial effect that was not sustained at 6 months follow-up [3]. When there is a discrepancy between the time frame of measurement and that of interest, whether to rate down by one or two levels will depend on the magnitude of the discrepancy. In this case, one could argue for either option.

Another source of indirectness related to measurement of outcomes is the use of substitute or surrogate endpoints in place of the patient-important outcome of interest. Table 2 lists a number of such surrogate measures that are common in current clinical investigation" 
Table 2. Examples of surrogate outcomes
ConditionPatient-important outcome(s)Surrogate outcome(s)
Diabetes mellitusDiabetic symptoms, hospital admission, complications (cardiovascular, eye, renal, neuropathic)Blood glucose, A1C
HypertensionCardiovascular death, myocardial infarction, strokeBlood pressure
DementiaPatient function, behavior, caregiver burdenCognitive function
OsteoporosisFracturesBone density
Adult Respiratory Distress SyndromeMortalityOxygenation
End-stage renal diseaseQuality of life, morbidity (such as shunt thrombosis or heart failure), mortalityHemoglobin
Venous thrombosisSymptomatic venous thrombosisAsymptomatic venous thrombosis
Chronic respiratory diseaseQuality of life, exacerbations, mortalityPulmonary function, exercise capacity
Cardiovascular disease/riskVascular events, mortality

Go to the orginal article for full text, or go to a specific chapter in the article:

See the Assessing Indirectness Training video from McMaster CE&B GRADE site:

Feedback and Knowledge Base