The Flaw of Averages

There’s nothing like a life-or-death issue to illustrate an analytical problem:

[A new paper concludes] there are fundamental flaws in the way researchers usually analyze and report the results of medical studies, especially randomized clinical trials that are seen as the “gold standard” method for studying the effectiveness and safety of new treatments....

“Most studies currently emphasize the average risk and average benefit found in the study, but the average trial participant might get much less benefit than average, or even be harmed,” says lead author Rodney Hayward, M.D. “If nine people are in a room with Bill Gates, the average net worth of people in the room will be several billion dollars even if everyone else in the room is in serious debt.”

The authors argue for a more sophisticated form of analysis, risk stratification, which they found in only 4% of papers reviewed from prominent medical journals. To make their point, they cite a major 1993 study that showed the clot-busting drug tPA to be, on average, significantly effective for heart-attack patients.

But when Hayward’s colleague David M. Kent, M.D., M.Sc., now at Tufts University, analyzed the data from this study in a risk-stratified way, he found major differences in effectiveness of tPA. In fact, his analysis shows that 25 percent of the patients in the original study accounted for more than 60 percent of all the benefit in the entire study. Meanwhile, half the patients received little or no benefit — and some had such a high risk of brain bleeding from tPA that there was net harm.

The full write-up about the paper is here. Those who know marketing analytics will recognize that risk stratification is similar to segmentation. Just as smart marketers no longer pursue a singular, average customer, the paper’s authors are urging the medical establishment to be wary of studies about the average patient.