Analytics That Explain Themselves

As computers have gotten more powerful, so too has the complexity of analytics they can do. But this power often brings a paradox: Complex and interesting analytics can go unused because few people know how to interpret the results.

When it happens, this failure is rarely due to the people. It’s usually due to a shortsighted view of analytics, a view that focuses on the underlying data processing at the expense of making the results understandable.

That’s the bad news. The good news is, computers can be used not just to “do” analytics but to explain them. For example, long ago at Personify, reports had a footer called “How to Read This Report.” It was a plain-English sentence that described the data using the top-left cell as an example. It was simple but effective.

The latest thing to remind me of this topic was one of the best display ads I’ve ever seen on the Internet. I was looking up Apple on a stock-quote site, and one of the ads was this:

The ad explains Apple’s current performance on a popular technical-analysis indicator for stocks. Traditionally, technical analyses are rendered as charts that require expertise to interpret. With this ad, Scottrade is demonstrating its SmartText technology that interprets the charts for you, using current data. (The reason I think it’s a great ad: Most ads promise something; this ad actually does it, in context.)

Would technical-analysis pros finds the explanations simplistic? Probably. But for the casual user, do the explanations begin to make sense of something that might be useful? I’d say yes.

I don’t follow the field of technical analysis for stocks, so I don’t know how unique or effective SmartText is—although it’s apparently unique enough to warrant its own ad campaign. What I do know is this: In business analytics the equivalent of SmartText’s functionality is a rarity. Analytics results that people don’t understand are not so rare.

In other words: Analytics, you’ve got some explaining to do.

How to Hold a HiPPO

At a recent conference, I saw analytics blogger Avinash Kaushik talk about the dangers of the HiPPO: the Highest Paid Person’s Opinion. From his speaker’s notes for a similar talk:

I can’t say it any better, HiPPO’s rule the world, they over rule your data, they impose their opinions on you and your company customers, they think they know best (sometimes they do), their mere presence in a meeting prevents ideas from coming up. The solution to this problem is to depersonalize decision making, simply don’t make it about you or what you think. Go outside, get context from other places. Include external or internal benchmarks in your analysis. Get competitive data (we are at x% of zz metric and our competition is at x+9% of zz metric).

Be incessantly focussed on your company customers and dragging their voice to the table (for example via experimentation and testing or via open ended survey questions). Very few people, HiPPO’s included, can argue with a customer’s voice, the customer afterall is the queen / king! : )

Although Avinash’s advice was about overcoming a HiPPO gone wild, I started wondering about the other side of the story: What if your opinion is the HiPPO in the room? How do you be a good HiPPO holder?

Before you say, “This doesn’t apply to me,” remember that the HiPPO holder is a relative position. In a meeting with your boss, you may be just one of the team; but if you in turn lead a team, or if you are influential with peers, you hold a HiPPO in some situations.

Realizing when you are the HiPPO holder is important because it will keep you on guard against the traps of being a bad HiPPO holder, the kind that stifles ideas just by being present.

What are the traps?

It’s not just about politics. You pride yourself on being non-hierarchical, open, and politics-free—and thus assume when people agree with you it’s because you have the best ideas. But even if you have minimized the politics, it’s still less work for your people to agree with you than to do the spadework of collecting and analyzing data. If they don’t have the time or incentive to dig for their own answers, they may not. (Promote the ones who do anyway.)

Metrics can have conflicts of interest. For example, company X has a call center where the key metric is call duration (lower being better for the company’s costs). Because it’s harder to measure, the company does not systematically track customer satisfaction with calls. As a result, “metrics-driven” decisions about the call center inadvertently favor churn-and-burn customer service practices. The point: Focusing on metrics doesn’t relieve you of understanding whether bias is still at play. Your company’s key metrics probably reflect those in charge’s worldview, so if you are looking for out-of-the-box thinking, ask yourself if your metrics already have you in a box.

Don’t mistake a lively debate for a good decision process. Open exchange of ideas is good. But in your team, do you really know whether the best ideas win, or whether the best debaters win? It’s an especially important question if you pride yourself as one of the best debaters.

Those are a few traps I’ve observed. I’m sure there are many more to avoid, but I’ll conclude by suggesting a principle for HiPPO holders to embrace: For big decisions, the HiPPO holder should focus on process, not outcomes. So if you own the final decision, be like a judge: Limit yourself to establishing and enforcing a fair process, then decide only at the end, based on the evidence.

This is consistent with Avinash’s advice to depersonalize the process, but with a twist. He was assuming that a team should depersonalize the process to overcome the HiPPO’s biases. If instead you use your HiPPO influence to incent and enforce an objective process, everybody is further ahead.