Organizations can easily do the wrong thing by mistaking the means for the ends. Following are a couple examples I ran across in the past few days.
Misleading Metrics
Airlines are subject to on-time rankings, a means to demonstrate reliability and thus customer satisfaction. However, these rankings are only based on non-stop flights, so they don’t consider flights that act as connections. And thus we have problems like what Ross Mayfield reports here:
It was abundantly clear last night when my connection was delayed that the airline industry is running on the wrong metrics. Half of the plane missed their connecting flight, most by minutes, when doors were still open, but gates closed — for sake of on-time-departure. The last planes left within a half an hour and we were left stranded in Virginia without hotel rooms in the vicinity.
Much of the airline industry operates via the “hub and spoke” method of multi-hop trips, so this kind of scenario is not a fluke. While it’s true that airlines will sometimes hold the connecting flight in close situations, the on-time metric creates a perverse incentive not to do so—despite the fact the metric exists as a proxy for customer satisfaction.
Method versus Mission
Former CIA Director George Tenet used to describe the CIA’s business as “stealing secrets.” In the CIA’s Studies in Intelligence journal, Stephen Mercado critiques this mindset for conflating a method (stealing secrets) with the organization’s mission, which is to provide actionable intelligence about national security. Mercado argues that another method—improving the CIA’s analysis of freely available information (“open sources”) such as from foreign newspapers—is more effective yet underutilized:
Despite numerous surveys putting the contribution of open sources anywhere from 35 to 95 percent of the intelligence used in the government, [open-sources intelligence’s] share of the overall intelligence budget has been estimated at roughly 1 percent.
Mercado argues that the “stealing secrets” mindset is so deeply ingrained in the CIA’s culture that the method has become the mission. After making an argument for open-sources intelligence, he advocates doubling its budget—to 2% of the overall intelligence budget—which is apparently a radical proposal.
On the airline topic, doesn't the rating includde both take off and landing? If so, the late arrival would offset the on time departure, no?
ReplyDeleteOn the CIA topic, I would equate information with ROI. While open source information may make up a large percentage of information, this does not make it more valuable. Perhaps the 5% of secrets equated to 90% of actionable information, in which case the investment in secrets produces a much larger ROI.
Thanks for the comment, Chad.
ReplyDeleteOn the take-off/landing issue: Although the stats track take-off and landing times, I believe the main on-time metric (the one you see quoted as a daily flight's on-time percentage) is whether a plane arrives within 15 minutes of its originally scheduled landing time, regardless of when it took off.
Regarding the ROI perspective on the CIA's information, you make a good point. However, I assume Mercado would say that the 99:1 ratio is still an artifact of the agency's culture, not an ROI analysis of *any* kind (quantity, quality, or otherwise).