On fairy stories and Key Performance Indicators

Everyone knows what fairy stories are. Snow White. Hansel and Gretel. Dragons. Wizards. Hobbits. And... well, fairies. In general, there are few things humans love more than a good story. Now, the popularity of mythology and what is often called fantasy has varied over the years, but there is a kind of story humans have consistently loved ever since it was first told.

The genre used to be called "Get Rich Quick" and was especially popular among the poor. You'd be amazed how easy it is to sell fake food to a starving man. It struggled to reach the more affluent classes though. Some clever marketing, however, has largely allowed it to transcend this boundary. Specifically, instead of stories belonging to the Get Rich Quick genre, they are being marketed individually as "Key Performance Indicators" (KPIs). Now bear with me, because we will be back to fairy stories in a little while, but we have some ground to cover on these Get Ri... ahem, Key Performance Indicators first.

The idea behind KPIs is that, by following this simple plan you'll double your income... no, wait. That's not it.

With this one secret trick... no sorry. That's not it either.

By measuring and strictly enforcing constraints related to some number or group of numbers, one can know with reasonable certainty how effectively a person or thing is contributing to a goal... yeah, that's the one.

So one example of a KPI which almost anyone will understand might be something like "number of emails sent." The idea could run something like "if you have two employees, you can track how many emails each of them is sending and get an idea of how much work they are getting done. If one sends 10 emails a day and the other sends 2, then the one sending 10 is communicating 5 times as much as the other and so is likely doing more work."

Ultimately it's a way of leaving as little as possible to the frequent ambiguity of reality and preferring instead the certainty of numbers.

Most people who like the KPI type of story tend to like to mash a bunch of them together in what is sometimes called a "holistic approach." The number of these stories which are coerced into playing with each other does not make a great deal of difference though (lest anyone be tempted to create a KPI of "number of 'interconnected' KPIs"), and one is easier to keep in our heads than many, so we will stick to the number of emails one. Working from the "number of emails sent" KPI then...

Let's consider our two employees. The diligent email sender and the one we don't hear much from. There are at least two major problems with our metric.

The first is that it invites what is called "gaming the system." If number of emails sent is the measure of how well a person is doing in his job, then all a dishonest person needs to do is send a bunch of meaningless emails. KPIs universally have this issue. Whether the metric is number of emails sent, number of lines of code written, average customer review, or anything else, once the KPI is known by the people or groups which are measured by it, there is no discernible difference between the one who is doing well and the one who is good at making the numbers say he is doing well. Likewise between the one who is doing poorly and the one who is more focused on what he is doing than on making the numbers say he is doing well. Almost anyone who has ever shopped online knows to be skeptical of too many good (though few apply the same skepticism to too many bad) reviews. Why? Because sellers know how important the "average review" KPI is for their bottom line, and it is hardly a secret that good reviews are cheap to buy.

KPIs necessarily function as aggregators, so we are not looking at 10 emails from the higher performing employee and saying "this employee has sent these 10 emails." We are looking at a report that says "this employee has sent some 10 emails." The whole genre collapses if that aggregation is skipped. The purpose of a KPI is to produce a number which is actionable at a glance. It is a shortcut before it is anything else. Context is the very thing a KPI seeks to compress out of existence as much as possible. At the point where we are looking at the emails themselves, we are looking at a lot more than just how many of them there are. We can see that 5 of them were strictly social. That 2 of them were forwarded by mail rules. And that the remaining 3, though real and relevant, did not convey anything especially meaningful. We can also see that the 2 emails sent by the under-performing employee were both immediately helpful in solving real problems. What we cannot do - unless we are very silly indeed - is say, with all that context in front of us "sure, but 10 is bigger than 2."

That actually brings us to our second problem, which is that the things which really do contribute meaningfully toward all but the shallowest goals are not reducible to mere numbers in that way. There is no way - without looking at the emails - to know whether 10 emails from one employee actually contributed more than 2 emails from another. If one programmer adds 100 lines of code to a program and another adds none but removes 2, the numbers by themselves do not tell us whether the addition of 100 lines or the removal of 2 lines was helpful or harmful, let alone to what degree. Just about the closest we can come to getting anything out of a KPI that is both trustworthy and meaningful is to measure things like "number of seconds to perform some specific task." And that does give us a number that actually means something, but unless we are perfectly certain that the person or program or whatever that completes the task in 20 seconds is also completing it as accurately as the one that completes it in 40 seconds, it still only touches the very most trivial part of the surface of what we actually care about.

The only time when the things we actually care about - whether profit or instances of cancer cured or anything else - can ever be reduced to trustworthy and meaningful numbers (and not even always then) is after end goals have been or failed to be achieved. No one cares about number of lines of code written or number of emails sent for their own sake. We care about whether the program runs better, whatever "better" means in our context. We care whether customer satisfaction is improved. We care whether down time is reduced. But these are outcomes, not processes. They are almost always the combined result of many kinds of work from many people, and they are almost never consistently, demonstrably tied to any KPI or group of KPIs.

So our KPI, whose main purpose is to identify and predict how effective a person or thing is at contributing to our goals, typically tells us less than nothing. It is easy for a clever person to fake and omits all the information which would otherwise have helped us to weigh it.

Ok, so I promised we would come back to fairy stories, and back to them we now come. Most of us find fairy stories enjoyable because of what might be called the fantastical elements within them. Very few adults - and a shrinking number of children - believe fairies, dragons, hobbits, dwarves, elves, and all their ilk are real, and that non-reality is exactly what makes the genre so easy to enjoy for most people. However, I put to you that CS Lewis, who is widely acknowledged as both a brilliant man and a great writer of what we call fantasy, was very open to the possibility that all of these things might be real.

Fairy stories necessarily and openly operate at the border between typical human experience and something else. That we do not know something to be real though is not the same thing as knowing that it is not real. Why should there not be unicorns? And if there are, why should you or I expect to have seen one. I have never seen a zebra, but no one would think very highly of me if, for that reason, I rejected their existence. KPI stories, on the other hand, get their entire popularity from the supposition that they operate at the center of knowable human experience. They deal with money and work and the mundane horrors of every day life. So when fairy stories deal with things which we cannot know to be purely fiction and KPIs deal almost exclusively with things which can be reliably distrusted and promise tangible results out of them, I ask you. Which of these two types of story rightly falls into the genre of fantasy?

Before I go, let me just say that I would not have anyone think I am too stupid or arrogant to appreciate how beautiful and useful data can be. Data, in itself, is a fantastic thing which forms one of the foundational parts of all rational thought. What it is not though is an oracle, a prophet, or an effective means of divination. It is those uses of it which I mean to condemn here.