Misleading Metrics That People Think Matter

Despite the title of this post, I like metrics. They have a lot of great uses. Metrics around spending can help people figure out how to get their finances in order. Businesses require good metrics in order to make decisions that help them grow (or even just stay afloat). Society as a whole can use metrics to determine where some of its greatest problems are (e.g. annual traffic deaths, crime rates, etc).

The problem with metrics is that they are only useful when generated in good faith.

Let's take software development. No one in their right mind uses lines of code as a performance metric. The reasoning is simple. Take a hypothetical task of printing numbers from 1-100. This can be done in 2-3 lines of code. It can also be done in 100 lines of code. The former is the proper way to do things and easier to maintain. The latter is what someone would pick if their wages were reliant on lines of code.

If that were the only example, it would be easy for engineering managers to notice when someone was gaming the metrics. The problem is when things get a little more complex. Enterprise Fizz Buzz is a wonderful piece of satire, but it hits a little close to home because a lot of software projects become inflated to this level when the real solution is only a few lines of code. The inflation usually has some seamingly reasonable justification as well such as better maintainability or scalability.

Engineering managers often feed into this as well. I've lost count of the number of meetings I've been in where someone suggests a performance metric, I ask "how is that better than lines of code? It seems like it has all the same problems" and the response is "Well, we need *something*"

"We need something"

I'd say that something is using good judgment. That judgment can be informed by metrics, but not decided by metrics.

That's not what "We need *something*" means most of the time though. Most of the time, people want a metric that they can use to make decisions for them.

It's honestly understandable. Metrics provide a visual indicator of progress. Humans like visual indicators of progress. This isn’t just a problem with your manager. We do this to ourselves in our personal lives as well.

Have you ever wished you had the time to read more books? That's often a metric used by us to determine how much progress we've made in being more productive or being smarter. It is a nice visual indicator. Look at the number of books on your bookself! Look at all that learning.

I too used this metric at one time. I've probably read maybe 300 non-fiction titles on software development, business, psychology, sociology, marketing, history, etc. I took pride in increasing my number of book cases. I was saddened every time I "dropped the ball" and went an extended period of time without finishing a book.

It was only after reading a really excellent 3 page essay that took the form of a 180 page book that I realized something. I spent my engineering career fighting metrics that didn't matter at work and here I was using a faulty metric in my own life.

It's not the number of books you read that matter. It's the quality of those books. It's how you change your behavior after reading a book. Or how you change your outlook on life. Or how you change what you do at work. The real metric is the number of books that have made an impact on your life.

That real metric is a hard number to come by. Number of books read is easy. You read a book from page one to the end? That's a read book. It is a fact. Something happened or it did not. There are no questions or nuances involved.

Number of books that have made an impact on your life? That's a judgment call.

If you read a book that was fun and interesting, but made little noticeable impact on your life otherwise, does that count? Maybe. It could have improved your mental health by being relaxing. Or it could have just been a waste of time. No one can make that determination other than you. Yet, if you're focused on seeing visual progress in the form of a metric, you may change your judgment to improve that metric.

This goes back to metrics only being useful if generated in good faith. I almost wrote how many of those 300 books actually made an impact on my life. I am choosing not to due to the temptation to change the number only to emphasize my point.

Metrics created in good faith are useful because they can provide indicators that you can use to back up your judgment. If the number of impactful books was 290 out of 300, then I should just continue reading books without worrying about the impact. If instead that number was 10 out of 300, then I need to figure out a better way of selecting books to read. If I have a target number in mind before making any judgments, I will likely change my judgments to meet my target.

We all have goals we'd like to achieve. Those goals are often hard to measure (How much smarter did the last book you read make you? 3 intelligence units? 10 intelligence units? What even is an intelligence unit?). Too often we replace our goals with something we can measure. Those metrics soon become our goal and we have neglected our original one.