5 Dangerous Assumptions in Using KPIs

The most common methods of analysing data in business these days, and for quite some time into the past, has included moving averages (or rolling averages), trend lines, annual trends (using as few as 3 to 5 years), stacked line charts (with each line representing data for one particular financial year), tables of “% difference” comparisons between this month and this month last year, budget and year to date.

These analyses dangerously influence the interpretation made by their users – and in fact, if each of these analyses is applied to the same set of data, the conclusions drawn are almost always different for each of the analyses!

This seeming-anomaly is due to the fact that each of these analyses are based on assumptions that aren’t all that sensible when you consciously examine them.

Dangerous Assumption 1: In “same month” comparisons, last year was normal.

This is when you take a look at performance this month, and compare it to performance in the same month last year, and say, “well things seem to have gotten worse (or better)!”

But honestly, it makes little sense to expect that every November should perform the same! To expect November’s variation should be within ±5% (or 3% or 10%) isn’t based on any sound reasoning. Just convenience. And how do you know that last November was normal, and a sensible benchmark for this November?

Dangerous Assumption 2: If Excel can calculate a trend line for a set of data, then there must be a trend.

Trend lines are calculated mathematically in a way that places a straight line as close to the middle as possible of a smattering of points in a time series. How far away the points are from the trend line is an indication of how reliable the trend line is in explaining the variation of the points over time.

Large variation means little reliability. And little reliability means no real trend.

Dangerous Assumption 3: The world starts anew on the 1st July every year.

Many analyses of performance focus only on the data from the current financial year, often because it is the overall end result of the financial year that people set targets for and are trying to therefore manage.

But usually this year’s performance is a product of last year’s performance, and the year before that, and so on. And it’s this big picture that we need to understand if we are going to validly interpret what’s really going on with performance this year.

Dangerous Assumption 4: The only probable cause for a difference is that something changed.

If you focus on point to point comparisons and your eye picks up a difference, that difference is not necessarily real. It doesn’t mean that something has changed.

It might mean that the level of data integrity is less than 100% (it’s rare to find any data that has 100% integrity). It might mean that the process or system producing the performance result has less than 100% control over that result (variation is a fact of life – no two things are ever exactly the same, mostly because we can’t control every single factor that affects it). Most differences are not the result of a change – most differences are the result of natural variation in stable systems and processes.

If you use time series of 4 or 5 points of data, then you are really only making point to point comparisons, as opposed to truly understanding trends over time (for which you need at least 20 points in the time series).

Dangerous Assumption 5: All changes happen gradually – at least for the data we choose to put moving averages through.

The purpose to moving averages is smoothing out variation (particularly seasonal or cyclical variation) with a view to picking up long term trends.

These analyses are only capable of picking up gradual trends – not sudden shifts. They are not capable of identifying when a shift or trend began or finished. In fact, any analysis that smoothes out or removes variation from your data is a danger to valid interpretation. You have to see all the data in order to assess if there is a real change, and what kind of change it is.

What should you do instead?

The common thread with all of these 5 dangerous assumptions is that none of them help you sort out the two types of variation in your performance values:

1. normal variation, which occurs even when there is no real change in performance (it’s just random noise)
2. abnormal variation, which occurs when a real change has happened to make performance better or worse (such as a process improvement you’ve made).

So there are two things you can start doing now to make sure you can see and distinguish both forms of variation in your performance measures:

• look at a larger window of values – say a time series of at least 20 values in a row (e.g. about 2 years of monthly values)
• graph the performance values as they are, don’t use trend lines or rolling averages or cumulative calculations

If you want to learn more, Donald Wheeler’s book “Understanding Variation: The Key to Managing Chaos” is a tremendous resource.

ABOUT THE AUTHOR

Stacey Barr is a specialist in organisational performance measurement, helping corporate planners, business analysts and performance measurement officers confidently facilitate their organisation to create and use meaningful performance measures with lots of buy-in. Sign up for Stacey’s free email tips at www.staceybarr.com/202tipsKPI.html and receive a complimentary copy of her renowned e-book “202 Tips for Performance Measurement”.