Guest post by John Hunter, founder of CuriousCat.com.
In Fourth Generation Management (I highly recommend this book, by the way), Brian Joiner provided an excellent summary of the options to get better “results” (as measured by the data used).
The options are to:
- distort the system
- distort the data
- improve the system
Obviously we hope to improve the system; and we would like to have confirmation we have done so shown with data. Using data isn’t as simple as just including data in reports and in meetings. That data must be understood and we must have an appreciation for the dangers of misinterpreting data.
Without an understanding of variation we can often mistake variation with evidence (of failure or success). Also without understanding the proxy nature of data we can be led astray when that proxy association is changed over time.
If a measured value improves over time it needs to provide an accurate measure of what we meant to measure, for that to be an accurate indication of success.
For example, if the time to fix identified bugs declined that seems to be a good thing. But if we have achieved that improvement by distorting the data or distorting the system it is not good evidence for an improved system.
We could, for example, distort the system by stopping new development. We could reassign all software developers to fixing bugs and to thoroughly testing existing software. We could put in quick fixes that fixed the issue raised, but only do that. So instead of doing sensible things like asking why to find solutions that may not only fix this bug but fix other potential bugs that haven’t been reported yet (or even fix processes that would reduce bugs from being introduced in the future) we just quickly fix the issue as narrowly as we can in order to keep the measured value (time to fix bugs) as low as possible.
Modifying the system is necessary to gain improvements. Sometimes a change could be seen as either a distortion or an improvement. Arguing over something that is close to distorting the system versus improving the system isn’t very useful. But often the distortions are fairly obviously causing harm to the overall system in order to achieve improvements in one measure.
Another way to improve the data is to change the system to adjust how things are measured. Don’t count a bug as being reported when the developer is told. Require that a form be completed and be found to have meet specific criteria before the clock starts counting. And also count a bug as fixed when the developer says it is. If the user finds it is not fixed force them to put in a new bug report and start counting the time all over again.
Distorting the data can take the form of outright lying. More often it will take the form of influencing the data through the judgements people make and in creating new criteria (maybe even including operational definitions, but often those are not even created).
The third method to getting better “results” is to improve the system. Making improvements that achieve better results (given the measure used) is often much more difficult than the first two options. This makes it very tempting to fall into using one of the first two methods.
As an organization matures in the use of evidence based management practices using the first two options becomes more difficult. Because the system is sophisticated enough to see attempts to claim improvement when actually only the limited data being looked at improved.
The problem is that many organization have not reached this level of an understanding of data and therefore often instances of distorting the data or the system outnumber instances of system improvement. Using data effectively requires understanding how measures capture the actual state of affairs and requires understanding the limitations of the specific measures we are using.
The desire is to improve not a number but the underlying system. When we are lucky the measures will nearly directly correspond to the system and distorting the data or the system is not going to work. But we are often not lucky. It is very easy to fall into the trap of focusing on the number without viewing the organization as a system or understanding variation or the proxy nature of data.
It is good to get in the habit of considering if the measured improvements are truly an indication of an improved system or merely the result of distorting the system or the data.
Related: Managing to Test Result Instead of Customer Value – Be Careful What You Measure – Brian Joiner Podcast on Management, Sustainability and the Health Care System – Problems caused by management by targets or goals – Where There is Fear You Do Not Get Honest Figures – The Defect Black Market (extrinsic motivation damaging the system)