“Change the rule and you will get a new number.”
Attributed to W. Edwards Deming. Direct quote from The New Economics “If you change the rule for counting people, you come up with a new number.”
Dr. Deming emphasized the importance of understanding what the data actually meant (and how easily we can be mislead when looking at data). Without understanding the operational definition for the data you take a great risk that you make mistaken guesses about what the data means.
“An operational definition is a procedure agreed upon for translation of concept into measurement of some kind.”
W. Edwards Deming, The New Economics, page 105.
Another quote attributed to W. Edwards Deming that is important for those using data to consider:
Nobody should try to use data unless he has collected data.
This echoes the idea of going to the gemba. If you have not collected data you often fail to understand the judgement calls that go into assigning and recording a result from real world conditions. If you don’t have operational definitions (which is a very common condition) there is a significant risk that data doesn’t provide a decent view of reality. Without an appreciation for the gemba, where the data was collected, it is easy to be mislead by the data.
The most common waste of effort in examining data is reacting to the expected variation of a system as if it is something special. We have discussed this in many previous posts (for example, We Need to Understand Variation to Manage Effectively). After that I think there is a good chance failure to appreciate what the data is (and what it is not) telling us (based on mistaken assumptions about what the operational definitions were – this happens implicitly not explicitly) is the next biggest source of wasted effort.
To counter the problem (of data that has introduced variation beyond the system itself due to the method of data collection), make sure you understanding what the operational definition used to collect the data was. If you do not have operational definitions for data provided in your organization, as you often will not, you need to figure out if the data is worth trusting (possibly by visiting the gemba) or whether you need to establish operational definitions and make sure new data is collected using the operational definitions.
Often the variation in the collection of data will hamper your ability to use PDSA. If you practiced PDSA as you should, this wouldn’t be a problem, because you would make sure data had operational definitions. But often data used in PDSA effort isn’t collected with operational definitions (so, it is in understanding this reality that the comments about using such data becomes relevant).
Without reliable data, the conclusions that will be drawn from results ads a great deal of potential for error to your effort. The potential for data discrepancies increases the larger systems get. Differences in performance between locations can often be due, not to actual differences in the processes themselves, but in the decisions made in collecting the data.