By John Hunter, author of the Curious Cat Management Improvement Blog.
Using data to understand the system and validate our theories and successful improvements is an important part managing well. In some cases it is fairly easy to understand and collect data that provides a clear and accurate measure of what we care about. But getting data that helps can also be very challenging.
Creating a management system that aims to use data while focusing on continually improving is a great start. But that requires not just that numbers are used: those numbers must be used properly. There must be an understanding of variation. There must be an understanding of the proxy nature of data. There must be an understanding of the organization as a system.
One of the challenges raised in the webcast that those with a background applying W. Edwards Deming will appreciate is the challenge of collecting accurate data based on the same operational definition. In business, often data is collected without even having operational definitions (“An operational definition is a procedure agreed upon for translation of concept into measurement of some kind.” W. Edward Deming). Without an operational definition there is a greater risk that the data is not accurate (since those collecting it might be using different criteria) and also that the data is not understood properly by those making decision.
Often formal health care studies give more care to collecting data, but even when they do well defining how data should be collected from the real world they run into another challenge – which is that even experts can’t agree which category the situation falls into (for example, how sick a patient is). If patients that are not in similar health conditions are compared to each other the data will (in addition to the various sorts of variation we know impact all processes) also include variation in the results that are due not, for example to the changes made in the process due to the PDSA, but just the variation of the baseline health of the patients in the experiment.
There certainly are challenges in using data in health care that are more complex than many other attempts to use data. But the types of issues with collecting accurate data, collecting accurate data on differing situations (for example comparing data from a PDSA to the current process), and making accurate conclusions about what we can learn from the data (just actually correctly understanding what it says, before we even then try to draw conclusions about what that means or what actions that should lead us to take).
This is an interesting webcast exploring the issues surrounding understanding data on medical errors and what conclusions we can draw from data about the impact of those errors. This is the latest webcast from Healthcare Triage, which is an excellent series for anyone interested in healthcare (not just for those in the field but for anyone interested in understanding human health).
In order to determine what actions should be taken we would like to know the impact of the current situation (how damaging are the large numbers of medical errors?). Yes, it is best to eliminate all errors. But to determine priorities to focus on it is important to know what impact the current situation has. If we have thousands of trivial errors it may well be that other items are more critical to focus on for patient health. If the impact of errors in the system are the largest cause of harm in the system then the issue of medical errors must be very prominent in the efforts to improve in hospitals and the healthcare system.
As the webcast makes clear it is hard to gage the true impact of errors within the healthcare system.
Obviously at the process level we want those working to make improvements continually including using concepts such as mistake proofing to reduced the possibility and occurrence of errors. And certainly some errors are catastrophic and critical to address. But using data isn’t as easy when you get to the implementation stage as it sounds. There are many complications that challenge us and require us to think.
Often people believe using data will make everything clear and obvious. It won’t. Using data well (with an understanding of variation) will greatly enhance improvement efforts but it is a challenge and requires thoughtful consideration. It isn’t as simple as plugging numbers into a formula and getting an answer.
Related: How to Use Data and Avoid Being Mislead by Data – Data can’t lie, but people can be mislead – Data is Important and You Must Confirm What the Data Actually Says – Statistical Techniques Allow Management to do a Better Job – Bigger Impact: 15 to 18 mpg or 50 to 100 mpg? – Process Behavior Charts are the Secret to Understanding the Organization as a System