By John Hunter, author of the Curious Cat Management Improvement Blog (since 2004).
This recent article, Has Uber Forced Taxi Drivers to Step Up Their Game?, explores how competition from Uber has pressured the legacy taxi business to improve customer service.
In exploring whether the data supports the idea that there has been an effort to improve customer service the article mentions that complaints about taxis increased in 2012, and then provides an explanation of a system change that is likely the cause. In that year taxis were required to prominently display the phone number to complain. Without knowing more than the article tells that seems like a logical explanation of the increase to me. And that understanding is very important to understanding what the data is telling us.
This highlights a very important factor when looking at data, you must understand the processes and system that generated the data. If you do not you will draw faulty conclusions.
If you bring in a new effort to focus on customers and solicit more feedback if you don’t get an increase in complaints that is likely not an indication of success but an indication of failure. One of the easiest way to reduce the number of complaints counted is to make complaining, in a way that is counted, difficult.
If you tie performance appraisals or bonuses to improved results, you will drive behavior to make the number look better (which isn’t the same as driving better results for the business and customers). Making the numbers look better through manipulation (of the data or system) is usually much easier to do (for example, by changing the process to make it harder to complain – or by just not recording verbal complaints even if the operational definition for the collection of data says those should be recorded) than it is to improve the process so people are actually happier with your service.
Data is important. Using data to measure the effectiveness of new efforts is important. But you need to understand the risks of being led astray. That risk is much greater if those analyzing the data are not intimately familiar with the processes generating the data and the operational definitions used to collect the data.
Related: Dangers of Forgetting the Proxy Nature of Data – Unknown and Unknowable Data – Customer, or User, Gemba – Executive Leadership