By John Hunter, founder of CuriousCat.com.
In this latest W. Edwards Deming Institute podcast, Dr. Sophronia (Frony) Ward, Managing and Founding Partner of Pinnacle Partners, (direct download) discussess what you need to consider when looking at data from surveys.
One of the points Frony makes is that often the survey data you see fail to provide the details that you really want to see when using the data. When you look at professionally done polls often they will be adjusted but those adjustments are not explained so the level of adjustment is often unknown.
For surveys done by businesses they often have big problems with survey methodology (dealing with issues such as response bias). Also businesses often have survey data that is collected where it is biased by pleas from those at the gemba for customers not to be honest on the survey.
Using data to help guide improvement efforts. But it is critical to know what the data does and does not tell you. There are many problems created when people attempt to use data but don’t understand what the data does and does not say. An understanding of variation is critical to using data effectively.
The Survey Handbook (created by the Navy TQL Office) is a good guide for using surveys to help learn about your organization system. As many of the readers of this blog will know Dr. Deming did a great deal of work with the U.S Navy and the Navy Total Quality Leadership office.
Another important concept for anyone using survey data is to consider the importance of stated preference versus revealed preference.
For those interested in political polling I strong suggest reading FiveThirtyEight. That site actually provides quite a bit of good thinking about data (on politics but also on sports, economics and more). One specific post on political polling: How FiveThirtyEight Calculates Pollster Ratings.
Related: Process Behavior Charts are the Secret to Understanding the Organization as a System (previous podcast with Frony Ward) – Look at All the Data and Be Wary of Unjustified Confidence – Operational Definitions and Data Collection – Enumerative and Analytic Studies