Guest post by Kim Melton
When many of the ads on the final round of the Masters Golf Tournament are about using data to make decisions, you know that data has become big business! In ads IBM touched on the use of data in transportation, healthcare, manufacturing, crime, love, education, farming, pollution, weather, energy, and many other areas!
The ability to collect large amounts of data is increasing daily, and organizations are attempting to use analytics to transform this data into usable information to help them gain better knowledge about their products, services, and customers. Successful transformation from data to information and knowledge touches on all four components of Deming’s System of Profound Knowledge (and the interactions among the components).
Unfortunately, there is no such thing as (truly) objective data. All data are subjective! Also, Deming’s System of Profound Knowledge can help us understand and improve the collection and analysis of data. The theory of knowledge helps us recognize that someone decides what data to collect and what operational definitions to use. Knowledge about variation helps us recognize that obtaining data (measurements) is a process and that variation is present in all processes.
As such, we must acknowledge that each individual measurement is actually an estimate of an unknowable value. Theory of a system leads us to see that collecting data is just one component of a larger system—that the measurements are intended to help us understand characteristics of the item or process that produced the data. Psychology influences what data individuals see as important to collect, how individuals react to the collection and analysis of data, and can influence the actual measurements obtained when data are collected.
Consider data collection in education. Data are collected to assess student learning, to evaluate teaching, to make decisions about college admissions, for budget allocations and financial aid decisions, and for many additional purposes. But, these uses of data are not independent. Measures used to assess student learning are often used as part of the evaluation of teachers, as input into making decisions about college admission, allocation of budgets, and financial aid decisions for academically based scholarships. And, the implications are not limited to one direction. For example, how a teacher is going to be evaluated may influence the timing and methods of assessing student learning as well as content coverage in courses.
Consider the events that occurred in the Atlanta Public Schools (APS) during the last decade. Over multiple years, student learning (as measured by the CRCT Exam) had increased at rates that were bringing national attention to the school system. This case provides an example where psychology, a failure to understand systems, and lack of knowledge about how the data were actually created interacted to create results that were inaccurate.
Reported results were used to make decisions related to passing students to the next grade, evaluate teachers, administrators, and systems, and allocate funding at the system, local, state, and national level. Over a several year period, the falsification of results (and advancement of students without the skills needed at the next level) led to a reinforcing cycle of misrepresentation. The state investigation of the scandal identified close to 200 educations involved and a grand jury legally indicted 35 employees of the system (including the superintendent when the scandal took place) of charges including racketeering. Of the 35 legally charged, only one was acquitted of all charges; and 2 others died prior to the end of the trial.
In summer of 2011 when the evidence related to influencing and/or changing student responses was coming to light the acting superintendent stated, “I am angered, and I’m sure you are too, that our students and parents were misled by inaccurate test scores that misrepresented students’ real proficiency.” Granted APS is an extreme example of measurement gone wild, but think about how measurements are impacted by other decisions made in education—from ones at the classroom level to the school system level and even at a national or international level. All of these will impact the perception of students’ real proficiency and can lead to inappropriate comparisons between groups.
- Course and curriculum design (each instructor, school, or school system may include different material in courses or programs with the same name)
- Instructor method of presentation of material (presenting material via lecture, discussion, small group exercises, … may result in different levels of learning)
- Choice of evaluation method for assessing individual students (multiple choice, fill-in-the blank, response to specific open ended question requiring more contextual knowledge, discussion tying multiple disciplines together to form a response require responses from the low to high end of Bloom Taxonomy of Learning)
- Timing and method of providing feedback to students (do students receive formative as well as summative feedback in the learning process; does feedback include information about why an answer is incorrect and how to improve; is feedback provided in a timeframe that allows students to integrate this into their learning,…)
- Stated learning outcomes and methods of student assessment at a course level
- Instructor reactions to student evaluations of instructor (making changes based on student/parent feedback rather than broader society expectations)
- Calculation of grades and use of GPAs (weighting of assignments, use of partial credit, use of +/- system of grading, “cut-offs” for letter grades, …)
- Comparisons across geographic areas to allocate funding (local, state, national expectations for curriculum content and student proficiency and the “teach to the test mentality” that tends to follow)
These are just a few of the sources of variation in education that influence the data that are obtained. Education is not unique. All organizations (businesses, social, or otherwise) need to recognize that many times data collection has implications far beyond the stated purpose.
Kim Melton is a Professor of Management at the University of North Georgia where she teaches Statistic for Business. In April 2015, she and Suzanne Anthony published “When Measurement Becomes the Mission, Don’t Trust the Measurement” in The Journal for Quality & Participation. She is a member of the American Statistical Association, Decision Sciences Institute, and a Senior Member of the American Society for Quality.
Related: What Role Does/Should The Deming System of Profound Knowledge Play in the World of “Big Data”? – Data is Important and You Must Confirm What the Data Actually Says – Analyzing Data Requires an Understanding of the System Generating the Data – Distorting the System, Distorting the Data or Improving the System – The Defect Black Market