But it Doesn’t Mean Anything

Guest Post by Dr. Doug Stilwell, Drake University

In my last paper, I wrote of a 1994 exchange between myself and my then supervisor, Dr. Doug Buchanan, in which during my evaluation at the end of my first year of being a principal, I shared I wanted to “change the educational world.” He paused and insightfully and correctly said to me I “needed to know something first” in order to have anything of value to share with anyone.  While I believe Dr. Buchanan was telling me I needed “profound knowledge,” the notion of “knowing something” also suggests having some degree of knowledge about, in my case, the world of education, specifically the school I was leading at that time.  Knowing things – possessing knowledge – helps one to begin the process of inquiry and improvement.

But what type of knowledge must one possess in order to make improvements, say, in student learning?  The universal mantra of being “data-driven” can lead one to believe that if we simply accumulate data, we will somehow divine what to do next.  But data or information according to Dr. W. Edwards Deming, is not the same as knowledge; something he tells us for which there is no substitute.  Quantitative data, the type we typically collect and analyze in education, is simply a set of numbers/values – results of some sort of assessment – and means, on its own, absolutely nothing.  The reader may disagree, but let me make my case for this by making a connection from a scene from the classic musical, The Sound of Music.

If you have seen the movie, or the play, you will recall the scene where Maria was attempting to teach the seven von Trapp children to sing; something that had been missing from their home since the death of their mother.  During the “Do-Re-Mi” sequence Maria sang a series of notes, “do, so, la, fa, mi, do, re,” based on a musical theory known as “solfege,” in which every note in the major scale has a certain unique syllable (do-re-mi-fa-so-la-ti-do), and asked the children to repeat what she had sung.  Maria continued to build on the complexity of her sequence, which the children, of course, repeated beautifully.  It was at the height of this complexity of seemingly disconnected notes that the youngest of the von Trapp children, Gretl, tugs at Maria’s skirt and bravely announces, “But it doesn’t mean anything;” and she was correct.  What the children were singing appeared to be random notes (data) that lacked meaning. . . until Maria emphasized this was a melody made up of the notes and then added the lyrics, “When you know the notes to sing, you can sing most anything.”  In other words, Maria provided a context through which to understand and make meaning of the data (notes).

Educators are asked – no; directed – to be “data-driven,” and I have come to believe that this mantra may be the wrong approach to advocate, for while it is well-intended, without a deeper understanding and appreciation for making meaning of data, it may result in “doing the wrong things ‘righter’” (see previous article with the same title).  We might look at the data, sometimes just one or two data points, believe we understand what it means (which may be a dangerous assumption) and then take action to improve the data.  This observation is not a criticism of educators, but of the educational system; one that is too heavily driven by numbers without context and meaning, and one that may not prepare educators to be “meaning-makers” of data.  As an example of the latter, in graduate school there is typically some requisite course in “educational research” and through the experience, educators learn how to manipulate statistical data using descriptive and inferential statistics employing statistical tools and processes that support their formal research.  And while these tools and methods have value in undertaking academic research, there is a question as to whether they prepare educators in the field – on the front lines – to make sense of student data.

So, “by what method” might we pragmatically develop knowledge and make meaning that leads to action and improvement when confronted with student data that we collect?  Let me offer two examples.  One method comes from category seven (Results) of the Baldrige Performance Excellence Framework, communicated in the form of an acronym – LeTCI (pronounced “letsee” – emphasis on “lets”).  The “LeTCI” framework focuses on the following:

• Le – current levels of performance
• T – Performance trends over time. In Baldrige, a minimum of three consecutive data points, at minimum, in any direction is required to be considered a trend.
• C – Making Comparisons with like students or groups
• I – Integration, to determine to what degree our results measure what matters.

Through the LeTCI method, educators determine at what level a student, or group of students, are performing.  They would then look at multiple data points to determine if any trends, positive or negative, are present.  Then, they compare the data with “like” students or groups, being careful not to use the data to rank, sort, or judge students, but rather, as Dr. Deming reminds us, to use results to help us coach and improve others’ performance.  And, finally, determine if what is being measured is actually what is important to the school district and its customers – parents and the community.  Through this examination and “meaning making” of the data, we can better understand what the data means in order to develop, implement, and monitor effective improvement plans.

Another method to assist in bringing meaning to data is through Walter Shewhart’s control chart and understanding the concept of variation.  In a control chart, data plotted and calculated over time returns three important results: the mean score of the data set, the upper control limit (three standard deviations above the mean) and the lower control limit (three standard deviations below the mean).  When results lie within the upper and lower control limits, the data, or system, is said to be stable.  When the data/system is stable, it tells us that the results/data are caused by the system.  When data lies outside the upper or lower control limits, there is something outside the system – something special – that has caused those results.  As an example, if student tardy data were entered into a control chart and one student’s tardies exceeded the upper control limit, it would be necessary to investigate the reason the student was so often late.  The investigation might uncover the fact that the student’s parent works the “overnight shift” and has difficulty getting up in the morning, causing the student to come late to school.  Understanding whether results are caused by the system (common cause) or something unique (special cause) then leads to either focus on improving the system or giving special attention to a special cause.  Without understanding what is causing the results (the system or a special cause), we may incorrectly attempt to either intervene our way out of a systemic issue or change the entire system for one unique situation.  Either way, our efforts toward improvement will likely fail.

Let me provide an example of a control chart and how it might be interpreted.  Below are 13 years’ worth of annual reading data results from different years of students at the same grade level in a school district in Iowa.  Note that the mean score is approximately 292 while the lower control limit is approximately 281 and the upper control limit is 304.

First, notice that all data points fall between the upper and lower control limits, which means the system is stable.  Without understanding variation while interpreting the data, focusing solely on the results from 2004 might provoke the following inquiry:  “What happened in 2004 and what was done to correct course?”  An understanding of variation tells us that while 2004 had the lowest result, it was still within the upper and lower control limits; meaning there was nothing particularly “special” about this group of students and their results.  In truth, no corrective actions were taken in response to the 2004 results, for what we see in this control chart is “normal variation,” caused by the system.  To have the system respond to a single data point, such as 2004, by changing the entire system – particularly since it falls within the upper and lower control limits – would be known as “tampering” with the system and might do more harm than good.  That said, because the 2004 group was close to being below the lower control limit, I would pay attention to the group as they moved through the system to ensure they were making adequate progress.

Keep in mind that having a stable system does not mean one must be satisfied with the results.  In the control chart above, there should be two goals over time:

1. Raise the mean
2. Reduce the amount of variation between results (aka reduce standard deviation)

The theory behind the control chart relates directly to what educators know as MTSS (Multi-tiered Systems of Support), which is based on the theory that:

• 80% of students (Tier 1) will have their instructional needs met, will be met through whole-class “universal instruction”
• 10-15% of students (Tier 2) will require small group interventions and
• 5-10% of students (Tier 3) will need intensive, individualized support

In other words, the vast majority of student needs will be met by the “system” (Tier 1).  Some will require a bit more support (Tier 2), and a very small percentage will receive more than the system provides in tiers one and two.  To make the control chart connection, students in Tiers 1 and 2 will have their needs met by what typically happens in the system, while Tier 3 students would be characterized as “special cause” requiring special supports that go beyond what the systems offers to everyone.

Control charts also help us to highlight the misunderstood and misapplied concept of “average” and that of being above or below average.  Notice two things about the data in the control chart in this light:

1. Not one data point lies exactly at the mean.  This supports the thesis of the book The End of Average in which author Todd Rose (2016) explains in greater detail that “average” is a mythical concept; an arithmetic calculation that leads to what he refers to as “averagarian” thinking, which espouses that we can understand individuals by looking at averages.  In other words, “The mathematical concept of averages has its purposes, but it’s irrelevant when applied to human nature.” (https://lifeclub.org/books/the-end-of-average-todd-rose-review-summary).
2. The reality is that some student and/or group results will be above and some will be below average, a fact that makes sense. . . once you think about it.  When data from which the mean is derived are plotted relative to the mean, arithmetically approximately half of the results should be above average and half below.  In the example above, six data points out of 13 fall “below average.”  What does it mean? – certainly not as much as we make of it, for if the entire population of the world was given the same assessment on any given day, approximately half of the world would score “below average.”  However it can be devastating for parents to hear their children are labeled “below average.”  The reality here is that the only place where “all children are above average” is in Garrison Keillor’s mythical town of Lake Wobegon.

So what can we do to make better sense of data to help guide our decisions as educators?  Here are some simple questions that may guide meaning-making of data:

1. Have we collected the “right” data?  In other words, are we actually measuring and collecting data based on what we value and wish to improve?
2. Have we collected ample data?  During my own tenure as a school principal, I can attest to the practice of developing annual building goals around student achievement by typically looking at the results of the most recent set of standardized tests, a practice in which I know I was not alone.
3. Do we have an assortment of data that helps to paint a more complete picture of performance?  One data source tells only part of the picture.
4. Is the system stable or unstable and how volatile is the variation?  If the system is unstable, a leader’s first job is to make it stable.
5. Can we determine root cause from the data?  The data are merely symptoms.  Simply examining scores and results does not tell us what caused the results.  Determining root cause of the results will direct improvement efforts at fundamental causes rather than symptoms.  Lower-than-desired results is not the problem; they are a symptom of a problem.  It’s like understanding that a high pulse rate taken on the wrist does not mean there is a problem with the wrist; it’s merely a symptom of something else, in this case, the heart.

Improving learning results, or any results for that matter, requires that that we know something first.  The first thing we need to know is that despite a call to do otherwise, data should not drive decisions in schools.  Dr. Deming would likely tell us that we need to go deeper than being “data driven.”  Data is simply information, and Dr. Deming tells us that what we need is knowledge.  At the most fundamental level we need to be able to answer the question, “What did the data help us to learn?”  The answer to this question then helps us to develop and operationalize an improvement theory.

What education needs is two-fold:

1. Adequate training for educators to make better sense and meaning out of data.  This will lead to…
2. Individuals who have the knowledge to understand what the data does and does not mean in order to inform decisions that are made regarding potential actions to be taken to improve student learning results.

Relative to these recommendations, I encourage educators to become familiar with Baldrige’s LeTCI and Shewhart’s control charts, for they can be key to developing the requisite knowledge to make wise decisions relative to improving student learning and will eliminate “knee-jerk” reactions to data that “doesn’t mean anything.”

To learn more about making meaning of educational data and the Drake Continual Improvement Network, contact Dr. Doug Stilwell at doug.stilwell@drake.edu.

References

Deming, W. (2018). The new economics for industry, government, education (3^rd ed.). Cambridge, MA: MIT Center for Advanced Educational Services.

Rose, T. (2016). The end of average: How we succeed in a world that values sameness. New York: HarperCollins.

Rose, T. (n.d.). The end of average summary and review. Retrieved from https://lifeclub.org/books/the-end-of-average-todd-rose-review-summary.

Wise. R. (Producer and Director). (1965). The sound of music [Motion Picture]. United States: 20^th Century Fox.