Guest post by John Hunter, author of the Curious Cat Management Improvement Blog (since 2004).
Transforming Resident Assessment: An Analysis Using Deming’s System of Profound Knowledge
The authors use Deming’s framework to illustrate the working components of the assessment system at the University of Cincinnati College of Medicine’s internal medicine residency program and draw parallels to the macrocosm of graduate medical education. Successes and failures in transforming resident assessments can be understood and predicted by identifying the system and its aims, turning information into knowledge, developing an understanding of variation, and appreciating the psychology of motivation of participants.
We abandoned the norm-referenced Likert scales we used for broad themes (e.g., patient care) in favor of a competency-based entrustment framework of discrete, observable skills (i.e. OPAs). In our OPA system, assessors rate skills using the five-level entrustment scale (described above) and a simple criterion-based question: At what level does the assessor trust the learner to perform the skill?
This seems like an excellent decision in my opinion. I am not a fan of the word “entrustment,” but that is a personal opinion so isn’t of much importance. My thoughts on the merit of what they are doing are explained in a previous post I wrote: A Powerful Tool: The Capacity Matrix. Their short article doesn’t go into detail on their system, but based on what they discuss, I do think the details provided by the Capacity Matrix concept are useful in efforts such as they mention.
We have made changes to develop the intrinsic motivation of our assessors. To foster a sense of autonomy, we engaged each division in the creation of OPAs [observable practice activities], giving ownership of each rotation’s assessment form to the faculty assessors. We also incorporated a “not observed” option on the form, so individual faculty assessors never feel forced to assess skills they did not observe.
This is a good effort to examine the overall system and change the system to improve results. The simple change of not forcing people to give responses when they don’t have the necessary background is excellent. I am frustrated with how often today’s electronic forms require false responses because accurate responses (such as “not observed”) are not allowed. That not only creates bad data, it also undermines confidence in the system as people know they are forced to input false information. Why should they not believe that other parts of the system are broken by design? And if they don’t have confidence that the system will work well. why waste their time?
They also looked to learn about the causes of responses not being received by assessors, rather than just more forcefully demanding that assessors respond on time. This is not that radical an idea, but it is fairly rare in practice in my experience. It is best to attempt to learn why the system isn’t working as you hoped when you designed it. Doing so lets you improve the system. The symptom you notice (rate of non-responses) may well share causes with other factors you do not notice: mindless completion by assessors to provide what is requested that can be easily measured [sending in some response], but not taking time to make it actually useful because they have no confidence in the system (being required to provide, for example, feedback on things for which they have no evidence).
Thank you, John, for bringing this article to our attention.
I immediately started thinking back to the last survey that came across my ‘desk’, and whether there was a similar “not observed” response. Certainly will keep this in mind as it isn’t uncommon to not be able to truly answer the options presented (wasn’t that a triple negative, so perhaps better phrased: is common to choose inaccurate responses given the options presented?!).
Even without reading the detail, it is great the authors reference Dr. Deming and his System of Profound Knowledge. Hopefully it will also prompt other readers unfamiliar with Dr. Deming to become curious and explore the System of Profound Knowledge.
After participating in a recent online workshop on Deming’s Leadership Principles, seeing an article like this spurs me on to work with my team in applying the System of Profound Knowledge in our healthcare domain and team.