Data dashboards for systems change in education

One part of a successful collective impact approach to systems changes requires the use of data to share progress, perspectives, and foster future actions. Surely in the course of systemic change in education, it is important to track data and make it publicly available and transparent for all. It’s also critical to ensure data is displayed in a way that’s helpful to motivate systemic and behavioral changes. However, which data is most appropriate to track and for what purpose is a question that remains.

Efforts often make the mistake of tracking and displaying too much data (data overload) or not tracking information that will make a difference. Some data, in fact, can throw us off track.

For instance, in early childhood, demonstrating the number of seats available for all children but not showing the quality of seats is misleading. It can lead the public to believe that simply creating more centers and thereby seats will improve the system, when in fact seats and quality are structured in tandem — the more seats we have without considering system resources (e.g., highly effective teachers) necessary to provide quality instruction for students can lead us to impact a system in an undesired direction.

What data is important to share and how can we provide information that is succinct enough to digest and yet helpful enough to improve a system that it’s worth tracking?

Here’s a preliminary list of important pieces to include in a data dashboard that is focused on systems change:

Longitudinal Data Graphs over Time. As a rule of thumb, whatever data is deemed important should be displayed over time. Cross-sectional datapoints of progress only demonstrate what’s happening now and not the trend in which one is moving. School data dashboards, for instance, should not simply demonstrate the current percentage of children absent on November 19, but also how that percentage has changed across the year. Data moving in the right direction over time can provide more texture to the story of what’s happening in the school and can help direct time and energy towards solving problems that are moving in the wrong direction.

Variables with High Sensitivity. While mapping the early childhood system in St. Louis, we quickly realized that though there were hundreds of data points collected on variables of common interest —number of providers, accredited centers, children in any given system, etc. — there was a dearth of data on characteristics of the system that were sensitive to model parameterization. In other words, data from variables that were driving the system were missing — we were not tracking them. Variables, such as the number of teachers gaining advanced credentials through the system, the average number of teachers entering the early childhood workforce and/or leaving, and the effect size of professional development on decreasing turnover — this type of data was not easily available or publicly accessible.

Of course, we didn’t realize that having these variables would have such an impact on the system — it was through the listening and computational modeling process that the importance of these variables became apparent. Future data dashboards will be smart to incorporate the variables that drive a system — even if they seem to be less related to children’s outcomes.

We cannot change what we don’t acknowledge.

Substratified data for oppressed/marginalized demographic groups. Data can tell stories. And people can manipulate how data is presented to tell the story they want to share. “Things are going well in our district,” is probably true when we look at districts from a 50,000 foot level. However, we learn more when we start substratifying data in a meaningful way. For instance, many schools in St. Louis demonstrate acceptable outcomes for the general school population. However, when substratifying data by meaningful demographic groups, such as by race, we find that some districts do a better job of serving historically socially marginalized and oppressed groups than other.

This is because math is agnostic. When the majority of a school population represents the general trends of a group, they are powered by the number of children driving those trends. The status of smaller subgroups, however, can be hidden due to their respective size. A great example of what we find when we start tracking outcomes by race, sex, and disability status in schools, for instance, can be found in the Falling Through the Cracks Data report by Forward through Ferguson. This report, by Karishma Furtado and Alexis Duncan et al., demonstrate the importance of looking closer and providing data that matters to others.

Accountability and Data Dashboards.

Of course, this list is just the start — but if we want to move towards greater systemic change, it’s important to start looking at the data that matters. Having a dashboard that no one accesses is a bigger problem — if no one is paying attention to the data, there’s likely to be little action. We see this happening in the COVID response in many communities where data does not lead to change and has disastrous effects on people in the systems.

In schools, if no one is paying attention or holds accountability or responsibility for the outcomes of note, there is little incentive to do anything to make change. Who is held accountable for the outcomes demonstrated in data and how they are held accountable is a touchy subject — teachers being held accountable through performance pay for student growth has had some disastrous outcomes in Georgia.

Therefore it’s also critical to consider the design of a social structure that benefits or is most affected from the information on a data dashboard. Designing the data dashboard is just one step — figuring out what people will do with it and how it will be used to foster systemic change is another that requires attention to detail, feedback thinking, and an understanding of social systems and motivation.