Three ways edtech usage data can yield classroom insights

To understand how to make the most effective use of their edtech applications, educators and researchers must look closely at the available data. And for better or worse, there’s more data available to conduct that analysis than ever before, executives from the edtech research firm Evidentally recently told the audience of a webinar hosted by edWeb.net.

In years past, educators could count the number of edtech programs they used on one hand. Usage eventually grew and schools began using spreadsheets and written reports that could take months or years to yield actionable data. But in 2019, said Denis Newman, co-founder of Evidentally, data is available to educators and researchers nearly in real-time.

Along with Evidentally’s Kylene Shen, Newman explained three reasons that this sort of usage research is critical for educators and developers alike.

1. Data uncovers opportunities for further study

Usage data shows where a comparison study might be feasible. For instance, in one case study shared by the presenters, a teacher communication tool had an unusual pattern. Upon consulting with the schools using the software, researchers found two distinct types of users and were able to look deeper into each group’s habits, yielding previously undiscovered insights.

2. Data can highlight which users are important

Edtech usage data can also show where a study population might be meaningful. One professional development company cited by Newman, for instance, had many users in New York. This realization afforded researchers the opportunity to conduct an in-depth study of those teachers to better understand the usage patterns that sat underneath.

3. Data can underscore important demographic differences

Newman said usage data can also reveal usage patterns across various metrics that are useful for educators and researchers. In one program used widely across California that was studied, researchers found achievement gains, but wanted to understand the gains in greater detail. They were then able, Newman said, to break the results into sub-groups to gain a better understanding of what achievement gains meant for various demographic populations among the users.

A shared interest

Schools can ask to be part of pilot studies and have any results shared with them, Newman said. Developers should look for several schools to conduct these studies so that they can accumulate data from a variety of sources. For both, subgroup analysis and avoiding selection bias are key, he said.

It’s not good enough to know if a product worked on average, Newman said — developers need to know how the product drives outcomes for subgroups, also. He warned about collecting data solely from the top users. Such a methodology doesn’t reflect how the product is used in a school, which includes users with a wide variety of skill levels. He also noted that several small studies typically yield a more reliable result than a single large study.

Finally, Newman proposed that developers and educators work together to perform these types of studies because both groups have a vested interest in improving edtech.

About the presenters

Dr. Denis Newman, lead author of the software industry’s guidelines for research on edtech impact, is the CEO of both Empirical Education Inc. as well as Evidentally, a spin-off research organization that conducts dozens of RCTs and other evaluations of school programs. He has 35 years of experience improving student-teacher learning processes and instructional technologies, and is a pioneer in applying internet to student learning, professional development, and school administration. His Ph.D. in developmental psychology is from the City University of New York.

Kylene Chinsio Shen brings more than 18 years of experience in the education industry to her role as VP of Marketing for Evidentally, Inc. Since receiving her MSEd from Northwestern University, she worked as a curriculum developer, an elementary school teacher and math specialist, an instructional designer, and product manager. At Empirical Education, she has overseen the development of company products, and at Evidentally, is leading the company’s marketing efforts.

Join the community

Inside K12 is a free professional learning community on edWeb.net that seeks to empower K-12 stakeholders with knowledge of hidden factors affecting our schools.

The edWeb webinar referenced above, sponsored by Evidentally, Inc., can be found here.