The study titled “Over-the-Counter Data’s Impact on Educators’ Data Analysis Accuracy” has concluded and the manuscript has been published by ProQuest. The findings hold significant implications for anyone involved in communicating data to educators.

Study Basics

The quantitative study was inspired by the notion that data reports should offer educators more support in analyzing the reports’ data, as when such highly educated and intelligent individuals are making data analysis errors, the tool they are using warrants improvement. Essentially, data reports should communicate more effectively and educators should not have to shoulder all the responsibility for improving their data analyses.

The researcher facilitated causal inferences concerning the degree to which including different forms of data usage guidance within a data system reporting environment can improve educators’ understanding of the data contents, much like including different forms of usage guidance with over-the-counter medication is needed to improve use of contents. The study’s primary independent variables included brief, cautionary verbiage in (a) report footers, (b) report-specific reference sheets, and (c) report-specific reference guides. These three data analysis supports (defined below), which can be generated within a data system, were each framed in 2 different formats. The dependent variable was accuracy of data analysis-based responses measured by a survey.

211 elementary and secondary educators employed at 9 schools, 6 school districts, 6 cities, and 3 counties in California generously participated. These educators answered data analysis questions while viewing 1 of 7 report sets of student data, which all featured identical data but varied levels of embedded support. The study was pilot-tested first, subscribed to all Institutional Review Board (IRB) and ethical guidelines, and reflected precautions to avoid or overcome threats to external and internal validity.

Definitions of Investigated Supports

• Footer – brief line(s) of text (containing up to 58 words, 269 characters without spaces, and 324 characters with spaces) at the bottom of each report communicating only the most important information an educator would need to know in order to correctly understand and analyze that particular report’s data

• Reference Sheet – a 1-page sheet (accompanying a report), often called an abstract, to communicate the report’s description, image, purpose (key questions the report will help answer), and focus (intended audience, content reported, and format in which data is reported), and warning (vital, cautionary information an educator would need to know in order to correctly understand and analyze that particular report’s data)

• Reference Guide – a 2- or 3-page guide (accompanying a report) that looks like the reference sheet on the 1st page, but follows this with instructions (how to read the report), essential questions (walking the user through the process of specifically where to look on this report – and what to look for – to answer each question listed in the purpose area of the guide’s 1st page), and a “more info” section (such as where and how to get additional information on such topics as finding additional analysis guidance or help generating the report in the data system)

Results

All supports used in the study had a significant, positive impact on educators’ data analysis accuracy. In terms of relative difference, educators’ data analyses were:

• 307% more accurate (with a 23 percentage point difference) when a footer was present and 336% more accurate (with a 26 percentage point difference) when respondents specifically indicated having used the footer,

• 205% more accurate (with a 12 percentage point difference) when a reference sheet was present and 300% more accurate (with a 22 percentage point difference) when respondents specifically indicated having used the reference sheet,

• 273% more accurate (with a 19 percentage point difference) when a reference guide was present and 436% more accurate (with a 37 percentage point difference) when respondents specifically indicated having used the guide, and

• 264% more accurate (with an 18 percentage point difference) when any one of the three supports was present and 355% more accurate (with a 28 percentage point difference) when respondents specifically indicated having used the support.

Overall, the 211 study participants indicated they used supports 58% of the time. 87% of participants who received no supports indicated they would have used footers, reference sheet, or reference guides if the supports had been available.

When no supports were used, data analysis accuracy was 11%. In cases where respondents indicated they used an available support, data analysis accuracy was 39%. Secondary independent variables, spanning a range of educator demographics and school demographics, had no significant impact. This means educator analysis performance was generally the same regardless of educator and school demographics, and the supports were equally effective regardless of educator and school demographics.

Significance

The findings of this study filled a gap in education field literature by containing evidence that can be used to identify how data systems and data reports can help improve educators’ data analyses by providing analysis support within data systems and their reports, and by rendering examples and templates for real-world implementation. Improvements data system and report providers make in light of this study have potential to improve the accuracy with which educators analyze the data generated by their data systems. This improvement will likely benefit students.

Free templates, modeled after the supports proven effective in this study, have been added to the Resources page of this website to assist anyone interested in adding reference sheets or reference guides to his or her data system and/or to accompany his or her data reports. The footer guidelines provided in this article (above) can also assist anyone wanting to add footers to reports. The study manuscript (http://pqdtopen.proquest.com/#abstract?dispub=3575082) offers more specific details. Please note its Acknowledgements section, which offers deep gratitude to all who encouraged and participated in the study.

Visit www.jennyrankin.com/subscribe to subscribe to this blog.