AJE special issue on the practice of data use

The use of data to improve classroom instruction and student outcomes is at the crux of nearly all recent school reform initiatives. Yet in spite of these initiatives, and the mounds of data they have produced, there is surprisingly little empirical research on how teachers and administrators actually make use of data, and to what effect.

The February 2012 (vol. 118 no. 2) issue of the American Journal of Education takes on the question of data use in education, and provides a crucial blueprint to guide future research on this emerging topic. The University of Chicago Press has made several of the articles freely available to all at the journal’s website: www.journals.uchicago.edu/aje

“In this special issue, we present a series of articles that focus on uncovering and investigating the practice of data use: what actually happens when people in schools, school districts, and higher education interact with data in the course of their ongoing work in the situated context of their workplaces,” write Cynthia E. Coburn (U.C., Berkeley) and Erica O. Turner (University of Wisconsin) in their introduction to the issue. “The articles review and reframe the small body of research that examines what we know about how individuals interpret and make meaning of data of various sorts and what happens when new data interventions, processes, and protocols enter into the complex ecology of the classroom, school, and administrative offices.”

The issue was guest edited by Coburn and Andrea C. Bueschel of the Spencer Foundation.

Article titles and abstracts are as follows:

The Practice of Data Use: An Introduction, by Cynthia E. Coburn, Erica O. Turner

Freely available for a limited time at: http://www.jstor.org/stable/10.1086/663272

Data in Practice: Conceptualizing the Data-Based Decision-Making Phenomena, by James P. Spillane

Freely available for a limited time at: http://www.jstor.org/stable/10.1086/663283

“Data use” and “data-based decision making” are increasingly popular mantras in public policy discourses and texts. Policy makers place tremendous faith in the power of data to transform practice, but the fate of policy makers’ efforts will depend in great measure on the very practice they hope to move. In most conversations about data use, however, relations between data and practice have been underconceptualized. In this essay, I identify and discuss some conceptual and analytical tools for studying data in practice by drawing on work from various theoretical traditions. I explore some ways in which we might frame a research agenda in order to investigate data in everyday practice in schools. My account is centered on schoolhouse work practice, but the research apparatus I consider can be applied to practice in other organizations in the education sector and indeed to interorganizational practice, a critical consideration in the education sector.

Understanding Data Use Practice among Teachers: The Contribution of Micro-Process Studies, by Judith Warren Little

Despite the growing volume of research on data use systems or data use activities in which teachers engage, micro-process studies—investigations of what teachers and others actually do under the broad banner of “data use” or “evidence-based decision making”—remain substantially underdeveloped. Starting with a review of the extant research on teachers’ data use practice in workplace and professional development contexts, this article argues for a more conceptually robust, methodologically sophisticated, and extensive program of micro-process research on data use that also anticipates the ways in which local practice both instantiates and constructs institutional and organizational structures, processes, and logics.

Performance Metrics as Formal Structures and through the Lens of Social Mechanisms: When Do They Work and How Do They Influence? by Jeannette A. Colyvas

Freely available for a limited time at: http://www.jstor.org/stable/10.1086/663270

Our current educational environment is subject to persistent calls for accountability, evidence-based practice, and data use for improvement, which largely take the form of performance metrics (PMs). This rapid proliferation of PMs has profoundly influenced the ways in which scholars and practitioners think about their own practices and the larger missions in which they are embedded. This article draws on research in organizational sociology and higher education to propose a framework and research agenda for analyzing the relationship between PMs and practice. I argue that PMs need to be understood as a distinctive form of data whose consequences in use relate to their role as formalized systems of abstractions and the social mechanisms through which they exert change in practice.

School–Central Office Relationships in Evidence Use: Understanding Evidence Use as a Systems Problem, by Meredith I. Honig, Nitya Venkateswaran

Research on evidence use in school districts overwhelmingly focuses within schools on how school staff work with evidence including student performance data, research, and information about teaching quality. While important, this focus on schools reflects a mismatch with federal and state policies that demand not only that school staff work with evidence but that school district central office administrators do as well. This school focus also downplays how complex, social school-level change processes such as evidence use may typically involve central office staff in implementation and vice versa. To what extent do central offices matter to school-level evidence-use processes, and do schools matter to such processes in central offices? We explore these questions with a review of research on evidence use in schools and central offices with a focus on school–central office relationships in the process. We find that central offices and schools influence each other’s evidence-use processes in specific respects. We elaborate what extant research teaches about these relationships and argue that future research should aim to understand how evidence use plays out not solely within schools or central offices but across district systems and through interactions between central office and school staff.

Exploring the Macro-Micro Dynamic in Data Use Practice (Commentary), by Pamela A. Moss

In this commentary on the reviews of data use research in this special issue of AJE, Moss analyzes the ways in which each author illuminates the evolving relationships between micro processes of data use and macro institutional logics. She then considers the complementarities and challenges these articles raise for conventional approaches to research that seeks generalizable causal conclusions about the effectiveness of educational interventions. She suggests ways these learnings might be used to understand and improve educational research and practice more broadly.

Data, Data, and More Data—What’s an Educator to Do? (Commentary), by Paul Goren

Freely available for a limited time at: http://www.jstor.org/stable/10.1086/663273

Goren makes three key points in his commentary on the reviews of data use research. First, he writes, data do not, by themselves, lead to improvement. The context, the setting, and the environment in which data are delivered all matter. Second, interpreting evidence is not a solo act—meaning comes from how a variety of individuals at different levels of the education system understand and make sense of data. Third, he states that while researchers continue to explore and understand data use in multiple educational contexts during this era of data driven decision-making, practitioners and policy makers will need to interact with the data and evidence before them with a healthy sense of constructive skepticism.