Archived: Chapter 2 - Study Aims and Study Questions

A r c h i v e d  I n f o r m a t i o n

Systemic Reform - October 1996

Chapter 2
Study Aims and Study Questions

The systemic reform literature summarized in the preceding chapter pointed to three areas meriting further study. First, we need to learn more about how states are changing the way they make policy as they adopt more systemic reform approaches. While the CPRE researchers have been studying state systemic reform activities in nine states through its "Core States" project (Fuhrman, Massell et al., 1992; Massell and Fuhrman, 1994), this study provides an opportunity to expand our understanding of how states design and implement systemic education reform policies and to examine strategies states have developed to overcome this structural fragmentation and political instability.

Second, we know little about how teachers, schools and school districts are responding to state systemic initiatives or about the impact of coherent state policies on what gets taught, how it is taught, what students are learning and on how student learning is assessed. This study is one of several currently looking at education reform at different levels of the educational system. Researchers at Michigan State University and the University of Michigan, for example, are examining how new instructional guidance policies are enacted at the school, classroom and/or student level in four states, paying particular attention to teachers of educationally disadvantaged students (Ball, et al., 1994a; 1994b; Jennings and Spillane, 1995; Spillane, 1994). The National Science Foundation's national evaluation of its Statewide Systemic Initiative will provide information on strategies that states, school districts and schools are using to reform mathematics and science education (Shields, Corcoran and Zucker, 1994; Zucker, et al., 1995). And CPRE's "Core States" study includes an examination of state and local relationships and local views of state reform in each of its states (Hertert, 1994). This study was designed to complement these other research activities by focusing on the response of reforming schools and school districts to state systemic initiatives, and on the factors that appear to support and/or hinder school and district-based reform activities.

Third, the research that was available at the time of the study design indicated that state policies to guide instruction often falter on the issues of teacher knowledge and capacity. Based on these findings, our preliminary research design focused on policies that linked K-12 instructional guidance and teacher pre-service and professional development policies. In a paper commissioned for this study, however, David (1993) argues that capacity-building in support of systemic reform goes well beyond the traditional teacher preparation and professional development activities in place today. "Systemic reform asks everyone in the education system to change their roles and relationships, not just teachers and students" (David, 1993, p. 2). Building the capacity to change teaching and learning means creating the opportunity for administrators, researchers and policymakers, as well as educators, to learn new ways of doing their jobs. Therefore, we expanded the focus of the study from looking narrowly at teacher preparation and professional development to the broader issue of capacity to support systemic reform. The case study component of the project collected information on the needs for assistance at different levels of the system, strategies and activities developed to address these needs, and barriers to building capacity and areas of unmet needs.

We these issues in mind, our study had five general objectives:

Research Questions

The following questions guided the collection and analysis of data in this study:

Elements of Systemic Reform

  1. What are the characteristics of coherent systems of instructional guidance in elementary mathematics and language arts in three states undertaking systemic reform?

    a. What policies do states include in their instructional guidance systems (e.g., curriculum, assessment/accountability, teacher policy, governance)?

    b. Why have states chosen to include certain components and exclude others?

    c. Are there gaps, conflicts, and misconceptions about state policies that would impede coherent state policy effects?

  2. What is the scale and scope of state and local policies? To what extent do they reach all students, schools, school districts, and teachers?

  3. What is the mix and balance of state policy instruments? How much reliance do states place on mandates? inducements? technical assistance? capacity-building?

  4. What sources of information do policymakers use in constructing coherent approaches to state and local policy?

Implementation

  1. What are district and school goals and priorities? What is the relationship of these goals and priorities to state education reforms in elementary mathematics and language arts?

  2. How are teachers, schools and local school districts using state curriculum frameworks and assessments?

  3. How have teachers changed their instructional practices and what influence over their practice do they attribute to state instructional guidance policies? What other factors drive their instructional practices?

  4. What is the relationship between systemic reform, including instructional guidance and governance reforms, and the school context? How does the school context influence systemic reform strategies?

  5. How, and to what extent, do federal policies facilitate or inhibit the implementation of systemic reform?

Capacity

  1. What capacity do local school districts, schools and teachers need to implement education reforms in elementary mathematics and language arts?

  2. What is the capacity of states, local school districts, schools and teachers to implement these reforms?

  3. Who provides professional development to local districts, schools and teachers? To what extent do these programs meet the current needs?

  4. What factors facilitate and inhibit the provision of sound professional development?

  5. What lessons can state and local policymakers in other states learn from these experiences? What are the necessary conditions for the provision of professional development in support of systemic reform? What steps should be taken to facilitate the development and provision of these services?

An Overview of the Study Methodology

Our study methodology is detailed in Volume III of this technical report.1 In brief, we conducted case studies in three states that were undertaking significant systemic reform activities. Based on the findings of the literature review, the deliberations of the national conference and research on systemic reform conducted by CPRE and others on developments in the states under consideration, we chose to study California, Michigan and Vermont. Within each state, we drew a purposive sample of two school districts with reputations for active use of state reform and the capacity to support education reform.2 In each study district, we then chose two schools, one at the elementary level and one at the middle school/junior high school level. Where the district had more than one school at these grade spans, we again chose schools with reputations for reform activities. Table 1 summarizes the characteristics of the school districts and states in our sample.

TABLE 1
Characteristics of Study Sites


 Enrollment, 1993 Community Type Racial/ethnic composition, 1992 % of students in poverty1
California 5,285,000
44% White; 36% Latino; 11% Asian; 9% African-American; (22% LEP) 26%
CA1 5-10,000 Suburban/rural 55% White; 39% Latino; (15% LEP)
CA2 >50,000 Urban 14% White, 35% Asian; 20% Latino; 20% African-American (30% LEP) 30%
Michigan 1,613,700
78% White; 17% African-American; 2% Hispanic 18%
MI1 5000 Suburban 94% White 20%
MI2 3300 Suburban Predominately White
Vermont 105,000
98% White 24%
VT1 1,500 Rural Nearly all White
VT2 2,300 Rural Nearly all White

1 Percent of students participating in the free and reduced school lunch program.

The focus of our study was on mathematics in grades K-8 across the three states and a second subject area, also in grades K-8, that was the focus of reform in each state--reading in Michigan, writing in Vermont and California. This approach enabled us to examine capacity-building in one subject (mathematics) that has been the subject of reform nationally, while examining a subject that has been of particular concern to each state. Within the K-8 grade span, we targeted instruction in grades 4 and 8.

Data Collection

In 1993-94, we conducted structured interviews with state policymakers, teacher educators and other providers of professional development, and district and school administrators in each of our sites. We asked state policymakers about the most recent changes in education reform, especially in mathematics and language arts, the capacity of the state, local school districts, schools and teachers to carry out the reforms, sources of support and professional development, and the state role in providing this support. At the local level, district and school administrative staff were asked about district goals and their relationship to state reform efforts, local curricula and assessment policies, perceptions of the capacity of their district, schools and teachers to implement state and local reforms, sources of support and professional development, and the state and local roles in providing this support.

We also interviewed five teachers in each of our twelve schools. In the elementary schools, we focused on teachers in grade four, but also included a teacher in grades three and five to capture curriculum and instruction in the grades "surrounding" the target grade. In middle or junior high schools, we targeted eighth grade teachers of mathematics and language arts, generally adding a seventh grade teacher in one of the subject areas. In place of classroom observations, which were beyond the financial scope of this study, we asked each teacher to complete a content coverage/instructional strategy questionnaire (in mathematics and reading or writing in the elementary grades; in their area of instruction in middle school grades).3 The purpose of the interviews and questionnaires was to obtain information on teachers' instructional practices (specifically topic coverage and sequence, time allocated to these topics and instructional approaches they use for teaching the topics), how state and local policies affect content and instructional strategies in the classroom, the extent to which teachers have changed in their teaching, and reasons for these changes. We also asked teachers where they looked for support as they reformed their teaching of mathematics and language arts and their professional development opportunities and activities. Since we did not conduct classroom observations, our data are limited to intended, rather actual curriculum and instruction. That is, we identified what and how teachers say they teach, not what they actually do in the classroom.4 However, our data collection enabled us to explore how teachers implement instructional guidance systems, and, as important for policymakers, what they view as problems with the policies and major barriers to implementation.

All respondents were guaranteed anonymity. We also agreed not to publish the names of the participating schools or school districts.

Data Analysis

This study used a qualitative case study methodology, supplemented by quantitative data analysis of the teacher questionnaires. Analysis of the qualitative data involved three stages. The first stage involved "within case" analysis for each school site and resulted in an unpublished case study for each school. The second stage entailed "cross-site" analyses across schools within a state, where the researcher compared and contrasted across school findings and analyzed state- and district-level data to generate state-level case studies. The state case studies, included in Volume II of this technical report, describe the components and characteristics of systemic reform and coordination among policies at the state and local level, and changes in teachers' instructional practices and the sources of these changes, and analyze the capacity to support systemic reform and factors facilitating and inhibiting reform and the capacity for reform. The third stage of analysis involved comparisons across the three states and focused on common challenges to implementing systemic reform (reported in Chapter 4) and issues related to capacity and capacity-building strategies and policies (Chapter 6).

Data from the teacher questionnaires were analyzed by subject area and grade level within each state and then across the three states. The questionnaires included questions drawn from three nationally-representative5 and one purposive survey6 of teachers, enabling us to relate the data from our small, purposive sample of teachers to national patterns. The analyses presented in Chapter 5 of this report describe the instructional practices reported by the teachers in our sample and compare these practices to state and national standards in mathematics and language arts, across states, and to data from the national surveys. We also analyzed teacher reports of what factors influenced their teaching, how much control they had over aspects of their school or classroom, and amount and generic sources of professional development activity. The small size and non-representative sample of teachers included in this study do not allow us to generalize about teachers' instructional behaviors within their schools or districts, or within or across the three states. The questionnaire data, however, taken together with the interview data, do provide insights into the connections among systemic reform policies, capacity building and instructional practice.

1 Volume III contains a more detailed description of the study methodology and copies of all of the interview protocols and teacher questionnaires.

2 "Active use" entails anticipating state responses or doing more tat the state requires. See Fuhrman, Clune and Elmore, 1988; Firestone, 1989; and Firestone et al., 1991.

3 The questionnaire component of the study was funded by the Carnegie Foundation a part of a CPRE study of professional development and systemic reform.

4 We feel that we collected accurate information on teacher instructional practices using teacher interviews and a content coverage/instructional strategies questionnaire, rather than brief classroom observations. Porter (1993) found considerable agreement between teacher logs and self-report through questionnaires seems a promising and less expensive alternative...for measuring opportunity to learn."

5 The National Education Longitudinal Study of 1988 First Follow-up [1990] Teacher Questionnaire, the Schools and Staffing Survey of 1990, and the NSF 1993 National Survey of Science and Mathematics Education--Mathematics Questionnaire.

6 Porter et al., 1993.
-###-

[Chapter 1: Summary Review of the Literature] [Table of Contents] [Chapter 3: State Profiles ]