A r c h i v e d  I n f o r m a t i o n

Designing Effective Development: Lessons from the Eisenhower Program - December 1999

Chapter 1

Approach to Evaluation of the Eisenhower Program

In February 1997, the U.S. Department of Education?s Planning and Evaluation Service commissioned the American Institutes for Research (AIR) to conduct a three-year evaluation of Part B of the Eisenhower Professional Development Program, Title II of the Elementary and Secondary Education Act, as amended by the Improving America?s Schools Act (IASA). The evaluation had two purposes:

  1. To describe Eisenhower-assisted activities and evaluate their effects. The 1994 reauthorization of the program instituted a number of far-reaching changes described above. ED wanted AIR to describe program-funded activities and to obtain an understanding of whether the changes were occurring.

  2. To provide information related to performance indicators that ED developed for the program in response to requirements of the Government Performance and Results Act (GPRA). GPRA requires ED to determine the program?s performance in relation to its goals and objectives.

To meet ED?s information needs, the evaluation team designed an integrated set of data collection activities. The evaluation used a multi-method strategy to collect quantitative and qualitative data about Eisenhower-assisted activities. These data come from a variety of sources?state and district officials, directors of grants awarded to institutions of higher education and nonprofit organizations, and teachers. The evaluation was designed to obtain national data about program-funded activities, to obtain a deeper understanding of how the program works in selected locations, and to collect information about how professional development activities affect teacher practice.13 Exhibit 1.0 displays a timeline of the evaluation?s major activities.

As Exhibit 1.0 shows, the evaluation is based on three strands of data collection. The first strand, a National Profile, provides information about program goals, strategies, operations, and activities nationwide. During the 1997-98 school year, we conducted telephone interviews with a national probability sample of Eisenhower coordinators in 363 school districts and SAHE-grantee project directors in 92 institutions of higher education or nonprofit organizations (IHE/NPOs).14 We also collected data from a mail survey of a national probability sample of 1,027 teachers who participated in 657 Eisenhower-assisted activities. 15 These Teacher Activity Survey data describe the types of professional development supported with Eisenhower funds, and compare activities sponsored by school districts to those sponsored by higher education institutions and nonprofits. Appendix A provides additional detail about the sampling design and methodology of the three components of the National Profile

A second strand of data, a set of Case Studies, provides detailed information about how the Eisenhower Professional Development Program operates in selected states, school districts, and schools. During the 1997-98 school year, AIR conducted In-Depth Case Studies in 10 school districts two school districts in each of five states: Kentucky, New York, Ohio, Texas, and Washington. The districts were selected to represent a diversity of region, urbanicity, poverty level, and ethnic composition.16 The sites also were selected because they supported diverse approaches to professional development, instead of, or in addition to, traditional workshops or conference attendance. For example, a number of the in-depth case districts used their Eisenhower funds to provide professional development to "lead" or "resource" teachers who became mentors to other teachers in their schools or districts. Other in-depth case study districts used Eisenhower funds to support workshops or institutes that were unusual because they extended over many days or weeks. Still other districts were selected because they supported school-based professional development activities with Eisenhower funds. Through site visits to the in-depth case study districts, we explored how decisions are made about the use of program funds, and the reasons that the goals, operations, and activities vary across states and districts. The case studies have been a critical source of information about how Eisenhower-assisted activities relate to other professional development and education reform efforts, and about the degree of coherence and consistency among these efforts.

Exhibit 1.0
Timeline for the Evaluation

[Graph not available]

The information from the in-depth case studies expands upon information we obtained from Exploratory Case Studies during the 1996-97 school year. Like the in-depth cases, the exploratory case studies, conducted in six school districts, were selected to vary by region, urbanicity, poverty level, and ethnic composition. The first report from this evaluation, the Eisenhower Professional Development Program: Emerging Themes from Six Districts (Birman, Reeve, & Sattler, 1998), was based on these exploratory case studies, and we also use information from the exploratory cases in this report. Exhibits 1.1 and 1.2 provide pseudonyms and demographic information about the in-depth and exploratory case studies. More detail about site selection, design, and methodology of the case studies, including their approaches to professional development, is contained in Appendix B.

The third strand of this evaluation, a Longitudinal Study of Teacher Change, examines the effects of Eisenhower-assisted and other professional development on teacher practice in mathematics and science. For the in-depth case studies, we interviewed and conducted classroom observations of teachers in three schools an elementary, middle, and high school in each of the 10 districts. We also surveyed all teachers who teach mathematics or science in those schools. We asked these teachers detailed questions about the topics they covered, their goals for student performance, and their participation in professional development activities at three points in time: the fall of 1997, the spring of 1998, and the spring of 1999. The three waves of the survey provide data pertaining to the 1996-97, 1997-98, and 1998-99 school years. The Longitudinal Study of Teacher Change will enable us to examine the extent to which teachers? participation in Eisenhower-assisted and other professional development activities improves instruction over time. Appendix C provides additional information about the sampling, design, and methodology of the Longitudinal Study of Teacher Change.

The three strands of the evaluation were designed to produce an integrated portrait of the Eisenhower program. The study does not describe the program simply from the perspective of its administrators, but also from the perspective of the teachers who participate in program-funded activities. The study supplements self-reported information from teachers and administrators with rich contextual information from case studies. Finally, the study does not rely only on teacher reports of changes in their classroom practice, but also is collecting data on teaching practice at three points in time. Thus, because the evaluation involves a variety of research methods and collects data from groups of individuals who view Eisenhower-assisted activities from different vantage points, it is able to provide an accurate description of program-funded activities and analyses of the features of these activities and their effects on teacher practice.

EXHIBIT 1.1
In-Depth Case Studies

District Pseudonym

Characteristics

Richmond

An urban district in New York State, Richmond serves approximately 10,000 students from a predominantly minority population. Nearly 60 percent of the students are African American, and half of the students qualify for a free or reduced-price lunch.

East City

An urban district in New York City, East City serves a largely immigrant population. Between 25 and 30 percent of its children are taught in Spanish, while many of the other children speak Creole, African languages, or French. One hundred percent of the students qualify for a free or reduced-price lunch.

Maple City

A large metropolitan school district in Ohio, Maple City serves nearly 64,000 students. Nearly 55 percent of these students are African American, and 60 percent are eligible for a free or reduced-price lunch.

Buckeye

A suburban school district in Ohio, Buckeye serves approximately 11,000 children. About 90 percent of the students are white. A very small percentage of students qualifies for a free or reduced-price lunch.

Rhinestone

A metropolitan school district in Texas, Rhinestone serves approximately 13,000 students from a predominantly minority population: 43 percent of the students are Hispanic, 35 percent are African American, and 20 percent are white. Fifty-six percent of the students qualify for a free or reduced-price lunch.

Lone Star

In a large city in Texas, Lone Star serves approximately 65,000 students from a predominantly minority population. Seventy-seven percent of students are Hispanic, and 18 percent are non-Hispanic whites. Thirty-two percent of the students have limited English proficiency, and 67 percent of the students qualify for a free or reduced-price lunch.

Lake Riverside

In an urban fringe of a large city in Washington, Lake Riverside serves 20,000 students in a relatively affluent community. Eighty percent of the students are white, and 15 to 20 percent of the students qualify for a free or reduced-price lunch.

Rainforest

A small logging community in Washington, Rainforest serves 1,570 students, E85 percent of whom are white. Ten percent of the students are Native American, and nearly half of the students qualify for a free or reduced-price lunch.

Weller

A poor rural district in Kentucky?s Appalachian Mountains, Weller serves 500 students. Ninety-nine percent of the students are white, and 60 percent qualify for a free lunch.

Boonetown

An urban fringe of a mid-sized city in Kentucky, Boonetown serves 8,000 students, 95 percent of whom are white. A very small percentage of students qualifies for a free or reduced-price lunch.

EXHIBIT 1.2
Exploratory Case Districts

District Pseudonym

Characteristics

West City

A large, urban district on the west coast, West City serves a predominantly minority population of 61,889 students; fewer than 15 percent of students are white, while nearly half are Asian, and another 20 percent are Hispanic. About half of the district?s students qualify for a free or reduced-price lunch.

Middle City

An urban district in the midwest, Middle City serves nearly 100,000 students. Nearly 60 percent of these students are African American, and almost two thirds qualify for a free or reduced-price school lunch.

South City

A large, urban district in the southeast, South City serves predominantly minority students, many of whom are not native English speakers. Nearly half of the district?s 333,817 students qualify for a free or reduced-price lunch.

Commuteville

A large, suburban county school district in the mid-Atlantic region, Commuteville serves an ethnically diverse population. Just over two thirds of students are white, with almost equal representation of African American, Hispanic, and Asian students. About 12 percent of the students qualify for a free or reduced-price lunch.

Northtown

Northtown is a small city in New England. Its population is predominantly white (about 80 percent of students), and over a third of students qualify for a free or reduced-price lunch.

Countryplace

Countryplace is a consortium of seven school districts in a rural part of the midwest. The population is fairly homogeneous; virtually all of the 6,000 students served in the consortium are white, and less than 20 percent qualify for a free or reduced-price lunch.

13 The evaluation was designed to collect data about six research questions: 1) What types of professional development activities does the Eisenhower program make available to teachers, and to what extent do these activities represent best practices? 2) Who participates in Eisenhower-assisted professional development activities? 3)As designed, planned, and implemented at the state, district and school levels, how does the Eisenhower program fit into the mosaic of professional development and other systemic reform activities? 4) From the teacher's perspective, how do Eisenhower-assisted and other professional development activities combine to provide a coherent, integrated set of learning opportunities? 5) Do teachers' experiences in Eisenhower-assisted professional development activities, in the context of other professional development activities, contribute to teaching practice? and 6) How is the Eisenhower program planned, coordinated, and evaluated at the state, district, and IHE levels? While these research questions guided the evaluation, they do not serve as the framework for reporting findings in this report. Rather, we report analyses from each of our data sets in turn, as depicted in Exhibit 1.3.

14 The telephone interviews of 363 district coordinators represent a response rate of 88 percent of sampled coordinators; the telephone interviews with IHE/NPO project directors represent a response rate of 87 percent of sampled project directors. Details regarding sampling are described in Appendix A.

15 The mail survey ot teachers represents a response rate of 72 percent of sampled teachers. Details regarding sampling are provided in Appendix A.

16 In addition to being selected for their demographic features, the six exploratory sites also represented districts with features that we believed might influence the program's implementation. The six sites were selected to include: one district that participated in the Eisenhower program through a consortium; at least two districts that had IHE-supported projects working in schools int he district; and two districts located in states that received waivers from ED allowing greater proportions of Eisenhower funds to support professional development in areas outside mathematics and science.
-###-


[The Eisenhower Program and the Goals of the 1994 Reauthorization]		 [Table of Contents]
[Overview of This Report]