A r c h i v e d I n f o r m a t i o n

Designing Effective Development: Lessons from the Eisenhower Program - December 1999

Chapter 3

Teachers' Experiences in Eisenhower-Assisted Professional Development Activities

The goal of professional development is to improve teaching. In the last chapter, we discussed the strengths and weaknesses of current teaching practice in mathematics and science, and we identified areas in which professional development appears to be needed. In particular, many teachers do not appear to give sufficient attention to advanced topics, and many attempt to cover too many topics in too little depth. In addition, although many teachers have begun to employ active, discussion-oriented teaching strategies, the use of these strategies is frequently superficial. We now begin to examine the features of professional development that are associated with changes in teaching.

Title II of the Elementary and Secondary Education Act, as amended by the Improving America?s Schools Act of 1994, states as its purpose to increase the quality of professional development and provide professional development to teachers of at-risk students. The legislation states that its purpose is to provide teachers with access to "sustained and intensive high-quality professional development" (Section 2002(1)) that among other things "is of sufficient intensity and duration to have a positive and lasting impact on the teacher's performance in the classroom" (Section 2002(2)(E)). The legislation also seeks to provide professional development that "reflects recent research on teaching and learning" (2002(2)(B)) and "includes strong academic content and pedagogical components" (2002(2)(C)).

The Eisenhower Program also places an emphasis on providing professional development to teachers in Title I schools. The legislation provides that LEAs and SEAs describe in their applications how activities will address the needs of teachers in Title I schools (Section 2205(b)(2)(E) and Section 2208(d)(1)(B)) and includes them in the needs assessment process (Section 2208(b)(2)).

In this chapter, we examine the extent to which Eisenhower-assisted activities meet these provisions of the legislation. We also examine the extent to which supported activities reflect the characteristics of high-quality professional development that have emerged in the literature. Finally, we report on the extent to which Eisenhower-assisted activities meet the standards that have been set in the performance indicators for the program established by the Department of Education, in compliance with the Government Performance and Results Act (GPRA).

EXHIBIT 3.0
Conceptual Framework for This Evaluation

The purpose of this chapter is to identify features of professional development that might be appropriately incorporated in Eisenhower-assisted activities because they have been shown to be effective in improving teaching practice. To this end, we first describe the features of effective, high-quality professional development identified in the literature. Second, we present results from our study describing the extent to which Eisenhower-assisted activities reflect these features. Third, we use our data to assess the extent to which these features are related to teachers? reports of improvement in teaching practice. Our analyses have implications for federal legislation and regulations, which call for Eisenhower-assisted activities to have particular qualities and features. In effect, this study assesses the effectiveness and appropriateness of the current legislation?s language and requirements, given our findings on the importance of particular characteristics of professional development. The framework that we use to model the features and impact of professional development is based on both the professional development literature and the analysis of our survey data.

High-quality Professional Development

Over the past decade, a considerable literature has emerged on professional development, teacher learning, and teacher change.¹ The research literature contains a mix of large- and small-scale studies, including intensive case studies of classroom teaching, evaluations of programs designed to improve teaching and learning, and surveys of teachers about their pre-service preparation and in-service professional development experiences.² In addition, there is a large literature describing "best practices" in professional development, drawing on expert experiences. Despite the size of the literature, however, relatively little systematic research has been conducted on the effects of professional development on improvements in teaching or on student outcomes.

Although relatively little research has been conducted on the effects of alternative forms of professional development, the research that has been conducted, along with the experience of expert practitioners, provides some preliminary guidance about the characteristics of high-quality professional development. (See, in particular, Loucks-Horsley et al., 1998.) Recently, for example, the U.S. Department of Education, drawing on the literature on teachers and teacher learning, identified six features of "best practice" in professional development. According to the Department (1999b, p. 63), effective professional development:

Reflects an image of teaching and learning that embraces high standards for all students. Those who provide professional development must have a clear image of effective classroom learning and teaching. Good professional development is grounded in approaches to teaching that enable all students to achieve high standards, and makes these approaches real and accessible.
Focuses on deepening teachers? knowledge of content and how students learn specific content. Good professional development develops teachers? skill and knowledge regarding the disciplines that they teach and children?s ways of learning at different ages and in different contexts. It is rigorous, relevant, and research-based.
Provides extended, in-depth learning opportunities for teachers. Good professional development promotes learning through, among other things, modeling the methods to be used with students and showing how methods are adapted for different types of students. Like all good teaching, such experiences build on existing knowledge, immerse learners in stimulating processes, allow for teamwork, and spread out over time to permit learners to digest new ideas, try them out, and re-gather for critical feedback. Professional learning is embedded in the life of the school whenever appropriate.
Supports expanded roles for teachers as leaders and colleagues. Good professional development provides opportunities for teachers to serve as mentors, peer coaches, leaders, designers, planners, and facilitators. Such roles encourage collegial relationships in a community of learners. These, in turn, could entail changes in allocations of authority, responsibility, and time in schools.
Links to an educational system?s programs and standards. To strengthen the effectiveness of schools, professional development should be linked to other federal, state or district initiatives. It should be tied to relevant curricula, assessments, and standards.
Is accountable for results. Professional development should be regularly evaluated for its impact on teaching and learning, and evaluation results should be used to support continuous improvement.

Similar characteristics appear in other recent examinations of mathematics and science teaching. For example, James Hiebert, in a review of the research on mathematics teaching and learning conducted for the National Council of Teachers of Mathematics, calls attention to the importance of high standards, content focus, and in-depth learning opportunities for teachers, although he uses somewhat different language. According to Hiebert,

Research on teacher learning shows that fruitful opportunities to learn new teaching methods share several core features: (a) ongoing (measured in years) collaboration of teachers for purposes of planning with (b) the explicit goal of improving students? achievement of clear learning goals, (c) anchored by attention to students? thinking, the curriculum, and pedagogy, with (d) access to alternative ideas and methods and opportunities to observe these in action and to reflect on the reasons for their effectiveness? (1999, p.15).

Although lists of characteristics such as these commonly appear in the literature on effective professional development, there is little direct evidence on the extent to which these characteristics are related to positive outcomes for teachers and students. Some studies conducted over the past decade suggest that professional development experiences that share all or most of these characteristics can have a substantial, positive influence on teachers? classroom practice and student achievement.³ Several recent studies have begun to examine the relative importance of specific dimensions or characteristics of professional development. For example, a number of recent studies suggest that the intensity or duration of professional development is related to the depth of teacher change (Shields, Marsh, & Adelman, 1998; Weiss et al., 1998). Furthermore, there is some indication that professional development that focuses on specific mathematics and science content and the ways students learn such content is especially helpful, particularly for instruction designed to improve students? conceptual understanding (Cohen & Hill, 1998; Fennema et al., 1996). However, few studies have explicitly compared the effects of different forms of professional development on teaching and learning.⁴

Thus, there is a clear need for new, systematic research on the effectiveness of alternative strategies for professional development. The National Research Council, for example, in a review of recent research on the cognitive sciences, teaching, and learning, argues that

Research studies are needed to determine the efficacy of various types of professional development activities, including pre-service and in-service seminars, workshops, and summer institutes. Studies should include professional development activities that are extended over time and across broad teacher learning communities in order to identify the processes and mechanisms that contribute to the development of teachers? learning communities (Bransford et al., 1999, p. 240).

We designed the evaluation of Eisenhower-assisted professional development activities to enable us to examine the relationship between features of professional development that have been identified in the literature and change in teachers? knowledge and skills and classroom teaching practices, as reported by teachers. We integrated and operationalized the ideas in the literature on "best practices" in professional development to create a set of measures or scales describing the characteristics of Eisenhower-assisted activities.

In our analysis of the characteristics of high-quality professional development, we begin by focusing on what we term "structural features." These are characteristics of the structure or design of a professional development activity. We focus in particular on three structural features:

The form or organization of the activity-that is, whether the activity is organized as a reform type, such as a study group, teacher network, mentoring, committee or task force, internship, individual research project, or teacher research center, in contrast to a traditional workshop or conference;
the duration of the activity, including the total number of contact hours that participants are expected to spend in the activity, as well as the span of time over which the activity takes place; and
the degree to which the activity emphasizes the collective participation of groups of teachers from the same school, department, or grade level, as opposed to the participation of individual teachers from many schools.

In addition to these structural features, we focus on three dimensions of the substance or core of the professional development experience. We examine three "core features":

the degree to which the activity has a content focus--that is, the degree to which the activity is focused on improving and deepening teachers? content knowledge in mathematics and science;
the extent to which the activity offers opportunities for active learning, such as opportunities for teachers to become actively engaged in the meaningful analysis of teaching and learning; for example, by reviewing student work or obtaining feedback on their teaching; and
the degree to which the activity promotes coherence in teachers? professional development, by incorporating experiences that are consistent with teachers? goals, aligned with state standards and assessments, and encouraging of continuing professional communication among teachers.

Through our study of the literature and the analysis of our survey data, we have formulated a model characterizing the relationships between the quality of professional development, defined by the structural and core features, and their role in improving teacher outcomes. In this model, the three structural features--type, duration, and collective participation--are conditions that enable or facilitate desire core features of professional development. That is, they set the parameters or context within which teacher learning takes place. The three core features of the professional development activity--content focus, active learning, and coherence-are factors that characterize the processes that occur during a professional development experience. By focusing on specific mathematics and science content, by engaging teachers in active work, and by fostering a coherent set of learning experiences, we hypothesize that a professional development activity is likely to enhance the knowledge and skills and improve the classroom teaching practice of participating teachers. This, of course, serves the ultimate goal of improved student learning.

Data Sources

For the analyses in this chapter, we use information from the Teacher Activity Survey conducted as part of the evaluation. In the spring, summer, and fall of 1998, we surveyed a nationally representative sample of teachers who had attended Eisenhower-assisted activities over the period from July 1 through December 31, 1997.⁵ We carried out the survey by drawing a national probability sample of districts and SAHE grantees. SAHE grantees include institutions of higher education (IHEs) such as universities, four-year colleges, or two-year colleges, and not-for-profits (NPOs), which are organizations such as zoos, museums, and libraries. Districts were sampled in proportion to the number of teachers in the district, and SAHE grantees were sampled in proportion to the size of their Eisenhower grant, based on the logic that SAHE grantees with larger grants would serve larger numbers of teachers. For each district and SAHE grantees drawn into the sample, we collected a complete list of all professional development activities conducted with Eisenhower funds over the period from July through December, 1997. We then drew a sample of two activities in each district or SAHE grantee, with the probability of an activity being selected in proportion to the number of teachers attending the activity. We then randomly subsampled two teachers who attended each activity. We received responses from 1,027 teachers, representing activities supported by Eisenhower funds in 358 districts and SAHE grantees. This produced an overall teacher response rate of 72 percent. ⁶

The survey asked each teacher to provide detailed information about the specific Eisenhower-assisted professional development activity that we drew in our sampling process and that led the teacher to be selected for our sample. Responses are self-reports of teacher experiences and behavior.⁷

In discussing the structural features, quality, and outcomes of Eisenhower-assisted professional development in the chapter, we also include information from our In-Depth Case Studies, where appropriate. The case study data we present include information from six exploratory case studies that we conducted in the spring of 1997; and case studies of ten districts, two from each of five states, that we conducted during the 1997-1998 school year. We chose sites to allow variation according to efforts in state-level reform, the district?s approach to professional development, and demographic and geographic characteristics. ⁸

Organization of Chapter

In the sections that follow, we draw on our national sample of teachers, supplemented with data from the case studies, to examine the extent to which Eisenhower-assisted activities incorporate practices that reflect high-quality, in terms of the three structural features and the three core features of professional development.⁹ Then, we use the data to examine the relative strength of the relationship between these quality features and teachers? reported improvement in knowledge and skills and teaching practices.¹⁰

In many of the exhibits in this chapter, we contrast teachers? experiences in district and SAHE-grantee activities. These two components of the program operate in somewhat different organizational contexts. In particular, as we discuss in more detail in chapters 4, 5, and 6, districts receive Eisenhower funds through an allocation formula, while SAHE grantees compete for funds. In addition, districts generally offer a range of Eisenhower-assisted activities serving a relatively large number of teachers, while SAHE grantees tend to offer a few activities focused on a small number of teachers. As we discuss further in later chapters, these differences in organizational context may help explain differences in teacher experiences in these two components of the program.

This chapter is organized in five sections. The first section examines the structural features of Eisenhower-assisted professional development activities, and the second examines the core features of teachers? professional development experiences in the activities in which they participated. The third section examines the extent to which teachers report that participation enhanced their knowledge and skills and improved their teaching practices, and the fourth examines the relationship between structural features, core features, and teachers? reported enhancement of knowledge, skills, and teaching practices. The fifth section examines the extent to which teachers in Eisenhower-assisted activities teach in high-poverty schools, and the final section summarizes our evidence from participating teachers and draws some initial conclusions about the Eisenhower program.

¹ See V. Richardson & Placier, in press, for a comprehensive review of the literature on teacher learning and professional development.

² See, for example, Cohen (1990) for a recent intensive case study of change in mathematics teaching; Carey and Fretchling (1997) for a program evaluation of exemplary professional activities in science; and U.S. Department of Education (1999a) for a national survey of teachers focused on teacher preparation and qualifications.

³ See, for example, Fennema et al. (1996), an experimental study examining the effects of Cognitively Guided Instruction, an intervention in elementary school mathematics; Wilson and Ball (1991), an intensive case study of two teachers who participated in the Summer Math program; and Cohen and Hill (1998), which describes the relationship between participation in professional development, teaching practice, and student achievement, using survey data from California. See Kennedy (1998) for a review of available randomized studies examining the effects of teacher professional development on student achievement in mathematics and science. See Shields, Marsh, and Adelman (1998) for a recent examination of the effects of the NSF SSIs on classroom practice in mathematics and science; and Weiss et al. ( 1998) for an examination of the effects of the NSF LSCs.

⁴ Kennedy (1998) and Cohen and Hill (1998) are among the few examples of studies that compare the relative effectiveness of different forms of professional development. Both studies conclude that professional development focused on the teaching and learning of specific mathematics and science content is more effective than more general professional development.

⁵ The term "Eisenhower-assisted activities" reflects the fact that district Eisenhower funds can support professional development activities in a number of ways. Eisenhower funds may be used to support all costs associated with activities, provided that these activities are allowed in the legislation (See Section 2210). Alternatively, Eisenhower funds may pay for only some of the allowable costs associated with an activity. This is a common occurrence, since the legislation encourages cost sharing of Eisenhower-assisted professional development activities with those funded by other programs (Section 2209).

⁶ See Appendix A for a more complete discussion of the sampling plan for Teacher Activity Survey. All parameter estimates reported in the chapter incorporate weights reflecting the sampling plan. Reported p-values and the standard errors on which they are based, however, do not reflect the clustering, stratification, and variance in weights incorporated in the design. Analyses that take these elements of the complex sample design into account have been carried out, and the results are nearly identical to those reported in the chapter.

⁷ The study was designed to maximize our capacity to provide a national description of Eisenhower-assisted activities and to assess the relationship between characteristics of activities and teachers' self-reported change in knowledge and skills and teaching practices. The study was not designed to allow us to examine the relationship between professional development and student achievement.

⁸ See Appendix B for a more complete discussion of the design of the In-Depth Case Studies.

⁹ Results for some analyses reported in this chapter were reported earlier in U.S. Department of Education (1999b). The earlier results differ from results in this report because they were preliminary, unweighted, and did not include the full sample of teachers and districts. Results are considered to be statistically significant if the p-value is .05 or smaller.

¹⁰ The survey results in this chapter are based on teachers' self-reports on change in knowledge, skills, and teaching practices. We are also collecting longitudinal data on a sample of teachers as part of the evaluation. These longitudinal data will provide stronger evidence of change in classroom teaching, and they will permit us to test the relationships that we discuss in this chapter. Analyses of the longitudinal data will appear in the third evaluation report, scheduled to be available in the spring of 2000.

-###-

[Chapter 2 Summary and Conclusions]

[Structural Features]