A r c h i v e d  I n f o r m a t i o n

Designing Effective Development: Lessons from the Eisenhower Program - December 1999

Chapter 7

Teachers' Experiences in Eisenhower-Assisted Activities

We examined teachers' experiences in Eisenhower-assisted activities in three ways. First, we assessed the effectiveness of Eisenhower-assisted activities by asking teachers to report on the extent to which participation enhanced their knowledge and skills. Next, we examined the quality of Eisenhower-assisted activities, be assessing the extent to which supported activities share features of quality identified in the literature on professional development. Finally, we used our data to assess the strength of the relationship between features of the activities in which teachers participated and teachers' self-reported outcomes.

Effectiveness of Eisenhower-Assisted Professional Development Activities

To measure the effectiveness of Eisenhower-assisted professional development activities, we asked teachers to report on the extent to which participation enhanced their knowledge and skills in six domains: (1) in-depth knowledge of mathematics/science; (2) curriculum; (3) instructional methods; (4) approaches to assessment; (5) use of technology; and (6) approaches to diversity.³

When asked directly, many teachers in SAHE-grantee activities and somewhat fewer teachers in district activities report that participation in Eisenhower-supported professional development has led to enhanced knowledge and skills and changes in their classroom teaching practice. For enhanced knowledge and skills, we asked teachers questions about six domains, ranging from enhanced knowledge of mathematics and science, to instructional methods, to approaches to diversity. For example, 48 percent of teachers in district activities and 68 percent of teachers in SAHE-grantee activities report enhanced in-depth knowledge of mathematics or science; and 63 percent of teachers in district activities and 79 percent of teachers in SAHE-grantee activities report enhanced knowledge of instructional methods. Percentages are lower for use of technology and approaches to diversity. Similarly, we asked about improvement in classroom practice, and the percentages were parallel. Once again, there is considerable variability among activities in reported change in practice, and teachers in SAHE-grantee activities report somewhat more change in practice than do teachers in district activities. Overall, our data on knowledge and skills and change in teaching practice suggest that teachers believe they are benefiting from their participation in Eisenhower-assisted professional development.

A comparison of our data for SAHE-grantee activities with the results obtained for 34 exemplary summer institutes in mathematics and science, supported by the National Science Foundation, the Department of Education, and other agencies, indicates that teachers participating in SAHE-grantee activities report enhancement of knowledge and skills in mathematics and science content roughly comparable to the results for the 34 exemplary activities.⁴ SAHE-grantee activities meet benchmarks for reported teacher outcomes set by other exemplary professional development programs. Teachers participating in district Eisenhower activities, however, show weaker results than do teachers in the 34 exemplary activities.

The Quality of Eisenhower-assisted Activities

Our analysis of the quality of Eisenhower-assisted professional development activities, which appears in Chapter 3, is grounded in the recent literature on the characteristics of professional development. Over the past decade, a considerable literature has emerged on professional development, teacher learning, and teacher change (Corcoran, 1995; Darling-Hammond, 1995; Hargreaves & Fullan, 1992; Hiebert, 1999; Lieberman, 1996; Little, 1993; Loucks-Horsley et al., 1998; Richardson, 1994; Sparks & Loucks-Horsley 1989; Stiles, Loucks-Horsley, & Hewson, 1996). The research literature contains a mix of large- and small-scale studies, including intensive case studies of classroom teaching, evaluations of programs designed to improve teaching and learning, and surveys of teachers about their pre-service and in-service preparation and in-service professional development experiences. In addition, there is a large literature describing "best practices" in professional development, drawing on expert experiences.

Despite the size of the literature, however, relatively little systematic research has been conducted on the effects of professional development on improvements in teaching or on student outcomes, and very little has been conducted comparing the effects of alternative forms of professional development. The research that has been conducted, however, along with the experience of expert practitioners, does provide some preliminary guidance about the characteristics of high-quality professional development (See Loucks-Horsley, 1998.) In particular, several recent studies suggest that professional development that focuses on specific mathematics and science content and the ways students learn such content is especially helpful (Cohen and Hill, 1998; Kennedy, 1998). To measure the quality of Eisenhower-assisted activities, we integrated and operationalized the ideas in the literature on "best practices" in professional development.

As we reported in Chapter 3, we focus on three "structural features," or characteristics of the structure of a professional development activity. These structural features include:

• the form or organization of the activity—that is, whether the activity is organized as a reform type, such as a study group, teacher network, mentoring relationship, committee or task force, internship, individual research project, or teacher research center, in contrast to a traditional workshop or conference;

• the duration of the activity, including the total number of contact hours that participants are expected to spend in the activity, as well as the span of time over which the activity takes place; and

• the degree to which the activity emphasizes the collective participation of groups of teachers from the same school, department, or grade level, as opposed to the participation of individual teachers from many schools.

In addition to these structural features, we focus on three "core features" or characteristics of the substance of the professional development experience itself:

• the degree to which the activity has a content focus—that is, the degree to which the activity is focused on improving and deepening teachers? content knowledge in mathematics and science;

• the extent to which the activity offers opportunities for active learning—that is, opportunities for teachers to become actively engaged in the meaningful analysis of teaching and learning, for example, by reviewing student work or obtaining feedback on their teaching; and

• the degree to which the activity promotes coherence in teachers? professional development, by incorporating experiences that are consistent with teachers? goals, aligned with state standards and assessments, and encouraging of continuing professional communication among teachers.

We used these structural and core features to judge the extent to which the Eisenhower program supports high-quality professional development. We found that most Eisenhower-assisted activities are traditional in form, such as workshops, courses, or conferences; and relatively few Eisenhower-assisted activities are reform types of activities, such as study groups, networks, or mentoring relationships. About 79 percent of teachers in district activities are in traditional types of activities—primarily workshops and conferences. About 76 percent of teachers in SAHE-grantee activities are also in traditional activities—primarily workshops and college courses. In contrast, approximately 22 percent of teachers in district activities and 26 percent in SAHE-grantee activities are in reform activities, including teacher networks, study groups, mentoring, committees and task forces, internships, and resource centers.

On average, SAHE-grantee activities are of longer duration than district activities. The average amount of time teachers in district activities report spending in Eisenhower-assisted activities is 25 hours, compared to 51 hours for teachers in SAHE-grantee activities. However, while district activities are shorter than SAHE-grantee activities, the average length of district activities, in hours, has approximately doubled since the last evaluation was conducted in 1988-89.⁵ In addition, a higher percentage of SAHE-grantee activities than district activities span an extended period of time. For example, 46 percent of teachers in SAHE-grantee activities are in activities lasting at least six months, including 2 percent lasting more than one year, 20 percent lasting 10-12 months, and 24 percent lasting 6-9 months. Twenty percent of teachers in district activities are in activities lasting at least six months.

On the third structural feature, collective participation, districts outperform SAHE grantees. Although relatively few district or SAHE-grantee activities emphasize the collective participation of teachers from the same department, grade level, or school, district activities give more emphasis to collective participation than do SAHE-grantee activities. Twenty percent of teachers in district Eisenhower-assisted activities report participating with other teachers in their department or grade level, compared to 7 percent of teachers in SAHE-grantee activities; and 19 percent of teachers in district activities report participating with all teachers in their school, compared to 11 percent of teachers in SAHE-grantee activities.

On the last three dimensions of quality—content focus, active learning, and coherence—we find that SAHE-grantee professional development is more likely to place an emphasis on these features than district professional development.

Two thirds of teachers participating in SAHE-supported Eisenhower activities participate in activities that place a major emphasis on content, compared to fifty-one percent of teachers in district Eisenhower-assisted activities. Similarly, the percentage of teachers reporting a major emphasis on each of 18 separate indicators of active learning is higher for SAHE grantees than for districts, but relatively few teachers in either district or SAHE-grantee activities report some elements of active learning. For example, only 5 percent of teachers in district activities and 16 percent of teachers in SAHE-grantee activities report that their teaching was observed by the activity leader.

Teachers in both district and SAHE-grantee activities report that the activities have elements that promote coherence with other aspects of their professional experiences. For example, more than three quarters of teachers in both district and SAHE-grantee activities report that their activities are aligned with state and district standards. Somewhat more teachers in SAHE-grantee than district activities, however, report that their Eisenhower activities built on prior professional development (39 percent compared to 31 percent) or were followed up with later activities (70 percent compared to 53 percent).

Our main finding is that there is enormous variability in quality among Eisenhower-assisted activities. This is not surprising, given that the Eisenhower program operates as a funding stream within broad guidelines. A great deal of discretion is left to districts and SAHE grantees to decide on the nature of the professional development that they will provide, though SAHE grantees have to design programs within the constraints of the SAHE's competition. The variability in quality indicates that some districts are providing the same kinds of high-quality professional development activities that SAHE grantees provide. For example, although many district activities are short, 2 percent of teachers in district activities are in activities spanning more than one year, 8 percent are in activities lasting 10-12 months, and 10 percent are in activities lasting 6-9 months. That some districts provide activities of extended duration, with collective participation, a major focus on content knowledge, a major emphasis on active learning, and coherence with teachers' other experiences, represents an "existence proof" that it is possible for districts to provide such activities.

We do not know, however, whether all districts possess the organizational capacity and staffing to provide quality professional development. Districts lacking capacity may well be able to purchase high-quality professional development (e.g., from institutions of higher education) if they are aware of and insist upon the dimensions of quality we have identified.

Finally, the average differences in quality we observe between district and SAHE-grantee activities are associated with parallel differences in cost. Thus, we conclude that high-quality professional development is more expensive than lower-quality professional development. SAHE grantees spend over twice as much per teacher participant as do districts. We estimated that SAHE grantees spend about $512 per participation, in comparison to $185 per participation for districts.⁶ High-quality professional development is more expensive.

Why the SAHE-grantee activities are of higher quality, on average, than the district activities is not completely clear. One possible explanation is that districts receive a formula allotment while the SAHE grantees must compete for their award. Both the competition and the SAHE guidelines for the competition may push in directions of higher quality.⁷ In addition, IHE/NPOs may, on average, possess more state-of-the-art knowledge about the attributes of high-quality professional development than districts do, and they may have greater capacity to deliver professional development with these attributes. Finally, as noted, high-quality professional development is considerably more expensive per participation. Undoubtedly, districts feel a responsibility to provide professional development to all of their teachers. This may push them in the direction of professional development with lower costs per participation.

Features of Professional Development that Promote Improvements in Teaching

We used our national probability sample of teachers who participated in Eisenhower-assisted professional development to estimate the strength of the relationships among features of the professional development and self-reported teacher outcomes?enhanced knowledge and skills and changed teaching practice. Our analyses, reported in Chapter 3 (Exhibit 3.21), estimate standardized path coefficients while controlling for the subject area of the professional development experience (mathematics or science); school poverty, percent minority, and grade level; and teacher gender, certification, and years of experience.

We find that activities with more positive structural features tend to provide professional development experiences with more positive core features, which in turn tend to produce more positive teacher outcomes. In particular, activities of longer duration, both in time span and in contact hours, tend to place more emphasis on content than shorter activities, provide more opportunities for active learning, and provide more coherent professional development. Activities with more collective participation also tend to provide more opportunities for active learning and offer more coherent professional development. The three core features, in turn, all have independent effects on teachers? reports of enhanced knowledge and skills. Teachers who participate in activities that place a major emphasis on mathematics and science content, for example, are substantially more likely to report enhanced knowledge and skills than teachers in activities that do not emphasize content. (For a more complete description, see Chapter 3.)

Thus, our results expand the existing literature by providing empirical support for the contention that professional development that emphasizes content knowledge, active learning, and coherence leads to enhanced teacher knowledge and skill and change in teacher practice. In addition, our results highlight the importance of professional development that engages teachers for an extended number of contact hours, and that spans several months, as opposed to being concentrated in a few days or weeks.

Our results also support the literature in indicating that reform types of professional development tend to produce more positive outcomes than traditional types, but the effect is smaller than we had anticipated. In our analyses, we find effects of the distinction between traditional and reform activities, and they are in the predicted direction, but they are generally not direct effects on teacher outcomes. Rather, the effect of reform versus traditional professional development activities operates indirectly through the structural and core features identified above. That is, reform activities tend to produce better outcomes primarily because they tend to be of longer duration. Traditional and reform activities of the same duration tend to have the same effects on reported outcomes. Further, we find a large amount of variability within both traditional and reform-type activities. Traditional activities sometimes have positive characteristics, such as opportunities for active learning, and reform activities sometimes have negative characteristics, such as limited duration.

Further, our analyses show that the difference between districts and SAHE grantees, in terms of enhanced teacher knowledge and skill and greater change in teacher practice, is explained almost entirely by the SAHE grantees' placing a greater emphasis upon duration, subject-matter content, active learning, and coherence. In short, if districts placed a greater emphasis upon these characteristics, we would hypothesize that teachers would report their knowledge and skill enhanced to the same extent as teachers in SAHE-grantee activities, and that they would be just as likely to report changing their teaching practice.

⁷ We were not able to conduct a systematic analysis of SAHE competitions.

-###-

[Current Teaching Practices in Mathematics and Science]

[District Management and Operation of Eisenhower-Assisted Activities ]