A r c h i v e d  I n f o r m a t i o n

Designing Effective Development: Lessons from the Eisenhower Program - December 1999

Chapter 6

Implementing the Vision: IHE/NPO Procedures for the Continuous Improvement of Professional Development Activities

Section Findings

• For SAHE grantees in states or districts with indicators, state indicators have a much greater effect on SAHE grantees than district indicators do.

• About two-thirds of SAHE grantees conduct needs assessments of teachers; the most popular methods of conducting these needs assessments are teacher surveys and informal conversations with teachers.

• Almost all SAHE grantees evaluate their activities. Of those that evaluate, almost all use teacher surveys as a method of evaluation; the majority also use counts of teacher participations and observations of teachers; less than a third use student achievement as an evaluation measure.

• Research/doctoral universities use fewer continuous improvement methods than other types of IHEs, and IHE projects housed in mathematics or science departments use significantly fewer continuous improvement methods than IHE projects housed in either education or other (i.e., non-mathematics, science, or education) departments.

In addition to alignment and coordination, another method of shaping professional development activities involves the use of indicators, needs assessments, and evaluations to help design and implement activities. As we noted earlier, the legislation?s overall purposes apply to SAHE grantees; the law calls for SAHE grantees to provide professional development that "creates an orientation toward continuous improvement throughout the school" (Section 2002(2)(F)). The law requires that SAHE grantees establish performance indicators (Section 2208(a)(2)), conduct an assessment of teachers needs (Section 2208(b)(1)), and report to the state on the progress toward meeting the indicators (Section 2401(b) and Section 2208(d)(1)(G)). These provisions reflect the "continuous improvement" paradigm that the federal government has adopted, embodied by the Government Performance and Results Act. Research has suggested that such continuous improvement methods are important for high-quality professional development. The quality of professional development increases when teachers and providers are held accountable for outcomes of professional development (Loucks-Horsley et al., 1998) and when professional development is evaluated based on teacher and student outcomes (Guskey, 1997).

We examined SAHE-grantee continuous improvement efforts by asking the project directors about their use of indicators, needs assessments, and evaluations in the design and implementation of their professional development activities.

Presence and Use of Performance Indicators

We asked SAHE-grantee project directors whether they are aware of any performance indicators for professional development set by the state and/or district, and if so, whether these indicators affect their Eisenhower projects. Exhibit 6.21 illustrates that 50 percent of teachers participating in SAHE-grantee projects are in projects whose directors report being aware of state performance indicators; of these, 77 percent are in projects whose directors also report that the Eisenhower project is affected by the state indicators. Forty percent of participating teachers are in projects that work with districts that have performance indicators; of these, only 22 percent are in projects whose directors say that the indicators affect the project.¹⁵ Thus, state indicators seem to have more effect on SAHE-grantee projects than do district indicators. This is consistent with findings reported in Chapter 5, which show that districts are also more affected by state than district indicators, and with findings reported earlier in this chapter, which show that state standards and assessments play a larger role in project design than district standards or assessments.

EXHIBIT 6.21
Percent of Teachers Participating in SAHE-grantee Projects, by Status and Effect of District and State Performance Indicators on the Project (n=92)

In our case-study interviews, an IHE project director in Kentucky said that state assessments have affected their projects more than district assessments, because schools in the state are held accountable for meeting the state goals, and the IHE projects aim to help teachers in this effort. Also, the IHE director said that district-level indicators do not affect the Eisenhower project directly, because the project works with many districts. Nevertheless, district indicators may have an indirect influence, because they are usually in line with the state indicators. An IHE project director in Texas supported this view, reporting that district assessments have not affected the design of the Eisenhower program because their project is based on the national standards.

Assessing the Needs of Teachers

Another dimension of continuous improvement is assessing teachers? needs in order to identify potential areas for improvement. About 66 percent of participating teachers are in projects that say they formally assess teachers? needs for professional development (data not shown). We asked SAHE-grantee project directors how they assess teachers? needs for professional development. Options were 1) with a survey of teachers, 2) with meetings of teacher representatives, 3) with a survey of principals or department chairs, 4) with measures of student performance, and 5) with informal conversations. Exhibit 6.22 shows that the most common methods are surveying teachers (51 percent of participating teachers are in projects that use this method) and informal conversations (50 percent). Meetings with teacher representatives (41 percent) are less frequent. Less than a quarter of participating teachers are in projects that indicate that they use measures of student performance (24 percent) or surveys of principals or department chairs (19 percent) to assess teachers? professional development needs. Thus the most common source of needs assessment information is teachers. While it is important to consider teachers? perceptions of their own needs as well as their students? needs, it may also be useful to combine this with information from other sources, such as classroom observations.

EXHIBIT 6.22
Percent of Teachers Participating in SAHE-grantee Projects That Use Different Strategies to Assess Teachers? Professional Development Needs (n=92)

Compared to the SAHE-grantee project directors, district coordinators report much higher levels of needs assessments of every kind (see Exhibit 5.12 in Chapter 5). There are several possible explanations for the higher levels of needs assessment in districts. One explanation is that, as shown in Exhibit 6.17, 65 percent of participating teachers are in projects that rely on district needs assessments to design their professional development program, rather than designing their own. Other reasons may be that districts have better access to teachers than do SAHE grantees, making assessment methods such as "informal conversations" much easier to implement. Districts can more easily identify the pool of teachers for which the professional development will be designed, whereas SAHE grantees may be working with several districts and it may not be clear from where participants will be drawn. Further, many IHE-sponsored activities are college courses, which may be designed to fulfill needs of teachers who are students at the college, rather than to fulfill the needs of teachers in districts with which the IHE may be working. Also, college professors may rely on theory and research to identify teachers? needs, rather than surveying teachers themselves.

Evaluating Professional Development Activities

The third component of continuous improvement that we measured is evaluation of the professional development activity. We asked SAHE grantees if they evaluate their Eisenhower-assisted professional development; ninety-three percent of participating teachers are in projects whose project directors say that they evaluate these activities. We then asked SAHE grantees how they evaluate their professional development. Alternatives include 1) by the number of teachers participating in professional development, 2) with a teacher satisfaction survey, 3) with observations of teachers, and/or 4) with student achievement. As Exhibit 6.23 illustrates, the most common method of evaluation is a teacher satisfaction survey, which almost all SAHE grantees use; ninety-two percent of participating teachers are projects that use surveys. To a lesser extent, SAHE grantee project directors report using a count of the number of teachers who participate (61 percent of participating teachers are in projects that use this method) and observations of teachers (59 percent) as methods for evaluating their Eisenhower-assisted professional development activities. The least common method of evaluation is using student achievement as a measure of the professional development activity; 31 percent of participating teachers are in projects that use this method.

One IHE project director in Texas offered a possible explanation for the relatively infrequent use of student achievement as a method of evaluating professional development. She explains that science is addressed infrequently on state tests and therefore the only regular indicator of student achievement in science that is available is student grades. This may explain why student achievement measures are not used for science, but it would not explain why they are not used for mathematics. It may be that IHE/NPOs do not use student achievement measures because of the complexity involved in identifying comparable measures over time and separating the effects of professional development from other effects. Using student achievement to determine effects of professional development is a complex, long-term undertaking involving longitudinal studies of teachers and students; SAHE grantees may not have the resources or capacity to launch such evaluations. Given time, expertise, and resource constraints, a more appropriate method of evaluation is for SAHE grantees to evaluate their professional development activities based on the characteristics of high quality that we have shown in this report to be indicative of effective professional development. For example, SAHE grantees could document the duration of their activities, and the opportunities for active learning and collective participation. In addition, SAHE grantees might evaluate the extent to which participants gain the intended knowledge and skills or show improvements in classroom teaching.

EXHIBIT 6.23
Percent of Teachers Participating in SAHE-grantee Projects That Use Different
Methods for Evaluating Activities (n=92)

Overall Continuous Improvement

Taken together, indicators, needs assessments, and evaluation strategies can be components of a process of continuous improvement in the design of professional development. To examine whether SAHE grantees? use of continuous improvement methods differs significantly by institution type or affiliation, we created scales for indicators, needs assessments, and evaluation, rescaled each to have a range from 0 to 1, then created an additive composite of all three scales, with a range of 0 to 3. Exhibit 6.24 shows the results for this continuous improvement composite. It indicates great variation in the extent to which SAHE grantees use continuous improvement strategies. The data also indicate that nonresearch/doctoral-granting institutions use significantly more continuous improvement strategies than do research/doctoral-granting institutions. As described earlier, research/doctoral-granting institutions may be more likely to design their Eisenhower projects to reflect current research agendas of the faculty, and therefore may be less likely than other types of colleges and universities to work with districts to provide activities tailored to the needs of their teachers, which would require more needs assessments and evaluation. Our data also show that IHE projects housed in education departments engage in significantly more continuous improvement activities than IHE projects housed in mathematics or science departments. As we discussed earlier in the chapter, professors in education departments may have closer relationships with districts than mathematics or science professors. This in turn may increase their access to teachers in the district, for purposes of getting feedback on how professional development activities might apply to district indicators, and for gaining access to teachers to assess needs and follow-up with evaluative measures. Also, education professors are more likely than mathematics or science professors to be social scientists and to use social science methods upon which the process of continuous improvement is based.

Exhibit 6.24
Extent of SAHE-grantee Continuous Improvement Efforts, Overall and by
Institution Type and Departmental Affiliation (n=92 )

Summary: Continuous Improvement

Our data from SAHE-grantee project directors show that grantees are much more likely to use state indicators than district indicators. This parallels our finding reported earlier in this chapter that SAHE-grantee projects are more likely to be aligned with state than district standards and assessments. These findings may indicate that the SAHE competitive-proposal process helps to ensure that SAHE-grantee-provided professional development is linked to statewide reform. But the results also imply that the law?s requirements that SAHE grantees develop cooperative agreements and work closely with the district do not translate into the use of district standards, assessments, and indicators. Further, we find that while most SAHE-grantee project directors conduct needs assessments and evaluations, these processes typically are based on some form of feedback from teachers, such as surveys or informal conversations. Observations of teachers or measures of student achievement, which may be more objective ways of assessing needs and evaluating outcomes, are less commonly used for these purposes. This may, in part, be due to the difficulty of obtaining these measures, especially student achievement. Finally, analysis of our continuous improvement composite indicates that projects in research/doctoral institutions and mathematics/science departments use continuous improvement processes significantly less than other institutions and education departments, respectively.

[Building a Vision for Professional Development]

[Differences in SAHE-Grantee Projects by Institution Type and Departmental Affiliation]