A r c h i v e d  I n f o r m a t i o n

Designing Effective Development: Lessons from the Eisenhower Program - December 1999

Chapter 5

Implementing the Vision: District Procedures for the Continuous Improvement of Professional Development Activities

Section Findings

• Less than one-third of the nation?s teachers are in school districts that have developed performance indicators to help them track the quality and effectiveness of professional development activities. Of those teachers who are in districts that do have indicators, less than a quarter are in districts that collect data on the indicators.

• The majority of the nation?s teachers are in districts that assess teachers? needs formally; nearly all teachers are in districts that use teacher surveys to do so, and most also use less formal methods such as meetings or conversations.

• Most of the nation?s teachers are in districts that evaluate the effectiveness of Eisenhower-assisted professional development activities. These evaluations almost always involve a teacher satisfaction survey; they are less likely to involve formally observing teaching practice or assessing effects on student achievement.

• Districts provide more types of guidance (e.g., assistance in interpreting Title II rules and in developing professional development plans) to schools than to other providers of professional development. Among all types of support that districts provide, they are least likely to focus on data-driven support, such as evaluations and indicators.

The Eisenhower legislation reflects the "continuous improvement" paradigm adopted by the federal government for all of its programs. Specifically, the legislation states that "a local educational agency shall set specific performance indicators for improving teaching and learning through professional development" (Section 2208(a)(2)). This provision applies to all of the district?s professional development, not just the activities supported with Eisenhower funds. The legislation also requires that each LEA "submit a report to the State every three years? regarding the progress of such agency toward performance indicators? as well as on the effectiveness of [the LEA?s] activities?" supported with Eisenhower funds (Section 2401(b)). These requirements are consistent with experts? views that accountability for the outcomes of professional development is a key component of high-quality professional development (Loucks-Horsley et al., 1998).

Performance indicators are designed to evaluate district professional development efforts by establishing measurable benchmarks to track progress toward the district?s goals and objectives. Of course, having such indicators then implies a process of decision making in school districts that is grounded in objective data. Advocates of education reform are increasingly promoting the value of school districts? using data to make decisions about directions for teaching and learning (e.g., Bernhardt, 1998). Two provisions of the Eisenhower legislation that have the potential for encouraging districts to produce data regarding their professional development activities are the provision for assessing the needs of teachers, and the provision for assessing Eisenhower-assisted activities.

The Eisenhower legislation requires that the LEA include in its application an assessment of local needs for professional development, as identified by local education agency and school staff (Section 2208(b)). This type of needs assessment of teachers is an important first step in planning a professional development strategy that accurately addresses the strengths and weaknesses of teachers in a particular district (Loucks-Horsley et al., 1998).

Evaluating the effectiveness of Eisenhower-assisted professional development activities also should contribute to districts? tracking of progress toward professional development goals. As mentioned above, the legislation requires districts to both assess progress toward established indicators for the district as a whole, as well as to report on the effectiveness of Eisenhower-assisted district activities. Guskey (1997) notes that commonly used measures for evaluating the effects of professional development include participants? reactions to the experience; participants? actual use of knowledge and skills they have gained; and the impact of participants? changes in knowledge and skills on student learning. Guskey argues that studies of professional development ought to focus less on teacher perceptions and reactions and place greater emphasis on teacher and student outcomes of professional development.

Thus, Eisenhower requirements for establishing indicators, assessing the needs of teachers, and evaluating Eisenhower-assisted activities are some of the ingredients in the legislation that could support a data-based continuous improvement process. Knowledge and use of indicators should provide target goals and benchmarks for measuring progress. Information about teachers? needs should assist in setting the goals and objectives for professional development. Evaluation data should provide one means for determining whether professional development activities are moving teachers toward these goals.

However, to have an effect on the design of professional development activities, continuous improvement means more than establishing goals, measuring progress toward these goals, and evaluating professional development activities. Continuous improvement also means communicating with schools and teachers about state and district standards and assessments, sharing the data collected from needs assessments and evaluation, and establishing goals and indicators for professional development. The term continuous improvement implies a "feedback loop" in which data about progress are part of continuous communication, and where data become part of a discussion about strengths and weaknesses, and future strategies and decisions.

Thus, in addition to establishing indicators, assessing teacher?s needs, and evaluating the effectiveness of professional development activities, another important role that districts may have is to offer guidance, support, or technical assistance to those who are involved in planning and implementing professional development. Schools often have a role in planning and implementing professional development activities, and districts may play a role in helping to familiarize schools with the standards and expectations of districts, state administrators, and policy makers, as well as with data from district data collection efforts. Furthermore, in addition to school staff, district staff, outside consultants, or individuals from other organizations such as teacher centers or regional service centers operated in some states may be providers of professional development activities. Some providers may be unfamiliar with the district?s vision and goals. In these cases, district data about their performance, and guidance about district goals and policies, could help these providers integrate their activities with district goals and standards. District support to school staff and other providers of professional development can help to enable teachers and schools to have professional development choices that fit within the state and districts? overall vision and focus. In requiring districts to establish goals and objectives for professional development, assess needs of teachers, track progress toward goals, and evaluate the effects of professional development activities, the Eisenhower legislation highlights the role of the district in supporting professional development activities. In these provisions, the legislation is consistent with recent literature on school reform, which also emphasizes the critical role of school districts in guiding professional development (Elmore & Burney, 1996; Spillane & Thompson, 1997).

In this section, we use our national profile of Eisenhower coordinators to examine the aspects of Eisenhower-assisted professional development that relate to performance indicators, needs assessments, evaluations of Eisenhower-assisted activities, and the provision of guidance to schools and professional development providers. We describe how successful districts are in meeting the Title II requirements for having performance indicators in place and collecting information on these indicators, and examine whether districts know about and use state-level performance indicators in their evaluation and improvement process. We describe the ways in which districts collect information about needs for professional development, and whether and how districts evaluate their Eisenhower-assisted professional development activities. Finally, we describe the ways in which districts provide guidance to schools and professional development providers regarding Eisenhower-assisted professional development activities.

Presence and Use of Performance Indicators

We asked Eisenhower coordinators if their district had developed, or was currently developing, performance indicators for professional development. Data from the national survey of district Eisenhower coordinators are somewhat discouraging with respect to district response to the requirements for having and using performance indicators. Some district Eisenhower coordinators continue to be unaware of the requirements, or, if they are aware of them, they have not yet acted on them. As Exhibit 5.4 illustrates, less than a third (32 percent) of the nation?s teachers are in districts that have performance indicators in place.

Of teachers who are in districts that have already developed performance indicators (32 percent), 60 percent are in districts that report that they collect data on those indicators to measure progress; an additional 40 percent of teachers are in districts that say they have plans to do so (data not shown). In other words, fewer than one in five, or about 18 percent, of teachers are in districts that currently collect data on performance indicators that they have established to guide their professional development efforts.

While most of the nation?s teachers are in districts that have not yet developed district-level performance indicators, district efforts to plan, evaluate, and track the progress of their professional development activities could benefit, in principle, from indicators developed by their states. However, a majority of teachers are in districts where Eisenhower coordinators are not aware of any performance indicators developed at the state level; only 34 percent of teachers are in districts where coordinators report that they knew of state-level indicators. Of these, 70 percent of teachers are in districts in which district Eisenhower coordinators report that the state indicators affect their use of Eisenhower funds or other district activities (data not shown). In other words, less than 25 percent of the nation's teachers are in districts that report being affected by their states' indicators for professional development.

EXHIBIT 5.4
Percent of Teachers in Districts According to Status of District Performance Indicators for Professional Development (n=363)

Just under 85 percent of teachers are in districts where coordinators say that they formally assess teachers? professional development needs (data not shown). As Exhibit 5.5 shows, most of the districts use several methods for assessing needs. Teacher surveys are the most popular method; 80 percent of teachers are in districts that formally assess needs using this method. Between 70 and 76 percent of teachers are in districts that use meetings with teacher representatives, surveys of the principal or the department chairperson, and informal conversations, and 65 percent of teachers are in districts that use measures of student performance to determine teachers? professional development needs. Our survey data do not yield information on how often needs assessments of the various types are used or how heavily districts rely on these different methods.

In our case-study districts, however, we find examples of how these approaches to assessing needs are used. Six of our case sites conduct some type of teacher survey to identify teacher needs for professional development. For example, needs assessment appears to rely exclusively on a teacher survey in Weller, Kentucky. There, teachers are asked about future needs for professional development at the end of their evaluation of professional development activities. Last year, the following were among the most frequently mentioned needs: more planning time in order to implement new ideas; information on how to use and integrate technology into the curriculum; strategies to work with students with special needs; information on working in classrooms with students of varying ages and ability levels; help in collaborative and cooperative learning; time management; new and current literature on teaching and learning; and stress management. Such information, and evaluations of teacher satisfaction with professional development activities, are the basis for planning Eisenhower-assisted activities, which in this district is done almost entirely by the Eisenhower coordinator herself.

EXHIBIT 5.5
Percent of Teachers in Districts That Use Different Methods to Assess Teachers? Professional Development Needs (n=363)

These examples of the variation of needs that teachers identify highlight the challenge in aligning teachers? needs with district standards, and addressing potential differences between the two sources. Relying exclusively on teachers? assessments of their own needs may be problematic, because teachers? perceptions of their weaknesses are influenced by their instructional philosophy and goals, which may not be consistent with district standards and goals.

Student test scores are the main source of needs assessment data in some districts, though sometimes they are used in combination with other sources of information. For example, in Rhinestone, Texas, the needs of teachers are determined by a combination of informal conversations, in person and by telephone, "walk throughs" by district staff in order to observe and talk with teachers, and an analysis of TAAS scores. Teachers call district staff to express their needs, and their opinions about the types of professional development that they would like. But, according to district administrators, academic achievement, as measured by the Stanford 9 and TAAS scores, often determine who is targeted for professional development and the type of professional development that is needed. District administrators indicate that reliance on student achievement tests to guide professional development needs is more possible for mathematics than science in some districts, because students are tested more often in mathematics than in science.

Evaluating Professional Development Activities

Another mechanism for continuous improvement is evaluation. Most teachers (93 percent) are in districts in which district Eisenhower coordinators report that they evaluate their Eisenhower-assisted professional development activities (data not shown). Given that assessment of these activities against performance indicators is required in the authorizing legislation, it is perhaps surprising that as many as seven percent of teachers are in districts that acknowledge that they do not evaluate Eisenhower-assisted professional development.

Districts report that they evaluate Eisenhower-assisted professional development in a number of ways. We asked district coordinators to indicate which of the following ways they evaluate the activities: 1) by number of teachers participating in professional development, 2) with a teacher satisfaction survey, 3) with observations of teachers, and 4) with student achievement scores. We did not ask the frequency with which districts use these evaluation methods; districts may use some of them only occasionally.

Exhibit 5.6 shows that 85 percent of teachers are in districts that evaluate Eisenhower professional development activities using a teacher satisfaction survey.

EXHIBIT 5.6
Percent of Teachers in Districts That Use Different Methods to Evaluate
Eisenhower-assisted Activities (n=363)

Approximately three-fourths of teachers are in districts that evaluate activities by the number of teachers who participate (70 percent) and by observing the classroom practices of teachers who have participated (71 percent). The least popular method is using student achievement data to evaluate the effects of professional development, although 60 percent of teachers are in districts that report using this method.

Our case districts appear to reflect our national data in the evidence for reliance on teacher satisfaction surveys to evaluate the outcomes of professional development activities. For example, Maple City, Ohio, uses a standard teacher satisfaction form to evaluate professional development activities. The form asks teachers whether they agree with the following statements:

1. This inservice dealt with a priority issue.

2. The goal(s) of the inservice were clearly stated by the presenter(s).

3. The stated goals(s) of the inservice were achieved.

4. The information was clearly presented.

5. I will modify my teaching behavior as a result of this inservice.

6. The information presented will assist me in my job.

7. A portion of the inservice time was set aside for questions and answers.

8. I believe there was sufficient input from staff in planning this inservice.

Teachers are also asked to respond to three open-ended questions:

1. What was the most valuable part of this inservice?

2. In what ways could this inservice have been improved?

3. Additional comments.

In other case sites, as well, evaluation of professional development involved teacher satisfaction surveys that asked questions similar to those on the Maple City survey.

While our national sample of district Eisenhower coordinators frequently cites the use of observation of teachers and student achievement data as a means for evaluation, our case-study data indicate that districts do not observe teachers or use student data in a formal, systematic way to evaluate the outcomes of professional development activities. Boonetown, Kentucky, is not unusual in its use of student assessments to evaluate professional development activities. There, in addition to teacher satisfaction surveys, professional development is considered successful if KIRIS scores increase in the targeted areas. While this is an outcome-based strategy for evaluation, it can be considered a remediation approach rather than a capacity-building approach; it appears to neglect two critical central steps—an objective assessment of the quality of the professional development activity, and the evaluation of the effects of professional development on teaching itself. In neglecting these two areas, the attribution of test score increases to Eisenhower professional development is tenuous. In the sequence of steps from needs assessment, to setting goals, to designing and participating in professional development, to changes in teaching, to student learning, to student outcomes, we find that none of our case districts systematically observe teachers? practices after they have participated in professional development activities in a systematic way.

Some coordinators in our case districts state that they observe teachers, but their observations appear to be informal. For example, in Rhinestone, Texas, the Eisenhower coordinator visits classrooms to observe teachers who have recently been through training to see if new ideas are in use in the classroom. On the whole, her evaluation seemed cursory to our case visitors. While she visits each classroom and uses an NCTM checklist of effective classroom practices for criteria, she relies primarily on anecdotal notes to record her observations. She explains that her goal is not to make teachers feel that they are being evaluated, but rather to send a message of assistance and support to teachers. Further, as we reported earlier, in this and other districts, one criterion for evaluating professional development in the schools is students? academic achievement. However, without rigorous, longitudinal, well-designed studies, it is difficult to attribute students? test score increases to teachers? professional development experiences.

District Support and Guidance to Schools and Professional Development Providers

Continuous improvement means setting goals, collecting and reporting measurable indicators of progress, assessing the needs of teachers, and evaluating professional development activities. However, it also means communicating goals and evaluation results with those who provide professional development, so that they are able to incorporate district goals into their professional development plans, and know how their activities will be judged. Professional development activities often are provided by schools, or by district staff or individuals hired by the district. Institutions of higher education and nonprofit organizations also frequently provide professional development to teachers in districts; and districts may work with these organizations in various ways to offer them guidance and support.

To find out about the types of support and guidance that districts offer to schools and professional development providers, we asked district Eisenhower coordinators a number of questions about the types of communication they have with schools and with providers of professional development and the guidance they provide to schools and other professional development providers. Specifically, we asked coordinators in which of the following ways district Eisenhower staff exchange information with schools regarding professional development: regular visits and observations, telephone calls, required reports, and/or required evaluations. We also asked them which of the following five types of assistance they provide to schools: guidance in interpreting Title II rules, help conducting needs assessments, help developing professional development plans, help developing specific activities, and help developing performance indicators for professional development.

On average, districts provide more guidance to schools than to providers of professional development. Districts report providing 6.2 of the nine types of support to schools and 4.6 to providers. Standard deviations are quite large (2.6 for schools and 3.4 for providers), which suggests that there is a large variation in the number of ways that districts provide guidance (data not shown).

As illustrated in Exhibit 5.7, districts are about equally likely to provide support to schools as to providers in the form of classroom visits, phone calls, and required reports and evaluations. However, teachers are much less likely to be in districts that provide the other types of guidance (i.e., interpreting Title II rules, conducting needs assessments, and developing professional development plans and activities indicators) to providers of professional development than to schools. Also, districts provide fewer data-related types of support (e.g., reports and evaluations) than other types of support, to both schools and professional development providers. Although these data provide information about whether or not district coordinators engage in particular types of support and guidance, we have no information on the quality or frequency of these activities.

EXHIBIT 5.7
Percent of Teachers in Districts in Which Eisenhower Staff Provide Different Types of Guidance about Professional Development to Schools and Professional Development Providers (n=363)

To measure the extent to which districts differ on all continuous improvement measures, we created a composite of indicators, needs assessment, evaluation, and guidance to schools and professional development providers. We standardized the variables comprising each of these five measures, adjusted the values to be on a zero to one scale, and then summed them to form an overall continuous improvement composite. Exhibit 5.8 shows that there is significant variation in continuous improvement according to both district size and poverty level. Small districts employ significantly fewer continuous improvement efforts than either consortia or large and medium districts, and medium districts employ significantly fewer continuous improvement efforts than large districts. Similarly, low-poverty districts have fewer continuous improvement mechanisms than high-poverty districts. Again, these results may reflect that large districts have more staff and a more comprehensive professional development program, which requires more systematic monitoring and evaluation. Similarly, high-poverty districts are more likely to have support from other federal programs, such as Title I, Part A, which also emphasize the use of indicators, needs assessments and evaluation.

EXHIBIT 5.8
Extent of District Continuous Improvement Efforts, Overall and by District Poverty and District Size (n=363)

Summary: Continuous Improvement

Most districts do not have performance indicators and many remain unaware of the requirement to develop performance indicators and of the purposes underlying state indicators and data collection activities, although Congress? intent is clear in these provisions to establish and measure progress on performance indicators. Strategic planning that includes developing goals and objectives and delineating how progress toward achieving these goals will be measured is key to a successful professional development program (Loucks-Horsley et al., 1998). Clearly, there is room in this area for a heightened federal role in assisting states and districts to learn how to develop well-constructed indicators, how to collect meaningful data, and how to use the indicators as part of a system of continuous improvement.

Our national survey data also show that while most districts collect information about the needs of teachers, and evaluate Eisenhower-assisted professional development activities, as required by law, their approaches to these activities are not as strong as they could be. Our survey and case-study data reveal that district needs assessments and evaluations are typically based on teacher surveys. This heavy reliance on teacher reports does not take into account the potential differences between teachers? instructional goals and district goals and standards. Further, teachers may not be able to identify their own weaknesses, and even if they can, they may not be able to identify appropriate measures to address the weaknesses.

Districts also tend to use classroom observations and student achievement data in needs assessment and evaluation, but not in rigorous and systematic ways. Most districts do not have the resources to use these approaches effectively. Studies that link professional development with student outcomes would require costly long-term data collections and an evaluation capacity that many districts do not have; these types of evaluations are better conducted as part of well-designed research studies. It seems more appropriate and effective for districts to focus their evaluations on assessing how well their professional development activities reflect high-quality structural and core features. This can be done through carefully crafted teacher surveys that ask teachers specific questions about the contact hours, duration, active learning opportunities, content focus, and coherence of the activity, as well as through direct observation of the activity.

Districts do often provide guidance to schools and professional development providers. However, despite the need to use data to make decisions about professional development activities, districts do not appear to communicate often to schools and professional development providers about indicators and evaluations. There are a number of possible reasons for this. First, district Eisenhower coordinators themselves may not be trained as evaluators, and may not be familiar with uses of data to judge their progress. Second, a number of Eisenhower coordinators in our case districts indicate that they are aware of the deficiencies of their approaches, but do not have adequate resources to conduct more thorough evaluation and data collection activities. Considering the critical nature of the requirements to establish indicators and to evaluate Eisenhower-assisted activities, it may be helpful to determine more definitively what the barriers are to establishing indicator systems and high-quality evaluations, and to provide guidance and assistance to help the districts overcome these barriers.

[Building a Vision for Professional Development]

[The Role of Teachers in Planning Professional Development]