A r c h i v e d  I n f o r m a t i o n

Designing Effective Development: Lessons from the Eisenhower Program - December 1999

Chapter 7

Strengths of the Data

Before considering the findings of the National Evaluation of the Eisenhower Program, it is important to take stock of the quality of the data on which the findings rest. The data set has a number of important strengths. First, our descriptions of the nature and quality of professional development provided through the Eisenhower program are based on national probability samples, each with an excellent response rate. The national probability sample of district programs and SAHE grantees has an 88 percent response rate for district program coordinators and 87 percent for SAHE grantees. The national probability sample of teachers who participated in Eisenhower professional development activities has a response rate of 72 percent. The 72 percent response rate is especially high when considering the multistage process necessary to complete the sample.1

Second, the two probability samples are complementary. Data from our telephone interviews with district program coordinators and SAHE-grantee project directors are backed up by teacher participant accounts of what they experienced and its quality.

Finally, we have taken a number of steps to maximize the validity and reliability of the national survey data. For example, although the telephone interview and teacher survey data are based on self-reports, most of the data represent an accounting of behaviors, not direct judgments of quality that might be more likely biased in a positive direction. In addition, the survey results are cross-validated through case study data that are rich in potential to explain the descriptive statistics and path analyses from the surveys.

The three strands of the evaluation are designed to produce an integrated portrait of the Eisenhower program from many perspectives. Because the evaluation involves a variety of research methods and has collected data from groups of individuals who view Eisenhower-assisted activities from different vantage points, it is able to provide an accurate description of program-funded activities and analyses of the features of these activities and their effects on teacher practice.

The data we analyze in this report do not provide direct estimates of changes in teaching practice over time. Our conclusions about the effectiveness of Eisenhower-assisted professional development are based on teachers' reports of the extent to which participation enhanced their knowledge and skills and improved their teaching. Data from the second and third waves of our longitudinal study, to be examined in our third report, will provide additional information on teacher change.2

None of our analyses address directly how teacher participation in Eisenhower professional development leads to gains in student achievement. The work reported here, however, is not completely divorced from student achievement. The characteristics of professional development on which project directors and teacher participants reported in the surveys are carefully grounded in the available literature on professional development and student achievement. To the extent that the Eisenhower program provides professional development with characteristics identified as effective in the literature, we conclude that they are of high quality, and we tentatively infer that they should lead to benefits for students.

1 District coordinators and SAHE-grantee project directors had to submit the complete list of professional development activities provided during the prior year and the number of participants. Two activities were selected from each district with probability in proportion to size, and from those, complete rosters of teachers were collected from which two teachers were randomly selected and surveyed.

2 We also took a number of steps to maximize the validity and reliability of the evaluation's national survey data. For example, most of the survey questions ask teachers and administrators to provide an accounting of behaviors, not direct judgments of quality that might be more likely to be biased. The substantial variation in the responses teachers and district administrators provided to these behavioral items, as well as the consistency in teacher and district administrator responses, tends to bolster our confidence in the validity of the data.
-###-


[Chapter 7 - Conclusions and Lessons for the Eisenhower Program]		 [Table of Contents]
[Current Teaching Practices in Mathematics and Science]