Evaluation Primer
Documenting and Analyzing Program Installation and Operations
Evaluation of the implementation of a program is a good example of process evaluation. The object is to understand how program plans and objectives are translated into action. Process evaluation may be undertaken in order to monitor project activities, ensure accountability (e.g., demonstrate that planned services are delivered in a timely manner), improve service delivery (e.g., identify obstacles and adjust activities), or set the stage for assessment of project outcomes. Some audiences may be most interested in accountability, but others may be more interested in improving service delivery. Monitoring the implementation and operation of a program usually involves identifying and tracking specific program activities or objectives. For example, if a first step in implementing a program is teacher training, the evaluation could examine the implementation of the training component. Here are a number of specific evaluation questions that could be asked about implementation of teacher training:
- Did the activity (or procedure) aimed at bringing about program implementation (in this case, the training) occur as envisioned? If not, what barriers or obstacles prevented parts or all of the activity from being executed?
To document that an activity is conducted as planned, the evaluators need a means of monitoring the activity. Monitoring of teacher training, for example, might entail no more than checking the sign-in sheets to find out how many staff attend, whether the staff who attend are those who are supposed to implement the program, and whether they stay for the entire training session. A more ambitious plan might involve exit questionnaires or enlisting an objective party (someone unconnected with the training) to observe the training and to write an assessment of how well the content of the sessions reflects program objectives. These observations could be carried out by the drug program coordinator, a district administrator, or someone hired by them specifically for this purpose.
- Did the clients (or participants) find the activity useful, and did they plan to use what they had learned?
Asking people to implement a curriculum or staging a training session to introduce a program is no guarantee that implementation will occur. To get a better fix on the likelihood of use, the evaluators could ask the participants about the quality and usefulness of the information designed to aid implementation (such as a curriculum guide or trainer presentation). That information would allow the persons charged with fostering implementation to get immediate feedback on how well they are teaching or otherwise transmitting information and to make any necessary adjustments. A survey of training participants, for example, could provide a measure of the training's effectiveness if the teachers say that they plan to use the information, the in-service session may be deemed effective, at least in a limited manner.
- Did the training result in the program operating as planned?
This is clearly the most critical question about the implementation of a program, since it asks whether training approaches, curriculum materials, or other information or techniques are being used by teachers in the classroom. It is both an implementation question (e.g., determining the result of initial training activities) and an intermediate question for a student outcome examination (because it must first be demonstrated that teachers implement the program before student outcomes can be measured). There are many ways that program operation could be measured. All teachers (or some teachers) could be asked a few questions about what they are doing in their classrooms, such as how often they use certain curriculum materials, whether they have completed particular curriculum units, or how many class sessions have been spent teaching particular skills to students. An unobtrusive measure of program operation could be a review of teacher lesson plans submitted to the school or district. A more ambitious plan might entail classroom observations and interviews of teachers or administrators by district staff or an outside evaluator.
The choice of approach will depend, in part, on the reasons for conducting the assessment. If the district's goal is to document implementation (e.g., to show a funding source that the program was adopted), then a short questionnaire filled out by a sample of teachers (or a review of randomly selected lesson plans or disciplinary records) may be sufficient to demonstrate the extent of program operations. Based on the findings, the evaluator might conclude that "Eighty percent of the teachers who attended training report that they used the curriculum as prescribed" or that "Suspensions for alcohol offenses increased 60 percent after the introduction of the new disciplinary policy." If the district's goal is to gain high (or improve previous) rates of implementation, the inquiry might not only want to determine whether most staff are carrying out the program at some level, it might also pinpoint where and why the program is not being implemented. The barriers to implementation could then be addressed.
If a further goal is to find out how program exposure is related to student behavior, ascertaining the degree of program implementation will require more detailed information. If a goal is to find out how program exposure is related to student behavior, ascertaining the degree of program implementation will require detailed information.
-###-
Next: Observing Behavioral Outcomes and Attributing Changes to the Program
Back: Why Conduct Program Evaluation?
Return to Education Evaluation Page
Last modified -- September 21, 1998, (lyp)