As a middle school coordinator, you are responsible for facilitating the collection of both process and outcome data. Most people tend to think about outcome evaluation when discussing research-based programs, since they are referred to by that term largely because of their capacity to produce positive outcomes. Yet, process evaluation -- which asks, Was the intervention implemented as intended? -- can play a key role in helping you monitor how well your program is being implemented. A well-designed process evaluation can yield information that will help you do the following:

• Know what actually happened. For example, it can tell you whether you reached your target audience and delivered all of the program's core components. If you have decided to adapt your program, monitoring can also help you document and track any changes that you make along the way.

• Understand how a program operates, such as how the different pieces of a program fit together in reality versus on paper, how those involved in the program interact, challenges that emerge, and strategies employed to resolve them.

• Provide feedback for continuous quality improvement. Monitoring can help you identify problems as they arise. This puts you in a better position to refine activities accordingly, which will increase the program's potential to produce intended results.

• Document compliance with legal and ethical guidelines, including state standards. A variety of school, district, and state policies dictate the number and type of health-related programs schools should offer to students. Careful documentation of program delivery can help a school demonstrate the extent to which it is meeting these requirements. Also, in the event of an incident or tragedy among students involving substance use and/or violence, it is helpful for a school to have evidence of prevention programming.

• Document compliance with MSC and related grants. Through careful program monitoring, your school will be able to demonstrate how funds are used and perhaps even make the case for extensions or additional funding.

• Provide regular updates to school and community partners. As different program components are delivered, you can keep school and community partners informed and invested by sharing information about these achievements.

• Support conclusions about the program's role in producing change. A school may judge a prevention program to be ineffective if an outcome evaluation reveals little change among participants. Yet, a lack of results may be due to poor implementation rather than the program itself. Also, monitoring data may help explain differential outcomes among participants (e.g., why positive changes occur among some participants and not others).

• Share knowledge about how to successfully replicate the program. Process evaluation data can yield valuable information about the realities of using research-based programs. If you have adapted your program, you will be able to describe how and why you did so, as well as the impact of these changes on program outcomes.

As an MSC, your implementation monitoring system should be designed to assess how much of the program was implemented, as well as how well each program component was conducted .For each classroom-based session you deliver, be sure to track the following:

• Who is delivering the session. This will help you see patterns and track progress across sessions.

• Who is participating in the session. Taking a head count and documenting who is there (e.g., parents, students) can help you determine the extent to which you are reaching your target population.

• Who is missing. Knowing who is missing may help you explain why certain populations demonstrate fewer or weaker program outcomes.

• How much time is spent on each activity or component. You will also want to note whether or not activities are completed.

• Any changes that were made. Document all program modifications, including any alterations to program sessions, additions of new activities or components, or omissions of activities or components.

• Problems that arose. Also note the strategies that were used to address these problems.

• Program success. Too often, aspects of implementation that work well are overlooked. Make sure to note aspects of program delivery that educators felt particularly comfortable with, as well as activities that participants seemed to respond well to.

Your program may include components that extend beyond the classroom, such as service learning, schoolwide activities to promote a positive school climate, and events to build school-family partnerships. Work with your evaluator to develop a system for monitoring these components if your program does not provide tips or tools for doing so.

Finally, there may be circumstances when you will want to monitor prevention activities that are not part of your initiative. For example, if your outcome evaluation is designed to include a control or comparison group, you will want to document any prevention activities in which students in that group participate. This will increase the accuracy of your own assessment efforts.

Do I Need to Hire an Evaluator?
USED requires that you conduct process and outcome evaluation studies so that you can measure the extent to which you have met the program objectives proposed in your grant application. Individual districts may hire outside evaluators or use district staff if the required expertise is available. Click here for resources related to program evaluation.

There are many ways to monitor implementation. Some programs include tips and/or tools that can help you to monitor their delivery. If yours does not, consider contacting the program developer for recommendations. Also, be sure to include your evaluator in the development of an effective assessment process.

Collaborate closely with school personnel to develop an assessment plan that will work in your school setting. As your plan takes shape, you will need to identify staff members who can help monitor program implementation and resolve any problems that arise.

The two most prominent methods for monitoring the implementation of a prevention program are observations and self-reports.

• Observations. The most credible information about program implementation tends to come from ratings by outside observers who are trained to watch the intervention "in action." Observers might take notes and/or use audio- or videotapes to document their observations, which they will then rate against an established set of criteria.

  If you are working with an evaluator, he or she might be able to help you recruit and train observers. Often, graduate students in education, public health, or psychology enjoy participating in this type of activity. School and community partners may also be willing to fill the observer role, though coordinating this activity with their other work-related responsibilities may prove difficult.

• Self-reports. This involves asking people who are directly involved in implementing program activities to provide regular self-assessments. While self-reporting is often the most practical way to obtain information, data collected in this manner may not be as valid as data collected by independent observers. Educators may have a difficult time being objective about how they are doing. They may also skew their reports toward the positive to avoid documenting anything negative or critical about their performance. Finally, self-reporting adds one more responsibility to a teacher's already-full workload.

  Some strategies for encouraging educators to complete their assessments honestly and objectively (and to increase the likelihood that they get done!) include the following:

  • Emphasize the importance of the self-assessments and explain how the information will be used.

  • Make it known that the self-assessments will have no impact on their job performance ratings.

  • Consider making the self-assessment reports anonymous (though keep in mind that this may limit your capacity to intervene if problems arise).

  • Suggest that educators include time to complete the self-reports when they schedule the program session.

  • Remind educators to complete their reports before moving on to their next task.

  • Create and adhere to a schedule for picking up completed self-assessments from program implementers.

Click here for a sample log that educators can use to record their implementation experiences.

As you establish your monitoring system, be sure to allow for multiple implementation checks over time. Many factors influence program delivery, and you will not be able to accurately capture what is really going on with only a few periodic assessments. Educators may be faced with unpredictable events that interfere with implementation. Also, since people tend to alter program activities over time, you will want to document these occurrences in order to prevent further "program drift."

"Data indicate that between 41% and 84% of teachers who begin preventive programs may modify them over time or discontinue their use altogether."

Joseph Durlak, Loyola University

Finally, collecting information about program implementation at multiple points in time provides those involved with the program with more opportunities to understand what is going on, pursue additional training or assistance, and refine program activities if necessary.

Please read the following scenario, then share your reactions in the Discussion Area.

After working as an MSC for over two years, Jane was very happy to finally be implementing a research-based prevention program. Yet, after monitoring program activities for only two weeks, she was distressed to discover that several teachers who had signed on to deliver the program were skipping over some important program activities and rushing through others. Jane knew that she needed to address these issues quickly, but was worried that this would alienate the teachers.

Discussion Questions

• What reasons might the teachers have for altering the program activities?

• What strategies could Jane use to approach the teachers and improve program implementation?

• If you have spent time monitoring your school's prevention activities, what other implementation challenges have you observed? How have you handled them?

References

Domitrovich, C.E. & Greenberg, M.T. (2000). The Study of Implementation: Current Findings from Effective Programs that Prevent Mental Disorders in School-Aged Children. Journal of Educational and Psychological Consultation, 11(2), pp. 193-221.

Durlak, J.A. (1998). Why Program Implementation Is Important. Journal of Prevention and Intervention in the Community, 17 (2), pp. 5-18.

ETR Associates. (1997). Dissemination Workbook for Programs That Work. Santa Cruz, CA: Author.

Greenberg, M.T., Domitrovich, C.E., Graczyk, P., & Zins, J. (January 2001). A Conceptual Model of Implementation for School-Based Preventive Interventions: Implications for Research, Practice, and Policy. Report to the Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, US Department of Health and Human Services.