Whether you work with an evaluator from your school or district or an outside consultant, it is important to understand the fundamentals of the evaluation process so that you can fully participate in it. The more you know about the purpose and nature of evaluation, the better prepared you will be to work with your evaluator and discuss the process and findings with your school and community partners.

While many of you may be well acquainted with this topic, it is still helpful to start at the beginning -- by defining evaluation and reviewing why it is so important.

• What is evaluation?
• Why is evaluation necessary?

What is evaluation?

We are all evaluators, carefully examining and assessing the value of everything around us. When purchasing produce, we smell the cantaloupes and squeeze the tomatoes before making our selections. When choosing a pain reliever, we consider price, reliability, and potential side effects. Sometimes we even consider other data sources, such as Consumer Reports or the opinions of friends and family members.

What distinguishes program evaluation from these everyday evaluation activities is the fact that it is systematic -- it involves the application of methods and techniques that are designed to increase our certainty about the validity of the results. We can use the information we collect through evaluation to improve the effectiveness and make decisions about the future of a prevention program. Although all types of interventions can be evaluated, this event will focus on program evaluation.

Generally speaking, there are four different types of program evaluation: formative, process, outcome, and impact. The type of evaluation you choose to conduct will depend on the current state of your prevention activities and the nature of the decisions you need to make about those activities.

• Formative evaluation involves collecting data to inform program development and delivery. An example is conducting a focus group with students to shape the development of a tobacco prevention media campaign. Formative evaluation activities are an excellent way to obtain feedback about the feasibility of proposed activities and their fit with the intended settings and participants. Although some people use this term synonymously with process evaluation, most use it only within the context of program planning and development.

• Outcome evaluation measures the direct effects of program activities on targeted participants, such as the degree to which a program increased knowledge about alcohol and other drug use among students. The emphasis here is on the program's short-term effects.

• Impact evaluation assesses the ultimate effects of program activities on targeted participants. The emphasis here is on the program's long-term effects. For example, impact evaluation might look at the extent to which program activities contributed to a reduction in risk factors (e.g., school drop out) associated with a program's intended outcomes (e.g., reductions in alcohol and other drug use among youth) among students.

Why is evaluation necessary?

Evaluation serves a variety of purposes, including the following:

• It helps keep you on track. Despite everyone's best intentions, program activities often change over time. These changes can sometimes compromise program effectiveness. A clear understanding of the implementation process can help you bring actual delivery in line with intended delivery. Such adjustments can ensure that program activities continue to reflect the program's original goals and objectives.

• It can improve program efficiency. Evaluation may reveal opportunities to streamline program delivery or enhance coordination between program components. Increased efficiency can help reduce cost or allow you to provide more services to a larger audience at the same cost.

• It may reveal unintended effects. Your program may contribute to many different changes among participants, including changes that your planning team does not expect. For example, some drug prevention programs have been shown to reduce the prevalence of one type of substance while simultaneously increasing the use of another.

• It enhances accountability. Accountability is the hallmark of the No Child Left Behind Act of 2001 -- and for good reason. Because resources are limited and the education and well-being of young people are at stake, we must do everything we can to ensure that the academic and health-related activities students experience are of high quality in design, delivery, and outcomes. Prevention activities must be held accountable to the following stakeholders:

  • funders who provide resources to support a program
  • school staff who spend their time and energy implementing a program
  • parents who entrust their children to schools
  • students who rely on schools to help them reach their full potential
  • other school and community partners invested in the well-being of youth

• It can advance knowledge. Documenting and sharing information about your program's successes and challenges can help advance the larger field of prevention and improve outcomes for young people beyond the walls of your schools and the borders of your communities.

Traditional vs. Participatory Evaluation

There are two primary approaches to conducting an evaluation: traditional and participatory. In the traditional approach, the evaluator is responsible for all decisions about how to conduct the evaluation. The evaluator's contact with program staff is minimal: program staff may be unaware of the evaluation's design and excluded from evaluation tasks.

A relatively new but increasingly valued approach involves a partnership between the evaluator and those who develop and deliver program activities. This approach, known as collaborative or participatory evaluation, relies on an evaluation team composed of one or more individuals trained in evaluation, program staff, and other stakeholders (e.g., students who are receiving the program, parents, etc.). The diverse members of the evaluation team work together to plan and implement all evaluation tasks.

Not only is the participatory approach to evaluation more closely connected to your own approach to prevention work -- namely collaborative, but it can also help you produce a better, more enjoyable, and less expensive evaluation. Click below to learn more about the advantages and disadvantages of participatory evaluation.

Traditional and participatory approaches to evaluation embody different values and result in different relationships, processes, and outcomes. In the end, you must work with your planning team and other key school and community partners to determine the best approach for the evaluation of your program.

References for Day 1 materials:

Harding, W. (2000). Locating, hiring, and managing an evaluator. Newton, Mass.: Northeast CAPT.

Jackson, E. T. (1998). Introduction. In E. T. Jackson & Y. Kassam (Eds), Knowledge shared: Participatory evaluation in development cooperation. (pp. 1-20). West Hartford/Ottawa: Kumarian/IDRC.

Muraskin, L. (1993). Understanding evaluation: The way to better prevention programs. Rockville, MD: Westat.

Scheirer, M. A. (1994). Designing and using process evaluation. In Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.), Handbook of Practical Program Evaluation. (pp. 40-68). San Francisco: Jossey-Bass.