With your evaluator on board, you are ready for the planning phase. Your evaluator will work with you and your team members to complete the following steps:

• Involve Key Stakeholders

• Focus Your Evaluation

• Craft Your Research Questions

• Select the Right Design

Involve Key Stakeholders

As with all aspects of prevention programming, it is critical to identify and involve key stakeholders in the evaluation process. When stakeholders are not appropriately involved, evaluation findings are likely to be ignored, criticized, or resisted. When stakeholders are involved, they can provide valuable assistance during the evaluation process and become advocates for your evaluation's findings.

Any or all of the stakeholder groups below may be interested in participating in the evaluation process and/or learning about the evaluation results:

• Program funders
• Program staff
• Program volunteers, collaborators, and supporters
• Participants' parents and other caregivers
• School administrators and other nonprogram school staff
• School board members
• County board members and elected officials
• Community leaders and activists
• Media
• Program participants

Focus Your Evaluation

Many prevention initiatives include multiple components (e.g., student education, policy enforcement, community awareness, and information dissemination). The second step in evaluation planning is working with your evaluator to determine exactly what you want to study: the whole program or just one part. To make this decision, consider the following questions:

• Which prevention activities will you be able to evaluate? Some prevention activities are more difficult to evaluate than others. For example, media campaigns that reach hundreds or even thousands of homes are more difficult to evaluate than a classroom-based curriculum. Your team will need to think carefully about how to use limited evaluation resources.

• Which elements of the program are most likely to demonstrate measurable effects? For example, a year-long prevention curriculum is much more likely than a one-time motivational speaker to influence youth outcomes.

• Which part(s) are key stakeholders interested in evaluating, and why? Different stakeholders will have different perspectives on and questions about your program, which need to be considered before you can clarify the purpose of your evaluation.

• Which elements require the most resources? It is important to understand whether or not program components that are very expensive, time-consuming, or labor-intensive are actually working well.

• Which elements have a strong research basis? If strong evidence shows that a particular component will be effective, you may want to monitor the implementation of that component.

Another method for defining the scope of your evaluation is to use a logic model to identify the most important program elements to evaluate and the data that will be needed. A logic model is a visual representation of the theory underlying a program. It displays the relationships between program activities and intended short- and long-term outcomes. Most of the exemplary and promising prevention programs identified by the Safe and Drug-Free Schools Program have logic models that can be adapted for local use following program selection.

Craft Your Research Questions

For any single program component, there are innumerable issues you can examine and evaluate. Most fall under one of two areas: implementation issues and participant outcomes.

• Implementation issues. Before assessing program outcomes and/or impact, you must determine the degree to which actual program delivery matches intended delivery. As mentioned on Day 1, this type of evaluation is known as process evaluation. Common process evaluation questions include the following:

  • Did the program recruit and serve individuals from the target population?
  • Did the program deliver services to participants according to the program design?
  • How much of each program service did the typical program participant receive?
  • Who delivered each service?
  • How long did participants stay in the program?
  • Why did participants drop out of the program?

  Click here for more information about process evaluation.

• Participant outcomes. Questions related to outcomes should assess the degree to which a program directly affects participants and how it goes about doing so. Outcome questions must be linked to the intended program outcomes. You can ask separate questions about each intended outcome or one broad question that addresses a variety of related outcomes (e.g., Did the program reduce risk factors and increase protective factors related to substance abuse among program participants?). The following are some questions that you might use to evaluate a school-based drug or violence prevention program:

  • To what extent has X behavior (e.g., alcohol use, fighting) decreased among students over the duration of this project?
  • To what extent has academic failure declined over the duration of this project?
  • To what extent has school attendance improved over the duration of this project?
  • To what extent has the number of disciplinary referrals decreased over the duration of this project?

  It may also be beneficial to ask questions that will help you better understand the outcomes you identify. For example, you might want to ask whether the program is more effective for some types of participants than for others, whether some program activities or services account for more of the program's effects than others, or whether length of participation is related to outcomes.

  Click here for some tips on developing good evaluation questions.

Select the Right Design

If your team decides to examine participant outcomes, then you will need to select an evaluation design. Four research designs are commonly used to assess program outcomes: post-test only, pre- and post-test, pre- and post-test with comparison group, and pre- and post-test with control group. These designs vary in their capacity to produce information that allows you to link program outcomes to program activities. The more confident you want to be about making these connections, the more rigorous the design and costly the evaluation. Your evaluator will help determine which design will maximize your program's resources and answer your team's evaluation questions with the greatest degree of certainty.

As the coordinator of this planning process, it is important for you to make sure that everyone is on the same page before proceeding from one step to the next. To manage this process effectively, it is a good idea to check in with evaluation team members and other stakeholders from time to time to make sure that they have a clear sense of the evaluation mandate and are comfortable with how the evaluation plan is coming together.

References for Day 3 materials:

Center for Substance Abuse Prevention. Evaluation Basics PreventionDSS 3.0. Available on-line at: http://www.preventiondss.org/Macro/Csap/dss_portal/
templates /start1.cfm?sect_id=1&page=/macro/csap/
dss_portal/portal_content/eval_intros /eval-nug8-30b.htm
&topic_id=5&link_url=processevalintro.cfm&link_name=
Evaluation%20Basics.

Harding, W. (2000). Locating, hiring, and managing an evaluator. Newton, Mass.: Northeast CAPT.

Kantor, G. K. & Kendall-Tackett, K. (Eds.) A guide to family intervention and prevention program evaluation. Edited and prepared for electronic dissemination by C. M. Allen. Available on-line at: http://www.fourh.umn.edu/evaluation/evaluationfiles/family/default.html.

Wandersman, A., Imm, P., Chinman, M., & Kaftarian, S. Getting to outcomes: Methods and tools for planning, self-evaluation, and accountability. Volume 1 available on-line at: http--www.stanford.edu-~davidf-GTO_Volume_I.pdf. Volume 2 available on-line at: http://www.stanford.edu-~davidf-GTO_Volume_II.pdf.

Western Center for the Application of Prevention Technologies. Building a successful prevention program; Step 7: Evaluation. Available on-line at: http://www.unr.edu/westcapt/bestpractices/eval.htm.