Evaluation Primer
Steps in Planning Evaluations

Identifying the Evaluation's Consumers

This point was made earlier, but it is important to reiterate that identifying the potential users will help to determine what questions are most important, what data will be viewed as credible, what analyses should be conducted, and how results should be transmitted and displayed. It is a good idea to solicit the views of other consumers, along with program staff, in drawing up the evaluation questions. Depending on the programs and setting, consumers may range from parents and the school board to other institutions, funding agencies, or broadcast media.

Choosing the Important Evaluation Questions

There is rarely enough time or resources to answer all of the questions about program practice and effects that consumers pose. A way must be found to establish priorities and to limit the number of questions. The most desirable method is to agree on a limited number of evaluation questions when the program goals and objectives are first established, but often the evaluation questions are drawn up after the fact or the program has multiple goals. Under these circumstances, the number of possible questions may be very large. One device to limit the inquiry is to ask each critical evaluation user to complete the statement, "I need to know ____ because I need to decide____."

As noted earlier, some programs may choose only questions that describe the intervention they carried out, while others may go further and examine the impact of the intervention. When the range of possible questions is agreed upon, estimates can be made about the benefits and costs of answering or not answering each of them. Some questions may be too costly to answer. Others may require a level of expertise in evaluation that is not available to the school district or agency. There may be simple measures that can achieve adequate results for some questions, making them candidates for inclusion. For example, a postcard follow-up survey of teachers who attended an in-service program might be sufficient to ask a few key questions about use of program materials. On the other hand, extensive questionnaires or classroom observation might be necessary to determine the extent to which, or the manner in which, teachers used particular instructional techniques.

Mapping Out an Evaluation Work Plan

It is critical to create a step-by-step work plan for conducting the evaluation. The first step will be to review the questions and group them in some logical manner by subject area, by the data needed to address them, by process/outcome/impact, or in some other manner. The plan should then outline the data that will be collected and how the information gathered will relate to each evaluation question. Suppose the questions are: how many hours of instruction and practice in a set of skills did the typical student receive during a semester, and did the number of hours vary substantially by classroom or school? What method(s) will be used to document the extent and nature of the services provided? Will evaluators review student records, review teacher logs or lesson plans, interview all the teachers or a sample of teachers, or administer student questionnaires? If teachers' lesson plans are used to document the amount of instruction, what data will be collected from those records and what evidence will indicate that refusal skill instruction took place on a particular day? What if lesson plans and teacher questionnaires disagree?

One device to limit the inquiry is to ask each critical evaluation user to complete the statement, "I need to know ____ because I need to decide____."

Making Sure Adequate Resources are at Hand to Carry Out All Functions

Evaluation is labor intensive. At a minimum, an evaluation of a school or community prevention program will require that the following resource-intensive functions be performed:

• Evaluation Planning formulating the overall evaluation strategy and identifying or developing the necessary evaluation instruments. A study of outcomes also may necessitate establishing participant and comparison groups and gaining parental consent for student questionnaires;
• Data Collection administering questionnaires, conducting interviews, observing program operations, or reviewing or entering data from existing data sources;
• Data Coding collating the information gained through data collection, ensuring that it is accurate, and translating collected data into usable formats for analysis; and
• Data Analysis conducting any statistical analyses related to evaluation hypotheses, preparing summary statistics, charts, tables, and graphs.

Some problems arise so often in conducting evaluations that they are noted here. There may be governmental or institutional research regulations to meet in drawing up an evaluation plan. In addition, states have rules regarding research with human subjects, especially when programs ask questions about behavior. These rules may deal with confidentiality or parental consent. Evaluators must ensure that all such requirements are addressed when the evaluation is being planned.

There may be difficulty in determining appropriate groups with which to compare program participants in order to study program outcomes or impact. If there is no group in the school, district, or agency that does not receive the service, it may be difficult to find a group with which to compare participants. Although establishing a comparison group to study outcomes may be problematic, comparisons with such groups are one way of demonstrating program impact.

Existing data sources may not be of sufficiently high quality to yield meaningful evaluation results or may not be kept in a usable format. For example, school records may be incomplete or lacking in detail. Knowing the limitations of available data early on will allow evaluation planners to gauge the amount of effort and time needed to collect additional information. Knowing in advance that available data are not sufficient can enable project staff to seek additional financial support and administrative or other approval for further data collection.

Evaluation tasks will invariably take more time than is originally anticipated. Establish a realistic schedule, and a budget that includes a little more resources than are initially anticipated to be necessary. It is always easier to reallocate unneeded resources than to find additional resources to fill an underestimated need. For example, procedures for obtaining written informed parental consent for student data collection can take a great deal of time. Or questionnaire responses may be incomplete and additional follow-up may be necessary. Evaluators should not expect everything to run smoothly.

Not every data collection strategy will be implemented as planned, so evaluators should prepare for contingencies. In other words, evaluation planners should not put all of their information "eggs" in one data collection "basket." It is useful to begin an evaluation with multiple data collection strategies or alternatives in mind. For example, a well-written survey can still have a low response rate because of high absenteeism on the days set aside for data collection. Or a comparison group can become "contaminated" by inadvertent exposure to the program, reducing the group's value. Program participants may drop out of school subsequent to the program, making it difficult to find them for a follow-up survey. Even if substitute approaches are more limited, they will be better than not completing the evaluation.

-###-

Next: Documenting and Analyzing Program Installation and Operations
Back: Why Conduct Program Evaluation?
Return to Education Evaluation Page