Evaluation Primer
Interpreting and Reporting Evaluation Findings

An analysis plan links the evaluation questions with the data collection and spells out the analyses that will be conducted when data become available. It should be written in conjunction with the drafting of data collection instruments rather than afterward. If the analysis plan is written too late, key questions may not be asked or may be phrased in unfortunate ways, or critical information from other sources may be overlooked.

As questionnaire items are prepared or data entry forms developed, the analysis plan links these data collection elements to the analyses that will be conducted. The analysis plan can also identify the specific range of answers that will be solicited. If items are specified in detail, and analyses are linked before instruments are completed, it will avoid disappointment later. An analysis plan links the evaluation questions with the data collection and spells out the analyses that will be conducted when data become available.

Analyzing Evaluation Findings

Once data collection is completed, the process of data analysis begins and the effects of the program emerge. A basic tool for analyzing data is descriptive analysis. Descriptive analysis may be as simple as summing or averaging results: What was the mean score of participating students on a knowledge items? What was the mean score of the comparison group?

In a process evaluation, statistics will likely be relatively straightforward: for example, the number of persons served with this program in place (possibly compared with the number of persons served before this program was in place), the number of instructional hours, the number of counselor contact hours or individual sessions, the number of staff trained, etc. Descriptive information should be presented objectively, in quantitative terms where possible.

Descriptive analysis also extends to characterizing the relationships between different measurable aspects of the program. By examining relationships using statistical tools as well as common sense, it may be possible to show, for example, whether attending in-service training sessions is associated with a greater likelihood that teachers adopt a new curriculum, or whether teachers who believe that a school that has a serious problem refer more students for treatment. Carrying out this type of descriptive analysis requires cross tabulations, correlations, and other statistical techniques designed to depict relationships between variables. These techniques cannot establish causality.

Analysis can also be inferential in nature. In an outcome or impact evaluation, there are a variety of questions about effects of a treatment on an observed result. A simple inferential analysis might seek to determine whether observed differences in outcomes between treatment and comparison groups are statistically significant, or whether it is likely that they could occur by chance. A more complex question might be whether differences between treatment and comparison groups are significant when the background or previous education of the two groups are taken into account (i.e., "held constant"). The help of a statistician or evaluation specialist may be needed to carry out this type of analysis.

Evaluators are also called upon to interpret data. For example, even if a statistically significant difference between groups is observed, is that difference meaningful in a practical sense? What does it tell program personnel that can help them to improve the program? Let's say that among 10th-grade students whose teachers indicated that they implemented a new curriculum in its entirety, 3 percent were suspended or expelled during the evaluation year. By contrast, 5 percent of the 10th graders who were not taught these skills were suspended during the same time period. While a statistical test shows that the difference between the suspension rates of the two groups is statistically significant (i.e., the difference is not likely to have occurred randomly), is that difference meaningful in a practical sense? Is the difference sufficient to continue the program in its present form, or should the staff make changes? Should every high school student in the district receive the new program if substantial additional teacher training costs will be incurred?

A final word on data analysis: Evaluators should not be discouraged by findings indicating that a program demonstrated few effects on participant behavior. It is difficult to evaluate a program that involves such a complex, sensitive issue as drug use. It is particularly hard because behavioral changes may occur for only small percentages of participants or changes may be difficult to measure. Programs that are new, or that are recently adopted, may not be sufficiently developed or implemented to show behavioral effects. One way to avoid disappointment is to make sure that the evaluation questions address changes that could reasonably have been expected to occur in the time frame under examination.Programs that are new, or that are recently adopted, may not be sufficiently developed or implemented to show behavioral effects.

Reporting Evaluation Findings

Once the findings are in, evaluators need to develop strategies to report the results. The form of the report depends on the audience. There may be several audiences for a program evaluation, each requiring different information or a customized presentation format. An in-house evaluation conducted to improve a program's implementation will lead to a report that is very different from that required in an "accountability" evaluation conducted for a funding source or the school board. A report intended for media use will be different still; it may, in fact, be a press release.

While most evaluations include a detailed written report, this approach is not always necessary or appropriate. Verbal presentations with supporting tables, graphs, and charts or case studies and targeted qualitative results may be enough for some needs. But most evaluations will call for a report summarizing the goals and history of the program, methods of evaluation, findings, interpretations, conclusions, and recommendations.

It is important to review the results with colleagues and program staff before completing an evaluation report. This review can be accomplished by circulating an interim or draft report and holding a meeting to discuss it together. The evaluator will gain additional perspective on the meaning of the data from the reviewers before he or she writes the final draft. For example, colleagues can discuss and interpret puzzling findings. Perhaps participants reported increased use of drugs after the treatment but less intention to use drugs in the future. Discussions with staff can bring out new perspectives on the meaning of these findings, perspectives that can be included in the final document.

It is also a good idea to brief important political actors before a report is released publicly. The briefing gives district or agency officials or other policymakers some time to digest the findings and to think about the policy implications. It may also provide them with an opportunity to prepare a response if they so desire. Through this process, evaluators will also learn what appear to be the most important findings from the perspective of the groups that will use them.

The final report can be written as a short summary document with a technical appendix. Most busy audiences look for brevity and clarity. In summarizing findings, the evaluators should not be afraid of appearing too elementary. Those who want more statistical or other detail can find it in the back-up technical report. Evaluators or program staff may also be called upon to make oral presentations before various groups. Speakers should prepare a set of perhaps half a dozen simple summary graphics (charts and tables) on the most important findings to show on an overhead projector and to distribute. They can also prepare copies of a summary sheet of results to give to the audience.

Whether oral or written, the report should begin by pointing out why the evaluation was conducted and asked the questions it did. It should state the purposes of the program and how it was developed or selected for the school or agency. The report should indicate what the board, staff, or others wanted to learn from the evaluation and why, and explain the methods and the procedures undertaken to collect and analyze data. The evaluator should share the highlights of the results and describe what the findings imply for program maintenance, expansion, redirection, funding, etc., as appropriate. The report may also advance recommendations for future steps, short- and longer-term actions that can be taken to improve the program further.

It is crucial to report evaluation findings objectively. Most evaluations have both positive and negative findings as well as findings about which the evaluator is less than fully certain. Most studies have methodological and other limitations that constrain the ability of the evaluators to reach definitive conclusions. Evaluations also encounter external events that impede the data collection or analysis. These circumstances need to be discussed fully and honestly, so that audiences can judge the degree of confidence to place in the results. A good scientist works hard to discredit his or her own findings.

Finally, the report should not portray the results as the final word on the program but should present the evaluation results as part of a cumulative and evolving process. Evaluation data represent one kind of input into decision making, but there are other sources of information as well. And the evaluation is ultimately a device for program staff to make adjustments, to improve their effectiveness. The change process will probably continue.
-###-

Back: Ensuring Evaluations Yield Valid and Reliable Findings
Return to Education Evaluation Page