Program evaluation is a basic tool for program management and improvement. Central questions which an evaluation can help answer include:
Evaluation involves the systematic and objective collection, analysis, and reporting of information or data. Using the data for program improvement and increased effectiveness then involves interpretation and judgement based on prior experience.
The evaluation process can be described as involving six progressive steps. These steps are shown in Exhibit 1, and are discussed below. It is important to remember that initiating an evaluation cannot wait until a program is nearing completion. An evaluation should be incorporated into overall program planning, and should be initiated when the program begins operation. In this manner, program processes and activities can be documented from their beginning, and baseline data on program participants can be collected as they enter the program.
The first step in planning an evaluation is to define its purpose and scope. This helps set the limits of the evaluation, confining it to a manageable size. Defining its purpose includes deciding on the goals and objectives for the evaluation, and on the audience for the evaluation results. The evaluation goals and objectives may vary depending on whether the program being evaluated is new and is going through a try-out period for which the planning and implementation process needs to be documented, or if a program has been thoroughly tested and needs documentation of its success before information about the program is widely disseminated and adoption by others encouraged.
Depending on the purpose, the audience for evaluation may be restricted to the project director and his/her staff, or may include a wider range of individuals, from agency administrators and decision-makers, to planners and other officials at the local, state, or national level.
The scope of the evaluation depends on the evaluation's purpose and the information needs of its intended audience. These needs determine the specific components of a program which should be evaluated and on the specific project objectives which are to be addressed. If an evaluation of a program has recently been conducted, a limited evaluation may be designed to target certain parts of the program which have been changed, revised, or modified. Similarly, the evaluation may be designed to focus on certain objectives which were shown to be previously only partially achieved. Thus, this step will define exactly which aspects of the program are to be evaluated. Costs and resources available to conduct the evaluation must also be considered in this decision.
Evaluation questions grow out of the purpose and scope specified in the previous step. They help further define the limits of the evaluation. The evaluation questions will be answered in the evaluation report and should be formulated to address the needs of the specific audience to whom the evaluation is directed. Evaluation questions should be developed for each component of the program which falls into the scope which was defined in the previous step. For example, questions may be formulated which concern the adequacy of the curriculum and the experience of the instructional staff; other questions may concern the appropriateness of the skills being taught in relation to employment opportunities in the local community; additional questions may relate to the appropriateness of the individuals being recruited for the program with respect to literacy and employment status; and finally, evaluation questions may relate to the extent to which participants are achieving the goals of the program, such as obtaining a GED, high school diploma, or specific job skills, stable employment and increased earning capability.
A good way to begin formulating evaluation questions is to carefully examine the program's objectives; another source of questions is to anticipate problem areas concerning program implementation. Importantly, the audience for or expected users of the evaluation should be involved in developing the evaluation questions. This should never be left solely to the outside evaluator, no matter how familiar he or she is with the program. Once the evaluation questions are developed. they should be prioritized and examined in relation to the time and resources available. Once this is accomplished, the final set of evaluation questions can be selected.
This step involves specifying the approach to answering the evaluation questions, including how the required data will be collected. This will involve:
The design and data collection plan is actually a roadmap for carrying out the evaluation. An important part of the design is the development or selection of the instruments for collecting and recording the data needed to answer the evaluation questions. Data collection instruments may include record-keeping forms, questionnaires, interview guides, and assessment measures of literacy and vocational skills. Some of the instrumentation may already be available, i.e., forms used for recordkeeping and management purposes, such as recruitment or intake forms, etc. Some of these forms will have to be modified to meet the evaluation needs. In other cases, new instruments will have to be created.
In designing the instruments, the relevance of the items to the evaluation questions and the ease or difficulty of obtaining the desired data should be considered. Thus, the instruments should be reviewed by the project director and staff members to ensure that the data can be obtained in a cost-effective manner and without causing major disruptions, or inconveniences to the project.
Data collection should follow the plans developed in the previous step. The individuals assigned to the various data collection tasks need to be thoroughly trained in the data collection requirements and procedures. Only by following standardized procedures will the data be reliable and valid. Following training, the project director needs to monitor the staff to ensure that they are accomplishing their data collection assignments according to the specified time schedule. The data should be recorded carefully and neatly so they can be read and interpreted during the analysis stage. Proper record-keeping and filing are similarly important so that the data are not lost or misplaced. Problems should be immediately brought to the attention of the project director and evaluator. Deviations from the data collection plan should be documented so that they can be considered in analyzing and interpreting the data.
This step involves tabulating, summarizing, and interpreting the collected data in such a way as to answer the evaluation questions. These procedures should be compatible with the type and amount of data which were collected, and the goals and objectives of the evaluation. Appropriate descriptive measures (frequency and percentage distributions, central tendency and variability, correlation, etc.) and inferential techniques (significance of difference between means and other statistics, analysis of variance, chi-square, etc.) should be used to analyze the data. The local evaluator should have responsibility for this aspect of the evaluation.
The evaluation will not be completed until a report has been written and the results communicated to the project director and other appropriate administrators and decision-makers. In preparing the report, the writers should be clear about the audience for whom the report is being prepared. Two broad questions need to be considered: (1) What does the audience need to know about the evaluation results? and (2) How can these results be best presented? Different audiences need different levels of information. Administrators need general information for policy decision-making. while program staff may need more detailed information which focuses on program activities and effects on participants.
The report should cover the following:
Importantly, the report should be organized so that it clearly addresses all of the evaluation questions specified in Step 2.
The evaluation should not be considered successful until its results are used by program managers and decision-makers for program improvement. After all, this is the ultimate reason for conducting the evaluation. The evaluation may indicate that a program activity is not being implemented according to plan. or it may indicate that a particular objective is not being met. If so, it is then the responsibility of the project director to make appropriate changes to remedy the situation. Project directors should never be satisfied with their programs. Improvements can always be made. and evaluation is an important tool for accomplishing this purpose.
The evaluation should be conducted by an independent, experienced evaluator. This individual will provide the expertise for an evaluation which is comprehensive, objective, and technically sound. The project director and her/his staff must work closely with the evaluator beginning with the planning stage to ensure the evaluation meets the exact needs of the program.
Adequate time and thought for planning an evaluation is essential, and will give the project director and staff an opportunity to develop ideas about what they would like the evaluation to accomplish. The evaluation should address the goals specified in the project application and management plan. In some projects, however, one or more goals or objectives may require special attention. Some activities or instructional strategies may have been recently implemented, or the staff may be aware of some special problems which should be addressed. For example, there might have been a recent breakdown in communication between instructors; or the characteristics of students in recent instructional groups might have begun to differ significantly from the past, having implications for vocational training or the approach to literacy instruction. These are examples of things which should be considered when the project director selects an evaluator. The evaluator must then familiarize himself or herself with the special issues of concern on which the evaluation should focus.
Thus, the initial step of the evaluation process involves thinking about any special needs which will help in planning the overall evaluation and selecting the evaluator. Problems identified and evaluation questions which focus on curriculum and instructional materials might suggest that an evaluator is needed with particular expertise in those areas. Similarly, if the project needs an evaluation which requires frequent on-site observation of teaching methodologies by the evaluator, then this will help the project director focus on hiring someone located nearby so that travel costs can be kept to a minimum.
In summary, defining the scope involves setting limits, identifying specific areas of inquiry and deciding on what parts of the program and on which objectives the evaluation will focus. The scope does not answer the question of how the evaluation win be conducted. In establishing the scope, one is actually determining which components or parts of the program will be evaluated, and implies that the evaluation may not cover every aspect and activity.
Selecting an evaluator is one of the most important elements in ensuring a technically sound and useful evaluation. The basic criteria suggested for selection are as follows:
The evaluator must be available for the complete time period required for the evaluation at a rate that fits basic budget resources and the number of evaluator-days projected in the initial planning. The evaluator should also reside within reasonable distance of the project so that travel costs are minimal and scheduling work sessions is not a problem.
Once an individual has been selected and has agreed to become the local evaluator for the project, a contract and work plan must be developed so that expectations, roles, and responsibilities are clear to all parties.
In order for the evaluation to be planned and carried out effectively, the roles of the project director, staff, and evaluator must be made clear to all parties. The evaluator should be responsible for specifying the design and developing the data collection instruments, training project staff to collect the required data, analyzing the data, and preparing a comprehensive report. The project director should work with the evaluator in specifying the objectives and scope of the evaluation, and assigning and supervising the project staff in carrying out their assigned data collection and record-keeping tasks. The project staff should be assigned the major data collection and record-keeping tasks for the evaluation. This will increase their job responsibilities, but is the most cost-effective way of collecting the required data. More specifically, the suggested roles of the project director, the evaluator, and the project staff are listed below:
The project director will:
The evaluator will:
The project staff will:
Critical questions in planning the evaluation concern how much staff time and financial resources can be expended on the evaluation, and what resources are needed to actually carry it out. These two questions need to be addressed by the project director and the evaluator early in the design process. Many evaluators will raise these questions in the initial exploratory discussions concerning the evaluation. Preliminary planning will have to be accomplished before a final decision can be made on both the role and level of effort of the evaluator and each project staff member.
A variety of design decisions must be made during the planning stages that affect the allocation of both financial and staff resources. Each decision affects the staff/evaluator roles and functions, level of effort, and resources, and ultimately determines the overall scope of the evaluation. For instance. a record-keeping procedure must be developed to maintain accurate evaluation data. A tracking system is also needed to manage follow-up surveys of former students. These record-keeping and tracking systems, as well as the actual data collection activities, involve staff and material costs. These costs need to be considered in defining the final scope of the evaluation.
To summarize, this chapter presented a general process which focused on how an evaluation may be planned and carried out. The next chapter, and in fact the remainder of this Guide, discusses what program components, evaluation questions, and data should be included in an evaluation of a local vocational or adult education program.
-###-
III. Evaluation Framework