This chapter presents a framework for evaluating local programs which combines outcome evaluation with process evaluation. An outcome evaluation attempts to determine the extent to which a project's specific objectives have been achieved. On the other hand, the process evaluation seeks to describe the program which was implemented, and through this, attempt to gain an understanding of why the objectives were or were not achieved.
Evaluators have been criticized in the past for focusing on outcome evaluation and excluding the process side, or focusing on process evaluation without examining outcomes. The framework presented here incorporates both the process and outcome side. In this manner, one can determine the effect (or outcome) of a program, and also understand how the program produced that effect and how the program might be modified to produce that effect more completely and efficiently.
In order to focus on both program process and outcomes, an evaluation should be designed in which evaluation questions, and data collection and analysis, address the following:
These components may be thought of as being linked in the following manner:
Using this framework, descriptions are prepared of the environment, the participants, and the program activities and services which are implemented. Outcomes of the program are also assessed. The description of the environment, participants, and activities and services are used to explain how the outcomes were achieved and to suggest changes which may produce these outcomes more effectively and efficiently.
Each evaluation component is described below.
This component defines the environment in which a program is implemented. This includes the community in which the program is located, the economy of the community, the job opportunities in the community, and the educational and job training needs of the program's target population. Understanding the environment in which the project is located is important to assessing and interpreting the outcomes of the program, especially such outcomes as job placement, job retention, or job advancement. Data concerning this component are collected through a needs assessment which should be completed prior to applying for funding. These data are generally presented in a grant application in order to make the case for the need for the program in the first place.
This component defines the characteristics of the participants, including literacy, educational level, previous vocational training, and work history. Data on the characteristics of participants may be collected as part of program recruitment and intake activities. In addition to their use for descriptive purposes, these data are useful for comparisons with similar data collected at program completion and at follow-up data collection periods.
This component describes how the key activities of the program are implemented, including recruitment, intake and assessment; instruction; counseling and other support services; and job development and job placement, if applicable. In this manner, the outcomes or results achieved by the program can be attributed to what actually has taken place, rather than what was planned to occur. This component also addresses the questions of what services and activities of the program have been fully implemented, partially implemented, and not implemented.
This component concerns the effects that the program has on its participants, and to what extent the program has met its stated objectives for program participants. At program completion, data may be collected on such variables as on rates of program completion, achievement of instructional objectives and competencies, vocational skills and knowledge, English language proficiency, placement in a job, continued education/instruction, and satisfaction with the instruction received.
If applicable, follow-up data may be collected at appropriate points in time following program completion on employment rates of former participants, wages per hour, number of hours worked per week, number receiving job promotions, number receiving wage increases, ratings of job satisfaction, and ratings of employer satisfaction with vocational and literacy skills.
Using the above four evaluation components, a comprehensive assessment of a program may be designed. Not only will this evaluation approach allow the project staff to determine the extent to which project goals and objectives are met, but will also enable them to understand how those outcomes were achieved and to make changes for program improvement purposes in the future.
The evaluation framework, presented above should be implemented using the six-step process described in Chapter II. The framework describes what should be included in the evaluation; the six-step process describes how the evaluation is to be planned and carried out. Guidelines for defining the scope of the evaluation, specifying evaluation questions, and developing the data collection plans for each of the four evaluation components are discussed in the following chapters.
-###-
IV. Program Environment