Program outcome data are used to determine the extent to which a program is meeting its goals and objectives for students or trainees. These goals and objectives may include:
As with the other evaluation components; the project director and evaluator must work jointly to define the scope of the outcome data to be collected. This should be accomplished by developing a set of evaluation questions to assess the extent to which the project's goals and objectives are met. Examples of evaluation questions directed at the outcomes of programs are shown in Exhibit 5. Also shown are the relevant variables which relate to the questions and possible sources from whom data maybe collected. Project directors and local evaluators should carefully examine these questions and select those which are applicable to their program. Additional evaluation questions may also be specified which address any special issues and concerns of the local program.
Depending on the goals of the program, data required at program completion may include:
Additional data will also be required depending on the final set of evaluation questions specified by the project director and evaluator.
The data on achievement of educational and vocational training objectives and competencies should be available from project records. The measure(s) should be the same as were used at program entry. In this manner, pre/post-test differences can be calculated to measure change. Data on job placements and wages may be obtained from program records or collected directly from the former trainees.
Data collected at follow-up may include:
As with the program completion data, other data may be required depending on the evaluation questions which are specified.
Follow-up data can be collected from former students or trainees or from their employers. First priority should be given to collecting data from former students, rather than from busy employers. It is suggested that contacts with employers to obtain project evaluation data be kept to a minimum.
Data collection instruments which may be useful for evaluation purposes include a Student Follow-Up Telephone Survey, a Job Placement Information Form and Follow-Up Questionnaire, and an Employer Follow-Up Questionnaire. Samples of these data collection instruments are found in Appendices A4, A5, A6, and A7. These examples may be adapted to fit the needs of each local program.
Appropriate data collection procedures and record-keeping systems need to be developed to ensure that program completion, placement, and follow-up data are collected, recorded, and filed. This will ensure that the data are readily available for tabulation and summary to meet the time schedule for the evaluation. Early planning by the evaluator and project director is one of the key elements to success in any data collection effort. This is especially true for the development of a record-keeping system for the collection of follow-up data. First, a tickler file or calendar must be established so that program staff can keep track of when each student or trainee has been placed in a job, and when follow-up is required. Without such a system, the staff will not be able to keep track of the appropriate time to follow-up each program completer. Second, an address file and a placement file for program completers need to be continually up-dated. Addressed and stamped post-cards may be given to students when they complete training so they can notify the program of address and job changes in a convenient manner. Alternatively, students can be asked to notify the project staff by telephone when they change addresses or jobs. Without these procedures, follow-up will be extremely difficult.
Follow-up data may be collected via telephone interviews with former students. Before completing the program, each student should be told to expect these calls, and to keep the program informed of address and telephone number changes. On the other hand, collecting follow-up data from employers is a very difficult task, as employers are reluctant to give out personnel information, and generally are busy and do not like to be bothered. However, if collection of data from employers is required, program staff should lay the groundwork for follow-up by building a good rapport with employers when students are placed. By establishing good relationships at these times, follow-up data can be more easily collected.
The evaluator, with input from the project director, should be responsible for developing the instruments. procedures, and record-keeping systems. The project staff, under the supervision of the project director, should have the responsibility of collecting and recording the required data. Training the staff in all data collection and record-keeping procedures should be the responsibility of the evaluator.
| Evaluation Questions | Variables | Possible Data Sources | |
| A. | Program Completion | ||
| 1. | How many students/trainees completed the training program? | Number of Completions | Program Records |
| 2. | To what extent were students' objectives met? What is degree of completers' satisfaction with training program? | Ratings of Satisfaction | Completers |
| 3. | To what extent did students increase their literacy skills? | Pre/Post Measures of Literacy Skills; Achievement of Training Objectives | Program Records |
| 4. | To what extent did students/trainees increase their vocational skills and knowledge? | Achievement of Vocational Training Objectives and Competencies; Pre/Post Measures of Vocational Skills and Knowledge | Completers; Program Records |
| 5. | How many completers were placed in jobs following training? How many were placed in full-time and part-time training-related jobs? | Number of Completers Placed in Jobs | Completers; Program Reports |
| 6. | What is the average gross wage per hour of completers who were placed in jobs? | Wages Per Hour | Completers |
| 7. | What is the average number of hours worked per week of those placed in jobs? | Number of Hours Worked | Completers |
| 8. | How many completers were placed in or enrolled in further education or training programs following completion of training? | Number of Completers Placed or Enrolled for Further Education or Training. | Completers |
| B. | Follow-Up | ||
| 1. | How many program completers were employed six months following initial placement in training-related and non-training related, Full-time and part-time jobs? | Number of Completers Employed | Completers; Employers |
| 2. | What is average gross wage per hour of completers six months following initial placement? | Wages Per Hour | Completers; Employers |
| 3. | What is the average number of hours worked per week of completers six months following initial placement? | Number of Hours Worked | Completers; Employers |
| 4. | How many completers are enrolled in school or other training program six months following initial placement? | Number of Completers Enrolled in School or Training Program | Completers |
| 5. | How many completers are unemployed six months following initial placement? | Number of Completers Unemployed | Completers |
| 6. | How many completers were employed at any time during the six month period following initial placement? | Number of Completers Employed at Any Time During Follow-up Period | Completers; Employers |
| 7. | How many completers obtained job promotions during the six month period following initial placement? | Number of Completers Who Obtained Job Promotions or Higher Level jobs | Completers; Employers |
| 8. | How many completers received wage increases or obtained a higher paying job during the six month period following initial placement? | Number of Completers Who Received Wage Increase | Completers; Employers |
| 9. | How satisfied are former participants with the jobs six months following initial placement? | Ratings of Job Satisfaction | Completers |
| 10. | How satisfied are employers with the vocational and literacy skills of the program completers they hired? | Ratings of Employee Satisfaction with Vocational Skills and English Language Skills | Employers |
VIII. Presentation of Findings