A Study of the Talent Search Program (1995)
Analysis and Highlights

Past studies of Talent Search have revealed difficulty in measuring the overall effectiveness of the program due to variation across projects in the target population, program emphasis, and data collection and reporting practices. Therefore, in the Fall of 1993, instead of launching an evaluation of the program, the Planning and Evaluation Service (PES) in the U.S. Department of Education awarded a contract to Decision Information Resources, Inc. to conduct study that would produce the following things:

• Information on the needs of the target population and effective interventions;

• Recommendations on performance measures that could be used by projects, in a national evaluation effort, and on the Annual Performance Report (APR) forms; and

• Recommendations on ways to improve the Annual Performance Report forms that projects submit to ED.

Main highlights from the report:

First Generation College Students: A Review of Needs and Effective Interventions:

Performance Measures and the APR for the Talent Search Program:

Every year each Talent Search project submits an Annual Performance Report (APR) form that contains information on the demographic characteristics of participants, the services provided, and the number of participants who enrolled in postsecondary education. However, over the past several years PES has been concerned about the small amount of useful outcome data provided on the forms. For example, the number of participants who enrolled in postsecondary education and the percentage of participants who enrolled in different kinds of postsecondary institutions, (public, two-year schools, private 4-year schools, etc..), are the only outcome measures for which projects provide data on the APR. No outcome data on the progress of middle school students have ever been provided or requested by ED.

In addition, PES is concerned about the validity of the outcome data because of the way that projects collect and report the information. Many projects report the number of students who were accepted by postsecondary institutions -- not the number of students who actually enrolled.

DIR's Recommendations:

• Include all performance measures on the APR. DIR visited seven case study sites and found that most of them limit their performance measurement system to the information needed for the APR. Therefore, DIR believes that efforts to encourage internal, project-level assessment and to improve the quality of data obtained from the projects, should start with a revision of the APR. DIR believes that by including all performance measures on the APR, projects will more diligently collect and report data.

• Add input measures and increase the number of process and outcome measures. DIR recommends the expansion and improvement of the two kinds of performance measures that are currently included on the APR:

  1. Process measures. These are measures of the activities and interventions that are used to achieve the program's goals, as well as the characteristics of the participants. DIR recommends that ED require the projects to provide descriptive data on a greater number of program services, as well as information on the frequency of student participation in each service.

  2. Outcome measures. These are measures that indicate the extent to which the program has achieved its goals. DIR recommends that a wider set of program outcome measures be included on the APR. Their recommendations include measures of the high school graduation rate of Talent Search participants, and the percentage of middle school participants who successfully complete the school year and advance to the next grade.

  DIR also recommends inclusion of Input measures that provide information regarding program resources such as budget, staff, administrative arrangements, and characteristics of the target population. DIR's proposed input measures include "the number of full-time staff" and the "percentage of high school starters who complete high school in the target schools."

• Provide projects with more assistance. DIR estimates that most of the Talent Search projects would need the equivalent of one additional staff person in order to collect and report most of the proposed performance measures. They will also need technical assistance and guidance from ED on standard definitions and data collection practices.

• Include additional measures in a national evaluation: DIR recommends that a national evaluation include all of the measures recommended for the APR, as well as additional measures that would be too burdensome for routine data collection by Talent Search projects. An example of such measures are those that include a comparison of the outcomes of Talent Search participants to the outcomes of a comparison group of students who did not participate in Talent Search.

  Implications of DIR's Findings and Recommendations:

  Before adding any performance measures to the APR and encouraging projects to assess their own performance, ED will need to address two concerns:

  • Which population of participants will be measured? Should all participants be included in the population to which the measures are applied? Students often participate in Talent Search on an intermittent basis and some only receive one or two hours of service.

  • Will the cost of collecting and reporting performance data be excessive? Talent Search resources are already thinly spread; projects spend approximately $230 per participant per year. The addition of many performance measures to the APR will force projects to spend a much greater portion of their resources on the collection and reporting of data. ED will need to make sure that the cost of implementing performance measures does not outweigh the value of the information that they provide.

-###-

Return to Higher Education Evaluation Page