Archived: U.S. Department of Education Strategic Plan, 1998-2002: Appendix B

A r c h i v e d I n f o r m a t i o n

U.S. Department of Education Strategic Plan, 1998-2002 - September 1997

Appendix B: Descriptions of Program Evaluations and Other Studies (continued)

Descriptions of Program Evaluations and Other Studies

Study or Data Collection Who sponsors?
Contractor or
grantee? Who collected on? What collected? When collected?
(Years collected,
frequency)

Postsecondary, Vocational, School-to-Work, and Adult Program Evaluations

11. College Preparation Studies PES Middle-grade students and their parents Research studies are being done using the National Education Longitudinal Survey (NELS) to identify key factors affecting which eighth grade students enrolled in college within two years of expected high school graduation and which did not. In addition, a series of focus groups and a survey of middle school parents are being conducted to develop a better understanding of what information parents of middle school students want and need concerning the benefits, academic requirements, and cost of college and the availability of student aid. These studies will be used to help formulate the national campaign aimed at encouraging middle-school students and their parents to prepare for college. Research analyses: 1996 and 1997
Surveys and focus groups: 1997
Future studies are planned for 1998

12. Evaluation of Effective Programs and Practices for Adult ESL Students PES and OVAE (Office of Vocational and Adult Education)
Contractor: American Institutes for Research, Inc.
Adult participants in English as a second language programs who have low native-language literacy Data on learning gains and persistence, and such longer-term outcomes as advancement to higher level English-language instruction, securing employment or job promotions, and removal from welfare. 1998-2000

13. Evaluation of Effective Adult Basic Education Programs and Practices PES and OVAE
Contractor: Abt Associates, Inc.
Participants in low-level Adult Basic Education classes Data on learning gains and persistence, and such longer-term outcomes as advancement to higher level literacy instruction, securing employment or job promotions, and removal from welfare. 1998-2000

14. Longitudinal Study of the Vocational Rehabilitation (VR) Service Program PES and OSERS (Office of Special Education and Rehabilitative Services)
Contractor: Research Triangle Institute Inc.
A sample of 8,500 current and former VR consumers at 37 VR offices Information on the characteristics of VR consumers and local VR agency offices; the impact of local economic and environmental factors on services and outcomes; and short and long-term outcomes achieved by VR consumers and their satisfaction with the program. 1994-1998

15. National Direct Student Loan Evaluation PES
Contractor: Macro, Inc.
Representative samples of postsecondary institutions, students, and parents participating in the Direct Loan and Federal Family Education Loan (FFELP) programs The evaluation centers on three major issues:
(1) Institutions' Administration of the Direct Loan Program: The study is identifying areas where Direct Loan institutions are having problems and also on identifying and disseminating best practices. These data will be collected through mail surveys of Direct Loan institutions and site visits to schools.
(2) Borrower Understanding and Repayment Experience: The study is surveying Direct Loan and FFEL borrowers to determine their satisfaction with their loan program as well as their knowledge and understanding of the loan origination and repayment processes. Information will be obtained through telephone surveys of Direct Loan and FFELP borrowers who are in-school as well as in-repayment.
(3) ED's Administration of the Direct Loan Program: The study will focus on the outcomes of ED's administration rather than internal processes, with the exception of certain key areas. Outcomes will be assessed mainly through institutions' and borrowers' satisfaction with ED provided information and services obtained in surveys of Direct Loan and FFELP institutions and borrowers as well as interviews of ED and other Federal Officials, and reviews of documents and reports.
Case studies annually from 1994 to 1998
Institutional surveys annually from 1995 to 1998
Borrower surveys 1996, 1997, and 1998
ED interviews annually from 1994 to 1998
Key aspects of the evaluation will be continued in future years

16. National Evaluation of School-to-Work PES
Contractor: Mathematica Policy Research, Inc. (MPR)
Surveys of 12th grade students; transcripts; follow-ups, surveys of local partnerships; case studies of eight states and 39 communities Information on participation, high school programs of study, postsecondary enrollment and employment; STW implementation 1996-2000

17. Postsecondary Gatekeeping Studies PES Purposive samples of postsecondary institutions, accrediting agencies, and states chosen either because of the type of instruction being provided or to identify and disseminate best practices Several studies have been or will be conducted assessing how the gatekeeping process is functioning. Examples of these studies include: examination of state oversight practices since the 1992 reauthorization of the Higher Education Act; assessment of the standards used in accrediting Associates in Applied Sciences Degrees; exploration of the use of secondary data sources, such as unemployment insurance files, to determine employment outcomes for students attending vocationally-oriented programs; and analysis of the standards employed by accrediting associations in reviewing graduation and placement rates in proprietary institutions. Variable depending on emerging issues

18. Postsecondary Customer Satisfaction Surveys PES and Office of Postsecondary Education (OPE) Representative samples of postsecondary institutions and students participating in the postsecondary education programs Surveys have been or are being conducted of postsecondary institutions satisfaction with the operation and management of the student aid programs, TRIO, and the National Student Loan Data System and student aid applicants' satisfaction with the application process. Institutional surveys, 1995 and 1997
Applicant survey, 1997
Additional surveys will be conducted in future years

19. Postsecondary Secondary Data Analyses PES Analysis of information previously collected through NCES or Census surveys or in ED administrative files Studies attempt to assess the effectiveness of the student aid programs by analyzing a number of issues including the educational attainment of low- versus high-income students, trends in the affordability of postsecondary education, debt burden, and the distribution of aid across student characteristics. Annual, with the type of analysis dependent on emerging policy issues and data availability

20. Targeted Student Aid Studies PES Representative samples of specific populations who are affected by changes in legislation or policy Small, one-time surveys of specific populations are conducted to assess the effects of major changes in student aid policy. Examples of this type of study are surveys of students affected by changes in the reauthorization of the Higher Education Act (HEA) and students and institutions participating in the Federal Work-Study (FWS) program. Study of 1992 HEA reauthorization was completed in 1997. FWS study: 1997-98
Planned study of next HEA reauthorization to begin in 1999

21. Title III (Higher Education Act) Performance Measurement Study PES
Contractor: Math Tech, Inc.
Institutions participating in the Title III (Higher Education Act, HEA) program This study is designed to develop a system of performance indicators for judging the success of the Title III programs. The four goals of the study are to: (1) define program goals in measurable terms; (2) determine how federal management activities contribute to program goals; (3) determine how institutional activities contribute to program goals; and (4) establish an annual progress report. Information to accomplish these goals will be obtained from a survey of all Title III participating institutions and case studies at between 30 and 40 selected sites. Both the survey and case studies will be conducted in 1997

22. TRIO-Student Support Services (SSS) Evaluation PES
Contractor: Westat, Inc.
A nationally representative sample of 2,900 students participating in SSS projects at 30 colleges and a control group of 2,900 students consisting both of students at the 30 SSS institutions who did not participate in the program and students at 20 matched institutions not participating in the SSS program. Case studies were conducted at the 50 schools. Data are being collected to assesses the effects of the SSS program on college retention and graduation, GPA, and transfer behavior (from 2-year to 4-year institutions). A second and equally important focus of the evaluation is the identification of effective services and practices for program improvement. A baseline survey was conducted in 1991-92 with a follow-up in 1994 and another planned for 1997. College transcripts were/will be collected with each survey.

23. TRIO-Upward Bound Evaluation PES
Contractor: Mathematica Policy Research, Inc.
A nationally representative sample of 67 Upward Bound (HEA) projects hosted by two and four year colleges, from which 2,700 eligible applicants were randomly assigned to Upward Bound or to a control group Data are being collected to assess the effects of Upward Bound (HEA) on students' preparation for college, high school graduation, college entry and achievement. Impacts are estimated by comparing students in Upward Bound to the control group on a range of measures, including grades, course-taking, attitudes and educational expectations. Additional data will be collected to assess longer term impacts on college persistence and graduation. A baseline survey was conducted beginning in December 1992, with follow-up surveys undertaken in 1994 and 1996, at which time high school transcripts were also collected. Future surveys will capture data on college persistence and graduation.

-###-

[Appendix B (part 1 of 5)]

[Appendix B (part 3 of 5)]

Descriptions of Program Evaluations and Other Studies
	Study or Data Collection	Who sponsors? Contractor or grantee?	Who collected on?	What collected?	When collected? (Years collected, frequency)
Postsecondary, Vocational, School-to-Work, and Adult Program Evaluations
11.	College Preparation Studies	PES	Middle-grade students and their parents	Research studies are being done using the National Education Longitudinal Survey (NELS) to identify key factors affecting which eighth grade students enrolled in college within two years of expected high school graduation and which did not. In addition, a series of focus groups and a survey of middle school parents are being conducted to develop a better understanding of what information parents of middle school students want and need concerning the benefits, academic requirements, and cost of college and the availability of student aid. These studies will be used to help formulate the national campaign aimed at encouraging middle-school students and their parents to prepare for college.	Research analyses: 1996 and 1997 Surveys and focus groups: 1997 Future studies are planned for 1998
12.	Evaluation of Effective Programs and Practices for Adult ESL Students	PES and OVAE (Office of Vocational and Adult Education) Contractor: American Institutes for Research, Inc.	Adult participants in English as a second language programs who have low native-language literacy	Data on learning gains and persistence, and such longer-term outcomes as advancement to higher level English-language instruction, securing employment or job promotions, and removal from welfare.	1998-2000
13.	Evaluation of Effective Adult Basic Education Programs and Practices	PES and OVAE Contractor: Abt Associates, Inc.	Participants in low-level Adult Basic Education classes	Data on learning gains and persistence, and such longer-term outcomes as advancement to higher level literacy instruction, securing employment or job promotions, and removal from welfare.	1998-2000
14.	Longitudinal Study of the Vocational Rehabilitation (VR) Service Program	PES and OSERS (Office of Special Education and Rehabilitative Services) Contractor: Research Triangle Institute Inc.	A sample of 8,500 current and former VR consumers at 37 VR offices	Information on the characteristics of VR consumers and local VR agency offices; the impact of local economic and environmental factors on services and outcomes; and short and long-term outcomes achieved by VR consumers and their satisfaction with the program.	1994-1998
15.	National Direct Student Loan Evaluation	PES Contractor: Macro, Inc.	Representative samples of postsecondary institutions, students, and parents participating in the Direct Loan and Federal Family Education Loan (FFELP) programs	The evaluation centers on three major issues: (1) Institutions' Administration of the Direct Loan Program: The study is identifying areas where Direct Loan institutions are having problems and also on identifying and disseminating best practices. These data will be collected through mail surveys of Direct Loan institutions and site visits to schools. (2) Borrower Understanding and Repayment Experience: The study is surveying Direct Loan and FFEL borrowers to determine their satisfaction with their loan program as well as their knowledge and understanding of the loan origination and repayment processes. Information will be obtained through telephone surveys of Direct Loan and FFELP borrowers who are in-school as well as in-repayment. (3) ED's Administration of the Direct Loan Program: The study will focus on the outcomes of ED's administration rather than internal processes, with the exception of certain key areas. Outcomes will be assessed mainly through institutions' and borrowers' satisfaction with ED provided information and services obtained in surveys of Direct Loan and FFELP institutions and borrowers as well as interviews of ED and other Federal Officials, and reviews of documents and reports.	Case studies annually from 1994 to 1998 Institutional surveys annually from 1995 to 1998 Borrower surveys 1996, 1997, and 1998 ED interviews annually from 1994 to 1998 Key aspects of the evaluation will be continued in future years
16.	National Evaluation of School-to-Work	PES Contractor: Mathematica Policy Research, Inc. (MPR)	Surveys of 12th grade students; transcripts; follow-ups, surveys of local partnerships; case studies of eight states and 39 communities	Information on participation, high school programs of study, postsecondary enrollment and employment; STW implementation	1996-2000
17.	Postsecondary Gatekeeping Studies	PES	Purposive samples of postsecondary institutions, accrediting agencies, and states chosen either because of the type of instruction being provided or to identify and disseminate best practices	Several studies have been or will be conducted assessing how the gatekeeping process is functioning. Examples of these studies include: examination of state oversight practices since the 1992 reauthorization of the Higher Education Act; assessment of the standards used in accrediting Associates in Applied Sciences Degrees; exploration of the use of secondary data sources, such as unemployment insurance files, to determine employment outcomes for students attending vocationally-oriented programs; and analysis of the standards employed by accrediting associations in reviewing graduation and placement rates in proprietary institutions.	Variable depending on emerging issues
18.	Postsecondary Customer Satisfaction Surveys	PES and Office of Postsecondary Education (OPE)	Representative samples of postsecondary institutions and students participating in the postsecondary education programs	Surveys have been or are being conducted of postsecondary institutions satisfaction with the operation and management of the student aid programs, TRIO, and the National Student Loan Data System and student aid applicants' satisfaction with the application process.	Institutional surveys, 1995 and 1997 Applicant survey, 1997 Additional surveys will be conducted in future years
19.	Postsecondary Secondary Data Analyses	PES	Analysis of information previously collected through NCES or Census surveys or in ED administrative files	Studies attempt to assess the effectiveness of the student aid programs by analyzing a number of issues including the educational attainment of low- versus high-income students, trends in the affordability of postsecondary education, debt burden, and the distribution of aid across student characteristics.	Annual, with the type of analysis dependent on emerging policy issues and data availability
20.	Targeted Student Aid Studies	PES	Representative samples of specific populations who are affected by changes in legislation or policy	Small, one-time surveys of specific populations are conducted to assess the effects of major changes in student aid policy. Examples of this type of study are surveys of students affected by changes in the reauthorization of the Higher Education Act (HEA) and students and institutions participating in the Federal Work-Study (FWS) program.	Study of 1992 HEA reauthorization was completed in 1997. FWS study: 1997-98 Planned study of next HEA reauthorization to begin in 1999
21.	Title III (Higher Education Act) Performance Measurement Study	PES Contractor: Math Tech, Inc.	Institutions participating in the Title III (Higher Education Act, HEA) program	This study is designed to develop a system of performance indicators for judging the success of the Title III programs. The four goals of the study are to: (1) define program goals in measurable terms; (2) determine how federal management activities contribute to program goals; (3) determine how institutional activities contribute to program goals; and (4) establish an annual progress report. Information to accomplish these goals will be obtained from a survey of all Title III participating institutions and case studies at between 30 and 40 selected sites.	Both the survey and case studies will be conducted in 1997
22.	TRIO-Student Support Services (SSS) Evaluation	PES Contractor: Westat, Inc.	A nationally representative sample of 2,900 students participating in SSS projects at 30 colleges and a control group of 2,900 students consisting both of students at the 30 SSS institutions who did not participate in the program and students at 20 matched institutions not participating in the SSS program. Case studies were conducted at the 50 schools.	Data are being collected to assesses the effects of the SSS program on college retention and graduation, GPA, and transfer behavior (from 2-year to 4-year institutions). A second and equally important focus of the evaluation is the identification of effective services and practices for program improvement.	A baseline survey was conducted in 1991-92 with a follow-up in 1994 and another planned for 1997. College transcripts were/will be collected with each survey.
23.	TRIO-Upward Bound Evaluation	PES Contractor: Mathematica Policy Research, Inc.	A nationally representative sample of 67 Upward Bound (HEA) projects hosted by two and four year colleges, from which 2,700 eligible applicants were randomly assigned to Upward Bound or to a control group	Data are being collected to assess the effects of Upward Bound (HEA) on students' preparation for college, high school graduation, college entry and achievement. Impacts are estimated by comparing students in Upward Bound to the control group on a range of measures, including grades, course-taking, attitudes and educational expectations. Additional data will be collected to assess longer term impacts on college persistence and graduation.	A baseline survey was conducted beginning in December 1992, with follow-up surveys undertaken in 1994 and 1996, at which time high school transcripts were also collected. Future surveys will capture data on college persistence and graduation.