Postsecondary Education Institutions' Satisfaction
with Student Financial Assistance Programs
Higher Education Surveys Report
Survey Number 20, July 1996
This report presents the results of a survey of postsecondary institutions concerning their satisfaction with the delivery of federal student financial assistance programs (SFAP). The purpose of the survey was to determine both institutions' overall level of satisfaction and their satisfaction within several categories of the U.S. Department of Education's (ED) customer support services (program materials and publications, training, the application process, electronic processing, inquiry and information services, funds management, program reviews and audits, and program operations). The data will be used to examine the current quality of ED's support of institutions' financial aid processing and to provide baseline data for measuring changes in satisfaction over time. In addition, institutions were asked for their impressions of student satisfaction with the programs to provide ED with preliminary information until a student survey is conducted.
- Executive Summary
- Background
- Overall Levels of Satisfaction
- Levels of Familiarity and Use of Services and Products
- Background
-
The Types of Services That Were Least Used or Least Familiar
- The Characteristics of Respondents with the Least Familiarity
- Levels of Satisfaction in Specific Areas
-
Program Materials and Publications
- Training
- Application Process
- Electronic Processing
- Inquiry and Information Services
- Funds Management
- Program Reviews and Audits
- Program Operations
- Impressions of Student Satisfaction
- Training
-
Differences Among the Respondents in Their Evaluations
- Summary
- Appendix A: Detailed Tables
- Appendix B: Technical Notes
- Summary
-
Higher Education Surveys
- Survey Methodology
- Reliability of Survey Estimates
- School Type Relationships
- Survey Methodology
-
Appendix C: Survey Questionnaire [not available in HTML format; please see PDF version of report]
LIST OF FIGURES
- Percentage of institutions agreeing they were satisfied overall with ED assistance
- Percentage of institutions not using or unfamiliar with service
- Evaluation of program materials and publications
- Evaluation of training
- Evaluation of application process
- Means of forwarding information electronically
- Evaluation of ED's data management and communication software for forwarding information electronically
- Evaluation of electronic processing
- Evaluation of inquiry and information services
- Evaluation of funds management
- Evaluation of program reviews and audits
- Evaluation of program operations
- Impressions of student satisfaction
Executive Summary
Introduction
This report presents the results of a survey of postsecondary institutions concerning their satisfaction with the delivery of federal student financial assistance programs (SFAP). The purpose of the survey was to determine both institutions' overall level of satisfaction and their satisfaction within several categories of the U.S. Department of Education's (ED) customer support services (program materials and publications, training, the application process, electronic processing, inquiry and information services, funds management, program reviews and audits, and program operations). The data will be used to examine the current quality of ED's support of institutions' financial aid processing and to provide baseline data for measuring changes in satisfaction over time. In addition, institutions were asked for their impressions of student satisfaction with the programs to provide ED with preliminary information until a student survey is conducted.
The survey is one of a series of surveys examining various aspects of ED's student financial assistance programs. Two earlier surveys focused on institutions' satisfaction with the first year of implementation of the Federal Direct Loan Program and, for comparison purposes, on institutions' satisfaction with the Federal Family Education Loan Program (FFELP). Followups to the Direct Loan and FFELP surveys are currently being conducted. In addition, a new survey is being considered to assess students' satisfaction with the financial assistance programs. This survey differs from the Direct Loan and FFELP surveys in its broader focus-covering all student financial assistance programs-and in its focus on the financial aid process rather than overall program operation.
The survey questionnaire was mailed to a nationally representative sample of 971 postsecondary institutions. Of these, 136 either had closed or were not eligible for federal financial assistance programs; 774 of the remaining 835 institutions responded, for a response rate of 93 percent. The data were weighted to provide national estimates for the roughly 5,500 eligible institutions represented in the survey.
Findings
Overall Satisfaction
An estimated 73 percent of higher education institutions were pleased overall with the assistance they received from ED in managing and administering the federal student financial assistance programs (Figure 1). Another 13 percent were neutral, and 14 percent expressed some degree of dissatisfaction. Few schools (4 percent) were very satisfied; the remainder of the 73 percent were either satisfied or slightly satisfied.
The areas receiving the greatest overall satisfaction were program materials and publications (79 percent), the application process (76 percent), and electronic processing (74 percent). The areas receiving the least satisfaction were inquiry and information services (58 percent), assistance in program operations (55 percent), and reviews and audits (50 percent). The percentage that expressed dissatisfaction ranged from 8 percent to 19 percent across the eight areas, while the remainder were neutral.
Figure 1. Percentage of institutions agreeing they were satisfied overall with ED assistance
Full Size Figure 1
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
Familiarity with ED Services
Often, institutions were not familiar with or had not used many of the individual services offered by ED (Figure 2). This was especially true of the inquiry and information services-where three of the four listed items showed high levels of non-use (74-92 percent)-and program operations-where all three items showed either high non-use or unfamiliarity (48-67 percent). Since these same two program categories were among the lowest in overall satisfaction, the lack of use or familiarity may be a factor; respondents that were unfamiliar with the services or had not used them showed relatively low levels of satisfaction and relatively high levels of neutrality.
The ordering of items in Figure 2 partly reflects whether each questionnaire item was stated in terms of non-use or unfamiliarity. Seven of the top eight areas (all but program systems services) were phrased in terms of non-use.
Figure 2. Percentage of institutions not using or unfamiliar with service
Not included in electronic version.
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
The Characteristics of Services That Were Viewed the Most Favorably
Institutions provided considerable information about their satisfaction with specific products and services. Following are some of the general patterns that appeared. In those cases where a range of percentages is provided, the statistics summarize institutions' responses across more than one questionnaire item where the dimension was examined.
- The characteristics of program materials and publications that were viewed most positively were their clarity (88-90 percent), organization (86-90 percent), and ease of use (83-87 percent). The characteristic that was viewed least positively was timeliness (52-61 percent).
- Respondents most often evaluated training positively with respect to offering helpful information (75-82 percent), providing ample opportunity to ask questions (77-80 percent), and providing information used in day-to-day operations (74-84 percent), and least often with respect to the frequency of the training (62-69 percent).
- The most positively viewed areas of electronic processing were understandable hardware requirements (74-88 percent) and ease of installation (72-87 percent), while the least positively viewed areas were the ease of customization (36-47 percent) and compatibility with other software (46-52 percent).
- Program reviews and audits received the highest satisfaction concerning institutions knowing what to expect (62 percent), and the least concerning the level of burden (31 percent said they were not burdensome, and 47 percent said they were).
- Assistance in program operations received the highest satisfaction on having individuals who were courteous (69-75 percent), knowledgeable (69-73 percent), and who provided helpful information (67-73 percent); the lowest satisfaction was with accessibility (48-53 percent).
Differences Among Respondents in Their Level of Satisfaction
In general, respondents at for-profit institutions were more satisfied with the support services than those at other institutions. Respondents were also more satisfied if they were in financial aid offices than in business offices, and if they had 5 years or less of experience than if they had over 10 years of experience. They were less satisfied if they were at research/doctoral institutions than if they were at other types of institutions.
Background
Within the business world, customer satisfaction is a key concept in marketing. For businesses that depend on having repeat customers, keeping those customers satisfied is an important goal. Even those businesses that primarily sell one-time goods or services, and thus may not expect frequently returning customers, often depend on a reputation for good service or good products in order to win new customers. Accordingly, customer satisfaction has been a major topic within marketing research, with over 15,000 academic and trade articles published on the topic over two decades.1
The federal government generally does not need to market its goods and services in the way businesses do. Because the services it provides are often free, the demand for services may be greater than the ability to provide them. In fact, often the goal of a service is to make a continuation of that service unnecessary; for example, the provision of services to pursue education or to provide welfare support is intended to help the recipient to become self-sufficient.
While federal agencies may not need to compete for customers, often their purpose is to provide a set of services to a defined clientele. Public opinion concerning the government depends in part on the degree to which agencies perform that job effectively. In Executive Order 12862, President Clinton wrote,
Putting people first means ensuring that the Federal Government provides the highest quality service possible to the American people.
Some of the specific actions that the Executive order called for were that agencies identify their customers, survey them to determine what they need and their level of satisfaction, and establish service standards and measure results against them.
In response to Executive Order 12862, this study was requested by the U.S. Department of Education's (ED) Office of Postsecondary Education, Student Financial Assistance Programs (SFAP) in conjunction with the Planning and Evaluation Service. This office is charged with helping students and postsecondary education institutions apply for and work with the federal financial aid programs; collectively these programs each year process about 8 million applications, disburse over $30 billion, and work with about 8,000 postsecondary education institutions. There actually are two sets of customers for whom a customer satisfaction survey would be appropriate: students who receive or apply for financial aid, and postsecondary education institutions that administer much of the financial aid programs at the local level, and that often act as an intermediary between the students and the financial aid programs. Primarily, this study was directed at the satisfaction of postsecondary institutions with the ED student financial assistance services provided to them; a short section of the questionnaire was also devoted to ascertaining institutions' perceptions of student satisfaction, but a future student survey is being considered to obtain more accurate statistics on student satisfaction. The general topics covered in the study were program materials and publications, training, the application process, electronic processing, inquiry and information services, funds management, program reviews and audits, program operations, and impressions of student satisfaction. Within each section, questions were asked about specific services and products, including about such qualities as timeliness, comprehensiveness, accessibility, and ease of use. Detailed statistics on the survey responses are provided in the tables and graphics of this report, while the text primarily is concerned with the overall patterns.
The survey is one of a series of surveys examining various aspects of ED's student financial assistance programs. Two earlier surveys focused on institutions' satisfaction with the first year of implementation of the Federal Direct Loan Program and, for comparison purposes, on institutions' satisfaction with the Federal Family Education Loan Program (FFELP). Followups to the Direct Loan and FFELP surveys are currently being conducted. In addition, a new survey is being considered to assess students' satisfaction with the financial assistance programs. This survey differs from the Direct Loan and FFELP surveys in its broader focus-covering all student financial assistance programs-and in its focus on the financial aid process rather than overall program operation.
The survey questionnaire was mailed to a nationally representative sample of 971 postsecondary education institutions, including institutions from a Higher Education Surveys (HES) panel that has participated in repeated studies related to higher education and a supplemental sample of less-than-2-year institutions that were not in the HES panel. Of the 971 institutions, 102 were considered out of scope because they were eligible for neither the federal Pell Grant program nor the Stafford loan program, and 34 institutions had closed; 774 of the remaining institutions responded, for a response rate of 93 percent. Additional information on the survey methodology is provided in Appendix B of this report.
The survey was designed so that each section of the questionnaire included both detailed questions about specific services or products and a question about overall satisfaction within that category. There also was a final question about overall satisfaction with all of the services and products provided by ED. The purpose of this design was to provide both general and specific measures of customer satisfaction and to facilitate statistical analyses to determine which specific services and characteristics were most strongly related to the respondents' level of overall satisfaction.
This report first details the overall levels of institutional satisfaction with ED's customer support services, followed by a discussion of respondents' familiarity with or use of ED's services. The next section discusses patterns of satisfaction with specific services and products, and the final section discusses patterns between types of respondents and their levels of satisfaction.
Overall Levels of Satisfaction
Roughly three-fourths (73 percent) of the responding institutions expressed overall satisfaction with the assistance they received from ED in managing and administering the federal student financial assistance programs (Figure 1). The single most common response was that they agreed that they were pleased with ED's assistance (43 percent), with an additional 4 percent strongly agreeing and 27 percent slightly agreeing.2 The remainder were split between those who were neutral (13 percent) and those who disagreed that they were satisfied overall (14 percent).
Figure 1. Percentage of institutions agreeing they were satisfied overall with ED assistance
Full Size Figure 1
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
2To simplify the presentation of this report, these three response categories will generally be combined into a single percentage of institutions expressing agreement with a questionnaire item. Generally, there was little variation in the percentages that strongly agreed. Though all three categories are shown in Figure 1, only the sum is reported as a percentage; the detailed breakdowns can be found in Appendix C. Sometimes, as in this particular case, the more detailed statistics may not sum exactly to the total reported in the graphics because of rounding. Besides the three categories of agreement listed above, respondents could also say they were neutral, slightly disagreed, disagreed, or strongly disagreed. Sometimes a substantial percentage said they were neutral, which may reflect true neutrality, mixed feelings (i.e., satisfied with some aspects and dissatisfied with others), or being uninformed about the particular item. One should not assume that a failure to express satisfaction necessarily means that the respondents were dissatisfied.
Five categories of customer services all received levels of satisfaction between 71 percent and 79 percent. SFAP program materials and publications were at the top of this group (79 percent), with the application process (76 percent) and electronic processing (74 percent) close behind. At the other end, three categories of customer services had overall satisfaction levels between 50 percent and 58 percent: inquiry and information services (58 percent)-many of which were unfamiliar to the respondents; assistance in program operations (55 percent)-another area where the services were often unfamiliar to the respondents; and reviews and audits (50 percent)-an area more directed to meeting ED's need to prevent fraud than to fulfilling the needs of postsecondary education institutions. The relatively low satisfaction levels with the latter three areas were less an indication of dissatisfaction than of neutrality; only reviews and audits had somewhat higher dissatisfaction levels (19 percent versus 8-13 percent for the other seven areas), while all three areas received a large percentage of neutral ratings (31-34 percent versus 11-19 percent for the other five areas). The neutrality in turn was related to some respondents' lack of information about the areas; it is the respondents who had reviews or audits conducted the least recently, or who were unfamiliar with essentially all of the services in the category, who were especially likely to assign neutral ratings.
Because of the large number of items on the survey questionnaire, it was not judged practical to ask respondents to state which factors were most important to them. However, it is possible to create a measure of interrater agreement that indicates which types of satisfaction are most strongly correlated with the respondents' overall level of satisfaction, thereby providing an indirect measure of what is considered the most important. By this measure, respondents' satisfaction with assistance in program operations and satisfaction with program materials and publications had the greatest correspondence with their overall satisfaction (kappa=0.48 and kappa=0.46, respectively), while satisfaction with the application process had the least (kappa=.30).
Levels of Familiarity and Use of Services and Products
As suggested in the previous section, a respondent's level of familiarity with a service or product may affect his/her rating of ED's services. The three areas receiving the lowest levels of satisfaction were also areas where many respondents were uninformed (either not having experienced a recent audit or review or not being familiar with or not having used essentially all of the services in the category) and gave neutral evaluations rather than positive ones. In fact, one of the strongest findings from the survey was that many services or products were unfamiliar or had not been used by the respondents (Figure 2).
Figure 2. Percentage of institutions not using or unfamiliar with service
Not included in electronic version.
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
For most items on the questionnaire, respondents could indicate either their level of satisfaction with the service or product or that they were not familiar with or did not use the item. Most often, the questions were phrased in terms of respondents' familiarity with the item, but some questions were phrased in terms of respondents' use of the item; one question (concerning the SFAP Electronic Bulletin Board) provided both alternatives. These concepts can be very different, as the responses on the SFAP Electronic Bulletin Board indicated: only 8 percent had used the Bulletin Board, but another 49 percent were familiar with it (Appendix C, question 36). More generally, one would always expect the percentage that used a service to be smaller than (or equal to) the percentage that were familiar with the service: an institution that used a service would presumably be familiar with it. On the other hand, an institution could be familiar with a service but have no reason to use it (e.g., the institution might not experience any problems that would require the help of the service, or the institution might not be involved in the particular kind of activity that would lead to use of the service) or lack the resources to use it (e.g., if use of the service requires particular computer resources).
Given the differences between familiarity and use of a service, it is not surprising that seven of the top eight areas in Figure 2 (all but Program Systems Services) were phrased in terms of lack of use. In any case, however, the overall lesson of Figure 2 seems clear: many services or products were not used by a large number of respondents, and many were not even familiar to them. Looking only at the issue of familiarity, two-thirds (67 percent) of respondents were unfamiliar with Program Systems Services, roughly half (48-57 percent) were not familiar with three other services (fiscal officer training, electronic processing training, and Campus-Based representatives), and roughly one-third or more (30-42 percent) were not familiar with five additional services. In a sense, respondents indicated a similar problem in their comments about some of the written publications; they had difficulty in finding what they were looking for and expressed the need for indexes within the publications. Thus, the major difficulty that some institutions have with ED's customer service may not be with the quality or usefulness of the materials and services, but with knowing what is available and where to find it. Possibly one of the most needed publications from ED is a regularly updated guide that lists all publications and services, how they would be used, and how they could be obtained.
The Types of Services That Were Least Used or Least Familiar
As noted earlier, two categories of services were especially likely to be either unfamiliar or unused. Among the four items examined within the inquiry and information services, three showed extremely high levels of non-use: the SFAP Electronic Bulletin Board (92 percent), the Pell Grant Program Institutional Access System (82 percent), and the Technical Assistance Hotline for Pell Grant software (74 percent). And among the three items examined within program operations, all showed high non-use or unfamiliarity: 67 percent had never contacted the Pell Grant Financial Management Specialists, 67 percent were unfamiliar with Program Systems Services, and 48 percent were unfamiliar with Campus-Based state representatives.
The Characteristics of Respondents with the Least Familiarity
One obvious potential explanation for institutions' lack of familiarity with a service is that if an institution is not eligible for a particular kind of aid (say, Pell Grants), the institution would have little reason to be familiar with or make use of a service that is specifically directed toward that program. However, the levels of unfamiliarity or non-use were too great for this to be the only factor at work. Only 1 percent of the respondents were not eligible for Pell Grants, and only 12 percent were not eligible for Stafford loans (not in tables).
Another logical hypothesis is that respondents with relatively few years of experience with federal financial aid programs would be less familiar with the various services, and this is confirmed by the survey data (Appendix Table A-1). Those respondents who had 5 years or less of experience were more likely than those with over 10 years of experience to be unfamiliar with such services as Delivery System Training (30 percent versus 10 percent), electronic processing training conferences (60 percent versus 39 percent), and Campus-Based state representatives (64 percent versus 40 percent).
There also were differences based on the role of the individual completing the questionnaire. In general, most questions were completed by individuals within institutions' financial aid offices; funding questions were sometimes completed by individuals within the business office, however, and sometimes other individuals (e.g., from the institutional research office) completed all or part of the questionnaire. On funding-related questions, those in business offices were less likely to be unfamiliar with an item than either those in financial aid offices or those in other offices for items such as the Blue Book (10 percent versus 22-27 percent), electronic funds transfer (13 percent versus 27-34 percent), and ED's funds management process (8 percent versus 27-40 percent). On the remaining questions, respondents in the financial aid offices were less likely to be unfamiliar with the items. For example, 45 percent of those in financial aid offices were unfamiliar with Campus-Based state representatives, versus 69 percent of those in offices other than business or financial aid.
Finally, large differences often appeared based on institutional type and control, with less-than-2-year, and private, for-profit institutions being more likely to be unfamiliar with or not use a service than other institutions. For example, private, for-profit institutions were more likely than other institutions to be unfamiliar with Delivery System Training (25 percent versus 8-12 percent), Campus-Based state representatives (67 percent versus 29-36 percent), the renewal application process (19 percent versus 2 percent), and FISAP software (56 percent versus 11-14 percent).
Levels of Satisfaction in Specific Areas
Program Materials and Publications
This section discusses some of the general patterns that appeared in respondents' evaluations of specific customer support services and publications. Because of the large number of items covered by the questionnaire, no attempt is made to discuss each item individually, but summary statistics on all items are provided in the accompanying graphics and in the appendices.
Of the eight areas of ED's assistance to postsecondary education institutions, the area receiving the highest overall satisfaction was SFAP program materials and publications (79 percent; Figures 1 and 3). All three publications covered by the questionnaire received overwhelmingly positive evaluations of their clarity (88-90 percent), organization (86-90 percent), and ease of use (83-87 percent). The area that least often received positive evaluations was timeliness (52-61 percent), though it still received more positive evaluations than negative ones. Also, several respondents in their written comments expressed the need for indexes within the publications so they could better find the information they were looking for.
Figure 3. Evaluation of program materials and publications
Not included in electronic version.
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
As in the analysis of overall satisfaction above, the measure of interrater agreement shows which areas of satisfaction are most important in determining the overall level of satisfaction with the program materials and publications. Not surprisingly, given the high overall satisfaction with the publications and the significantly lower satisfaction with timeliness, timeliness was the dimension least related to overall satisfaction (kappa=.29-.39, depending on the publication). No single dimension clearly stood out as being the most strongly related to overall satisfaction, but the clarity of the publications (kappa=.47-.51) and the ease of use (kappa=.45-.49) were somewhat more consistently among the factors most related to overall satisfaction than the other dimensions.
Training received positive overall evaluations from 71 percent of the respondents (Figure 4). The aspects of training that were most positively evaluated were offering information that was very helpful (75-82 percent), providing ample opportunity to ask questions (77-80 percent), providing information that institutions used in their day-to-day operations (74-84 percent), and conducting the training at convenient locations (68-78 percent). Respondents' satisfaction was lowest for the frequency of the training (62-69 percent), the clarity of Delivery System Training (64 percent), and the timeliness of fiscal officer training (64 percent).
Figure 4. Evaluation of training
Full Size Figure 4
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
The dimensions that were most strongly related to overall satisfaction with training were offering information that was very helpful (kappa=.55-57) and very clear (kappa=.52-.55), while conducting the training at convenient locations was the least related to overall satisfaction (kappa=.26-.39).
About two-thirds (68 percent) agreed that it would be helpful to reinstate automation training for Pell Grants, while only 4 percent disagreed (Appendix C, question 8).
Respondents' overall satisfaction with the application process was the second highest of any of the eight categories (76 percent; Figures 1 and 5). The Free Application for Federal Student Aid (FAFSA) received the most consistently high satisfaction, with three of the four listed dimensions showing 82 percent satisfaction or higher. By contrast, three of the four listed dimensions for renewal applications ranged from 72 to 73 percent, and three of the five dimensions for the CPS User Services Hotline ranged from 73 to 74 percent. The item receiving the least agreement was the statement that the verification process is not burdensome (51 percent).
Figure 5. Evaluation of application process
Not included in electronic version.
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
The single item showing the strongest relationship to overall satisfaction with the application process was the ease of understanding the application corrections process (kappa=0.52), while other items showing a similarly strong relationship were the statement that the FAFSA asks for the right amount of information (kappa=.51) and the ease of completing the FAFSA (kappa=.50). By contrast, items relating to the renewal application showed weaker relationships (kappa=.35-.40).
In several ways, respondents indicated a lack of experience with some aspects of the application process (Appendix C, questions 16-18). Most had no opinion on whether Delivery System Forums were a convenient forum for offering changes to the application process (61 percent), had not used the CPS User Services Hotline (65 percent), and had not used ED's new service of printing all the renewal applications for the upcoming year and forwarding them to the schools (74 percent).
Those institutions that did use ED's new service of printing all the renewal applications provided some information about how that service might be improved (Appendix C, questions 19-23). Roughly half (56 percent) considered sending the packages earlier than November and sending the packages in two mailings (e.g., in November and in January) to be of at least some help, and 26 to 31 percent said it would be a great help. A large majority of the institutions indicated that sorting the applications in alphabetical order was very helpful (83 percent), 23 to 28 percent considered sorting by graduate/undergraduate status or Social Security number to be very helpful, and 17 percent considered sorting by the number of renewal applications to be very helpful. Relatively few institutions (13-15 percent) had experienced problems with the transmissions of the student selections or with lost or damaged forms. Close to half (43 percent) made use of the electronic record of all the renewals that accompanies the application forms.
In addition to questions concerning satisfaction with ED's services and products, institutions were asked about their support of electronic processing of financial aid processing, the type of computer configuration used for processing financial aid, and their method for forwarding information electronically to the U.S. Department of Education.
Use of electronic processing. Roughly half of the institutions (55 percent) were very supportive of electronic processing, and one-fourth (27 percent) were somewhat supportive (Appendix Table A-2). For-profit institutions were more likely to be very supportive than public and private non-profit institutions (65 percent versus 47-49 percent), and research/doctoral institutions and less-than-2-year institutions were more likely to be very supportive than other types of institutions (61-63 percent versus 45-54 percent).
Overall, 90 percent of institutions used computers for processing financial aid. The three most common computer configurations were individual microcomputers only (32 percent), a mainframe computer with terminals and/or microcomputers (27 percent), and microcomputers connected through a local area network (24 percent). Some large differences appeared in the computer configurations depending on the institutions' control and type. For-profit institutions were much more likely to use individual microcomputers than other institutions (50 percent versus 16-22 percent), while public and non-profit institutions often used a mainframe computer with terminals (51 percent and 35 percent, respectively, versus 6 percent at for-profit institutions). Research/doctoral institutions and comprehensive institutions were especially likely to use a mainframe with terminals (69-74 percent versus 6-39 percent at other institutions), less-than-2-year institutions were the most likely to use individual microcomputers (50 percent versus 2-37 percent), and liberal arts and specialized institutions were the most likely to use local area networks (36-39 percent versus 16-27 percent).
Electronic forwarding of information. Overall, 79 percent of the institutions forwarded information electronically to the U.S. Department of Education, while 10 percent did not use computers for financial aid processing, and 11 percent used computers but filed information by paper only (Figure 6). The most widely used method of forwarding information was through an EDExpress linkage (57 percent); other means included the Recipient Data Exchange (8 percent), the Floppy Disk Data Exchange (7 percent), using a servicer to forward the information (4 percent), and using software developed by the institution or by outside parties (3 percent).
Figure 6. Means of forwarding information electronically
Full Size Figure 6
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
While all three of the ED linkages were viewed favorably, the Floppy Disk Data Exchange was viewed the most favorably (Figure 7). A larger percentage of respondents said it was easy to use (91 percent versus 70-75 percent), well documented (80 percent versus 62-59 percent), and along with the Recipient Data Exchange, fit well with other software (59-62 percent versus 46 percent). The greatest dissatisfaction was with customizing the software; for each of the three methods, 31-34 percent said that customization required considerable work.
Figure 7. Evaluation of ED's data management and communication software for forwarding information electronically
Full Size Figure 7
*For these items, respondents who agreed were expressing dissatisfaction rather than satisfaction. The percentages that were satisfied (i.e, that disagreed) were EDExpress, 40 percent; Recipient Data Exchange, 25 percent; and Floppy Disk Data Exchange, 47 percent.
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
Satisfaction with products and services. Electronic processing received the third highest level of satisfaction of the eight areas (74 percent), and was the only area where more than 10 percent strongly agreed that they were satisfied (11 percent; Figures 1 and 8). The areas receiving the highest satisfaction were understandable hardware requirements (74-88 percent) and the ease of installation (72-87 percent), while the weakest areas were the ease of customization (36-47 percent) and the compatibility with other software (46-52 percent).
Figure 8. Evaluation of electronic processing
Not included in electronic version.
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
In general, respondents also showed higher satisfaction with Fiscal Operations Report and Application to Participate (FISAP) software than with EDExpress software; for example, they were more likely to agree that the FISAP software had understandable hardware requirements (88 percent versus 74 percent), and that it was easy to install (87 percent versus 72 percent).
Other areas that were examined were the satisfaction with the EDExpress documentation and with training conferences on electronic processing. The EDExpress User's Guide received generally positive evaluations (53-64 percent); the strongest satisfaction was with the timeliness of the updates (64 percent), while the weakest was with its ability to meet all of the respondents' needs for information (53 percent). Electronic processing training conferences received the greatest satisfaction concerning their content (generally in the range from 66 to 70 percent), and the least satisfaction concerning their frequency (51 percent) and location (58 percent).
In general, the items that were most strongly related to overall satisfaction with electronic processing were those concerning the training conferences, with the single item that was most strongly related being whether the conference provided information that was used in day-to-day operations (kappa=.52; the next highest value of kappa was .45). In addition, the responses on EDExpress software were more closely related to overall satisfaction with electronic processing than the comparable responses on FISAP software.
Inquiry and Information Services
As noted earlier in this report, the inquiry and information services were one of the lowest rated of the eight categories examined (58 percent expressed overall satisfaction), possibly because the respondents often had no experience with some of the services listed, and thus may not have been aware of the assistance that was available (Figure 9). Because of the relatively small number of institutions that had experience with the services and were able to provide evaluations, the estimates for this category are not as reliable, and relatively large differences between services are required in order to achieve statistical significance.
The Technical Assistance Hotline for Pell Grant software tended to receive higher satisfaction than the other services that were listed, with a higher percentage saying that it provided clear information (78 percent versus 60-70 percent), provided complete information (75 percent versus 58-65 percent), and had knowledgeable personnel (78 percent versus 64 percent). By contrast, the SFAP Electronic Bulletin Board received lower satisfaction than the other services (55-60 percent versus 64-78 percent), though so few institutions had used it (8 percent) that these estimates are not very reliable.
There were not very clear patterns in terms of some aspects of the various services receiving higher satisfaction than others. For example, the Student Aid Information Center and the Pell Grant Institutional Access System each received higher satisfaction on ease of access than on providing complete information, but the Technical Assistance Hotline for Pell Grant software received higher satisfaction on providing complete information (75 percent versus 69 percent).
No clear patterns emerged in terms of which aspects of the services were most related to overall satisfaction, with large variations in the kappa scores along each dimension. Also, the service that tended to receive the highest kappa scores-the SFAP Electronic Bulletin Board-was used by too few institutions to be a reliable indicator of the factors most affecting overall satisfaction.
Figure 9. Evaluation of inquiry and information services
Not included in electronic version.
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
Overall, 71 percent of the respondents agreed that they were happy with the funds management process (Figure 10). The items receiving the highest agreement were that the electronic funds transfer provided funds quickly (89 percent), respondents were happy with the current electronic funds transfer process (81 percent), and ED's funds management process provides funds in a timely and efficient manner (81 percent). Some of the areas receiving the least agreement were that the Blue Book and Payments Recipient Guide were updated in a timely manner (49-60 percent), and the funds management process handles problems quickly and efficiently (57 percent). Roughly half (53 percent) agreed that they prefer Pell fund transfers to be made at the time the student accounts are credited.
The single item that showed the highest agreement with respondents' level of overall satisfaction with the funds management process was whether the funds management process fit easily with other financial aid functions (kappa=.53, with no other item showing a similar level of agreement; the next highest value was kappa=.43).
Figure 10. Evaluation of funds management
Not included in electronic version.
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
Program reviews and audits are more directed to meeting ED's need to prevent fraud than to fulfilling the needs of postsecondary education institutions. Thus, it is not surprising that they received the lowest level of overall satisfaction of the eight listed categories (50 percent; Figure 11). One respondent commented in the pretest that while the reviews and audits were burdensome, they were also necessary. The greatest area of satisfaction was in knowing what to expect (62 percent), while the greatest area of dissatisfaction was the level of burden (31 percent said the reviews and audits were not burdensome, and 47 percent said they were).
The areas showing the strongest agreement with the overall level of satisfaction were whether the reviews help institutions to better administer the Title IV programs (kappa=.65) and whether the ED reviewers are helpful in explaining problems (kappa=.64).
Figure 11. Evaluation of program reviews and audits
Full Size Figure 11
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
ED systems for providing assistance in program operations received the second lowest level of satisfaction (55 percent; Figure 12). As noted earlier, one reason may be that 48 percent were not familiar with Campus-Based state representatives, 67 percent were not familiar with Program Systems Services, and 67 percent had never contacted the Pell Grant Financial Management Specialists (Figure 2); thus, respondents did not appear to have a good understanding of the resources that were available to them.
The areas of greatest satisfaction were that the individuals providing services were courteous (69-75 percent), knowledgeable (69-73 percent), and provided helpful information (67-73 percent). The lowest levels of satisfaction were with accessibility (48-53 percent) and with Campus-Based state representatives' responsiveness in completing transactions such as reinstating or adjusting award amounts (54 percent).
Because of the low use and familiarity with the services, it is difficult to determine which factors were most strongly related to overall satisfaction. The respondents' evaluations of the Program Systems Services generally showed the highest agreement with their overall satisfaction (with kappa ranging from .53 to .70), but only one-third of the respondents were familiar with the Program Systems Services.
Figure 12. Evaluation of program operations
Not included in electronic version.
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
Impressions of Student Satisfaction
This study was not designed to provide a representative sample of students receiving financial aid, and it cannot provide accurate estimates of students' levels of satisfaction. However, the financial aid personnel at postsecondary education institutions who work with students can provide general impressions of student satisfaction based on the students' comments and requests for assistance. Thus, the respondents were asked for a few general impressions in order to provide preliminary data until a student survey can be conducted.
The institutional officials perceived students as having lower satisfaction than they typically reported for most of the areas covered by the questionnaire; roughly half of the respondents perceived the students to be satisfied in the five listed areas (Figure 13). Respondents most often perceived the students as satisfied with respect to their ability to get the needed information from the written materials (57 percent), and least often perceived them as satisfied concerning their ability to complete the forms without extensive assistance (44 percent).
Figure 13. Impressions of student satisfaction
Full Size Figure 13
SOURCE: Higher Education Surveys, Postsecondary Education Institutions' Satisfaction with Student Financial Assistance Programs (HES 20), U.S. Department of Education, 1996 (survey conducted in 1995).
Differences Among the Respondents in Their Evaluations
Some categories of institutions or respondents were more likely to be satisfied than others. Because of the large number of items in the questionnaire, however, it would require excessive space to discuss each item individually. Instead, this analysis will be limited to a few representative categories-two from the beginning of the questionnaire, and one from the section on funds management (in order to compare the responses of respondents in the business offices with those in financial aid offices)-and will discuss differences among the respondents on those items. An examination of additional tabulations revealed that the patterns discussed here were largely consistent throughout the questionnaire, unless the item was one that had such strong agreement among the respondents that there was insufficient variation for such differences to be important.
In general, for-profit institutions were often more positive in their evaluations than public institutions (Appendix Table A-3). For example, they were more likely to agree that the SFA Handbook guidelines were easy to implement (68 percent versus 54 percent) and the Verification Guide gets timely updates (68 percent versus 54 percent). Private, non-profit institutions typically fell somewhere in between, usually being closer to the public institutions in their satisfaction. The relatively positive evaluations by for-profit institutions may be an indication that for-profit institutions place a greater importance on federal financial aid because it directly affects their profitability, while other institutions may have a different mixture of goals and resources.
Research/doctoral institutions often were the least positive in their evaluations, especially when compared with less-than-2-year institutions (which largely consisted of for-profit institutions), but also as compared to other institutional types. For example, they were less likely than other institutions to agree that the SFA Handbook gets timely updates (34 percent versus 46-60 percent), that the SFA Handbook guidelines are easy to implement (41 percent versus 54-67 percent), that the Verification Guide gets timely updates (33 percent versus 50-69 percent), and that the Verification Guide guidelines are easy to implement (48 percent versus 69-77 percent).
Individuals with less than 5 years of experience with federal financial assistance programs tended to be more satisfied than those with more than 10 years of experience. For example, the former were more likely to agree that the SFA Handbook gets timely updates (65 percent versus 47 percent), the SFA Handbook guidelines are easy to implement (71 percent versus 55 percent), and the Verification Guides gets timely updates (70 percent versus 53 percent). One can speculate about the reasons for these differences. Those who are least experienced presumably might have more questions and difficulties, but their problems might also be easier to solve. On the other hand, experienced people might mainly seek help with the most difficult cases, where help might also be more difficult to provide. Experienced people also might tend to base their responses on services that were provided over a number of years, while less experienced people might use a shorter time frame; since several respondents commented that the quality of service had improved over time, the more experienced people might thus tend to be more negative.
For most questions, the attitudes of respondents in business offices cannot be compared with those in financial aid offices, because there were too few respondents in business offices to provide reliable estimates. However, on questions relating to funding, business offices provided responses for roughly one-fourth of the institutions, and comparisons are possible. On these questions, the business offices were generally less satisfied. For example, they were less likely to agree that the Blue Book meets all information needs (53 percent versus 68 percent), is easy to use (59 percent versus 72 percent), gets timely updates (38 percent versus 53 percent), and has guidelines that are easy to implement (46 percent versus 63 percent). The relatively low evaluations given by business offices were predicted by one pretest respondent, who suggested that business offices were more likely to see the financial aid process as burdensome, while financial aid offices were more likely to be satisfied because they were directly helped by SFAP programs in fulfilling their mission.
Summary
Overall, officials at postsecondary education institutions expressed a positive evaluation of the support services provided by ED. When looking at all products and services, 73 percent at least slightly agreed they were pleased with ED's assistance, with most of those saying they agreed they were pleased (43 percent). Only 14 percent said they were not pleased with ED's assistance overall, while the remaining 13 percent were neutral. The category receiving the highest overall satisfaction was SFAP program materials and publications (79 percent at least slightly agreed), while the category with the lowest satisfaction was reviews and audits (50 percent). Some commented that the support services have improved over time.
One of the strongest findings was the high percentage of institutions that had not used or were not familiar with particular services. In some cases, this lack of use or familiarity may have lowered institutions' overall evaluations: thus, two of the three categories receiving the lowest overall satisfaction were also categories in which most or all of the services were not familiar to the respondents or had not been used. The two categories for which a lack of familiarity or use were most an issue were assistance in program operations, and the inquiry and information services; respondents who were least informed about the services often gave neutral evaluations of the categories, thus lowering the percentage of positive evaluations. Similarly, for the third category receiving low evaluations-reviews and audits-it was the respondents who had experience reviews and audits the least recently who were less likely to give positive evaluations and instead give neutral evaluations.
Some of the areas in which respondents expressed the greatest satisfaction were with clarity, organization, ease of use, and offering information that was helpful. Some of the areas least likely to receive positive evaluations (though more respondents typically gave positive evaluations than negative ones) were timeliness, the ease of customization of software, and the frequency of training.
Often there were differences among institutions and respondents in their level of satisfaction. For-profit institutions, individuals in financial offices, and individuals with 5 years or less of experience with federal financial aid programs tended to be the most satisfied, while research/doctoral institutions, individuals in business offices, and individuals with more than 10 years of experience tended to be the least satisfied.
-###-
Technical Appendices
Return to Financial Aid Evaluation Page
Last modified -- June 17, 2003 (jer)
