Information For           Students Parents Teachers Administrators Information About Grants & Contracts Financial Aid Research & Statistics Policy Programs

DATE:	January 31, 1997
TO:	Blane Dessy Director, National Library of Education (NLE)
FROM:	Keith M. Stubbs Director, NLE/Resource Sharing and Cooperation Division (RSCD)
SUBJECT:	Internet Customer Survey Results

The Department's Internet services have grown far beyond their experimental beginning four years ago to become an important channel for information dissemination and customer service. Each month the Department's Online Library is visited by more than 200,000 individuals from all states and sectors of the United States and over a hundred countries worldwide. The Online Library contains more than 20,000 files and is growing faster than ever with contributions from nearly every principal office. Subscribership to the EDInfo listserv recently passed the 8,000 mark, and EDInfo messages reach additional thousands of Internet users through re-posting on other education listservs and state networks.

We have reached a point at which it is necessary to learn more about our customers and their needs so that we can focus our efforts on making the most useful information and services available in the most useful forms possible. Until recently, our knowledge about customer characteristics and preferences was limited to what we could derive from Web and Gopher usage logs and unsolicited email messages.

Two months ago, we received OMB approval of an Internet Customer Survey. The purpose of this memorandum is to report preliminary findings from the 1,300+ responses received to date.

Background

Web and Gopher usage logs provide much useful data -- the number of times each file is accessed, when, from which Internet address, and using which Web browser. From this data we can derive the kind of information included in RSCD's monthly reports -- the growth trend over time, the shift from Gopher to Web, the most frequently used files and menus, the sites and countries that visit us the most, and the most widely used Web browsers among our customers.

However, the usage logs do not provide any insight into some important customer characteristics which we need to understand better:

• What kinds of people and organizations use our information?
• How often do they visit ED's Web and Gopher sites? Do they subscribe to EDInfo?
• How satisfied are they with the information we provide -- its usefulness, timeliness, accuracy, comprehensiveness, and presentation?
• How satisfied are they with the way the information is organized and with the menus, links, and search tools we provide to help them find information of interest?
• What kinds of information are most valuable to them?
• How do they prefer to use various types of information, and what formats are most useful? What is their technical capacity to receive and use the information?
• What kinds of new services would be most valuable to them in the future?

The Internet Customer Survey (Attachment C) was designed to answer such questions. It was designed by RSCD, modeled on the National Center for Education Statistics' (NCES) Customer Satisfaction Survey, and refined with assistance from NCES (Nabeel Alsalam, Edith McArthur, and Bill Freund), the Office of the Under Secretary (OUS -- Kirk Winters), the Office of Interagency and Intergovernmental Affairs (OIIA -- Peter Kickbush), and the INet contractor, Decision Systems Technologies, Inc. (DSTI -- Ed Loeb and Blake Sobiloff). NCES and the Office of the Chief Information Officer (OCIO) handled the OMB clearance process. OMB approval was obtained in mid-November.

Conducting the Survey

The survey was made available on the ED Web site home page on Monday, November 25 and on the Gopher site and via the Almanac email server a few days later. It was not widely publicized until Thursday, December 19, when an EDInfo message containing the survey was distributed (Attachment D). The EDInfo message was subsequently re-posted on other Internet listservs, including:

• Net-Happenings (a general announcement listserv that reaches 10,000+ subscribers),
• CoSNDISC (the Consortium for School Networking's general discussion list),
• K12OPP (the Global SchoolNet Foundation's K-12 Opportunities list),
• ISTE (the International Society for Technology in Education's lists for general membership, executive board, and several special interest groups), and
• TENET (the Texas Education Network's listserv for Internet trainers).

As of Friday, January 24, 1,316 responses have been received and tabulated. The Web form accounted for 494 responses (38%); the email survey accounted for 821 responses (62%); one response was received via U.S. Mail. No responses have been received from the Gopher or email version.

What the Responses Tell Us

The survey cannot be considered a random sample, since respondents (a) were those people who were in a position to become aware of the survey either by visiting the ED Web site or by receiving an email copy via EDInfo or another education listserv; and (b) took the time to respond. Certainly the responses over-sampled EDInfo subscribers (73% of the responses, 12% of all EDInfo subscribers). Nevertheless, the information collected provides useful insight into our customers and their preferences.

Note: The survey was originally designed to be presented as a Web form. It was decided later, in order to increase the response rate, to distribute the survey itself in the EDInfo announcement rather than referring users to the Web form. While that strategy was very successful, it complicates interpretation of some of the responses, particularly to questions #7 through #12. It is apparent that some users responded to those questions solely in terms of the EDInfo listserv, rather than also considering the Web and Gopher. It is difficult to determine for which responses that is the case, but it is evident in the low response rate to Question #10 compared to Question #7 (64% vs. 88%) and in the high number of votes for plain ASCII text in all four sections of Question #12.

• The cumulative responses to a Questions #7 through 10, which ask users to rate their satisfaction in eighteen categories covering all eighteen aspects of the ED Online Library and EDInfo, show that overall satisfaction is quite high -- 3.98 on a scale of 1 to 5, where 5=very satisfied, 4=satisfied, 3=neither satisfied nor dissatisfied, 2=dissatisfied, and 1=very dissatisfied.

• Across all eighteen categories, 75% of the respondents were either satisfied or very satisfied; only 5% were dissatisfied or very dissatisfied.

• See attachment B for selected "Rants & Raves" representing the most positive and negative comments received. No responses were excluded from the results below, even those few that oppose the Department's existence and gave the lowest available rating in all categories.

• Our most frequent users are education administrators and managers (36%), teachers and professors (21%), and researchers and analysts (16%). To a lesser extent, we are also used by parents (4%), students (4%), librarians (3%), writers and reporters (2%), policy makers and legislators (2%), and miscellaneous others (13%) including counselors and consultants.

• In descending order of frequency, our users' organizational affiliation is elementary and secondary education (36%), colleges and universities (25%), associations and non-profit organizations (10%), state government (7%), junior and community colleges (5%), private individuals (3%), federal government (3%), local government (1%), media (1%), libraries (1%), and miscellaneous others (5%).

• Nearly 73% of respondents visit ED's Web site at least once a month; 41% visit it at least once a week. By contrast, only 25% visit ED's Gopher site at least once a month; 10% visit it at least once a week; 60% visit it less than once every six months or never.

• EDInfo subscribers are more likely than non-subscribers to be frequent users of ED's Web or Gopher.

• Writers and researchers tend to visit ED's Web site more frequently than the average respondent; teachers and students less frequently.

• Respondents with fast Internet connections tend to visit ED's Web site more frequently than those with slower connections.

• Nearly three-fourths of the responses come from EDInfo subscribers. Of the remainder, 80% are not familiar with EDInfo but are interested in learning more about it.

• The set of responses from EDInfo subscribers contains a higher percentage of researchers and managers than the set of responses from non-subscribers, and a lower percentage of librarians, students, and parents.

• The Online Library and EDInfo are trusted information sources for respondents. Accuracy received the second highest rating (4.31) of any category among the satisfaction-oriented questions. 87% of respondents are satisfied or very satisfied; only 2% are dissatisfied. Managers (4.41) and researchers (4.37) are the most satisfied; parents (3.95) are the least satisfied.

• The Online Library and EDInfo also receive high marks for timeliness (4.25), although comments indicate that some respondents believe we should devote more effort to making information about funding opportunities -- both individual announcements and the annual forecast -- available earlier. Some respondents have unrealistic expectations of timeliness. One respondent recommended that any information older than two months should be removed from the ED Web site!

• Overall usefulness, completeness/comprehensiveness , and relevance to the respondent's needs rated 4.07, 4.06, and 4.05, respectively, with writers the most satisfied and parents, policy makers, and students the least satisfied.

• A representative sample of comments:
  • My only problem is that I wish more research and survey information were available, but I know that takes time and effort. Keep up the good work. You have come such a long way since before the Clinton administration. It's impressive.
  • Sometimes the information is too comprehensive
  • The information I wished to find is not within your site. The download times are fantastic, and the database is comprehensive enough, I suppose. Though, as I said, I did not find the information that I was looking for.
  • There is very little about elementary schools.
  • A lot of your information is more relevant to K-12; I'm more interested in higher education information.
  • Your 1996 Educational report seemed to be based upon information older than 1995. It seems to me that your reports should be within the current-to-one-year-old range. Of course, I am not familiar with the obstacles you face in obtaining such info.
  • Your service provides a comprehensive, up to date information source. Please continue and expand!

• Clarity of the writing received the highest rating (4.35) of any of the eighteen categories in Questions 7-10. 92% of respondents are satisfied or very satisfied; only 1% are dissatisfied. Writers (4.46) and managers (4.41) are the most satisfied; parents (4.11) are the least satisfied. Based on respondent comments, some of the high rating reflects the high esteem in which EDInfo messages are held by most subscribers; but it is impossible to isolate that effect. One respondent commented "I have no idea how the service has evolved such a candid style. I hope it can be preserved."

• Respondents are satisfied with the layout of the material (4.05) and the clarity of the tables and charts (4.02). Parents and writers are less satisfied than other respondents.

• The two graphics categories earned somewhat lower marks -- 3.82 for clarity and 3.71 for amount. Parents and policy makers are the least satisfied. Comments indicate a lack of consensus among respondents; opinion appears to be evenly divided into two camps, exemplified by the following sets of polar comments:

  Could more data be presented in graphical rather than tabular form? A graphic goes a long way to explain concepts.

  Please don't load things up with slow-loading and superfluous graphics. When I go to your site, I'm after info and text. Thanks for not slowing down the world with useless visuals/graphics.

• There is no correlation between respondents' rating of the ED Web site's graphics and presentation and their Internet connection speed.

• The ratings and comments about ease of finding information of interest (3.85), ease of finding new material (3.83), and menus and categories (clarity, ease of use) (3.84) reflect the difficulty of organizing a large body of information into a scheme that works well for such a broad range of users. Some representative comments are:
  • Can't find the name of the Secretary of Education
  • After a topic is clicked on, the information about it needs to be presented next. There are too many clicks to get nowhere.
  • How the information is organized is problematic for all of us. I don't know how to suggest you improve since I have difficulty on my own hard disk. The organization is as good as any...
  • I know what your specific categories subsume, in general, so I find resources & some categories have better substructures than others. Your search mechanisms aren't any better or worse than others; pre-knowledge is necessary.
  • It took me 30-45 minutes to find the "Nation at Risk" report.
  • It takes a little while to find exactly what I need, but, if all else fails, I use the Ed Search engine and generally find what I need.
  • Well organized and easy to navigate.

• Search tools received the lowest rating for both their usefulness in finding information (3.76) and their responsiveness (3.66). 8-9% of respondents are dissatisfied. When we first implemented a full-text search of the entire site a year and a half ago, it was a leading edge service and much appreciated. Expectations among Internet users have risen a great deal since then. Now our search engine is considered average and some respondents are dissatisfied with its ease of use or relevance ranking algorithm. Some representative comments:
  • Is there a search tool for this web site?
  • Please keep the search tools and continue to upgrade them. They are especially helpful if one does not know how the U.S. DOE is organized and needs to find information.
  • Search tools are an incredible addition.

  Note: We are within a month of implementing a new search engine which should help address some of the respondents' concerns, including ease of use, retrieval of alternate word forms, date-range retrieval of grant announcements, informative display of hit list, and search speed. Two other projects that are underway, Topics A-Z and Cross-site Indexing, are also intended to improve users' ability to find information easily and on their terms. However, there is a gap between most users' wish for more powerful search features and their inability to exploit features beyond the basics in the existing search features. Some users appear to want software that reads minds.

• Respondent comments about the speed or sluggishness of Web/gopher screens (3.78) and searches (3.66) reveals that some of our customers cannot distinguish between slowness at our site, problems at their end, or general congestion on the Internet.

• Not surprisingly, satisfaction with the speed of the web screens and search engine is influenced by the users' connection speed. Users with direct Internet connections are more satisfied than users with modems, and users with fast modems are more satisfied than users with slower modems.

	Kind of information	Avg. Rating*	Rated highest by	Rated lowest by
H i g h	announcements of funding opportunities and information about grants and contracts	2.63	Students (2.71) Managers (2.70)	Parents (2.35) Policy makers (2.33)
	research findings (syntheses and summaries)	2.63	Researchers (2.78)	Parents (2.50)
	research reports	2.59	Researchers (2.79) Students (2.72) Writers (2.71)	Parents (2.33)
	updates on budget, legislation, and activities	2.59	Writers (2.78)	Parents (2.49) Policy makers (2.48) Teachers (2.45)
	directories of information centers, clearinghouses, and technical assistance centers	2.58	Librarians (2.74)	Policy makers (2.40)
	descriptions of effective and promising practices	2.54	Librarians (2.68)	Students (2.41) Writers (2.38)
	statistics	2.54	Librarians (2.73) Researchers (2.69) Policy makers (2.68)	Teachers (2.39) Parents (2.30)
M i d d l e	descriptions of exemplary schools and programs	2.47		Policy makers (2.25)
	legislation and regulations	2.43	Librarians (2.56)	Teachers (2.30) Parents (2.26)
	general guides to the Department of Education and its programs and services	2.42	Librarians (2.71)
	press releases	2.42
	publication announcements	2.42	Researchers (2.53)	Policy makers (2.32) Parents (2.22)
	descriptions of EDfunded projects	2.41	Librarians (2.55)	Parents (2.15)
	fulltext publications	2.41	Librarians (2.68)	Policy makers (2.21)
	conference calendars and announcements of upcoming events	2.36		Policy makers (2.20) Parents (2.18)
L o w	lesson plans and teacher guides	2.25	Librarians (2.60) Teachers (2.48)	Researchers (2.03) Policy makers (1.88)
	student aid information	2.25	Students (2.79) Librarians (2.54) Parents (2.49)	Writers (2.13) Policy makers (2.00) Researchers (1.89)
	activities for families, parents, and children	2.24	Parents (2.48)	Researchers (2.04) Policy makers (2.00)
	speeches and testimony	2.07	Parents (2.33) Writers (2.23)	Teachers (1.97) Policy makers (1.90)

* 3=very useful; 2=somewhat useful; 1=not useful

• Respondents consider most kinds of information offered by the Online Library and EDInfo to be useful or very useful -- an overall average of 2.43 on a scale of 1 to 3.

• Librarians and students tend to rate information categories higher than most respondents, while parents and policy makers tend to rate them lower.

• Student aid information shows the widest disparity (0.90) between the highest rating (2.79 by students) and the lowest rating (1.89 by researchers).

• Lesson plans and teacher guides show the second widest disparity (0.72) between the highest rating (2.60 by librarians) and the lowest rating (1.88 by policy makers).

• Surprisingly, nearly half of our users have direct Internet connections, a third use 28.8kbps or fast modems, and only 15% use 14.4kbps or slower modems.

• Although one might expect such high speed connections to indicate a desire for more sophisticated information presentation, the responses are mixed.

• HTML and plain ASCII text are the preferred data formats for all kinds of information -- short documents, long documents, tables, charts, and data sets. As mentioned earlier, the high number of votes for plain ASCII text is partly attributable to the fact that some EDInfo users responded to the survey solely in terms of the the email messages they receive via the EDInfo listserv rather than the Web and Gopher.

• Adobe Acrobat's Portable Document Format (PDF) tallied the third most votes, but provoked an equal number of strong pro and con comments.

• Respondents prefer to read short documents online and, to a lesser extent, to print them. They prefer short documents in HTML (49%), with ASCII text (45%) a close second, and PDF third (22%).

• Respondents prefer to print long documents and, to a lesser extent, to download them and view them offline. They prefer long documents in HTML (54%), with ASCII (40%)a close second, and PDF third (23%).

• By a three-to-two margin, respondents prefer that long documents be presented as a collection of short HTML pages rather than a single long HTML page. However, a number of respondents voiced dissatisfaction with having to download and print multiple files for a document and requested that, in addition to HTML for convenient browsing, long documents be provided as a single file for downloading or printing. Our efforts in this area to date have been hampered by the fact that the Department's publication production process is not completely electronic and most of the documents we receive are not easily converted to PDF format.

• Respondents prefer to view tables and charts online or to print them. Again, HTML (40%) is the favored format, followed by ASCII text (28%), PDF (23%), and standard Internet formats (GIF and JPEG) for graphics (20%).

• Respondents prefer to search data sets online, with ASCII (36%) and HTML (36%) as the preferred formats. The response rate was much lower than for the other formats and preferences less pronounced.

Overall response to the fourteen potential new services listed on the survey indicates that we're on the right track. On average, the fourteen services received support from nearly half the respondents (47%). The rankings confirm our project and funding decisions. Most of the services which received the highest number of votes will be addressed this year by projects already planned or underway.

In descending order of popularity, the fourteen new services listed on the survey are:

• Respondents express a preference for database and search capabilities and electronic forms over discussion forums and audio/video transcripts of Department presentations .

• Managers tend to vote for more future capabilities than most respondents, while policy makers and writers tend to vote for fewer.

• Student aid applications show the widest disparity (49%) between the highest percentage (58% by parents) and the lowest rating (10% by writers).

• Grant applications show the second widest disparity (39%) between the highest rating (71% by managers) and the lowest rating (32% by policy makers and writers).

• enhanced capability to search for grant announcements based on application deadline dates and applicant eligibility categories;

• hyperlinks from grant announcements to online application packages, program information, profiles of previous award recipients, and their winning applications

• ability to monitor status of bids and proposals

• ability to register an interest profile and be notified only of those items which pertain to interests listed in the profile (some respondents had funding opportunities in mind and cited the FEDIX Opportunity Alert service and NSF's FastLane project; others wanted to receive selected EDInfo messages on specific topics or for specific target audiences or beneficiaries; others asked to be notified automatically of new web postings in their area of interest);

• inclusion of contact information (name, phone number, and email) in all exemplary program profiles and all information about ED programs, and addition of email addresses to online ED staff directory; and

• easy and immediate access to basic documents such as NPRMs, regulations, Dear Colleague letters, rulings, interpretations, case law, and statutory provisions.

The survey itself, and the signal it sends that we value and respond to our customers, elicited a number of comments, nearly all favorable.

• Thank you for taking the time and effort to address your subscribers' needs and request their opinions.
• You are on the right track and this survey demonstrates that.
• Do more surveys like this one. Very well done.
• I think surveys like this are a great way to get feedback from users.
• I think this survey is a proper step in trying to upgrade what has become a good service.
• I had to go back to a previous email to find this page [the survey]. Why are you hiding it?
• This survey was too long.
• I believe this questionnaire should be put on the net as a permanent feature. It should be highlighted at least monthly as new users come online.
• You're doing a great job! When I hear people recommend eliminating ED altogether, I have to wonder whether they know how accessible INET is making all this important & useful info. I'm also impressed by your responsiveness, as demonstrated by this survey.

Attachments:

• A: ED Internet Customer Survey -- Summary of Responses
• B: ED Internet Customer Survey -- Selected Comments (Rants & Raves)
• C: ED Internet Customer Survey -- Web form
• D: ED Internet Customer Survey -- EDInfo message

[ Back to Customer Survey Executive Summary ]

This page last modified June 10, 2003 (kms).