Over the past several years, the federal government has spent millions of dollars to nurture the development of the National Information Infrastructure (NII), a collection of private and public technological investments that will electronically connect the nation's schools, businesses, hospitals, and homes. The intent of this effort is to apply advanced network technology to deliver a wide range of information products and services that will include on-line purchases and transfer of funds, information on demand, distance learning, and highly interactive entertainment. The goal is to improve the country's economic competitiveness and the quality of life of our citizens (Information Infrastructure Task Force 1993, 1994).
Several federal programs have been instituted to increase the likelihood that the NII will also contribute to educational improvement. These include the Department of Education Challenge Grants, the Department of Defense Computer-Aided Education and Training Initiative (CAETI) administered by the Defense Advanced Research Projects Agency (DARPA), the Department of Commerce Telecommunications and Information Infrastructure Assistance Program (TIIAP), and a set of National Science Foundation (NSF) programs that includes Networking Infrastructure for Education (NIE), Advanced Applications of Technology (AAT), and Project GLOBE. These public investments have been complemented by private efforts, such as Pacific Bell's California Research and Education Network and Education First projects, Apple's Classroom of Tomorrow, the Lightspan Partnership, and multiorganizational regional efforts, such as Joint Venture Silicon Valley's Smart Valley Project.
Over the next 10 years, it is anticipated that these public and private investments will result in dramatic changes in the relationships among schools, homes, and workplaces (Kozma, Grant, and the Center for Technology in Learning 1995). New technologies will be developed, test beds installed, software designed, and, in schools, new network-based curricula will be generated, along with the professional development programs needed to implement them. This infusion of technology into the lives of our citizens, teachers, and students promises to expand the boundaries of educational systems, extend participation to those traditionally not involved in education, and increase educational opportunities and access to educational experiences for all.
While the promise of these technology investments is great, there will be a crucial need to evaluate the educational impact of this infrastructure as it unfolds. Evaluations will be needed to determine the extent to which individual projects have achieved their goals, to identify unintended negative consequences, and to acquire information that can help project staff increase their effectiveness. In addition, the impact of entire network-based funding programs will have to be evaluated. Evaluations will be needed to help program staff identify the extent to which there are coordinations, redundancies, gaps, and conflicts among individual projects within and across programs. And evaluations will be needed to aggregate findings across projects and assess the net impact of entire funding programs on the quality of our educational system and our lives.
In this article, we identify some of the issues and problems in implementing good evaluation of network-based technology. We present a conceptual framework and an approach to evaluation that addresses these problems. We elaborate on three of the components of this framework and discuss their implications for designing project and program evaluations. And we describe the contribution that technology itself can make to these designs and their implementation.
While comprehensive approaches such as the Natchitoches project may increase the likelihood that significant results will eventually accrue (David 1994), they also significantly complicate the evaluation. Comparison groups are difficult or impossible to identify or create; the lack of experimental control makes the isolation of individual variables difficult; and various elements of the project interact with and influence each other. As a consequence, it is difficult for evaluators to make comparative statements and nearly impossible to attribute results to a particular component of a project--one cannot say whether the outcomes are due to the technology, the curriculum change, increased teacher skill, or the involvement of a new set of participants. Yet some stakeholders frequently demand such attributions, and evaluations must address these needs. We propose an approach that can provide some insight into causal relationships within a project.
Alternately, the TELSPAN Consortium--participants in Pacific Bell's California Research and Education Network--are using high-bandwidth networks and video technology to connect schools with the Jet Propulsion Laboratory and the California Museum of Science and Industry. While the TELSPAN Consortium also hopes, no doubt, to influence student learning, this would likely occur as an offshoot of some other change--such as increasing teachers' scientific knowledge, which in turn may increase students' scientific understanding. Evaluation designs must be able to measure intermediate products and partial results that can be used to trace indirect causal connections to distal outcomes; this is part of the approach we propose.
By clustering projects with similar goals, evaluators can identify common criteria and use similar instruments to collect comparable data across projects. This can be done in a hierarchical and flexible fashion. Evaluators can use similar instruments and outcome measures for the projects that have the most goals in common (Barley and Jenness 1995), and can aggregate these projects for most analyses and interpretations. For projects with fewer goals in common, evaluators may use only a subset of these measures, supplemented by others. These less similar projects would be aggregated together for certain analyses and purposes but not others. Shared measures can maximize the usefulness and cost-effectiveness of evaluations. For example, the Education Challenge Grant recipients in Natchitoches, Dover, Waukegan, Indianapolis, Baltimore, and several other communities all focus their large-scale projects on increasing student learning, as described in Goals 2000 (National Goals Panel 1994). Clustering these projects together for evaluation would allow sharing of outcome measures and instruments, which would reduce the costs of the evaluation, increase the possibility of making comparisons within the Challenge Grant program, and allow summative statements to be made about these projects as a group. For other purposes, projects within this larger group can be clustered together around other features, such as technology approach or target audience.
Clustering projects that have similar purposes and stakeholder needs promotes sharing of data and information across projects in ways that can improve project performance and effectiveness. Clustering all planning grants, or all grants targeted at the school district level, or all grants focused on at-risk students will allow for the development of communities of practice that can share effective project strategies and lessons learned.
Projects can be clustered around similar technological or pedagogical approaches, such as all network applications that support project-based science or all applications that connect schools with homes to increase parental involvement. When experimental designs are possible and quantitative measures are used, these groupings can form the basis for sophisticated meta-analyses (Kulik 1994). When standard experimental designs are not possible, however, evaluators may draw on the commonalties of these projects to develop working hypotheses or local theories that postulate the means or mechanisms by which project activities can have an impact on common outcomes (Herman 1994). Such theories can guide the collection of data to test the causal relationships between given components of the project and specific outcomes (Means 1996). Alternatively, some level of causality can be established by comparing the effects of projects with certain features or components with a group of projects that have some of these components but not others (Yin 1995; Strauss and Corbin 1990; Ragin 1987). By using these clusters to aggregate projects within a program, funders can document the range of impacts of the portfolio of their investments.
At the Center for Technology in Learning at SRI International, we have used this approach to develop a conceptual framework that can guide the evaluation of the educational impacts of the National Information Infrastructure. We place the broad range of education-related NII efforts within a framework that describes the major factors impinging on network-based education reform and the causal relationships among them. These relationships are diagrammed in Figure 1.
With support from the NSF NIE Program, SRI is working with project staff and stakeholders across the country to elaborate on this framework, develop evaluation designs, and identify criteria and data collection strategies that are appropriate to the common goals and activities of projects within clusters and are sensitive to the differences between clusters. SRI is using the framework to cluster network-based education projects in groups that share goals, purposes, or similar approaches to using network-based technology.
Network-based projects are similar and different in a variety of ways. From the breadth of possibilities, we have identified some key features that will guide our clustering. The most significant characteristics that can be used to cluster or differentiate network-based projects are:
This conceptual framework allows us to identify the most likely direct or proximal impact of a particular project. For example, the NSF NIE Community of Learners Network project in Maine will develop a regional data network to link schools, libraries, and community centers to individual households and the Internet. This project falls into the "educational systems" component of our model, which suggests that the most direct impact would occur on structures within the system rather than on student learning, for example. The likelihood that this project also would contribute eventually to improved educational practices and teacher and student outcomes would be moderated by other factors in our model, such as the availability and use of curriculum materials that take advantage of the network, the significance of the scientific content, and the access to needed technical support.
Within this framework, "advanced technology" projects are efforts that aim to develop educationally-oriented hardware or software systems, similar to the activities typically supported by the NSF AAT program. These systems are usually designed as proof-of-concept prototypes. While development is focused on educational applications, it is typically not within the scope of the project that these prototypes will be widely used in classrooms. Such projects include the Distant Mentor project at SRI, which develops and tests network technology and a pedagogical model that supports audio communications and shared computer applications; the goal is to create a real-time, distance learning environment.
"Testbed projects" provide the technological infrastructure that connects schools. Examples are the NSF AAT- and NIE-supported projects at Technical Education Research Center (the Alice project) and at Bolt Beranek and Newman, Inc. (the National School Network Testbed) to develop education-oriented, client-server models that support direct access of schools to the Internet.
"Planning and policy studies" and "research studies" are projects in our framework that also aim to restructure the educational system and, ultimately, educational practices. A sample planning project is the NSF NIE grant to Morse High School in San Diego to work with local industry on a plan for an infrastructure that will connect the school with companies and to implement a curriculum that will involve students in producing a biomedical product. An illustrative policy study is the NSF NIE grant that supports the project described in this paper: building a model and set of recommendations for assessing the impact of network-based projects on education. Research studies supported by sources like the NSF Research on Teaching and Learning Program develop basic knowledge about cognitive and social processes and the variables that influence them, including technological factors.
Advanced technology projects, testbeds, and planning, policy, and research projects support and enable other projects that are closer to what happens in schools and classrooms. In the remainder of this paper, we concentrate on those projects that focus most directly on "educational systems" and "educational practices." These projects are the most likely to have an impact on improved teacher and student outcomes. In the following sections, we elaborate on the features of projects that fall into these clusters and discuss criteria and methodologies that can be used to assess their impact. We conclude by describing how network-based technology can facilitate this assessment as it becomes more pervasive in schools and homes in the future.
Some projects in this cluster use the network to coordinate efforts with national, state, or district agencies. For example, Discovering Community Networks, an NSF NIE project at the Maine Mathematics and Science Alliance, is linking diverse sites participating in the Maine State Systemic Initiative. Other projects may connect homes, businesses, and professional groups to schools. These projects may emphasize support from and connections to groups and resources outside the classroom. For example, the Education Challenge Grant project at Capital School District in Dover, Delaware, will be using set-top boxes connected to home televisions to bring entertaining, interactive, classroom learning activities directly into the homes of schoolchildren throughout the state.
These projects have the potential to change the structure of the educational system. They offer resources not previously available to schools. They offer the possibility for new people to participate in the educational process--parents, scientists, business people, community leaders, other social service providers. They enable new relationships with other teachers, schools, businesses, and homes. As a consequence, conventional organizations of places, people, and time may yield to alternative configurations of classrooms and laboratories, actual and virtual field experiences, teaching teams and communities of practice, distant mentors and teleapprenticeships, and variable block schedules. These structural changes may be vertical, as well as horizontal; networks may connect teachers and schools to professional development and educational reform efforts across district, state, and federal levels. Given that network-based programs may have their most profound effect on what the educational system is, who is involved, and where, when, and for how long education takes place, evaluation designs will need to document and analyze these differences.
Among the criteria that might be used to assess the impact of these projects are:
Many network-based projects aim to change educational practice by supporting pedagogical innovations and new approaches to assessment. These projects provide teachers and students with access to authentic and challenging tasks, multidisciplinary work and extended blocks of time, opportunities for student-directed learning and collaborative relationships, and performance-based assessments that support educational reform (Means and Olson 1994, 1995). These reform efforts seek to change the role of students and teachers; students are more actively involved in structuring their learning, and teachers serve as coaches, models, and guides. For example, two NSF AAT projects, Kids as Global Scientists at the University of Colorado, and the Global Laboratory at the Technical Education Research Center, provide students and teachers with technologies and activities that support authentic scientific experimentation by students within local and distant collaborative communities. And in a number of Apple Classroom of Tomorrow schools, including sites in San Rafael, California, and Washington, D.C., groups of students are exchanging and discussing multimedia projects via satellite.
Projects in this cluster will require, and may help make possible, new approaches to teacher professional development. Teachers will need to learn how to orchestrate network-based activities, student groups, and alternative approaches to instruction and assessment. In turn, the NII can support teachers' collaboration with other teachers in professional development activities to help them acquire and apply these skills and to stimulate their reflections on effective practices. Electronic bulletin boards, e-mail, chat rooms, and videoconferencing can allow teachers to exchange ideas and concerns. In addition, teachers could post classroom materials, activities, and actual lessons, as well as reflections on their use. Teachers could also share resources and methods for adapting activities for diverse student populations. Immersive environments, such as multi-user domains, might offer teachers an electronic space, independent of time and place, within which they can build self-sustained communities of practice.
In this section, we specify criteria for judging the quality of project materials and activities. The curriculum of network-based projects would be evaluated according the quality of the subject-matter information and resources made available. Network-based curricula can be evaluated both by expert reviews and by observations of actual implementation. Lessons would be expected to engage students in thorough study of significant problems and to provide for collaboration, self-directed learning, and self-reflection.
Evaluations of network-based professional development programs will look for evidence that they are collaborative, focused, and ongoing (Little 1993). Professional development activities aimed at non-educators who participate in education will be evaluated also by the guidance given about pedagogy and communication with students.
Methodologically, expert reviews will provide one source of evaluation data on the quality of curriculum content and professional development experiences. Classroom observations and case studies can assess the changes in classroom practices of both teachers and students. Observations of in-service sessions and working groups can document the nature and extent of teacher participation and collaboration. Teacher reactions gained through surveys and interviews can also provide evidence of impact.
In the area of pedagogy, teachers are moving into roles as facilitators, models, and coaches, particularly for strategic inquiry. The appropriateness of pedagogical techniques employed by individuals from other professions and backgrounds who may serve as experts or mentors will need to be assessed. Scientists, for example, may be expert in their field but may not communicate their expertise to students in clear, developmentally appropriate terms.
Network-based projects require teachers to develop classroom organization patterns that allow students to collaborate with other students in their own classroom, in other schools, and in other communities of practice. Teachers need to be able to structure group memberships and roles to support productive collaboration, to monitor and provide feedback, and to assess the progress of students as individuals and as team members. Teachers are expected to use performance-based assessments of students' progress and to involve students and their parents in assessing portfolios of achievement. In addition, teachers increasingly are expected to collaborate with other members of their profession in renewing their teaching skills and improving their classroom practice.
These teaching outcomes can be assessed through combinations of standard tests (for knowledge), classroom observations, and structured interviews. Recent designs for teacher performance assessments ask teachers to assemble portfolios describing their classroom practices and to reflect upon them. These will be important sources of information on changes in teacher practice.
As network technology enables the participation of people, agencies, and resources that traditionally have not been accessible from the classroom, it will also automatically collect data on the extent and nature of this participation. By routinely monitoring on-line information about participation, patterns of access and use, and user reactions, project staff can follow the impacts of their initiatives and modify the design and features of their projects. Evaluators can use these data to document the breadth of impact on the educational system and the extent to which program activities are coordinated with and supportive of other reform efforts.
As more of the curriculum is on-line, the network can be used to assess its quality. Current approaches for judging the quality of educational curricula and conducting program evaluations rely, in part, on laborious forms of document analysis and review by panels of experts. On-line curricular materials and activities could reduce the logistical and economic costs of assembling expert panels. Criteria for judging quality could be shared and debated, and judgments of quality could be deliberated in multiple electronic arenas. Judgments could be directly linked to the artifact being judged, with confirming and dissenting commentary attached. As a result, the evaluation process could become more open to standard-setting groups, such as subject matter experts. In addition to inspecting curricular artifacts, panelists could conduct on-line interviews with developers. Because expert critics need not be physically convened, the frequency of reviews could be increased and the composition of review teams could be improved--for example by involving more scientists and experts.
Professional interactions and professional development activities will leave electronic traces on the network for evaluators to collect, describe, and analyze. Evaluators could analyze the number of participants, the frequency of interactions, the composition of groups, the contributions of individuals to group efforts, and the substantive focus of discussions or projects. These descriptive data could support judgments of the quality of professional development offerings. These data could also be used to evaluate professional development programs and their impacts on classroom practices and individual teachers.
Networks can also support assessments of project impact on teachers. Network-based projects can develop templates for teacher notebooks, logs, and journals that will invite teachers to document classroom practices and reflect upon them. Templates that structure teacher records of activities can also provide ongoing documentation of day-to-day activities and teaching that infrequent site visits and classroom observations can only sample, and they can do it in a standard form that allows for comparisons across classrooms and projects. Electronic teacher portfolios consisting of such entries as developed activities, assessments, and communications with students and parents could be designed to serve both program evaluation and teacher assessment purposes. In addition, the NII will allow evaluators to field more traditional instruments on-line, such as the electronic administration of teacher surveys. Advances in network conferencing capabilities will allow focus groups and interviews to be conducted electronically. Moreover, technology may permit classroom observations to be conducted by video and followed by electronic teacher interviews.
Finally, the NII will offer evaluators many more systematic, dynamic, and cost-effective ways to gather ongoing information about a broader range of student outcomes. Common assessment approaches such as student surveys and assessments can be administered on-line. These traditional instruments can become more useful by building in adaptive items and branching questions that customize the assessment and collect more precise data.
In addition to assessments of students' subject matter knowledge, network-based projects can yield rich documentation of students' reasoning processes. Networks can provide frequency data on which resources students access and how often and in which ways they access them. Networks can provide electronic logs of how often students interact with other students and/or experts, and allow researchers to collect sample excerpts of on-line work and electronic conversations and interactions. Analyses of these conversations and interactions can provide evidence about how well students are reasoning, how much their understanding about subject matter knowledge is deepening, and how effectively they are collaborating with other students.
Not only can the network archive student products for examination and assessment by evaluators, it also can permit the electronic publication and distribution of these digital artifacts to authentic audiences and users, including other students, parents, scientists, business people, and even "customers." These electronic audiences, in turn, can be involved in authentic evaluations of student products using the real-world criteria of interest, need, and usefulness.
These electronic data collection methods will make part of the evaluation job easier, less expensive, and more comprehensive. But evaluators will be confronted by a mountain of data to analyze. Here, too, technology will be able to help. As the electronic data become pervasive, network-based tools will be developed for searching large databases for specific types or patterns of data. These tools will need to be sensitive to a range of data that include numbers, text, and multimedia information. And they will need to assist evaluators in annotating, coding, and sorting data in ways that will support logical analysis, as well as statistical analysis. When available, such tools will dramatically increase the utility of the massive amounts of indigenous digital data that will be on the network and will facilitate the assessment of the NII's impact on education.
Barley, Z., and M. Jenness. 1995. Conceptual underpinnings for program evaluations of major public importance: Collaborative stakeholder involvement. In Footprints: Strategies for non-traditional program evaluation, ed. J. Frechtling. Arlington, VA: National Science Foundation.
Brown, A. 1992. Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences 2(2): 141-178.
David, J. 1994. Realizing the promise of technology: A policy perspective. In Technology and Education Reform, ed. B. Means, 169-190. San Francisco: Jossey-Bass.
Herman, J. 1994. Evaluating effects of technology in school reform. In Technology and Education Reform, ed. B. Means., 133-168. San Francisco: Jossey-Bass.
Information Infrastructure Task Force. 1993. The National Information Infrastructure: Agenda for action. Washington, DC: U.S. Department of Commerce.
Information Infrastructure Task Force. 1994. National Information Infrastructure: Progress report. Washington, DC: U.S. Department of Commerce.
Kozma, R., W. Grant, and the Center for Technology in Learning. 1995. Year 2005: Using technology to build communities of understanding. In Education and technology: Future visions, ed. K. Fulton, 121-145. Washington, DC: U.S. Government Printing Office.
Kulik, J. 1994. Meta-analytic studies of findings on computer-based instruction. In Technology assessment in education and training, ed. E. Baker and H. O'Neil, 9-33.
Little, J. W. 1993. Teachers' professional development in a climate of educational reform. Educational Evaluation and Policy Analysis 15(2): 129-152.
Means, B. 1996. Evaluating technology's role in state and local education reform. Paper commissioned by the U.S. Department of Education for a conference on the Design of Effective Evaluation of the Star Schools Program, Washington, DC.
Means, B., and K. Olson. 1995. Technology's role in education reform. Menlo Park, CA: SRI International.
Means, B., and K. Olson. 1994. Tomorrow's schools: Technology and reform in partnership. In Technology and Education Reform, ed. B. Means, 191-222. San Francisco: Jossey-Bass.
National Goals Panel. 1994. The national educational goals report: Building a nation of learners. Washington, DC: U.S. Government Printing Office.
Ragin, C. 1987. The comparative method: Moving beyond qualitative and quantitative strategies. Berkeley, CA: University of California Press.
Salomon, G. 1991. Transcending the qualitative-quantitative debate: The analytic and systemic approaches to educational research. Educational Researcher 20(8): 10-17.
Strauss, A., and J. Corbin. 1990. Basics of qualitative research. Newbury Park, CA: Sage.
Yin, R. 1995. New methods for evaluating programs in NSF's Division of Research, Evaluation, and Dissemination. In Footprints: Strategies for non-traditional program evaluation, ed. J. Frechtling. Arlington, VA: National Science Foundation.
[ Back to the Futures Page ]
Last modified May 1, 2002 (pas).