Archived Information

Further Reflections By The Panel

The following section reflects the work and thinking of the Educational Technology Expert Panel after the completion of the review process and the recommendation of programs to the Secretary. The material in this section should not be construed or interpreted as an endorsement by the U.S. Department of Education. After much discussion, the Panel leaves with you a definition of significant learning for your study and consideration. The Panel also offers for your consideration and use the following framework and guidance as you plan or review your educational technology program.

You may have noted that "learning" is cited many times throughout the Educational Technology Expert Panel's "Guidelines and Materials for Submitting Educational Technology Programs for Review" (June 1999). The Panel applied the criterion of complex learning and thinking skills as outlined on page 11 of this Report in making its program recommendations. After the proceedings, however, the Panel concluded that it needed to propose to the field a much more encompassing term than complex learning; thus, it offers to the field the term significant learning. As stated in the Report (page 3), "The objective of using educational technology should be to create an instructional environment that fosters significant learning for preK-12 students." The Panel's working definition of the term follows:

Significant learning is learning that stimulates the student to further inquiry and reflection, a process that teaches the student how to master ever-deeper levels of understanding. It entails a combination of mastery of basic skills (such as number facts and literacy) and acquisition of foundational information. Fundamental skills and knowledge include facility with and understanding of quantitative relationships; the ability to recognize and construct a persuasive argument using evidence based on observation and the vocabulary of a specific discipline; and sufficient familiarity with the natural and historical world to recognize patterns of behavior, causality, and meaning. Significant learning enables the student to know what questions to ask when confronting new information or circumstances and how to marshal appropriate processes and resources to solve problems.

Significant learning is ultimately what the teaching and learning process is aiming to accomplish with all learners in our nation's schools.

Revisiting The Criteria For Evaluating
Technology Programs

The preceding criteria and indicators (pages 9-13) were used to evaluate the educational technology programs submitted to the Expert Panel. Based on the knowledge gained by the Panel and as a result of a careful examination of the review process, the Panel proposes the following refinements in the criteria and indicators for greater clarity. You will also see below a set of rubrics developed and offered by the Panel to (l) clarify the scoring system; and (2) help you evaluate the soundness and effectiveness of your educational technology program. The Panel hopes that this refinement in the indicators and its proposed set of rubrics will help to stimulate meaningful conversations and thinking around what constitutes effective teaching and student learning in our nation's schools when technology is used as part of the educational process.

A. Quality of Program

Criterion 1. The program addresses an important educational issue or issues and articulates its goals and design clearly.

Please describe the program in detail. Readers should be able to understand the overall program, as well as what participants actually do, sufficiently well to explain the program to others. Include the items following each bullet point.

• The educational goals are significant. Include in description the
  • need or problem the program addresses and how it relates to teaching and learning in preK-12 schools;
  • program goals; and
  • content and learning goals.
• The program design is thoughtful and coherent. Include in description the
  • program design--how the parts relate to the whole;
  • technology used and how it is essentialfor achieving the program's goals;
  • professional development provided as part of the program; and
  • assessments used to determine the program's efficacy and achievements.
• The program goals and design are supported by research. Include in description the
  • relationship of goals and design to relevant research on educational reform, the science of teaching and learning, or technology.
• The program description is clear and complete. It should include
  • key learning activities for participants;
  • subject population(s), including ethnic, racial, socioeconomic, and gender percentages and the size of any special populations served (e.g., ESL, AP biology students, and students with disabilities);
  • overall size and maintenance costs (e.g., funding and staff requirements, and number of people in target population);
  • keys to the program's success; and
  • a specific, concrete example that best captures the changes achieved by this program.

Criterion 1 Rubrics

Level 1

There is little or no demonstration that the program addresses important educational issues or articulates its goals and design clearly. The program's goals, design, and description are vague and incomplete. The design for the use of technology may be an example of already-existing widespread practice and therefore will not significantly advance or inform the field. The design may be vague, incoherent, unclear, or unsupported by research; the program description may be unclear. In most cases no indicator is adequately addressed.

Level 2

There is insufficient or incomplete demonstration that the program addresses important educational issues or articulates its goals and design clearly. The goals, program design, and program description are often too general to be considered adequate. In some cases, however, the program is designed or described adequately, but the goals are not significant. The goals may be limited or local--not set in a broader educational or research context. Generally, only one indicator is adequately addressed.

Level 3

The goals are adequately significant and the design adequately thoughtful and coherent. The description, however, may be insufficiently clear or incomplete. The goals and design are often not supported by research. At least two indicators, including the significance of the educational goals, are adequately addressed.

Level 4

The goals, design, and description may be convincingly significant, thoughtful, coherent, clear, and complete. Alternatively, the goals and design may be compelling in their significance and thoughtfulness, but the description only adequate in its clarity and completeness. Program goals and designs are at least adequately supported by research.

Level 5

The goals, design, and description are convincing or compelling. The program goals and designs are convincingly supported by research.

B. Educational Significance

The Expert Panel considers the following three areas--learning, equity, and organizational change--essential to fulfilling the promise of educational technology. A sound program must address all three, and all three must be shown to have impact on or linkage to preK-12 student learning.

Criterion 2. The program develops complex learning and thinking skills.

If the participants are other than preK-12 students, the applicant should articulate the program's goals and their connection to student learning.

• Students increase their in-depth understanding and competence in at least one content discipline.
• Students develop the habits of self-directed, lifelong learning (e.g., the ability to collaborate, to direct one's own learning, to solve problems, to communicate ideas clearly, and to think flexibly and critically).
• Students become proficient and critical consumers and producers of educational technology.
• Students are prepared to enter a technology-infused workplace.

Criterion 2 Rubrics

Level 1

There is little or no demonstration that the program develops complex learning and thinking skills. It may be unclear just what activities learners engage in, and how and what they may be learning from these activities. The contribution of technology to the learning may not be well specified. At this level, there is insufficient demonstration of any of the indicators.

Level 2

There is insufficient demonstration that the program develops complex learning and thinking skills. The program may adequately demonstrate that it is addressing one indicator, or may provide more vague demonstration of more than one indicator. Programs may describe clearly the activities learners engage in, without making the case for how these activities lead to the development of complex learning and thinking skills. If the program is for teachers, it may be unclear how the program can lead to their students' developing complex learning and thinking skills. Programs that focus on the transition from school to work may not clearly demonstrate how they connect to content learning or to self-directed, lifelong learning.

Level 3

There is adequate demonstration that the program develops complex learning and thinking skills. The program adequately addresses at least two indicators. These programs describe program activities sufficiently, and make a case for how these activities contribute to learning, for two of the indicators. Programs that serve teachers may not fully clarify how the program will result in complex learning for students. Programs that focus on the transition from school to work may fail to demonstrate how they connect to content learning or to self-directed, lifelong learning.

Level 4

There is convincing demonstration that the program develops complex learning and thinking skills. The program addresses a minimum of three indicators, at least one convincingly (the other two at least adequately). The program provides a clear description of program activities, and makes a case, through both argument and examples, for how these activities contribute to learning. Programs that serve teachers clarify the links to student learning--that is, how teacher participation in the program is intended to result in complex learning for students. Programs that focus on the transition from school to work demonstrate their intended connection to content learning or to self-directed, lifelong learning.

Level 5

There is compelling demonstration that the program develops complex learning and thinking skills. The program addresses at least three indicators, at least two convincingly (the third at least adequately). The program provides a clear description of program activities and makes a case, through both argument and examples, for how these activities contribute to learning. Programs that serve teachers demonstrate the links to student learning. Specifically, they show, through examples of student activities and student work, how teacher participation in the program leads to complex learning for students. Programs that focus on the transition from school to work demonstrate their actual connection to content learning or to self-directed, lifelong learning.

Criterion 3. The program contributes to educational excellence for all.

• The program conveys high expectations for all learners.
• The program responds to the diverse needs of varied populations of learners. Learners may vary, for example, by achievement level, ethnicity, socioeconomic level, English language proficiency, gender, learning style, and handicapping conditions.
• The program includes active outreach and partnerships with the community or relevant organizations to encourage broad participation of diverse groups of learners.
• The program increases the participation or achievement of underserved learners so that the gaps between these learners and other categories of students diminishes.

Criterion 3 Rubrics

Level 1

There is little or no demonstration that the program contributes to educational excellence for all. It may not be clear who has access to or is served by the program. There is no evidence of outreach or collaboration. While the program may, for example, set high expectations for its learners, it may serve only students who are already well served without attempting to include and serve a broader range of learners.

Level 2

There is limited and insufficient demonstration that the program contributes to educational excellence for all. A program may address the needs of an underserved group of learners, but without conveying expectations that are high or clear. A program in teacher development may serve a diverse group of teachers without making any connection to the diverse needs of the students they teach. A software-based program may claim that the software is designed to serve diverse learners but not demonstrate how or that it does. There is no evidence of active outreach or collaborative partnership.

Level 3

There is adequate demonstration that the program contributes to educational excellence for all. Programs generally set high expectations and serve diverse groups of learners. Many do not have strong outreach programs. Programs may have partners, but be unable to show how these partnerships contribute to broader participation. Some programs are able to demonstrate that they are closing gaps in the participation of underserved learners.

Level 4

There is clear and convincing demonstration that the program contributes to educational excellence for all. Programs set high expectations for all learners, meet the needs of diverse and underserved learners, and have active outreach and collaborative partnerships. Programs show that they narrow the gaps in participation for underserved learners, although at this level they do not yet demonstrate an ability to narrow the gaps in student performance.

Level 5

There is complete and compelling demonstration that the program contributes to educational excellence for all. The program fully addresses all four indicators. Programs are able to demonstrate that they have increased both the participation and the performance of underserved groups of learners.

Criterion 4. The program promotes coherent organizational change.

• The program reflects a vision of educational renewal consistent with disciplinary content standards, recommendations from national commissions, findings from educational research, and documented best practices.
• The program has identified a set of goals that will lead to educational renewal within the organization.
• The program has identified and involved key constituents related to its set of goals--parents, professional groups, and community members.
• Through partnerships and professional development, the program enhances the human capacity necessary to accomplish its goals (e.g., allocates time for teachers' and administrators' collaboration and planning).
• As a result of the program, policies, procedures, funding, and practice have been changed to increase their alignment in support of sustainable change.

Criterion 4 Rubrics

Level 1

There is little or no demonstration that the program promotes coherent organizational change. Either the response to this criterion is too vague, or the program has not demonstrated the vision, goals, involvement of constituencies, enhancement of human capacity, or changes in policy that can promote organizational change.

Level 2

There is limited and insufficient demonstration that the program promotes coherent organizational change. For example, a program may claim to be guided by a vision without making clear what that vision is, or may recognize the need for capacity building to support change without actually building that capacity. Alternately, it may indicate that it has established partnerships without showing how they contribute to organizational change. The program may adequately demonstrate one indicator.

Level 3

There is adequate demonstration that the program promotes some organizational change. The program adequately addresses at least three of the indicators in ways that can result in organizational change. Change is not yet comprehensive or fully coherent.

Level 4

There is clear and convincing demonstration that the program promotes coherent organizational change. The program addresses four indicators, at least three convincingly.

Level 5

There is complete and compelling demonstration that the program promotes coherent organizational change. All five indicators are addressed, at least four convincingly.

C. Evidence of Effectiveness

Criterion 5. The program has rigorous, measurable evidence of its achievements for one or more among Criteria 2, 3, and 4 (learning, equity, and organizational change).

• The evidence is clearly related to the claims made for program effectiveness. For example:
  • framework for gathering evidence is closely tied to program goals; and
  • evidence bears directly on claims made for program effectiveness.
• The design for collecting evidence meetsgenerally accepted standards of quality and may include:
  • one or more comparison groups;
  • a quantified validation by an external authority;
  • a formal evaluation;
  • a quantified demonstration of positive change among participants as a result of the program (e.g., increased parental involvement in school governance; diminished gaps in achievement between groups; increased enrollment in rigorous mathematics courses or graduation rates among subject populations; changes in the base funding and requirements for professional development); and
  • an in-depth, qualitative analysis of change among participants as a result of the program (e.g., case studies, ethnographies, principled analysis of observations and interviews).
• The evidence is complete and convincing. This includes the following:
  • methods are fully described and explained;
  • sample sizes are appropriate;
  • results are clearly and completely documented;
  • data are disaggregated for relevant groups;
  • multiple methods are used to support key claims; and
  • results are generally positive.

Criterion 5 Rubrics

Level 1

There is no rigorous, measurable evidence for the program's achievements. If evidence is presented, it is not clearly related to program goals and claims, does not meet standards of quality, and is often anecdotal. Programs at this level are often in too early a stage to have collected valid evidence, although they may have plans for doing so.

Level 2

There is incomplete, insufficient evidence for the program's achievements. Although considerable amounts of data may be presented, the data do not constitute credible evidence of effectiveness. Often the program lacks a framework for gathering, analyzing, and interpreting evidence. A program may make numerous claims, for example, but present data that bear on only one or two, or none, of them. A program may be very limited in the kind of data it collects; it may, for example, rely solely on attitudinal evidence. A program may not sample adequately given the size and type of groups it serves. A program may not describe its methods with sufficient clarity. Generally, programs use only one method to support any claim. Often, programs are in too early a stage of development to present credible evidence of impact.

Level 3

There is credible, adequate evidence for the program's achievements. The evidence is clearly related to program goals and to claims of effectiveness. The evaluation design meets adequate standards of quality. The design includes most of the elements required to make valid inferences about the effectiveness of the program. The evidence is sufficiently well documented, analyzed, and complete to present an adequate case for the program's effectiveness with respect to at least one criterion.

Level 4

There is convincing evidence for the program's achievements. The evidence is clearly related to program goals and to claims for effectiveness. The evaluation design meets high standards of quality. The design includes most of the elements required to make valid inferences about the effectiveness of the program. The evidence is well documented, carefully analyzed, and complete. It presents a convincing case for the program's effectiveness with respect to at least one criterion. It further presents an adequate case for the effectiveness of one other criteria.

Level 5

There is compelling evidence for the program's achievements. The evidence is clearly related to program goals and to claims for effectiveness. The evaluation design is driven by these goals and claims. For all claims, clear and appropriate evidence is provided. The design meets high standards of quality. The design includes all the elements required to make valid inferences about the effectiveness of the program. The evidence is completely documented, carefully analyzed, and presents a compelling case for the program's effectiveness with respect to at least one criterion. A program at this level shows evidence of at least adequate effectiveness for the other two criteria.

D. Usefulness to Others

Criterion 6. The program is adaptable for use in multiple contexts.

• The program's technology requirements are easily available to potential users.
• The program is cost-effective relative to its benefits.
• After its initial implementation, the program is sustainable with existing resources (i.e., does not require extraordinary/unreasonable time, effort, or funding), and scalable (i.e., can naturally expand its scope to several teachers, multiple grade levels/subjects/sites, and different disciplines).
• The program is adaptable to a range of educational settings.
• The program provides clear and detailed guidelines about the conditions required for its successful implementation (e.g., philosophical assumptions and level of personnel training prior to start-up).
• The program provides clear and detailed guidelines about the start-up and annual ongoing costs required for its successful implementation (e.g., equipment and infrastructure; personnel; training costs; technical support; replacement or upgraded hardware or software; materials and supplies).

Criterion 6 Rubrics

Level 1

There is little or no demonstration that the program is adaptable for use in multiple contexts. In some cases, the program is too new to be able to show adaptability. In some cases, the program description is so vague or unfocused that what would be replicated or sustained is not clear.

Level 2

There is limited or insufficient demonstration that the program is adaptable for use in multiple contexts. Programs may demonstrate that their technology and cost requirements are reasonable and that they are locally sustainable, but not that they are scalable or adaptable to a range of settings;or they may provide no guidelines for implementation. Programs at this level may have made no effort to expand beyond a set of limited goals for a particular target population. Alternatively, they may be prohibitively costly, although their model is, in principle, adaptable.

Level 3

There is adequate demonstration that the program is adaptable for use in multiple contexts. Programs at this level have adequately available technology, are cost effective, and demonstrate adequate scalability or adaptability. Some may have clear guidelines for implementation, including costs and other requirements.

Level 4

There is convincing demonstration that the program is adaptable for use in multiple contexts. Programs at this level are at least adequate with respect to five of the six indicators.

Level 5

There is compelling demonstration that the program is adaptable for use in multiple contexts. Programs address all of the indicators and are able to show that they can be been widely used in multiple settings.

[The WEB Project]

This page was last updated March 15, 2002 (jer)