![[Table of Contents]](/icons/up.gif){kind=icon}
Background Paper for the Expert Panel on Educational Technology--August 31, 1998
The Federal Regulations for designation of exemplary and promising programs list four criteria for determining whether an education program should be recommended as exemplary, promising, or neither. These are evidence of success, quality of the program, educational significance, and replicability. While the Expert Panel on Technology must use these four general criteria as the basis for judgments, additional specific criteria and indicators within each of these headings can be developed, and the process of evaluation can be handled in a manner of the Expert Panel?s choosing. For example, the Expert Panel on Mathematics and Science organized these evaluation criteria for mathematics programs into three categories to be evaluated by Quality Review Panels; these were labeled quality of program, usefulness to others, and educational significance. The fourth criteria, evidence of effectiveness and success, will be evaluated by an Impact Review Panel comprised on experts in evaluation design and analysis. Within these four areas, eight specific criteria, each with several specific indicators, were developed.
Our review of lessons learned from past technology studies suggests considerations to be taken into account within each of the four evaluation criteria found in the Federal Regulations. We present these as food for thought as the panel develops their specific criteria, indicators, and procedures.
What is the intellectual substance of the program? Does it support content standards in the curriculum areas in which it is applied? Does it focus on challenging learning goals? What audiences is it appropriate for and under what conditions? Does it support authentic learning and problem solving in a context that is meaningful to the intended student audience? Is is based upon the latest research in the content area and draw upon experts in the field? Does it encourage depth of understanding, allowing students to "grow with it" according to their level of prior expertise? Is it free of bias and stereotyping of any kind?
As discussed above, to be eligible for selection, projects and activities should use technology in a way that goes beyond automating existing practices. While technology can make it possible to do things faster, easier, with more "pizzazz", and even more cheaply in some cases, any one of these factors alone is not enough to make an application rise to the level of significance suggested by the resource group. Ideally, several of these factors should be demonstrated in a practice; even better are those activities that go beyond what is today?s usage and make it possible to do entirely new things in the classroom that would not have been possible or feasible without the use of technology. While this is often a subtle distinction, it gets at the essence of the kinds of factors noted in the section above describing impacts of technology in support of school reform goals--that is, significantly changing classroom practice and interactions through the use of technology.
A project or practice that is unique to a particular teacher or setting, while interesting, will not have the impact of one that could be used by any teacher or school, under the right conditions. These conditions, (e.g. training, resources, and time) should be specified and appear reasonable for other sites considering adoption of the program or activity.
The question of sustainability might also be considered under the replicability criterion. That is, is what is demonstrated in the application something that sets up a framework that would last beyond a particular teacher or principal?s support if that individual were to leave the school? Identified projects and activities should show evidence that they have or will become institutionalized as a new way of doing business in that school. If originally funded by outside resources, has a program provided enough promise (for promising identification) or evidence (for an exemplary designation) to move forward under local or community support?
In considering metrics and impacts, multiple metrics are preferable to single measures, and both qualitative and quantitative measures should be valued. These might include:
Determining how these multiple and varied measures are correlated, validated, and equated will be a task for the evaluation and measurement specialists advising the Expert Panel. The assessment measure should be appropriate for the learning activity. For example, when assessing the impact of computers on improving writing skills, it makes no sense to conduct the test in a multiple choice pencil and paper format. Clearly, the assessment format is as important as its content.
The following outcomes are suggested as examples of the kinds of impacts the Expert Panel might consider in designating promising and exemplary practices . While not all of these are necessary in any one project, many of these outcomes have been found in what have been considered promising technology facilitated projects in the past, as described in section II above.
Is complex content mastered earlier in the curriculum by a broader range of learners?
This would be indicated by, for example, mastery of algebra, geometry, or calculus concepts earlier in the mathematics curriculum; earlier and deeper mastery of foreign language skills; or an increase in the specialized technological skills acquired by the students.
Are groups bypassed by traditional teaching approaches thriving under this activity or practice?
These groups might include low achievers, students with limited English proficiency, special education students, those with learning disabilities, gifted and talented students, or students with behavior problems.
Does the project make it possible for students to master material or develop skills that might not otherwise be taught (or understood) through traditional teaching methods?
Evidence for this could include such activities as systems thinking, computational science, research design and defense in a public forum, and working with abstract concepts and symbolic thinking.
What achievement gains can be documented?
These could be indicated by such measures as higher test scores, greater enrollment in advanced courses and a larger percentage of students going on to college.
Has the technology impacted students' attitudes and motivation?
Do they spend more time on task? Is their attendance better? Do they concentrate more in class and show a heightened interest in the material? Have discipline problems decreased? Do students work better with students from other social or ethnic groups? Do they demonstrate new or greater leadership and cooperative skills?
How has the project impacted teachers' teaching styles?
Assignments may become longer or more complex. Lesson plans may become more thematic and interdisciplinary in content, with assignments linked to activities outside the classroom. Teachers might show evidence of new or greater use of inquiry tasks requiring a depth of analysis not evident in earlier student work. They may encourage more use of collaborative learning, student independence and initiative . Teachers might demonstrate greater reflection on their teaching style and how it effects their students? learning.
What are the personal and professional impacts on teachers?
These might include such indicators as greater retention of good teachers by a school system, or a higher percentage of teachers applying for and meeting the standards of the National Board of Teaching Standards. Do teachers demonstrate increased involvement in professional activities and make presentations at conferences? Are their teaching materials used by other teachers locally or virtually? Are they receiving professional recognition outside the school?
What are the impacts on the school environment?
Examples might include such outcomes as greater collaboration among teachers or changes in the allocation of time and resources within the school. Schools might exhibit greater involvement with reform and research organizations or improved links with higher education institutions.
Are there indicators of improved school/community interactions?
These might include such indicators as improved parental involvement (more parents involved from all sectors of the school community) and greater business involvement (contributions of time, personnel, expertise, equipment and/or expertise). Another indication could be evidence of developing collaborative relationships (summer, part-time, or graduate hiring or internships) among students and teachers with local businesses and industries. Projects might demonstrate that they provide valued service to the community in a variety of ways.
[V. Conclusion]