Evaluating technology effectiveness in education poses special problems, because of additional educational changes, beyond the technology, needed to reveal the technology's full promise. Douglas Merrill of Rand spoke to this problem. Larry Frase volunteered that in the circumstance of rapid technological change, the much slower pace of educational software development, and the relatively glacial rate of change in new knowledge from research on software usability and on models of implementation in different subject matters, an engineering approach to evaluation might be preferable to the idealized scientific model of controlled experimentation. He subsequently wrote on the subject from which we quote below.

Douglas Merrill (RAND)

Merrill emphasized that the opportunity of using technology to improve student learning and school effectiveness could readily be thwarted by an evaluation approach that was collinear with existing educational practice and ignored factors affecting an effective technology implementation like:

• pedagogy, e.g., more student-focused than teacher-centered; and,

• time and curriculum framework, e.g., longer, multidisciplinary classes.

He went on to say that the metrics employed in an appropriate evaluation should focus on the interaction of student, institutional and task factors. From a paper by Merrill²⁸: Naturally, the student's personal characteristics interact with the educational intervention to produce differing outcomes. For example, there are students who exhibit higher ability than others, regardless of one's beliefs about the nature of the vague construct "ability". Higher ability students tend to react differently to feedback than students who are having more difficulty. ... differences in feedback behavior are often directed at maintaining students' motivation, another key personal characteristic affecting pedagogical outcomes. Thus one would expect ability--however defined--and motivation to be important factors to consider when evaluating educational technology.

Concerning institutional factors, Merrill writes: Student factors are not all that should affect the metrics used by an evaluation. The way the technology is integrated within the school setting also can change the set of appropriate measures. For example, it is often the case that schools do not place computer-based tools in many classrooms, but rather centralize them in a computer laboratory, run by a set of 'high priests'.

This design does not lend itself to a tight integration of technology into curriculum, because using the machines requires reserving the laboratory, getting the students there, ... For example, one common early measure of the effect of technology on education was the total time spent using the computers per class. Given a laboratory design, this particular metric is virtually guaranteed to be small--but interpreting this small amount of time on computer tasks as indicative of a failure of technology is probably short-sighted.

Again, Thus, another possible source of interference with evaluation of educational technology is variance in the training of teachers, and the consequent differences in comfort level and curricular integration.

On task factors, Merrill writes: The nature of the task put to students and the knowledge they are expected to acquire should also affect the evaluation metrics selected and their interpretation. A task could be primarily procedural in nature, requiring the application of a sequence of steps, but not necessarily drawing on many facts, known as declarative knowledge. Much of mathematics consists of procedural tasks, with the declarative portion devoted solely to selecting a procedure to apply. In contrast, the study of history involves the recognition of relationships between actors and events--declarative knowledge. Again, an evaluator must determine which of these two types of knowledge to measure and to interpret the results accordingly.

Finally, Future evaluations of educational technology should not focus single-mindedly on simple outcome measures, such as post-tests, without also attending to more complex metrics describing the learning process, such as cognitive modeling. Furthermore, the research must also take the institutional factors into account, such as where a changed curriculum fits and what other changes must accompany its introduction. Similarly, each different task proposed as part of a new curriculum could require different sorts of evaluations.

Larry Frase (ETS)

Frase wrote later summarizing his thoughts about the workshop. We quote on the issue of evaluation.

"The workshop showed that we can expect increasing demands for teacher training, partnering to achieve the best and most efficient educational implementations, demands for new research and measurement techniques and for more formal approaches to educational implementation, and demands for the development of broadly applicable telecommunications and educational software tools.

"The technologically derivative revolutions in education and research pose a peculiar problem, because of the temporal discontinuities associated with technological change. Consider the time scales within which progress takes place in technology, education, and research and what these discontinuities portend. The rapid advance of computing speed and miniaturization is well known. The development of software, to make use of those capabilities, appears to lag, but research on software usability and models for educational implementation in different subject matters lags even more. Developments in educational software, in fact, are impeded by archaic models of educational theory.

"The idealized model of scientific research, as controlled experimentation leading to the resolution of major theoretical issues, is probably the wrong model for education in an age of rapid technology development. A more appropriate model would be an engineering approach, using the tools and techniques of science to evaluate evolutionary (formative) changes in educational products and processes. Companies, such as AT&T and IBM, have already pushed their research communities in this direction. It is only a matter of time before the academic community does the same.

"The concept of educational achievement, as represented by the measurement of outcomes, must change. The outcome evaluations reported in this workshop were all the more impressive, because they represent weak tests--many of the evaluative instruments were quite general, only weakly focused on the concepts taught and quite probably containing irrelevant elements. In short, the concept of educational achievement, as represented by global outcome tests, lacks conceptual validity. We know that educational outcomes consist of more than what we can see on a few test items. If we can better define the processes involved in thinking in different subject matter domains, more reasonable instruction and assessment will follow. To the extent that this is true, the data on the effects of educational technology presented in this workshop were an answer to the wrong question. In cases where the elements of instruction were developed concurrently with tests of achievement, very strong educational effects were seen.

Previous	Table of Contents	Next