1. INTRODUCTION
2. WHAT ARE WE MEASURING?
3. MEASURING USE OR EXPOSURE
4. HOW DO WE KNOW IF TECHNOLOGY WORKS? MEASURING THE DEPENDENT VARIABLE(S)
5. CONCLUSION

APPENDIX

Ten Practical FAQ's (Frequently Asked Questions) about measuring IT effects

Selected Sources on Measurement of Instructional Technology

Evaluating the effects of technology use provokes the same evaluation challenges as does any other program intervention. The issues that I address in this paper are based upon my experience in evaluating the achievement effects of specific technology implementations. The five studies that have offered me the largest learning laboratory are listed in Table 1. Each required a careful description of the technology to be studied, a measure of how much students used the technology, and a measure of achievement gains.

As Mann has pointed out in "Documenting the Effects of Instructional Technology: A Fly-Over of Policy Questions", a variety of stakeholders are beginning to ask questions about technology use in schools. Many of these questions go no further than "Does technology work?" Or, "Does technology use improve student achievement?"; "Is technology in schools worth the money it costs?"; "Are there benefits to students beyond achievement?"

Many of the program administrators responsible for IT have not thought through the questions they want answered by documentation research, nor can they be expected to since operational responsibilities often preempt evaluation. Part of the job of the evaluator is crafting work that serves the needs of the stakeholders: Is this an evaluation for re-funding? For use in curriculum refinement? For analysis of classroom instruction? For public relations? For all of these?

Because stakeholder needs are not always clear, the first measurement challenge is to determine the technology "input" to be examined. Technology is lots of things: computers, CD-ROM and videodisc players, networked applications. If we focus on computers, it generally is not the use of the computer per se that is of interest, but rather a specific use, especially particular software.

For most readers of this paper, the "what is the technology" question will seem elementary. However, my experience has been that many stakeholders -- particularly school administrators, school board members, and legislators -- expect that if hardware is purchased, then improved achievement should follow. A common situation we have faced is being asked to determine achievement gains in schools where computers and word processing software are purchased. The notion that doing anything on a computer should lead to (any) achievement gains is widespread. (We were once asked to measure the math achievement impact of having provided Corel's WordPerfect word-processing software to all the elementary teachers of a district!) Therefore, identifying what technology use is being analyzed is a first step, and a step I would not bother to relate had I not learned the hard way that identifying the technology to be measured requires a considerable amount of interaction with stakeholders.

Is the technology question really a focus on the teaching efficacy of a particular software that students are using? If so, is there a relationship between the software design characteristics and student achievement? Do any of the following make a difference: instructional control, feedback, objectives and advance organizers, cognitive strategies, conceptual change strategies, scaffolding of learning support, still and animated graphics, dynamic visualization, video, navigational technique, text and story content, game context and visual metaphor fantasy context, Window presentation styles?

Or, is the question about multiple sites for technology use? The home? The school? Both? And if so, how much of what interaction in which site is related to achievement?

Do different technologies result in different kinds of achievement? For instance, do telecommunication distance learning technologies such as access to online resources, document exchange and discussion, or professional development on-line improve student achievement? If they do, is this be a direct relationship? How would we isolate these uses while examining student achievement?

It is easy to see how an initially simple question like, "What is the relationship between technology use and student achievement?" blossoms into refinements and further definitions. Carefully defining the technology to be studied then takes us to the next step.

Just because technology is present does not mean that the students are using it. How do we measure the intensivity of student use?

We faced this question in every study we have done. We have used observations, file server records, student reports, parent reports (thousands of telephone interviews, each logged and coded), teacher reports, and on-site observations. Because it isn't feasible to shadow every student every day, observational data, although probably both reliable and valid, is not often feasible. Metering and file server records, although able to record time on the computer or software, are not available in most schools. The next level of data is self report data from students, which can be verified by teachers and parents. If we are examining the relationships between the use of some technology and student achievement, we do sampled surveys of use. We ask students, teachers, and parents about the previous day or week's activity. We use e-mail, web-site, telephone, face-to-face, and paper and pencil surveys to document student use.

Not surprisingly, filling out surveys is not a priority for many educators, whether they are sent by e-mail, snail mail, or over telephone lines, but we have always had excellent cooperation that easily exceeds the minimum standards for sample size and response. Student reports of their own behavior tend to be more accurate than parent or teacher responses, although children younger than fifth grade often have difficulty estimating time. Teachers are usually able to tell us how much in-class time that students spend on the computer, although it often depends on which day, which class, and which student. Teacher reports are aggregate reports, while student reports are specific to the individual student.

Because student use (at least in schools) is related to teacher use of and comfort with technology, we include in the description of the technology the amount of teacher professional development and integration into the curriculum. We ask teachers and administrators about use. We examine teacher professional development participation, both in school and out of school, formal and informal. Self reports of technology literacy, faculty meeting agendas, lesson plans, and observations all help to describe what the teacher knows about technology, how comfortable the teacher is with technology, and how and how often the teacher is able to integrate technology into the curriculum.

While this paper is about measurement issues and student achievement, there are worthy reasons to use technology beyond bottom-line achievement. We have examined technology use and self efficacy, attitude about school, attendance, and discipline.

However, to understand the relationship between technology use and student achievement, we are most comfortable with examining gains in individual student achievement that would be reasonably expected because of the technology. Thus, we don't expect that time using music composition software would accelerate student learning in biology. The measures used must relate to the expectations of the technology.

We use the same data that schools use to determine achievement, even when we might not think it is the best form of measurement. We use these data because that is how the districts and their superordinate jurisdictions measure achievement. While we can argue that most achievement tests do not accurately or fully explain what students learn, the reality is that achievement data is often the best we have.

Thus, we often rely upon gain scores from September to May on norm referenced tests such as the Stanford 9, the Iowa Tests of Basic Skills, or CTB-Terra Nova. Since most districts don't test twice a year, this usually requires some negotiation. However, the result is that we have individual student gain scores to relate to the individual student use measures.

Additionally, we use grade, teacher developed tests, state achievement tests, district achievement tests, and authentic displays of student work. The more types of data, the better the understanding.

If you look across the measurement literature (and Jay Sivin-Kachalan and Ellen Bialo have, see sources below), you will find different methods to study different combinations of different interventions. It is hard to make those disparate studies add up in a way that compels belief. In part, that is the nature of decentralized science in a democracy. Still, we would like to see a short list of preferred evaluation methods or models, each for example, with two alternative methods for different intervention niches like early childhood literacy or gender studies of literacy applications delivered on the Internet. We would like to see those models developed and recommended (or even encouraged) by funding agencies. That way, at least some of what we do would add up in a more direct fashion than has so far been the case.

Measuring technology outcomes is undeniably messy and imperfect. It is also important for the practice-improving signals that can be developed even from this sometimes frustrating enterprise. It may also be helpful to recognize that just as instructional technology continues to evolve and to improve, so does our ability to document inputs and measure effects.

About the Author: Charol Shakeshaft, Ph.D., is professor in the Department of Administration and Policy Studies, School of Education, Hofstra University, Hempstead NY 11590. An internationally recognized expert in gender studies and women's leadership in school administration, Professor Shakeshaft's new book is In Loco Parentis: Sexual Abuse in the Schools (San Francisco, Josey-Bass, in press). Dr. Shakeshaft is a Managing Director of Interactive, Inc., 326 New York Avenue, Huntington, NY 11743-3360: p 516 5470464: f 516 547 0465.