A r c h i v e d  I n f o r m a t i o n

What Information Do I Need to Collect?

Now that you've identified goals and formulated research questions, the next task is to figure out what information you will need to address them. In evaluator's terms, you will need to define more clearly your indicators and measures. This section is designed to guide you through the process of figuring out what kind of information you need to collect.

Evidence

To meet requirements and to address questions, you will need evidence. Think about the questions that you and others would like to have answered about your technology program. First, what evidence would you need to have to answer your questions? What evidence will others be looking for? Examples of evidence might include:

• Improvement (or lack of improvement) in test scores
• Information on quality and quantity of professional development
• Data on numbers of computers and Internet connections
• Information on student use (e.g., types, amount, what technology is used for and in what subjects)

Types of Information

The type of information needed can vary from study to study. Although there are many variations, the type of information you will collect may be qualitative or quantitative and either may be appropriate. An example of strictly qualitative information would be a narrative description of a program or descriptions of how students are using computers. An example of strictly quantitative information would be counts of computers in school buildings, number of professional development hours, or number of hours students spend using computers. Qualitative and quantitative data can reinforce one another. For example, knowing both what students do when using computers, a qualitative measure, and how much time students spend using computers, a quantitative measure, provides a more complete understanding of student computer usage than either type of data individually.

Going From Questions to Evidence

Ok, I have my goals and questions and I know there are different kinds of information out there -- How do I know what information I should get? What you want to do is go from the goals and questions you have already thought about to "indicators," to "benchmarks," and then to "measures." Some of these terms you may have heard before (e.g., goals), others you may not have. Since they are used in a variety of ways by different researchers, it's easy to get lost in the language. For simplicity, we'll take each of the steps one at a time, defining how we use the terms as we go, beginning on the next page with an explanation of goals.

Goals

First, review your goals. In the last section, you devised a list of goals and research questions. Although goals can vary greatly, they can be thought of as generally falling into two broad categories: "intermediate" or "outcome."

Intermediate goals are the products or processes necessary to achieve outcomes. In one sense, intermediate goals can be described as a "means to an end." The national technology goals, as follows, are good examples of intermediate goals.

• All teachers in the nation will have the training and support they need to help students learn using computers and the information superhighway;

• All teachers and students will have modern multimedia computers in their classrooms;

• Every classroom will be connected to the information superhighway; and

• Effective and engaging software and online resources will be an integral part of every school's curriculum.

Intermediate goals such as these -- and the indicators, benchmarks, and measures developed from them -- are important for determining whether a program is up and running as it was intended and are vital for providing possible explanations for ultimate outcomes.

Outcome goals reflect the final product, or "outcome," desired. In many cases, in educational research, outcome goals are related broadly to improved teaching and learning. For example, one may want to know if access to word processing software will improve student writing. In this case, access to word processing software would be an intermediate goal and improvement in student writing would be an outcome goal.

The relationship between intermediate goals and outcome goals is not always straightforward. In some cases, intermediate results (e.g., more computers in the classroom, added professional development) work together to enable desired outcomes to occur. In other cases, one intermediate result will influence an outcome by way of another intermediate result (e.g., increased professional development leads to increased integration of technology into the curriculum, which then leads to higher student performance).

The ultimate goal of many technology programs is to increase various aspects of student achievement and, in some cases, teacher performance. For instance, a district may want to use technology to improve the quality of student writing. However, there are steps that must occur before determining whether the use of technology has improved student writing. There must be (a) enough technology available to make a difference, (b) teachers adequately trained to use that technology, and (c) integration of technology into the curriculum and classroom activities. In this case, the outcome goal is improved student writing and the intermediate goals, those "means to an end," are the availability of technology, professional development, and curriculum integration.

In this section, we will use three intermediate goals and two outcome goals as examples:

Intermediate
Professional Development	All teachers will have the training they need to help students learn using computers and the Internet.
Availability of Technology	All teachers and students will have modern multimedia computers in their classrooms.
Curriculum Integration	Teachers will use and view technology (e.g., effective and engaging software and online resources) as an integral part of the curriculum and daily classroom activities.
Outcome
Reading and Writing	Student reading and writing will significantly improve.
Computer Literacy	Both teachers and students will display increased computer literacy.

Although there are many different ways that the above goals could be related, the following diagram outlines some of the possibilities:

The relationship among the goals is not straightforward. For example, all of the intermediate goals can work together directly to influence the computer literacy of students, and both professional development and the availability of technology can influence computer literacy by means of their effect on curriculum integration. Every goal (intermediate and outcome) is important, and it is necessary to assess each of them. We now turn to the task of how to assess goals and, specifically, the task of developing indicators, benchmarks, and measures based on goals.

Indicators

Next, you will need to develop indicators. For our purposes here, indicators are statements that reflect specific goals that can be used to gauge progress. Indicators help orient you toward a measure of performance outcomes and typically focus on only one aspect of a goal. Some examples related to the goals described on the previous page are:

Goals	Indicators
Intermediate
Professional Development	Increasing numbers of teachers will be trained to integrate technology into their teaching. Teachers will learn how to integrate technology into their teaching.
Availability of Technology	Teacher:computer and student:computer ratios will steadily decline. Students will actively use computers for projects and assignments.
Curriculum Integration	Technology will increasingly be incorporated into the curriculum in all subject areas. An increasing percentage of teacher lessons will incorporate technology.
Outcome
Reading and Writing	Students will produce reports and presentations that teachers judge to be of higher quality. Students will display increasingly higher performance on tests assessing reading ability.
Computer Literacy	Both teachers and students will display increased computer literacy.

Good indicators should:

• Include relevant measures. Indicators should be designed with thought toward the information that could be collected to inform them.

• Address appropriate groups. Indicators should consider any subpopulations that need to be examined separately (e.g., school level, racial/ethnic group).

• Be few. You do not need a lot of indicators to convey a lot of information.

• Be actionable. Indicators should guide improvement by helping to identify programs that are working and those that are not, should help program administrators make decisions to improve results, and should help policy makers judge progress toward goals.

• Be timely. Indicators should be as useful next year as they are today.

• Be reliable. Indicators should be easily measured by a variety of people and across time.

• Be comparable. Measures developed from the present indicators should be able to be compared to measures developed from similar indicators in the past.

Benchmarks

A benchmark is a specific target. Benchmarks provide a sense of what the program is striving to attain, either on an incremental basis (e.g., "every year test scores will rise 2 percentage points") or as a final target (e.g., "all students will score at or above the proficient level"). Two important factors need to be considered when setting benchmarks: the resources available to the program and a starting point, or baseline. If, for example, you are evaluating a relatively small initiative that is funding the purchase of new equipment, you would not expect the student to computer ratio to increase to the same extent as it might in a program that provides more extensive funding. Baselines also are important. They help establish the amount of change one can realistically expect. Let's assume that the purpose of the program you are evaluating is to improve student reading. To set an appropriate benchmark, you would need to determine student performance before the program began. This is considered a "baseline," a point from which to measure change and to determine appropriate benchmarks.

Indicators	Benchmarks
Intermediate
Increasing numbers of teachers will be trained to integrate technology into their teaching.	By the end of the school year, 50 percent of teachers will have had technology training.
Teachers will learn how to integrate technology into their teaching.	After training, teachers will be able to devise at least three examples of how technology could be integrated into their lessons.
Teacher:computer and student: computer ratios will steadily decline.	By next year, teacher:computer and student:computer ratios schoolwide (counting all computers) will be 4:1 and 6:1.
Students will actively use computers for projects and assignments.	All students will use computers at school at least 4 hours per week.
Technology will increasingly be incorporated into the curriculum in all subject areas.	In two years, English, science, and social studies curricula will have at least 25 percent of lessons incorporating technology.
An increasing percentage of teacher lessons will incorporate technology.	All trained teachers will have at least 25 percent of lessons incorporating technology.
Outcome
Students will produce reports and presentations that teachers judge to be of higher quality.	Within two years, student grades on reports and presentations will increase, on average, by 10 points or one letter grade.
Students will display increasingly higher performance on tests assessing reading ability.	Within two years, student scores on the reading portion of standardized tests (e.g., ITBS, CAT) will increase by 10 percent.
Both teachers and students will display increased computer literacy.	After one year, at least 75 percent of teachers and students will display at least an intermediate level of computer literacy.

Measures

A measure is an item reflecting the information or evidence needed to answer a research question, inform an indicator, or determine how close one is to achieving a benchmark. If applicable, a measure typically includes gauges such as percentages and ratios. Measures are similar to indicators, but are more concrete and more specific. Often, an indicator will have multiple measures. Measures attempt to answer the question of what evidence you would need to have to assess your indicators. The following are examples of measures used to assess the benchmarks:

Benchmarks	Measures
Intermediate
By the end of the school year, 50 percent of teachers will have had technology training.	Number and percent of teachers requesting and receiving training by type of training
After training, teachers will be able to devise at least three examples of how technology could be integrated into their lessons.	Lesson plans produced at professional development activities Teacher responses from professional development survey
By next year, teacher:computer and student:computer ratios schoolwide (counting all computers) will be 4:1 and 6:1.	Teacher:computer ratios Student:computer ratios
All students will use computers at school at least 4 hours per week.	Percent of students who use computers at least four hours per week at school
In two years, English, science, and social studies curricula will have at least 25 percent of lessons incorporating technology.	By subject area, the percentage and frequency of lessons included in the curriculum that incorporate technology
All trained teachers will have at least 25 percent of lessons incorporating technology.	From classroom observations, the percentage of teacher lessons that incorporate technology Percentage of lesson plans that incorporate technology
Outcome
Within two years, student grades on reports and presentations will increase, on average, by 10 points or one letter grade.	Student letter or numerical grades on reports and presentations
Within two years, student scores on the reading portion of standardized tests (e.g., ITBS, CAT) will increase by 10 percent.	Student standardized test scores
After one year, at least 75 percent of teachers and students will display at least an intermediate level of computer literacy.	Student and teacher self-report of computer literacy Student and teacher results from a skills test requiring performance of various tasks on a computer.

Below you will find a summary of the goals, indicators, benchmarks, and measures discussed. This should help you to follow your own path from general goals to specific ideas of what information you will need to collect to measure them. Take it one step at a time. First, write down your goals and questions, then indicators, benchmarks, and measures.

Goals	Indicators	Benchmarks	Measures
Intermediate
Professional Development	Increasing numbers of teachers will be trained to integrate technology into their teaching.	By the end of the school year, 50 percent of teachers will have had technology training.	Number and percent of teachers requesting and receiving training by type of training
	Teachers will learn how to integrate technology into their teaching.	After training, teachers will be able to devise at least three examples of how technology could be integrated into their lessons.	Lesson plans produced at professional development activities Teacher responses from professional development survey
Availability of Technology	Teacher:computer and student: computer ratios will steadily decline.	By next year, teacher: computer and student: computer ratios school-wide (counting all computers) will be 4:1 and 6:1.	Teacher:computer ratios Student:computer ratios
	Students will actively use computers for projects and assignments.	All students will use computers at school at least 4 hours per week.	Percent of students who use computers at least four hours per week at school
Curriculum Integration	Technology will increasingly be incorporated into the curriculum in all subject areas.	In two years, English, science, and social studies curricula will have at least 25 percent of lessons incorporating technology.	By subject area, the percentage and frequency of lessons included in the curriculum that incorporate technology
	An increasing percentage of teacher lessons will incorporate technology.	All trained teachers will have at least 25 percent of lessons incorporating technology.	From classroom observations, the percentage of teacher lessons that incorporate technology Percentage of lesson plans that incorporate technology
Outcome
Reading and Writing	Students will produce reports and presentations that teachers judge to be of higher quality.	Within two years, student grades on reports and presentations will increase, on average, by 10 points or one letter grade.	Student letter or numerical grades on reports and presentations
	Students will display increasingly higher performance on tests assessing reading ability.	Within two years, student scores on the reading portion of standardized tests (e.g., ITBS, CAT) will increase by 10 percent.	Student standardized test scores
Computer Literacy	Both teachers and students will display increased computer literacy.	After one year, at least 75 percent of teachers and students will display at least an intermediate level of computer literacy.	Student and teacher self-report of computer literacy Student and teacher results from a skills test requiring performance of various tasks on a computer.

"What Information Do I Need to Collect?" Worksheet

Using the goals and research questions you came up with in the last section as a starting point, work through the following grid. You may have to revise some of your goals and questions as a result.

[What Questions Should I Ask?]

[What's the Best Way to Collect My Information?]