Evaluating The Use of Technology: What Are My Conclusions?

A r c h i v e d I n f o r m a t i o n

An Educator's Guide to Evaluating the Use of Technology in Schools and Classrooms - December 1998

What Are My Conclusions?

Review Research Questions

What's next? I've got some information, now what do I do with it? The first thing you should probably do is review your goals and research questions. Figure out what you wanted to know in the first place. It's easy to get lost in the amount of data you have collected and the details of individual responses. Reviewing what you wanted to know may help focus you on the task at hand -- figuring it out!

Organize Your Information

Next, organize your information by goal. If you filled out a chart of goals, indicators, benchmarks, and measures, you have a good start. Make a list of what information you now have that corresponds to each measure, and thus, addresses each of your goals. You will probably find that one instrument, such as a survey, addresses multiple goals and helps to answer multiple questions. This is expected. Do the best you can to separate out which measure corresponds to which goal and if there is overlap (e.g., one measure still addresses multiple goals), put the measure under both. Analyze

Like it or not, the next step is data analysis. Sometimes, when people think of data analysis, they assume that complicated statistics must be involved and are worried that they do not have the background to attempt such a task. In reality, there are two things to keep in mind:

Not all data analysis involves complex statistics.
Even if statistics are involved, they should be at the level that the intended audience will easily understand.

The most common way statistics are used in evaluation is for descriptive purposes. For example, if you want to describe the number of hours students spent using a computer at school, you would want to calculate either an average or the percentage of students who use computers more than a certain specified base. In addition, you may want to also compare the results of different groups of students (e.g., at-risk students, gifted and talented students) to see if technology has different impacts. In this case, you may want to use the same statistics (e.g., means, percentages), but report separate results for each group. Whether you use an average or percentage and how you want to group your information depends on the type of information you have collected.

Coming to a Conclusion

In the end, the important part of analyzing information is not the statistics, but rather the conclusions that you draw. This is what this section is meant to guide you through - reaching conclusions.

The process of coming to a conclusion can vary widely from goal to goal. One of the most difficult tasks is defining vague goals, such as what is "sufficient training"? Luckily, you have your indicators and benchmarks to help you with this. For example, if a certain benchmark was reached, then what you were measuring could be considered "sufficient" or "adequate."

To help you understand the process and give you some ideas, several examples of how goals were addressed are provided on the next page. The goals vary in topic and type of information used.

Goal Measure(s) Coming to a Conclusion

Professional Development
Number and percentage of teachers requesting and receiving training by type of training
Carlos in District A wanted to know about teacher access to training. The district wanted to make sure that all teachers had training within two years, so he developed a benchmark - 50 percent of teachers should have had training after the first year - to determine the extent of training. Carlos then obtained a list of all training activities offered at the state, district, and school levels from the district professional development coordinator and lists of persons who attended the activities. He found out that there was a three-day summer training given by the district (50 slots open) and three one-day workshops given by one of the schools in the district (30 spaces for each). The three-day summer training ended up being full (50 people attended), but only an average of 20 people attended the workshops. Carlos was then able to describe what was offered and the number of attendees and compare what he found with the indicators and benchmarks that he had established. With only 25 percent of teachers having had training, he concluded that, unfortunately, the district was not meeting its benchmark. He recommended that, since the summer workshop was well attended, the district should offer additional summer workshops in the coming year.

Curriculum Integration
By subject area, the percentage and frequency of lessons included in the curriculum that incorporate technology
Cecily in District B sat down and went through the district curriculum. Defining "incorporating technology" as either the teacher or students using technology during the proposed lesson or activity, and having set a benchmark of 25 percent of lessons as optimal, she counted all the lessons in three main subject areas: English, science, and social studies. She found that 30 percent of the science and social studies lessons contained technology, but only 5 percent of the English lessons contained technology. Thus, Cecily concluded that science and social studies are on target, but English is not.

Reading and Writing
Student letter or numerical grades on reports and presentations
Steve in School C asked all students in the fourth and fifth grades to write three reports at the beginning of the school year on different trips they would like to take within the United States. At the end of the school year, he asked these same students to write three reports on trips they would like to take to other countries. He not only examined these reports for content and quality of citations, but also incorporated information into his conclusions on the extent to which the teachers of these students used technology in their classrooms. (He had observed each of the fourth and fifth grade classrooms on at least six random occasions throughout the year.) Students whose teachers incorporated technology into their lessons did improve more than students whose teachers did not incorporate technology as much. Thus, Steve concluded that having computers available to students in classrooms was not sufficient; teachers had to use technology on a regular basis in their lessons in order for student performance to improve.

Goal	Measure(s)	Coming to a Conclusion
Professional Development	Number and percentage of teachers requesting and receiving training by type of training	Carlos in District A wanted to know about teacher access to training. The district wanted to make sure that all teachers had training within two years, so he developed a benchmark - 50 percent of teachers should have had training after the first year - to determine the extent of training. Carlos then obtained a list of all training activities offered at the state, district, and school levels from the district professional development coordinator and lists of persons who attended the activities. He found out that there was a three-day summer training given by the district (50 slots open) and three one-day workshops given by one of the schools in the district (30 spaces for each). The three-day summer training ended up being full (50 people attended), but only an average of 20 people attended the workshops. Carlos was then able to describe what was offered and the number of attendees and compare what he found with the indicators and benchmarks that he had established. With only 25 percent of teachers having had training, he concluded that, unfortunately, the district was not meeting its benchmark. He recommended that, since the summer workshop was well attended, the district should offer additional summer workshops in the coming year.
Curriculum Integration	By subject area, the percentage and frequency of lessons included in the curriculum that incorporate technology	Cecily in District B sat down and went through the district curriculum. Defining "incorporating technology" as either the teacher or students using technology during the proposed lesson or activity, and having set a benchmark of 25 percent of lessons as optimal, she counted all the lessons in three main subject areas: English, science, and social studies. She found that 30 percent of the science and social studies lessons contained technology, but only 5 percent of the English lessons contained technology. Thus, Cecily concluded that science and social studies are on target, but English is not.
Reading and Writing	Student letter or numerical grades on reports and presentations	Steve in School C asked all students in the fourth and fifth grades to write three reports at the beginning of the school year on different trips they would like to take within the United States. At the end of the school year, he asked these same students to write three reports on trips they would like to take to other countries. He not only examined these reports for content and quality of citations, but also incorporated information into his conclusions on the extent to which the teachers of these students used technology in their classrooms. (He had observed each of the fourth and fifth grade classrooms on at least six random occasions throughout the year.) Students whose teachers incorporated technology into their lessons did improve more than students whose teachers did not incorporate technology as much. Thus, Steve concluded that having computers available to students in classrooms was not sufficient; teachers had to use technology on a regular basis in their lessons in order for student performance to improve.

What Were Rivers' Conclusions?

Here is Kathy's attempt to come to conclusions on her research questions:

Goal Measure(s) Conclusion Process

Improved reading performance (Will the new technology and training improve reading performance?)
ITBS reading scores
State test scores in reading
Grades in reading
Kathy determined that the program has not been in operation long enough to reach a conclusion on this question yet. She acquired test scores and grades from the district and will track scores annually over the next 5 years.

Improved dropout and attendance rates (Will the new technology and training help lower the dropout rate and increase attendance?)
Dropout rate
Attendance rate
Again, Kathy felt that more time was needed to answer this question. She collected the information from district records and will continue to examine these records on an annual basis.

Cost-effectiveness (Is this program as or more cost-effective than other programs that may show similar results?)
Program budget
Budget of similar program
Program evaluations
Kathy got budgets of 5 programs she thought were similar in terms of expected results. She ranked them all by cost and expected results. The cost of the Rivers technology program ranked in the middle. It is too soon to draw conclusions about results, however.

Effective professional development activities (Will the professional development help teachers integrate technology into their teaching?)
Review of teacher lesson plans
Observation of classrooms
Teacher Survey
Kathy both reviewed teachers' lesson plans and observed classes. Kathy determined that teachers were incorporating technology into their teaching more often after they had participated in professional development. She also surveyed teachers and determined that teachers believed technology was improving their teaching, particularly in social studies.

Increased computer literacy (Will students and teachers become more computer literate as a result of the program?)
Assessment of teacher computer literacy
Assessment of student computer literacy
On both teacher and student surveys, there was a self-assessment of computer literacy. However, just as with test scores and attendance rates, and because baseline data are only now being collected, Kathy feels more time is needed to reach a conclusion as to whether there is change over time.

"What Are My Conclusions?" Worksheet

Try to fill out the same form that Kathy did:

Goals Measure(s) Conclusion Process

-###-

[What's the Best Way to Collect My Information?]

[How Do I Communicate My Results?]