A r c h i v e d  I n f o r m a t i o n

What's the Best Way to Collect My Information?

There are many different methods of collecting information. Any college textbook on research design and analysis will provide an overview of different methods of data collection (e.g., Rossi & Freeman, 1993). Here, we present the "highlights." We will begin with a few tips, or words of advice, and in the process define some basic terminology.

Baseline Data

Collecting baseline data may be important. As noted in the discussion on benchmarks on page 20, baseline data provide information at the start of a program. Baseline data are used to set benchmarks and to determine the amount of change at different points in the course of the program. For example, if you wanted to find out if teacher use of technology increases after professional development, it would be best to collect information on use both before and after the activity. In this case, information on the amount of use before professional development would be considered "baseline data." Baseline data are most important when the question you are trying to answer is one of change (e.g., has student computer use increased? Does training help improve teacher attitudes towards technology?).

Pilot Testing

Pilot testing may prevent costly mistakes. Another term that you may have heard is "pilot testing." Pilot testing is a trial run of procedures and instruments that you plan to use. For example, pilot testing of a mail survey to teachers could be done in a couple of different ways. You could mail the survey to a handful of teachers and then call them to discuss the questions. An easier, but less rigorous, method might be to distribute the survey after a professional development activity and ask the teachers there what they think.

The main purpose of pilot testing is to catch potential problems before they become costly mistakes. It is typically used if an instrument or method of data collection is being used for the first time or for the first time with a particular group (e.g., a survey used before with a different age group). Pilot testing provides information on how long data collection can be expected to take and a preview of how difficult items will be to complete. The latter is important as, with proper advanced notice, you can modify questions and possibly even the way you collect information (e.g., reading questions to people rather than having people read questions themselves).

Using Multiple Sources

It's best to use multiple sources. When looking for evidence, many sources are better than one. For example, if you are gathering information on student computer use and you obtain reports from students, observe students using the computer, and get computer logs detailing an increase in use, there is a good chance that the information is reliable. On the other hand, if only computer logs are collected, or only student reports or observations are used, you will need to make conclusions more cautiously. A general rule is, the more sources of evidence that you have, the more faith you can have in a result or finding.

Sampling

Sometimes you won't have the time or money to get information from everyone. If collecting information from everyone (e.g., all schools) is not feasible, you will need to sample or get information from a subset of persons or schools. Proper sampling techniques allow you to take information from a small group and generalize to a larger group (e.g., some students in a school to all students in the school).

There are many ways to sample, and the process can readily become difficult. By far, the easiest way to is to choose people randomly (e.g., pull names out of a hat or take every third person on a list). However, sometimes random sampling may not be possible (e.g., you need to rely on volunteers, groups may already have been assigned), or there may be a reason that you would not want to sample randomly. For example:

• You may be interested in results from a particular group. For example, you may be particularly interested in learning whether your technology program improves student achievement for students who transferred to your high school from a particular middle school, but there are not that many students who participated in your technology program from that school. If you selected a random sample, you may not select enough of these students from which to draw conclusions. To investigate the effects of your technology program on students from that particular middle school, you may need to sample these students separately or only look at these students.

• You may want to make certain that your sample reflects a range of students or teachers. For example, you may want to find out if a technology program works with students at all ability levels. A true random sample from the district as a whole may not provide you with sufficient numbers of low-performing students or students enrolled in honors classes. You thus may need to "oversample" low-performing students or students enrolled in honors classes.

Techniques of sampling that would not be considered random are used fairly regularly, but must be treated with caution. By not selecting a sample randomly, you will be able to make conclusions about the types of students or teachers that were sampled, but you may not be able to make conclusions about the group as a whole (e.g., all students in the district).

Sampling can be a valuable tool and there are many resources available to help you (e.g., books, statisticians or social scientists at your local university). A good piece of advice on sampling is to consult your library or call a local university if you have questions.

Comparison Groups

You may need to compare one group to another. Often, it is helpful when evaluating a program to compare one group to another. Typically, comparisons are made between students or teachers who have been exposed to a particular initiative and students or teachers who have not been exposed. For example, you might compare:

• Classroom lessons of teachers who have received training to lessons of teachers who have not received training;

• The computer literacy of teachers before and after professional development;

• The computer literacy of students in classrooms with four computers to students in classrooms with only one computer; of

Other types of comparisons can also inform evaluations. You might want to compare:

• The amount and type of computer use of girls to that of boys; or

• The achievement of students in the school to national norms.

As you can see, there are many ways to think about comparison groups. Some require looking at possible differences in the same group over time (e.g., comparing teachers before and after a professional development activity). Some require examining differences between groups differing in program implementation (e.g., number of computers in a classroom). Some require looking at how different groups (e.g., gender, ethnicity, at-risk, grade level) respond to the same program. All methods are used regularly, and the "best" method will vary from program to program. In general, there are three pieces of advice we would give:

• Compare groups that are as similar as possible in areas other than the difference you are investigating (e.g., someone interested in gender differences in computer use would make sure that students differed only on gender).

• Make conclusions cautiously about the role technology plays in differences (e.g., What else could explain differences?).

• Compare your results to those of other studies (e.g., Did other people find the same thing? Why or why not?).

To figure out which method of data collection to use, you will need to ask yourself some additional questions. Specifically, you will need to ask whom you will get your information from, whether you will need to compare your program with others, and what possible constraints there are.

Source of Information

Whom should I be getting information from? Your source of information will vary depending on the issue you wish to address. For example, if you are interested in professional development, it would be best to get information from teachers and/or administrators or coordinators. If you are interested in student use, then it would be wise to go to the source -- students. For technology programs, possible sources of data include administrators, teachers, students, parents, program coordinators, businesspeople (e.g., software manufacturers), and financial experts (e.g., information from budgets). There are three main things to keep in mind when deciding what source of information to use. First, think about the person(s) or materials that will give you the most accurate information. Second, assess the availability of this person or materials and use an alternate source if necessary, staying as close to your desired source as possible. Third, tailor your instruments to the level of knowledge of the person you are getting your information from. For instance, you will need to word a questionnaire very differently for adults than for children

Comparing to Other Results

Do I need to be able to compare my results to others? If you need to be able to compare across programs or compare the results of your evaluation with similar evaluations, you should look at how information was collected previously and what instruments were used. If the same method and instruments are available and will work, use them! Otherwise, you will need to stick as closely as possible to what was done previously and make comparisons cautiously.

Possible Constraints

What are some possible constraints? Many of the reasons that researchers choose one data collection method over another have to do with different constraints that were placed on them. Some typical constraints include time, money, availability of personnel, and legal or bureaucratic roadblocks (e.g., not able to use a copyrighted test, needing permission from parents and/or school officials but there is not time). Two common con-straints are the need for district approval (e.g., the district must approve any research done in the schools) and parental permission for student surveys (e.g., do you need signed permission slips?). Check out these policies early!

Designing for Analysis

What kind of analysis will the information require? When deciding what type of information to collect, another point to consider is the type of analysis that will be required to form conclusions about the information. For informa-tion such as numerical responses on surveys, you may want to think about the level of statistical knowledge required. If your background is not exten-sive, it may be wise to gather information that can be expressed in terms of counts or percentages (e.g., 46 percent of teachers said they had training). For qualitative types of information, time and complexity of analysis may both be a concern. Coming up with a method of looking at and coding narrative responses can be both a time-consuming and complicated task. On the one hand, qualitative information is useful as it provides a lot of detail and rich context. However, analyzing it is often not as straightforward as it seems.

Now that you have your potential measures and an overview of some basic data collection issues, it's time to look at your options for what method to use to collect information. Some common methods include surveys, interviews, focus groups, observations, and collection of already available materials (e.g., school records). Following is an overview of these different methods of data collection.

Surveys

Surveys are often used to collect information from large groups. Surveys can vary along a number of dimensions, such as how questions are asked and how the survey instrument is delivered. For instance, questions can be open-ended (no choices provided) or close-ended (choices provided). Surveys can also be mailed or distributed in teachers' boxes at school, or administered by telephone, or in a large group setting.

The question of what to ask on a survey and how to ask it is not an easy one. Like sampling, entire books and college courses have been devoted to the development and delivery of surveys. It is easy to get lost in the complexity, but there is a basic tenet that may help you out: Think about what you need to know and ask it as directly as you can. For example, if you want to know how often teachers use the Internet in their teaching, you may want to ask, "How often do you use the Internet in your teaching?" either in an open-ended format or by providing possible response options (daily, once a week, etc.).

Once you have survey questions that you feel will meet your needs, you will need to figure out the best method to administer them. The three most common ways to deliver a survey are by mail or distribution into "mailboxes" at a school, in a group setting, or by telephone. Mail surveys are one of the least costly methods of delivering surveys. However, you may run into problems getting people to return surveys, obtaining correct addresses and phone numbers. Administering a survey to a group, possibly at a teachers' meeting, is probably the quickest way to collect survey information. In addition, administering a survey to a group will allow more flexibility than mail surveys to answer respondents' questions (e.g., "Do I consider graphing calculators as a use of technology?"). Telephone surveys may be a good way to go if the group of people you wish to collect information from is not readily available (e.g., parents) or if the information requires extended explanation on either your part or the part of the respondent.

Overall, there are many advantages to using surveys. Most are easy to analyze, they can be quickly administered, and they can be easily adapted to fit specific situations. Using a previously administered survey will save you time and energy, may give you something to compare your results to (if results from previous administrations of the survey are available), and will give you confidence that some of the potential problems have already been addressed. One word of caution in using surveys that others have developed is warranted, though. It's tempting to look at an already developed survey and think that it is wonderful, but you need to look at what information you need and adapt surveys as you see fit.

Interviews

Just like surveys, interviews can vary greatly in form. Interview protocols can range from extremely unstructured (e.g., a list of general topics to be covered) to something that might resemble reading aloud a survey (i.e., lists of questions that must be asked). Interviews are somewhat like surveys - an interviewer asks questions of people - but since interviews occur on an individual basis, they are often more time-consuming and costly. In addition, evaluators need to be concerned about consistency across interviewers. However, interviews also have advantages. They are good for collecting detailed information or information from a limited number of people. In addition, interviews give people an opportunity to explain responses or give responses that you may not have predicted. For example, interviews may be useful in examining gender differences in signing up for computer classes

Focus Groups

Focus groups can be an easier way to collect information that otherwise would be collected through interviews. Focus groups can be thought of as group interviews. Just like interviews, they can be structured or unstructured. One of the most popular ways to use focus groups is in conjunction with surveys - either the focus group helps the researcher interpret the results of a survey or a focus group helps define and/or refine survey questions. An advantage of focus groups is that, since data are being collected from multiple people at the same time, they are more cost-effective than interviews. A disadvantage is that, in the group setting, you might not get to hear the opinions of all participants (e.g., some may not feel comfortable speaking in this forum). You also need to be concerned about how well those participating in the group represent the group you are interested in (e.g., are there only "honors" students in the group?) Another issue regarding focus groups, that can be both an advantage and a disadvantage, is that responses from one participant can sometimes trigger responses from others.

Observations

Another common method of collecting information is to observe. Sometimes the best way to collect information about people's behavior is to watch them. Observation allows the researcher to collect information without being a burden on the person providing the information. Typically evaluators develop guides that structure the observation process. As an example, observation can be used to collect data on teacher lessons and student use of computers. One drawback to observation, however, is that it measures only what you can see. Other types of data (e.g., opinions, reasons behind behavior) cannot be collected in this fashion. Another drawback is that it can be time-consuming as multiple observations are often required.

School Records

Don't forget school records. Schools routinely gather lots of information (e.g., attendance, grades, test scores) that could be useful to your evaluation. Getting this information directly from school records can save time. In addition, it can also be more accurate than asking students or teachers for the information and relying on their recollections. However, it is important to ensure the confidentiality of school records. In general, you will need to make sure that you don't report individual student names, that you have accessed records through the appropriate channels, and that you have received necessary permission. Even with these caveats in mind, though, records may be easier to get than you may think and, if they provide needed information, they are worth considering.

Collecting Materials

Finally, you could also look at other existing documentation. Looking at already existing evidence (e.g., teacher lesson plans) places little burden on people. Drawbacks, however, are that information may not be exactly what you need, in the form you need, or may be incomplete or need explanation.

In the Meantime...

Different pieces of evidence take different amounts of time to collect. Often stakeholders will want to see results quickly. For some measures (e.g., test scores), however, it could take years. What do you do in the meantime? Here are some ideas for evidence that doesn't take long to see results:

• Information on student use. For example, gather narratives on time spent at computers, how students are using technology (e.g., class projects), anecdotes of student excitement (e.g., "they are lined up during lunch period") or student problems (e.g., getting into inappropriate material).

• Student work. Soon after installing equipment, students will begin to use it to produce work. Collecting this work will provide examples of how the equipment can be used and the possibilities for improving student work.

• Teacher lesson plans or classroom observations. Soon after teachers have been trained, some will experiment with technology in their lessons. These beginning lessons can be used as either examples for further training or examples to give to stakeholders of technology use.

• Comments from students and parents. Like it or not, teachers and others will hear comments (either positive or negative) from students or parents. Ask around to find out what these comments are. Comments might give you ideas on areas you may want to investigate later or they may provide anecdotal evidence of student use and attitudes.

• Start collecting outcome data. Although it may be too soon to answer impact questions, you can start collecting information relating to outcomes early in the program?s history. Gathering information from the beginning of the program will allow you to track students' progress over time.

Collecting a variety of information throughout a program forms the core of an evaluation approach known as "portfolio assessment." For more information on portfolio assessment, consult the RAND website (http://www.rand.org).

Putting It Together

At this point, you have information on many possibilities. The next step is to put the information together to decide which of the many possibilities to use. On the following page is a summary of the different methods of collecting data that were discussed here, listing some advantages and disadvantages for each. It is not meant to be an exhaustive list, but rather a guide for things you should watch for or be aware of.

In addition, you will find, in Appendix C, examples of surveys. If they address the information you are looking for, great! Acknowledge where they came from and feel free to use them. For other materials, you need to make certain that there are no copyright restrictions.

After some final decisions, you should be ready to go. You have your goals and research questions, you know what kind of information you need, and you know how you're going to collect it. Remember to make sure you have all the permissions necessary (e.g., district, teachers, parents, children), and start interviewing, surveying, observing, or collecting materials.

Different Methods of Collecting Information

How Did Rivers Collect Information? Kathy listed possible ways to collect information pertinent to her measures and came up with the following: Review of teacher lesson plans before and after they took part in professional development activities Interviews - parents, teachers, students, administrators Collect reading test scores Collect information from district records on attendance and dropout rates Assessment of computer literacy - Observation? Self-Assessment? Test? Since her principal said that she had only had six months to come up with a report, she decided that she could not do everything she wanted - doing both interviews and surveys would take too much time. So, she had to set priorities. Well, since surveys are quicker, she decided to survey a random sample of 100 teachers, 25 administrators, 300 students, and the parents of the 300 students. While searching the Internet, Kathy found examples of surveys (in fact, many more than she needed!) that had previously been used both in high schools and in elementary schools. She found a student survey that looked like it would work well - it had some questions on attitudes toward computers, use of computers, and a self-assessment of computer literacy. Her district had assessed professional development programs before, so a survey was available that had been previously used. But in order to address her specific professional development concerns (e.g., will professional development help teachers integrate technology into their teaching?), Kathy needed information on both participation and its results. She therefore needed to review teacher lesson plans and observe teachers in the classroom as well as determine how much and what types of professional development teachers participated in. Kathy felt she came up with the right set of instruments and methods to use, and since most of the survey questions had been used before, she felt pilot testing was not necessary. However, she still had a few questions: What kind of parental and or district permission would be needed? Am I going to run into statistical analyses I can't handle? Will my results be comparable to other studies? (people want this) Kathy, luckily, has a friend at a local university. She faxed drafts of her surveys to him and asked his opinion on comparability, statistical analysis, and permissions needed. He assured her that, since the issues addressed were not sensitive (e.g., sex, drugs) and since she worked for the district, she did not have to go through a formal review process to get district approval. In addition, she could handle parent permission by sending the parent surveys out first and giving them a vehicle for declining to let their children participate (e.g., call her if they have a problem with it). Finally, Kathy's friend at the district said that she did the right thing by picking previously used surveys, and that since the district had used them before, there would be no problem with copyright. He suggested a few changes to the questions she had added, but basically she was "good to go" and would be able to compare her results to other programs.

What did you think of as possible measures?

____________________________________________________________________________

____________________________________________________________________________

What kind of information has been collected before and/or is already available?

____________________________________________________________________________

What kind of constraints do you have (e.g., time, money, needed permission from district)?

____________________________________________________________________________

Assess Quality of Measures. Will the measures give you the information needed? Are they doable given time and financial constraints?

____________________________________________________________________________

____________________________________________________________________________

Do you need to sample? If so, how will you do it?

____________________________________________________________________________

Do you need to use comparison groups? If so, how will you do it?

____________________________________________________________________________

Have the instruments been used before? If not, do you need to do pilot testing?

____________________________________________________________________________

What kind of statistical knowledge will be required to analyze the data?

____________________________________________________________________________

Any other problems you need to resolve?

____________________________________________________________________________

____________________________________________________________________________

-###-

[What Information Do I Need to Collect?]

[What Are My Conclusions?]