A r c h i v e d  I n f o r m a t i o n

What Questions Should I Ask?

Program Goals

First, review program goals. Figuring out what questions to ask is probably one of the most difficult tasks in designing an evaluation. One way to begin to think of questions is to review program goals. For our purposes here, a goal is defined as an "overall objective of a program or initiative."

Goals tend to be broad and general. They reflect the overall intent of a program or a shared vision of what the program is supposed to accomplish. For example, a district's technology plan could state that all teachers are to receive adequate professional development so that they can integrate technology into their teaching and help improve student performance. Technology goals will differ slightly from district to district and perhaps school to school. They are generally reflective, at least in part, of local concerns, state plans or goals related to educational technology, other educational reform initiatives, and possibly the national technology goals. Reviewing the specific purposes that you and others might have for evaluation discussed on the "Why Am I Evaluating" worksheet and reviewing written plans and documents are two places you might start in trying to find information on the goals of your program.

Your Priorities

Next, figure out what you would like to know about the program. Trying to answer the question "What questions should I ask?" can begin with asking yourself what you want to know or what you would like to find out about your technology program. Do you want to know what teachers thought of the train-ing? Do you want to know how often students are using the new equipment? Do you want to know the extent of student technology skills? What goals do you want to see accomplished? What questions do you want answered?

Others' Priorities

Then figure out what others would like to know about the program as well. Fortunately or unfortunately, your opinion is not the only one that needs to be considered. What goals do other people have for the program? What questions do others want answered? Some stakeholders that may need to be considered include:

• Funding agency (e.g., Department of Education, foundation, state agency)
• State officials
• School boards
• Legislators
• School administrators (e.g., principal, superintendent)
• Teachers
• Parents

You can find out what others want to know in two ways. First, you could simply ask questions. Ask the person who gave you the responsibility for conducting the evaluation about persons who were involved in decision-making for the program. He or she should have at least a fairly good idea of who the major stakeholders are. In addition, you could get some information on individual goals and questions that people want answered from reading various planning documents. For instance, if the technology plan includes a list of major program goals, then it is a fair assumption that people will want to know if these major goals have been or are in the process of being achieved.

Other Requirements

Also, keep in mind any other external requirements. Grant money often comes with certain "strings" attached. In particular, many grants come with legal or contractual requirements. For example, in the case of the Technology Literacy Challenge Fund awards, funds are to be targeted to areas with the greatest poverty and the greatest need for technology. Some types of requirements that you might run into include:

• Requirements on what the money can or cannot be used to purchase
• Requirements on how much money can be used for equipment
• Requirements for matching funds
• Requirements on certain types of reports or data that must be provided to the district or state (e.g., progress reports)

Thinking About Outcomes

Finally, think about how you may want to ask questions about outcomes. One of the most pressing questions you are likely to encounter in evaluating a technology initiative is whether the initiative produces measurable outcomes. For example, you may be asked whether student achievement has improved or you may be asked about other potential long-term impacts such as changes in classroom teaching methods. You will need to begin to think about how you will answer such questions as:

• Does the availability of technology in the classroom increase the quality and/or quantity of student work?

• Has professional development related to technology changed the way that teachers are organizing instruction? Are teachers teaching any differently than they did before participating in professional development?

• Does having access to the Internet increase student knowledge of other cultures and traditions?

• Does the infusion of technology into the curriculum help increase student performance on standardized tests?

In the hustle and bustle of trying to get a technology initiative off the ground, it may be tempting to put these types of questions low on your priority list and not think about them. On the other hand, you may feel the pressure of wanting to know the answers to these types of questions quickly. Neither of these approaches is helpful. In general, questions that measure a program's impact take time to answer. You may be more interested in getting your program off the ground (e.g., determining whether the basic features of your program are in place, whether the computers you were supposed to have operating in classrooms are actually there) than thinking about longer-term impacts. But don't forget: it is often these questions about impact that are of primary interest to policymakers.

What Questions Did Rivers Ask? Kathy had already asked herself why different stakeholders wanted an evaluation and what they wanted from an evaluation. She had also come up with two major reasons why she wanted an evaluation: To find out if the program is beginning to produce desired results (e.g., student reading scores are beginning to improve, professional development is effective, the program is cost effective) To get information on the implementation process Well, these reasons seemed very broad, so Kathy, using the general ideas and the list of specific things that others wanted from the evaluation, came up with general goals that people were interested in and, within these goals, more specific questions she wanted answered. The following is the result of her work: Goal Question Improved reading performance Will the new technology and training help improve reading performance? Improved dropout and attendance rates Will the new technology and training help lower the dropout rate and increase attendance? Cost-effectiveness Will this program be as or more cost-effective than other programs that may show similar results? Effective professional development Will the professional development help teachers integrate technology into their teaching? Increased computer literacy Will students and teachers become more computer literate as a result of the program? After coming up with the questions above, Kathy reviewed them to determine if there was anything she was leaving out. She assessed her priorities and made sure they were included. She ran through her list of stakeholders -- principal, superintendent, professional development coordinator, teachers, parents, herself. All stakeholders were accounted for. Finally, Kathy looked at the grant to see if there were any requirements she needed to assess. The grant was through the Technology Literacy Challenge Fund, so it was fairly flexible and many of the requirements were at the state level. Basically, one of the biggest requirements was that the district was responsible for reporting back to the state yearly on what they had done. In Kathy's opinion, assessing the goals that she had developed would satisfy the state. She would call her state technology coordinator to be sure, but it looked fine and she thought she was ready to further define her evaluation plan.

"What Questions Should I Ask?" Worksheet

What do you want to know? (You may want to look back to the worksheet on page 5 for ideas on why you are evaluating and what you want from an evaluation.)

____________________________________________________________________________

____________________________________________________________________________

What will others want to know (e.g., principal, superintendent, board members, funding agency, teachers, parents, students)?

____________________________________________________________________________

____________________________________________________________________________

____________________________________________________________________________

____________________________________________________________________________

____________________________________________________________________________

What other requirements are there?

____________________________________________________________________________

____________________________________________________________________________

After this exercise, what are your "top 5" or "top 10" goals and questions?

Goals	Research Questions to Address Goals
______________________________	________________________________________
______________________________	________________________________________
______________________________	________________________________________
______________________________	________________________________________
______________________________	________________________________________
______________________________	________________________________________
______________________________	________________________________________

-###-

[Where Do I Start?]

[What Information Do I Need to Collect?]