Evaluation Primer
Ensuring Evaluations Yield Valid and Reliable Findings
Whatever methods of data collection are used, the data collected must meet two conditions to be considered accurate: they must be valid and reliable. Respondents may be tempted to answer questions in ways that they think are expected of them or that do not place them in jeopardy. Evaluators will want to take steps to ensure that they have obtained the most accurate (i.e., valid and reliable ) responses they can get. A data collection item (such as a question on a questionnaire) is valid to the degree that it actually measures what it claims to measure.
A measure is reliable to the degree that its meaning is stable. A reliable item or set of items on a questionnaire would lead to similar responses by the same respondent (in an unchanging situation) each time the item was asked.
Reliability is an assurance that the instrument or measure is consistent. One of the simplest tests of reliability is whether the same questionnaire, administered to the same person twice in a short period of time, yields similar responses. If this does not happen, the questionnaire probably contains unreliable items. Consistent responses suggest reliability, and consistent responses to different items that seek to measure the same knowledge or behavior provide greater confidence that the questionnaire is reliable.
Can a measure be reliable but invalid? Yes, because reliable but invalid responses can be obtained consistently if a data collection instrument or procedure is poor. An evaluator administers an instrument that asks about current drug use to the same set of students twice, a month apart. The students give the same answers both times, so the questionnaire appears to be reliable. But the evaluator asks the students to sign their names to the questionnaires, so the students assume that if they reveal any drug use they will be disciplined. Hence, reliability is achieved because drug use rates at both administrations are consistently low. So the measure is very reliable, but still invalid.
A measure must be both valid and reliable to be useful. Establishing the validity and reliability of data collection methods and items is a technical area in which school staff may wish to obtain outside assistance. The use of validated existing questionnaires is a good way to minimize validity and reliability problems, provided they are used in a manner similar to the way in which they were used when their reliability and validity were established. 
-###-
Next: Interpreting and Reporting Evaluation Findings
Back: Observing Behavioral Outcomes and Attributing Changes to the Program
Return to Education Evaluation Page
Last modified -- September 21, 1998, (lyp)