![[Table of Contents]](/icons/up.gif){kind=icon}
Imagining Continuous EvaluationImagine that a group of parents, students, and school staff who are concerned about students' poor health and their health risks convenes community health and mental health providers. School staff report on the high absenteeism of students due to illness and the short time allotted in the school year to any explicit health curriculum. Students report results of a student council survey on smoking behavior and alcohol use. Health and mental health care providers report on the locations and operating hours of their clinics. The group states its shared belief that improved access to services that focus on health education and support healthy lifestyle choices will improve student health and reduce risk behaviors. Their school-linked strategy calls for the high school to create clinic space for health and mental health care providers that will:
Each step along the way should be evaluated. First, was the space provided? Was the clinic furnished, equipped, and staffed appropriately? Were services advertised? Did students come? If not, why not? The activities that were implemented (and the people responsible for implementing them) can be assessed on the basis of answers to these questions--in other words, the early evaluation results--and actions can be taken based on these results. In this way, the clinic's implementation is evaluated well before any changes in the lives of children and families can be expected. At later stages, an evaluation will examine different results. For example, after the clinic has been open before and after school and on weekends for a full year, student surveys could determine the clinic's accessibility, student use of the clinic, and the results of student use. Findings from these assessments would test whether the clinic's activities are having their intended direct effects on students--effects projected during the planning process. For example, imagine that by the end of its first year:
With these consumer survey results, changes in student health and related behaviors could logically follow and should be assessed--for example, using data from:
Adapted from Connell, personal correspondence, 1995. |
The continuous evaluation framework in this guide book is built around the themes, principles, and stages of evaluation highlighted in this chapter.
The following reminders can help you build a high-quality, credible, and relevant evaluation at all stages of the process:
Engage the community. Your comprehensive strategy will be most effective and information about it most useful when the strategy--and your plan for evaluating it--reflect the perspectives of everyone who has a stake in success. All groups must have a voice in determining goals, strategies, and measures of success. Too often, key groups such as parents or teachers are left out of the decision-making process. When this happens, the strategy misses out on the valuable experiences these groups can share, and the evaluation process may fail to meet their information needs.
Community participation also builds support for collaborative efforts. It is much easier to get things done when many people share your goals and are willing to work toward them. When you enlist all affected groups, every possible community resource is used to its fullest potential. For example, inviting business leaders into partnerships with public agencies can open access to a wider array of private resources to benefit children and families.
An example from Philadelphia demonstrates how local leaders used data on student conditions to mobilize parents and community members in support of change.
Using Information to Mobilize the Community for ChangeWhen educators and community leaders in North Philadelphia decided that major improvements were needed in the services provided to public school students, they set out to mobilize local parents and other community residents around their common desire to improve the lives of children and youth. They quickly discovered that their strongest tool was data?charts, tables, and numbers that told a story of poor academic achievement, high dropout rates, high rates of risky health behaviors, and limited employment prospects. Parents and others in the community at first were disheartened to see that the data bore out their personal observations. But their second response was to ask one another what they could do together to achieve large-scale change. With this new level of agreement around the need for improvement, parents and community members became active participants in designing the North Philadelphia Compact for Student Success, which now creates improved opportunities for children and youth in three high schools and their associated feeder schools and for North Philadelphia high school graduates attending Temple University and the Community College of Philadelphia. |
Reflect an understanding of community context. Your strategy will be strongest and its evaluation most relevant when both grow out of the needs, strengths, and resources specific to your community. You don't want to focus on reducing the number of high school dropouts if the graduation rate in your community is already acceptable or if parents, teachers, school administrators, and business leaders do not recognize dropout reduction as a priority. And you don't want to start a Saturday morning reading program to improve student performance if your community already has a popular children's activity on Saturday mornings.
Community events and trends inevitably affect the results of a comprehensive strategy, as it evolves. For example, a factory shut-down in the area may affect the apparent success of a job training program for youth by cutting the number of jobs available. Try to incorporate these contextual factors in your evaluation plan and in your interpretation and communication of information about your strategy.
Use varied perspectives to examine your comprehensive strategy. Viewing your strategy from varied perspectives will help you develop a better understanding of its successes as well as opportunities for improvement. For example, you may want to track results for several levels of impact, including child, family, community, and system changes. By doing so, you might discover that your strategy is effective at changing the way agencies operate but needs to be strengthened to create actual improvements for children. This type of multi-pronged evaluation will help identify strategic adjustments you can make as you pursue your ultimate goal: improving the lives of children and families.
Gather several types of information. Answering questions about the operation and impact of your comprehensive strategy is likely to require both numerical (quantitative) and narrative (qualitative) information. For example, your comprehensive strategy may target tuberculosis as an emerging local health problem. In addition to examining health records to determine the number of TB cases, you may also want to talk to doctors and patients to try to understand why the disease is spreading among some groups and not others.
As information needs change in the process of developing and implementing your comprehensive strategy, you are more likely to have the information you need if you build in procedures for producing several kinds of information. Multiple forms of information also are useful to policy makers and other stakeholders, who appreciate specific human stories but also want hard numbers.
Stay flexible. Communities are dynamic. As new information indicates changes in conditions and resources, remember to adjust the goals and activities of both your strategy and its continuous evaluation processes. For example, if your partnership is working toward improved school readiness as a goal and a new day care center opens in the neighborhood, this new resource might create an opportunity to add a new partner and expand your efforts to serve preschool children. This may require revising programs and the criteria by which they are evaluated. Flexibility means keeping your eyes open for chances to link with related activities or organizations and then incorporating changes into your continuous evaluation process.
The steps in planning and conducting a good evaluation parallel the steps in designing and carrying out a comprehensive strategy itself. These steps are outlined below and discussed at greater length in Chapters 3 and 4.
1. Identify goals and objectives for your comprehensive strategy and use them to guide continuous evaluation. By working with all relevant stakeholders, you can develop goals that articulate the shared long-term aspirations of your community. These statements can be broad and might reflect concerns in such areas as education, health, and employment.
For each goal statement, a thorough assessment or "scan" of community conditions and resources will tell you where you are relative to that goal, what you can use in moving forward toward the goal, and the barriers you are likely to encounter. Based on this scan, you can construct a list of specific, measurable objectives for children and families that relate to each goal. You can use these objectives to assess the progress of your comprehensive strategy. They become your indicators of long-term results.
2. Spell out the chain of assumptions connecting activities to goals. When you know where you want to go, you are ready to chart a course for getting there. You will need to articulate a chain of assumptions--the logical sequence of activities that you believe will lead to an improved result. This chain should reflect your understanding of what has hindered success in the past, so that your new activities can address those barriers. In constructing this chain, start with objectives and work backward, asking what has to happen to achieve each goal. The next link in the chain should address what has to happen to achieve the step before it, and so on.
As chains of assumptions for each objective take shape, you will recognize how activities fit together in coherent ways. If a chain of assumptions is weak, your strategy may need rethinking. Some pieces may fit well to answer your question of "what has to happen" to achieve an objective. Other activities you may be pursuing or planning to pursue may not find a natural fit with your chains of assumptions. Working backward from objectives, rather than starting with activities and justifying them in terms of your objectives, will help ensure that your resources are invested only in those activities that logically contribute to the results you seek.
3. Select indicators of results. Identify and use indicators of short-term, interim, and long-term results to measure whether the activities that make up each chain of assumptions are occurring and whether progress toward objectives is being achieved. The links in your chains inform the selection of indicators. You then use these indicators to determine how the logic of your sequence of activities holds up at various points along the path to improved results. Indicators should be both understandable and measurable.
4. Set up a system for managing information. When you know the types of information you need to measure progress, you can adapt existing procedures or initiate new ones for producing and managing that information. The information might include client characteristics, activities, staff time, costs, community perceptions, and, of course, results. Some information can be maintained manually, whereas some might be available in or added to an automated management information system (MIS). You will want to take advantage of information that is already available and minimize burden on staff and clients for collecting new information.
5. Analyze and use what you learn. Information must be used to be helpful. Usually, many stakeholders have invested time and money in a comprehensive strategy and will be interested in an on-going report of the activities and results associated with it. As you choose the types of information you will collect, you can also develop procedures for regularly reporting it to your audiences.
A good plan for using information takes into account specific decision points that might be informed by data (e.g., annual budget decisions that are made in May, monthly agency staff meetings). Your plan can ensure that your partnership regularly asks the questions: Are we doing what we intended? Are we accomplishing the short-term and interim results needed to achieve our longer-range objectives? Having the right information available at the right time will allow you to answer these questions and improve your comprehensive strategy. Experience with using information also will point up ways to revise your evaluation approach to make the information it produces as useful as possible.
6. Be patient. To put change in perspective, veteran collaborators suggest asking yourself: How long would it take to change your marriage? Just as most marriages do not change substantially overnight, changes in the lives of children, families, communities, and service systems do not happen quickly. It takes perseverance, dedication to the goals of the comprehensive strategy, and the simple passage of time for a strategy to make an impact. Although you cannot expect dramatic results immediately, take heart that you are using good information to document progress as you go along and that in the long run, you will obtain convincing evidence of what your strategy is accomplishing for children and families.
A Good Evaluation Involves
|
-###-
[Getting Started]