Archived: Making Information Work For You - Documenting Progress and Results

With goals, objectives, and activities in place, you're ready to ask questions that focus on progress in implementing your comprehensive strategy and on the results the strategy is achieving. This information can tell you about clients, services, staff and other resources, collaborative partners, and the community. It will also enable you to report to many interested audiences on the success of your strategy for children and families. When you cast your information-gathering net this broadly, you will create an overall picture of the status of your strategy as it evolves, and you will be able to identify the information to use in its continuous improvement.

This chapter provides you with tools for identifying, tracking, and using information on the progress and results of your comprehensive strategy. It guides you through four steps:

Selecting Indicators of Results

As discussed in Chapter 3, your chains of assumptions chart the paths you expect to follow in order to reach your goals and objectives. There are many checkpoints along each chain at which you can assess whether the steps you laid out are actually occurring and, perhaps, change course to a more productive direction. In evaluation language, these checkpoints are opportunities to gather indicators of short-term and interim (or intermediate) results.

Partnerships use indicators of all three kinds of results to improve their strategies. Although it is unlikely that a strategy can make big strides in meeting objectives quickly, you will want to demonstrate positive results to your community, partner agencies, and funders before the fundamental condition of children and families has had time to change significantly. Indicators of short-term and interim results supplement long-term indicators by allowing you to view early actions as essential results, even though they may not directly measure improvements in the lives of children and families. You can use these early results to show stakeholders that your comprehensive strategy is headed in the right direction.

Indicators of short-term results measure what you do. Indicators of short-term results address implementation issues: Are the activities specified in your chain of assumptions being implemented? At the intended level? With the intended clients? With what resources? Answers to these questions can help you hold collaborative partners, program staff, and other participants in your strategy accountable for their contributions.

Think back to the example in Chapter 3 of a chain of assumptions for reducing student absenteeism in an effort to improve student achievement. Activities on that chain include a program to reduce the incidence of older students being kept home to care for younger children. Some indicators of short-term results for this chain might be:

Indicators of interim results measure what happens. Indicators of interim results move beyond basic implementation questions to address changes in behavior and attitudes that might contribute to the attainment of your objectives. In the student absenteeism example:

These changes can be measured to assess interim progress toward reducing absenteeism. The more these activities occur, the greater the chance of reducing absenteeism due to children being kept home from school to care for siblings. Along the same vein, reduced absenteeism can be viewed as an interim indicator toward the long-term goal of increasing children's success in school.

Feedback from community members and collaborative partners about their awareness of, experience with, and perceptions of your comprehensive strategy also can provide valuable interim results. Documenting the fact that activities are engaging staff and satisfying clients can help you sustain support for your strategy as it works toward longer-term results.

Remember to look for information that tells you not only what is happening with clients, activities, and staff, but also what isn't happening. Consider the following questions:

Keep in mind that, although objectives can be considered ends in themselves, some might also be interim indicators for other objectives. For example, decreasing the rate of diseases preventable by vaccination may be a worthy child objective in itself. However, if a community perceives absences caused by mumps and measles as a barrier to academic achievement, then decreasing the incidence of preventable diseases can also be seen as an interim indicator for the child goal of increasing children's success in school. The more you can link different objectives and chains of assumptions, the closer you will come to realizing the benefits of a truly comprehensive strategy.

Long-term indicators measure results. An indicator of long-term results tells you the extent to which you are achieving your goals (see Chapter 3). In the student performance example, indicators of long-term results would reflect the goals of student academic success. Indicators might involve improved test scores or improved student grades.

How do Partners Select Indicators of Results?

As you choose the indicators for which you will collect information, consider similar criteria to those you used in choosing objectives:

Remember That Definitions of Indicators Are Not Always Consistent

Indicators of results for children and families are defined in many different ways, which makes it hard to determine what we really know about comprehensive strategies. For example, dropout rates are a common indicator of impact on older students, but that rate is variously defined as:

Several efforts are under way to standardize definitions of indicators (e.g., California Interagency Data Collaboration, 1995; Family Health Outcomes Project 1995). See the Tool Kit for an example of a "data dictionary" that state-funded, school-linked service sites in one state use to standardize the definitions of indicators they may use in measuring their results. You are encouraged to consult these sources or others that may be available in your state in defining your indicators so that the information you produce allows you to compare your results with other benchmarks.

Once you have chosen indicators, you are ready to collect data on the short-term, interim, and long-term results of your comprehensive strategy. In doing so, you will want to consider three factors:

What Types of Information Can You Collect?

Using multiple sources of information and multiple strategies for collecting it help you develop a thorough understanding of what your strategy is doing and accomplishing. Such information could pertain to:

Information on children and families. A basic question about children and families involved with a strategy is "how many are served?" If this is a short-term indicator for your strategy, event description logs can estimate the number of participants in activities of various kinds (see the Tool Kit for an example log).

An "intake" or "family profile" form, completed for each family within the first several weeks of their participation in activities, can provide additional information on clients who are more intensively involved with your strategy. Information on client characteristics will tell you who is attracted to or in need of your services and activities. Consider gathering intake data on the composition of the household, ages, school status, ethnic and language background, family strengths and needs, and participation in other programs such as Temporary Assistance to Needy Families (TANF) or bilingual education.

Enrollment information on children and families also can serve as a baseline measure of results. Ask families to report their current status on indicators of interim or long-term results that your partnership has selected. For example, if your partnership seeks to increase parent involvement in school activities, ask parents at enrollment about their current level of involvement. Use periodic follow-up interviews to assess changes in the level of involvement of enrolled families.

Surveys and focus groups are useful approaches for obtaining information from children and families. However, written questionnaires can be perceived as "tests" by some groups and can limit the participation of those with low literacy skills.

Information on activities and services. A log of ongoing services will help you document the services and activities provided by your initiative. A sum of such counts, categorized by type of service, can document the level of activity of the initiative and its reach into the target community. (See the Tool Kit for a sample log and a graphic format for reporting service information.)

Some initiatives also track services received by individual clients or families or a subset of them, such as those receiving multiple services or case management. This approach links the pattern of services for clients to their individual results and thus measures the impacts of the strategy at the child or family level. Service providers that track individual services usually record individual client names or identification numbers along with the date and type of service. (For a sample form, see the Tool Kit.) Computer analysis usually is required to correlate results of specific children or families with the services they received.

Some initiatives choose a mixed approach for recording information on clients or services. That is, they document the number of participants in community-wide activities (e.g., a school health fair) or ongoing activities (e.g., a parent education class) and also record more information for a subset of "core families" who receive a broader array of services over a longer period of time.

Information on staff and other resources. Knowing the roles and activities of school nurses, case managers, teachers, family advocates, and other service providers lies at the heart of understanding the operations of a strategy. Daily activity logs can record the activities required of program staff to make a comprehensive strategy a reality. Activity reports can be expanded to record the amount of time devoted to particular activities. (See Tool Kit.)

Time on Task

A youth-serving organization placed case managers in a local school to work with a group of high-risk young people, identified by their poor grades and citizenship marks. The evaluation for this program gathered data on how much time the case managers spent with the students in their caseloads. A look at the first three months of these data revealed that the case managers were working an average of 25 hours per month!

"What are you doing with the rest of your time?" asked the program evaluator. "Filling out the stupid evaluation forms!" replied the case managers. "Ah, we can record that time too," thought the evaluator. Indeed, the case managers' logs were altered to allow them to record the time it took them for "administrative tasks," including furnishing data to the program evaluator.

After another three months, the case managers were shown to be working 30 hours per month! What was going on?

The evaluation staff shadowed the case managers for a few days and discovered that instead of case management, these new and welcome workers in the school were being absorbed in tasks that the school considered essential: crisis intervention for students not on their caseloads; hall, cafeteria, and bus duty; and a myriad other tasks that the school suggested would help them fit in better.

These data prompted a renegotiation with the school administration about the purpose of placing these case managers there in the first place. Some staff were moved to new offices to keep them out of the main school traffic, and meetings were held with the teachers in the school to quell some resentment that these case managers did not have to "pull their load" on the routine school tasks.

What caused the project to become aware that the intervention was not being delivered and that these changes were needed? The simple act of adding up minutes spent on the core tasks of the program.

Source: Philliber, 1995.

Other participants in the comprehensive strategy, such as part-time or volunteer workers, can document their activities by creating a diary of project work from which you can extract the critical incidents in implementing a comprehensive strategy.

Implementing your comprehensive strategy will take other resources beyond staff time, including dollars, training, space, equipment. You eventually may be able to demonstrate that good things happen for children and families through your comprehensive strategy. When you provide that documentation of results, the next question you surely will face is "but what did it cost to do that?" You need to record information on resources used as you go along in order to be able to answer that question.

Documenting the origin and use of relevant resources will enable you and your partners to demonstrate to constituencies their contributions to your efforts. (See the Tool Kit for an example.) You can add monthly calculations of these resources to estimates of recurring costs (e.g., a half-time counselor's salary, office space) in estimating the costs of a strategy or its components.

Information on collaborative partners. Members of a partnership can document what they do to implement the components of a comprehensive strategy by creating a diary of collaborative work from which the critical events in implementation can be abstracted. (See the Tool Kit for a sample.) Understanding important events in the life of an initiative "can inform the initiative's attempts at renewal, such as during changes in staffing, board membership, or the mission" (Fawcett 1994, p. 51).

You can also develop a "report card" on collaborative activities by periodically surveying partners for their views of their work. Periodic surveys also can indicate changes in agency procedure, staffing patterns, or other aspects of systems change that are appropriate to the short-term, interim, or longer-term indicators you have selected.

Information on community members. Periodically soliciting feedback from community members will encourage broad community representation in all aspects of your comprehensive strategy. (See Chapter 2.) Ask community representatives about their awareness of, experience with, and perceptions of your initiative. Knowing where you stand relative to these stakeholders is essential in maintaining support for your activities.

For What Groups Should You Gather Information?

In collecting and interpreting information, you will face a persistent question: Can you attribute changes to your strategy, or is there some other explanation? The traditional way of answering this question comparing a randomly assigned group of people who experienced the strategy to a control group that did not participate is neither feasible nor appropriate for most comprehensive strategies, which are tailor-made for specific clients, organizations, or communities. But there are some procedures you can use to collect information that lends itself to valid comparisons. These include:

Matched comparison groups. A group of similar clients, a similar neighborhood or community, or a similar school sometimes can serve as a comparison for those involved in your intervention. For example, the evaluation of the Walbridge Caring Communities school-linked service strategy (Philliber Research Associates 1994) used a comparison school of very similar demographic characteristics in an adjoining neighborhood to isolate and identify the results of Walbridge's comprehensive strategy. Matched comparison groups are a persuasive method when the groups being compared are very similar except for the presence of the comprehensive strategy in one of them.

Trends in the general population. Existing databases can identify trends in the general population that you can compare to trends in the population participating in the comprehensive strategy. For example, if the rate of births to adolescents declines over a three-year period in a community that has launched a broad-based teen pregnancy prevention initiative, the fact that the community is bucking the national trend in adolescent births lends credibility to the conclusion that the initiative is working.

The closer to home you can get in defining the comparison population, the better. State trends are stronger comparisons than national trends; trend data for your county, city, or school district are better comparisons than data for your state.

Self-comparisons over time. Evaluations often include repeated measurements of the same indicators over time to determine if conditions are improving. For example, you can measure the condition of clients at the beginning of their involvement in a comprehensive strategy and then again at regular intervals perhaps every six months during their involvement. Similarly, you can compare aggregate measures of student achievement on standardized tests or crime rates for a given community over several years to identify whether student performance and community safety are improving.

Multiple comparisons. Various comparisons can be combined in some ways to offset the shortcomings of each. For example, you could assess changes over time for your families or community and compare that trend for your county, state, or for the country at large, using available databases.

Your strategy operates in a complex environment, possibly alongside several other efforts that are addressing the similar goals of improving results for children and families. At the same time, the more effectively you articulate your vision of children and families the more you lay out a logical path to achieving that vision, and the more you demonstrate that you are following that path the more willing stakeholders will be to embrace and support your comprehensive strategy.

Collecting Information Efficiently

Although you can document some aspects of the work of your comprehensive strategy manually, you may choose to track many of the types of data described above through an automated management information system (MIS) designed to meet a range of information needs.

An MIS can help you systematically compile data and share information with collaborating agencies. For example, software is available to support case management, including modules to record case notes, remind staff of pending follow-ups, schedule appointments, and produce standard reports for specific programs. The benefits of a good MIS are substantial, but the investment in equipment and training can be significant.

Even if you have developed a near-flawless process for collecting information, the data you need may not always be readily available. Because your chain of assumptions shapes your information needs--and those assumptions may change over time--it is important to build flexibility into your data collection processes.

Keep identifying data sources that are appropriate to your information needs. If time and resources permit, you can add these new information sources to your data collection. If you frequently have trouble finding the information you need to demonstrate results, reassess your processes. Ask other stakeholders in your community for advice on better ways to track and organize data on activities and the children and families they serve.

Using the Information You Collect to Improve Your Comprehensive Strategy

The information you collect will provide the tools you need to improve your comprehensive strategy. For example, you may discover that your strategy has succeeded in achieving one objective, but not another. For the objective you did not achieve, you can ask whether one or more of the designated steps in your chain of assumptions never took place, whether the activities were implemented inadequately, or whether the logic of the chain is flawed or incomplete.

You can use the information you have collected to determine which scenario applies. In our student absenteeism example, perhaps parents did not receive the information on school policies and the impact of absenteeism. Perhaps they received the information but could not understand it. You can adjust these activities and try again to get the results you desire. Collect information again on indicators and objectives to see if the missed step was the problem.

Of course, there is a chance that all the intended steps occurred but the strategy still did not achieve the objective. If this is the case, there may be a fundamental problem with the logic of the chain of assumptions. Perhaps an important event was not included on the chain, or perhaps an existing step is hindering attainment of the objective.

Revisit your chain and the links that compose it. Ask yourself: How else can we get from where we are to the results we desire? Consider the possibility of designing a new chain of assumptions, establishing new indicators, and collecting new information on them.

It may take longer to achieve some objectives than others. For example, if you focus resources intensively, you may be able to reduce in six months the portion of student absenteeism that is due to students' child care responsibilities at home. However, you may need a much longer and more multi-faceted approach to reduce unexcused absences among students who are poorly engaged in learning. Even with activities that take a longer time to reach objectives, you can measure achievement against indicators of short-term and interim results.

Explaining Results to Stakeholders

The many stakeholders who have invested time and money in your comprehensive strategy naturally want to know if the strategy has made a difference in the lives of children and families. They are also likely to want to know what resources have been needed to make that difference. You can be accountable to stakeholders by regularly reporting back information as it becomes available. For example:

Working proactively with the media can also generate public support and effectively disseminate the knowledge you gain from evaluation. Although it is natural to want to avoid sharing information with the media when findings reveal shortcomings in your comprehensive strategy or when your program's successes in some areas outstrip its progress toward other goals, open relationships with the news media can build community awareness and support for your efforts. The real winners will be community members who will become better informed and presumably more supportive of comprehensive efforts.

Here are some suggestions for developing productive relations with the media and other stakeholders:

Identify the communication tools you can use to disseminate your message. Include alternative and neighborhood news outlets as well as mainstream sources.

Pick one or two credible, accessible members of your team to be key media representatives, so that your message to the community is consistent. Media contacts should be able to serve as liaisons between reporters and sources within your group of evaluators.

Know your audience. Discuss how the community is likely to react to each program element. Work your way up the controversy scale, developing honest responses to issues. These responses should provide the media with accurate information and legitimate arguments to counter concerns.

Develop a few powerful, simple messages that support your program. For example, focus on your goals or the economic impact of clients' problems. Contact editorial writers, editors, and reporters to explain your messages. Provide them with a "snapshot" of local data and findings that they can use to justify coverage in the interests of parents, community members, and policy makers.

Be specific when you explain your strategy. Describe the activities, resources, responsibilities, schedules, and results in a straightforward way.

Be prepared to respond to community concerns with media savvy and compelling arguments. For example, provide testimony from a program client that provides a human angle for reporters, supported by additional facts. Member of your partnership may want to hold a joint press conference, so reporters can gain access to a variety of viewpoints.

Respond to media inquiries quickly, concisely, and accurately. Describe your results honestly, even if interim data indicate temporary setbacks, to build your evaluation's credibility. A reporter who senses evasiveness will be forced to focus the story on the problem rather than on broader issues.

Provide reporters with a chance to follow up on their stories. Call the reporters who have covered child care stories in the past, for example, and tell them when you begin to find solutions to the issues they have covered. Discuss your findings in a way that connects them to important policy issues.

Provide descriptive, local examples that illustrate your results. Help the reporter connect these stories to regulation, funding, or other local and state issues. Try to link the reporter with any other research on your own community that relates to your findings.

Make public relations an ongoing part of program operations. By keeping your efforts in the news, you will keep community members interested. You may want to: include media representatives on your program's advisory committee; solicit comments from community members frequently through focus groups, mail-in responses, and/or radio talk shows; and maintain regular contact with members of the media.

Publicize your results. Don't wait for reporters to discover that you have findings - let them know what you're doing as you progress from forming a partnership and setting goals to measuring and analyzing results. Issue press releases and hold news conferences.

A r c h i v e d I n f o r m a t i o n

Documenting Progress and Results

Selecting Indicators of Results

Sample Questions for Monitoring Progress