A r c h i v e d  I n f o r m a t i o n

Resources For Obtaining Good Information

Information usually isn't free; gathering and using good information requires resources. Your choices regarding the information you need and how you will collect it will determine the resources your initiative needs in order to gather good information. You may need one or more of the following:

• Staff time to computerize or hand-tally quantitative data--for one person who is responsible for all information management, or for several people who each will enter information on their own activities

• Training in data entry, compilation, and reporting skills for people involved in these activities; training in how to interpret and use information for managers and other decision makers

• Computer equipment and software for data entry, compilation, and reporting

• Focus group leaders to obtain qualitative information from participants, including facilitators who can conduct focus groups in the home languages represented in the community

Some important resources may exist within partners' organizations. Others may need to be obtained from outside the partnership and the program being evaluated. This chapter describes key internal and external resources and suggests ways in which they can enrich your evaluation.

Internal Resources Strengthen Evaluation

Your partnership inevitably has many assets that can contribute to evaluation.

Staff knowledge. An important first resource is the knowledge of your partnership's own staff members. On-site staff who gather information may be able to compile, enter, and report data using computerized or paper-based systems, or to conduct focus groups or ethnographic interviews that produce reliable feedback. Program leaders and decision makers can interpret and use the information to yield accurate assessments, guide program planning, adjust the course of current efforts, and satisfy funders.

Program staff who participate in information gathering can use their own knowledge about how a program does (and doesn't) work to identify many of the components that measure results. Staff input has several advantages:

Staff who believe that they are being seriously consulted about the measures that will be used to assess their work have an acute awareness of what is fair. Staff will sometimes... argue for more demanding standards than an external evaluator might. But the key is that they have an opportunity which is more than token to reflect on what they do every day and how it can be fairly measured--given the resources and external effects they deal with in their work (Young, Gardner, & Coley 1994).

Take a hard look at how collaborators collect and use information to assess their own progress. Staff from partner organizations may be able to gather data or train a program's own staff to collect and use information. Or, they may serve as a liaison between the project and a third entity that has expertise or infrastructure for evaluation. The same networking skills that support service delivery can build capacity for data collection and analysis.

Existing data systems. As noted in Chapter 4, data systems that can contribute to information collection include academic, behavioral, and discipline records from schools and community crime and child abuse reports. Other data systems that support evaluation include: (1) case management or client tracking systems; (2) shared information systems, which centralize data from multiple programs or agencies; and (3) data matches, which map client data across programs and agencies. For example:

• A partnership in California matched student records at eight schools against databases for 19 city and county programs to find out how many children used multiple services, whether students having difficulty in school were more likely to use public services, and whether service use varied among racial and language groups (Marzke, Both, & Focht 1994, p. 36).
• Ten agencies in Oregon participate in a shared information system to centralize data and produce summary reports (Marzke, Both, & Focht 1994, p. 37).
• An automated case file used by a program in Minnesota links home-based case managers with a database and action plan on the county mainframe computer, via laptops. The system includes information on family needs, assessment, referrals, follow-up, case planning, and client progress toward goals; caseworkers with shared cases can communicate and coordinate their efforts (Marzke, Both, & Focht 1994, p. 35).

These systems can collect information on short-term, interim, and long-term results (useful to evaluators as well as front-line practitioners); identify patterns in service use; place a program's efforts in context within community needs and resources; and, in some cases, generate interim summary reports. However, they require staff training and time to keep the systems updated.

Parent and community contributions to evaluation. Parents and community members can bring meaningful perspectives on services to an evaluation. By participating in information collection, they also learn new skills that strengthen them and establish positive relationships between them and professional staff. For example:

• Parents and students at a middle school in Philadelphia helped gather information on how a school reform effort improved students' readiness for high school. Students participated in focus groups and ongoing data gathering and analysis sessions; developed recommendations for school staff that related to the project's goals; and helped prepare a workshop for a national education forum. Parents participating in a research team conducted focus groups with teachers, analyzed data collected from students and alumni, learned ethnographic skills for "shadowing" and interviewing students, and helped revise the research plan to obtain a better understanding of students' perspectives.

If you involve student interns, parents, and/or community members in evaluation research, keep in mind that they may require some training and oversight.

Using internal resources builds local capacity for evaluation. It can save money, provide opportunities to develop on-site expertise and solidify partnerships between collaborators, and produce findings that are enriched by the first-hand knowledge of program staff.

Outside Resources Contribute Additional Expertise

Depending solely on the expertise and resources available within a collaborative effort can limit an evaluation's scope and impinge on scarce time and staff resources. To avoid these problems and to improve evaluation quality and credibility, outside resources are valuable.

Outside evaluators offer expertise and objectivity. External evaluators can augment limited resources within a program and bring a new perspective to your thinking about evaluation issues and options. Outside evaluators are not burdened by the responsibility of operating the program or by the political decisions and emotional investments that may affect program staff.

External evaluators should not simply descend on your program and conduct an evaluation. Evaluation is something that a program should do as a normal part of its existence; it is not something that should be done to you. Carefully define the scope of the external evaluator's role so you both understand the expectations and approaches. Ask the evaluator to explain in detail how he or she envisions each step of the evaluation, including staffing and budgeting.

Where can you find outside evaluators? You may find outside evaluators at local colleges or universities (e.g., graduate students or professors with experience in program evaluation) or through recommendations from programs that have already evaluated their efforts. Read other programs' evaluation reports to identify evaluators who produce useful, comprehensive information. Talk with your peers at other programs who may be able to identify evaluation sources.

How much does an outside evaluator cost? The primary cost for an outside evaluation is for the evaluator's time. A university-based evaluator may charge upwards of $250 a day; a private evaluator may charge considerably more. To estimate the cost of your evaluation, identify each phase of collecting and analyzing information, and estimate how many days it should take to complete each step.

To make an evaluation more cost-effective, consider whether program staff or volunteer resources can collect and compile much of the data for the outside evaluator. (The external evaluator should always be involved in designing the evaluation, however, to ensure that the information collected is useful and directly relevant to the evaluation.)  

Resources contributed from outside your partnership can be cost-effective. Key outside resources include the time and expertise of graduate student interns from local colleges or universities. Student interns who help programs with data collection and analysis offer fresh knowledge of evaluation techniques and approaches and extra eyes, ears, and hands to do much of the work of information collection so program staff can focus on providing services. At the same time, student interns gain valuable experience by participating in a real evaluation.

A Word about Confidentiality

Confidentiality is important not only to protect the privacy of children and families, but also to establish trust between program staff and the clients they serve. Client perceptions of confidentiality are as important as legal enforcement; if a client fears his or her needs are not private, he or she is less likely to seek services. At the same time, each partner involved in a collaborative program wants access to information about clients especially when that information will enable partner organizations to improve services and satisfy evaluation needs. In some cases, the issue is complicated by legal restrictions on information sharing across agencies, especially those that serve undocumented residents and clients seeking services for sexual abuse and mental health needs.

But confidentiality concerns need not prevent a partnership from sharing information. In many cases, confidentiality concerns are more perceived than real, and several strategies offer protection from breaches of confidentiality. These strategies include:

• Obtaining informed consent for information-sharing from clients
• Developing user identification codes or passwords that protect client information
• Developing shared policies, definitions, and standards for controlling and using client data

For further discussion of confidentiality issues and strategies, see the documents referenced in the "Resources on Resources" box.

-###-

[Documenting Progress and Results]

[An Evalulator's Tool Kit]