Section 2: Assess Confidence in the Accountability Theory of Action

Key questions:

How understandable are the components of your theory of action? Can individuals or entities be assigned to each component?

Why is this important?

If roles, responsibilities, and expected outputs are not clear in the theory of action (or, if no specific individuals or entities are assigned to them), it is less likely that individuals and entities will enact the expected behaviors. However, we recognize it is not possible to detail every aspect of a theory of action without it becoming unwieldy. Discuss the appropriate "grain size" for the theory of action that works for your state.

Key evidence checks:

• Review whether the language across the theory of action can be universally understood by educators and staff.
• Determine whether the language across the theory of action that refers to SEA resources, processes, policies, and outcomes aligns with state and local language.
• Determine whether the theory of action includes activities without specified individuals or entities to carry them out. In addition, consider whether any individuals or entities are over-represented beyond their actual capacity to conduct assigned activities.

Potential next steps:

• Better define the anticipated inputs or outputs so that key actors can understand what they should accomplish for the theory of action to work as planned.
• Identify individuals or entities who can be assigned to different components of the theory of action.

Key Questions:

Are the appropriate materials for supporting the state's accountability system ready and available? Are there avenues through which stakeholders can access resources or attend trainings?

Why is this important?

State accountability systems cannot stimulate the intended behaviors and outcomes if actors do not have sufficient resources, knowledge, and skills.

Key evidence checks:

Please consider the following evidence checks for each component in the theory of action as well as the overall theory of action:

• Consider whether the theory of action is viable in resource-rich as well as resource-limited districts in your state, including appropriate next steps.
• Consider whether supporting materials (e.g., manuals, software, websites) are in place and whether resources (e.g., curricula, learning materials, hardware) are sufficient to help people understand the role they play in a broad conceptualization of the theory of action (from school designations through supports and interventions for improvement).
• Consider whether available resources are equitably distributed and sufficient across all districts, schools, and students to support the theory of action. Is the educator preparedness level sufficient and equitably distributed across schools to support the theory of action?
• Determine what skills, experience, and preparedness are necessary to understand the decisions required in the theory of action. Do educators and staff have the required skills and experience to enact the next steps that are expected of them?
• Determine whether educators and staff have access (e.g., attainable and not cost-prohibitive) to the necessary information to inform their decision making and actions and whether it is available in a useful format.
• Determine whether there is support (e.g., training, coaching) in place to ensure that educators and staff can implement the component.

Potential next steps:

• Consult with other SEA colleagues, intermediary agencies, and federal resource centers to best leverage the supports available. Expectations regarding the impact of a state's accountability system are likely to be unfulfilled in districts and schools with insufficient resources, whether in terms of materials, knowledge, or skills.
• Consider what training is available to supplement educator skills, experience, and influence. This could include knowledge of the standards, ability to diagnose curricular gaps, and capacity to support school improvement team planning.
• Many districts still suffer from serious hiring challenges. Consider how your state is supporting districts that have persistent shortfalls in terms of hiring teachers and school leaders.

Key Questions:

Which offices in your SEA and which regional organizations support the components of the theory of action? Is the SEA collecting data necessary to inform steps in the theory of action? What protocols or systems support the flow of information?

Why is this important?

The anticipated functioning of the theory of action is dependent on a robust infrastructure, just as it is dependent on personnel (see Consideration 1.2).

Key evidence checks:

• Identify the data that are needed to inform and enact each component of the theory of action.
• Determine whether the following are present to support the theory of action:
  • Data systems are in place to inform actions associated with the theory of action; data are linked longitudinally and across platforms.
  • The necessary standard operating policies are established and in place.
  • The necessary inter- and intra-organizational partnerships are in place.
• Identify barriers within the SEA (and, if applicable, other state agencies) that are related to the exchange of information that can support the theory of action.

Potential next steps:

• Identify data needed but not yet available that are relevant to the theory of action to make components seem relevant and tangible. It is likely that different data and information will be required for different stakeholders within and across components so that they can relate it to their own work.
• Identify any prerequisite systems or processes that can impede how actors should implement activities or inputs in the theory of action. For example, are any key policies or resources missing? If so, how do these impede data collection, progress monitoring of components, or partnerships? If policies or resources are in place, what is the next component that is missing (e.g., data collection, partnerships between offices, understanding of policies). Target high-impact systems, policies, or organizational links that have downstream impacts in the theory of action.

Key Questions:

• Are the linkages in the theory of action based on research? Are there statistically significant correlations (where applicable)?
• Are the linkages between inputs and outputs or outcomes based on state or local experience?
• Are there conceptual gaps in key steps of the theory of action (e.g., from school identification to actions, and from actions to outcomes)?

Why is this important?

The underlying premise of any theory of action should be sequential, logical, and grounded in evidence, whether that evidence is from experience or research. Without this justification, it is more difficult to conclude that the theory of action will play out as anticipated.

Key evidence checks:

• Critically examine each logical step in the theory of action. Provide a reasonable, evidence-based justification for why one step would lead to another.
• Consult research reviews or meta-analyses to determine whether there is a sufficient research base to support an emphasis on the components highlighted in your theory of action.

Potential next steps:

• Add detail or provide support for the sequential steps in your theory of action. Where there are gaps, fill these in, both conceptually and in practice.

Key Questions:

• Are useful data shared with key educators and staff to help motivate intended behaviors?
• Are appropriate and strong incentives in place to drive behaviors? Are disincentives minimized?
• Are educators supportive of the intended behaviors included in the theory of action?
• Are there any barriers that could prevent educators and staff to respond in good faith?
• Are there other barriers created by institutional inertia?
• Is the relationship described among inputs, outputs, and outcomes likely to occur similarly across all student groups? Are there additional linkages to consider based on student abilities or status?
• Does the theory of action hold for schools of different sizes and settings (e.g., urban, suburban, rural)?

Why is this important?

Underlying the generic theory of action of accountability is the premise that the system will include elements that incentivize changes in behavior. If the system is not structured in a way that does so across variable settings, the promise of the policy will not be realized.

Key evidence checks:

• Critically examine any barriers to full execution of the theory of action, whether associated with regional variation, policy history, or resource allocation.
• Probe the extent to which hypothesized incentives are meaningful for the actors they are intended to motivate. Diagnose where incentives may be less effective than anticipated.
• Determine whether data are accessible to local administrators and educators and communicated in ways that enable them to make use of the data (also see Module 2B: Indicator Interaction in the State's System of Annual Meaningful Differentiation (AMD) and Modules 3A-3E: Indicators).

Potential next steps:

• Take actions to reduce any barriers identified.
• Enhance access to data and associated supports for interpreting data.
• Address underlying circumstances that are limiting the effectiveness of incentives associated with accountability mechanisms.

Key Questions:

• Are there potentially "negative" linkages or consequences that should be considered and accounted for in the theory of action?
• How will educators and staff respond to various components of the theory of action? What are the implications of their responses?
• Are there ways of "gaming the system" or distorting results that could significantly compromise the theory of action?

Why is this important?

Research on the implementation of states' accountability systems has documented frequent evidence of unanticipated consequences, such as gaming the system or focusing on a few "bubble students" whose performance may move the needle on school identification. These actions run counter to the broader policy intent of improving outcomes for all students.

Key evidence checks:

• Because evidence of unanticipated consequences is difficult to identify until after the fact, challenge yourselves to consider whether the incentives associated with accountability mechanisms would prompt counterproductive actions, and what information or supports would avoid such actions.

Potential next steps:

• Provide communication, data, or technical assistance that would enable local actors to embrace the spirit of the theory of action and not succumb to counter-productive actions.
• Put in place guardrails to help prevent counter-productive actions.

Key Questions:

• Does the theory of action promote the state's objectives for:
  • Student achievement?
  • College and career readiness?
  • Attainment of postsecondary education, training, and/or employment?
  • Attainment of a well-rounded education?
  • Equitable access?
  • School and student performance outcomes?
• Do the objectives articulated within the theory of action reflect the documented objectives and needs of districts and schools?
• Does the state receive positive feedback on the theory of action objectives from local constituents?

Why is this important?

Although some common themes shape educational objectives across states, each state's unique context will shape the needs to be addressed through the theory of action. Accountability policies will resonate with local educators if state objectives are aligned with local needs.

Key evidence checks:

• Gather input from a range of stakeholders (or revisit feedback from previous listening tours) to critically examine whether the theory of action is framed in ways that will support accomplishing state objectives and resonate with local educators.
• Review a sample of local needs assessments, local educational agency (LEA) consolidated plans, Title I plans, and school improvement plans to identify concerns and to assess alignment between state and local objectives.

Potential next steps:

• Consider modifications to the theory of action and associated mechanisms in your state's approach to accountability, in conjunction with lessons learned through other modules of this tool.