Module 2B, Section 3. Confidence in the Operations and Results of Combining Indicators

Module 2B: Indicator Interaction in the State's System of Annual Meaningful Differentiation (AMD)

This webpage is part of the Evaluating State Accountability Systems Under ESEA tool, which is designed to help state educational agency (SEA) staff reflect on how the state's accountability system achieves its intended purposes and build confidence in the state's accountability system design decisions and implementation activities. Please visit the tool landing page to learn more about this tool and how to navigate these modules.

A key part of validating a theory of action is to determine whether evidence confirms the assumptions and links between components that are designed to yield intended outcomes. A state's accountability system is a measurement instrument that helps the public understand the degree to which schools meet the state's educational objectives and priorities as well as a policy lever to incentivize actions that help achieve those same objectives and priorities.¹ To what degree is that happening? If a state can identify sufficient evidence that upholds the assumptions associated with indicator interactions, then it can more effectively argue that the results of the state's system of AMD are valid for identifying schools. The following self-reflection prompts provide an opportunity for a state to consider whether the interactions among the indicators or the decision rules of the state's system of AMD uphold the underlying rationale, as well as an opportunity to determine whether the SEA can be sufficiently confident that the elements of the state's system of AMD (i.e., indicator interactions and decision rules) are working as intended.

Respond to the following prompts to engage in the reflection around the way in which indicators interact:

Read the claim, consideration, and key evidence checks; then examine the specific evidence available in your state.
Reflect on whether you believe you have collected enough evidence to be confident in the claim stated or whether there is a need for further examination.
Finally, respond to questions that pose whether you have sufficiently explored the confidence claims below and believe that you have collected enough evidence that these claims can be confirmed.

Some questions may be based on opinion, whereas others will require an examination of data, supplemental analyses, or conversations with other members of your state department.

You may print this webpage and use it as a template for note-taking if working with colleagues.

For summative rating systems (e.g., index-based systems), please see the reflection prompts below in Table 6.

Table 6. Confidence in the Operations and Results of Combining Indicators for Summative Rating Systems Reflection

Claim 1: The indicator weights reflect the state's theory of action and stakeholder vision, as appropriate.
Consideration 1.1: Indicator weights or decision rules reflect appropriate stakeholder and constituent input.
Consideration 1.2: Indicator weights are coherent with the policy intent and intended incentivized behaviors for the state's accountability system.
The Elementary and Secondary Education Act of 1965 (ESEA), as amended by the Every Student Succeeds Act (ESSA) requires that each state consult with key stakeholder groups that represent the range of constituents across the state when developing their state plans, which include a description of their state's accountability system. However, some stakeholders' recommendations may not be appropriate to implement as is, given constraints such as high-stakes use, corruptibility, data access, and data collection. It is important that SEA staff supporting accountability have a clear understanding of how the indicator weights and interactions reflect the state's theory of action and stakeholder feedback gathered as part of the ESEA consolidated state plan development process. For each set of claims, consider the following statements and explore the suggested evidence for index-based systems or non-summative systems.
Reflection Prompts	Notes
Key questions for the indicator: How were stakeholder groups solicited for feedback? To what extent was this feedback incorporated when developing policy weights for the system?
Why is this important? Soliciting stakeholder feedback and input is an important design step to ensure representative viewpoints are included. It is important to blend this feedback with the overall policy objective and theory of action for how indicators are weighted. This feedback can also be incorporated to improve the system over time.
Key evidence checks: Determine the degree to which a sufficiently representative set of stakeholder groups was identified and consulted. Determine the degree to which strategic representation was included in design and feedback (e.g., advocacy groups, industry organizations). Determine the extent to which feedback was incorporated and how it was balanced with operational decisions based on constraints (e.g., available data, data collection efforts, policy constraints) when establishing indicator weights. Identify how to best reconvene stakeholders to support any redesign or additional buy-in efforts.
Recommended next steps: Stakeholder groups include both representative and strategic groups that bring role-specific perspectives, concerns, and recommendations. Consider the motivations of each stakeholder group as indicator reporting is designed. If necessary, develop talking points or outreach materials that highlight how feedback was incorporated in light of data or policy constraints. Stakeholder feedback is a powerful tool to enhance buy-in for the state's accountability system, which includes how indicators are weighted. Consider reconvening stakeholders after the state's accountability system is operational to obtain additional feedback to support communications, enhancements in reporting, and potential revisions to the system.
Reflection Prompts	Notes
Key questions for the indicator: What behaviors or next steps did you intend to promote based on the way in which indicators are weighted?
Why is this important? An important aspect of a state's accountability system design is considering how people will interpret, use, and act upon data from the system. It is important to consider how weighting decisions influence changes in awareness and behavior.
Key evidence checks: Review the weights of each indicator and determine whether they prioritize the state objectives as intended. Determine the kinds of behaviors or conversations indicator weighting and reporting should elicit. Identify ways to confirm these assumptions through feedback.
Recommended next steps: The relative weight of a given indicator compared to other indicators may drive focus and awareness to certain reports or performance at the school. For example, achievement and growth may be equally weighted, or growth may be weighted more heavily than achievement. Each of these design decisions should be reflected in communications, resources, and reporting strategies. Consider reconvening stakeholders to determine whether indicator weights promote conversations among data users as intended (e.g., student growth as a new part of the system is weighted heavily, college- and career-readiness measures are weighted enough to warrant attention). If needed, develop additional communications materials or consider revising reports or system weights.
Claim 1 Reflection Questions		Claim 1 Response
Reflecting on your notes above, consider your confidence in responding to the reflection prompts below. If you answer "no" or are not confident in your response, use the notes from your discussion to determine next steps.
We have sufficiently explored the confidence claims above to understand how our indicator weights reflect the state's theory of action and stakeholder vision.		Yes / No
We have collected enough evidence to sufficiently address key questions and can confirm that the considerations associated with Claim 1 reflect the state's theory of action and stakeholder vision, as appropriate.		Yes / No

Claim 2: The empirical indicator weights reflect the intended state priorities and promote valid, fair, and reliable school ratings.
Consideration 2.1: Indicators can be compared and contrasted based on technical characteristics of the data.
Consideration 2.2: Empirical indicator weights reflect intended policy weights and result in accountability signals as designed and intended.
How these indicators are combined in the form of weighting or decision rules plays a major role in how schools are differentiated and identified. Because of the variety of indicators and the diversity of data that may be used in systems of AMD, several technical considerations should be addressed. These might include examining the appropriateness of comparing and contrasting measures, ensuring policy weights are reflected in system operations, or identifying sources of volatility and error within and across indicators. For each set of claims, consider the following statements. and explore the suggested evidence for summative rating systems or non-summative systems.
Reflection Prompts	Notes
Key questions for the indicator: Are the measures and data that comprise each indicator functioning appropriately for use? Are the measures across indicators appropriate for use as part of a state's system of AMD?
Why is this important? These technical characteristics include both the process- and outcome-related characteristics of the data. Process-related characteristics may include things like policies related to data collection, data collection processes, ownership of data, corruptibility of data, and cleanliness of data. Outcome-related characteristics may include things like the shape, skew, range, mean, and mode associated with measures after data are collected and cleaned. Identified concerns associated with process-related characteristics can introduce uncertainty when trying to compare schools in the aggregate. Increasing consistency in policy interpretation, establishing systematic mechanisms for data collection, or increasing training in collaboration with the district can help address data entry and process concerns.
Key evidence checks: Identify any concerns associated with data-collection burden, points of failure in data entry, and who controls or owns data. Confirm that measures comprising indicators are not susceptible to corruption or fairness issues. Examine indicators to determine whether they have similar ranges, shapes, and variability.
Recommended next steps: Data that are subjective or lead to any unintended negative consequences can result in non-representative data, under-representing the construct, or reducing the fairness of comparisons. This may require clarifying the assumptions behind the theory of action, including certain clarifying data, or reducing the weight of certain indicators. The outcome-related characteristics of data are important for ensuring that composites of measures and comparisons across indicators can and should be made. If indicators—or the measures that comprise them—exhibit sufficiently different data characteristics, composites of indicators can lead to unexpected changes in ratings or rankings and can lead to challenges if compared or combined. Any identified issues should be addressed through transformation of measures, indicators, or composites (e.g., standardizing, norming, or indexing data or indicators).
Reflection Prompts	Notes
Key questions for the indicator: What empirical evidence is available to show that design decisions for policy weights are reflected in operational weights?
Why is this important? The ways in which indicators interact affect how the overall state's system of AMD differentiates and identifies schools. It is important to have a clear understanding of what indicators are most influential in the state's system of AMD, how changes within indicators affect differentiation over time, and whether any indicators have an unexpected amount of influence on the system.
Key evidence checks: Examine correlations among indicators to understand interaction among indicators and the stability of interactions year-over-year. Compare policy weights and empirical weights (i.e., those calculated through relevant inferential analyses) to determine the degree of match.²
Recommended next steps: Correlations can help you understand the overall direction and size of the relationship among indicators. Although a good first step, it should be used as a preliminary technique to determine whether relationships between indicators are too strong or weak. Use the results of correlation analyses to define a baseline for relationships, make comparisons to historical data, and make comparisons to future operational years of the state's system of AMD. After identifying the indicators that contribute the largest explanation of school rankings or scores using inferential analyses (e.g., regression with commonality analyses, factor analyses), determine whether any indicators contribute to the overall score more than anticipated. This may signal the need to revise weights to better approximate policy weights or transform data to increase or decrease the influence of certain indicators. If weights cannot be revised, it will be important to have materials and documentation that explain how to interpret how changes in indicator performance affect changes in the state's system of AMD.
Claim 2 Reflection Questions		Claim 2 Response
Reflecting on your notes above, consider your confidence in responding to the reflection prompts below. If you answer "no" or are not confident in your response, use the notes from your discussion to determine next steps.
We have sufficiently explored the confidence claims above to understand whether indicator weights promote valid, fair, and reliable results.		Yes / No
We have collected enough evidence to sufficiently address key questions and can confirm that the considerations associated with Claim 2 reflect the intended policy weights to promote valid, fair, and reliable results.		Yes / No

[If you wish to explore specific indicators in more depth, click here to continue on to Modules 3A — 3E: Indicators.]

[If you do not wish to further explore specific indicators, click here to continue on to Module 6: Reporting.]

[Click here to go back to the tool home page.]

Table 7. Confidence in the Operations and Results of Combining Indicators for Non-Summative Rating Systems Reflection

Claim 1: The decision rules reflect the state's theory of action and stakeholder vision, as appropriate.
Consideration 1.1: Decision rules reflect appropriate stakeholder and constituent input.
Consideration 1.2: Decision rules are coherent with the policy intent and intended incentivized behaviors for the state's accountability system.
The ESEA requires that each state consult with key stakeholder groups that represent the range of constituents across the state when developing the consolidated state plan, which includes a description of the state's accountability system. However, some stakeholders' recommendations may not be appropriate to implement as is, given constraints such as high-stakes use, corruptibility, data access, and data collection. It is important for SEA staff supporting accountability to have a clear understanding of how the sequence of decision rules and interactions reflect the state's theory of action and stakeholder feedback gathered as part of the ESEA consolidated state plan development process. For each set of claims, consider the following statements and explore the suggested evidence for index-based systems or non-summative systems.
Reflection Prompts	Notes
Key questions for the indicator: How were stakeholder groups solicited for feedback? To what extent was this feedback incorporated when developing decision rules for the system?
Why is this important? Soliciting stakeholder feedback and input is an important design step to ensure that representative viewpoints are included. It is important to blend this feedback with the overall policy objective and theory of action for how decision rules are designed and ordered. This feedback can also be incorporated to improve the system over time.
Key evidence checks: Determine the degree to which a sufficiently representative set of stakeholder groups was identified and consulted. Determine the degree to which strategic representation was included in design and feedback (e.g., advocacy groups, industry organizations). Determine the extent to which feedback was incorporated and how it was balanced with operational decisions based on constraints (e.g., available data, data collection efforts, policy constraints) when ordering decision rules. Identify how to best reconvene stakeholders to support any redesign or additional buy-in efforts.
Recommended next steps: Stakeholder groups include both representative and strategic groups that bring role-specific perspectives, concerns, and recommendations. Consider the motivations of each stakeholder group as indicator reporting is designed. If necessary, develop talking points or outreach materials that highlight how feedback was incorporated in light of data or policy constraints. Stakeholder feedback is a powerful tool to enhance buy-in for the state's accountability system, which includes how decision rules are ordered. Consider reconvening stakeholders after the state's accountability system is operational to obtain additional feedback to support communications, enhancements in reporting, and potential revisions to the system.
Reflection Prompts	Notes
Key questions for the indicator: What behaviors or next steps did you intend to promote based on the way in which decision rules are ordered?
Why is this important? An important aspect of accountability-system design is considering how people will interpret, use, and act upon data from the system. It is important to consider how design decisions around the order of decision rules facilitate changes in awareness and behavior.
Key evidence checks: Review the order of decision rules and the relative impact of each round of decisions and determine whether they prioritize the indicators as intended. Determine the kinds of behaviors or conversations decision rules and reporting should elicit. Identify ways to confirm these assumptions through feedback.
Potential next steps: The order or influence of certain decision rules may drive focus and awareness to certain reports or performance at the school or district. For example, achievement and growth may be the first indicators used to group elementary schools, followed by progress in achieving English language proficiency, and then college- and career-readiness indicators or other school quality/student success indicators. Ensure that each of these design decisions are reflected in communications, resources, and reporting strategies. Consider reconvening stakeholders to determine whether the order and impact of decision rules promote conversations among data users as intended (e.g., student growth is the first filter and student achievement is the second; graduation rate and achievement are the main drivers of high school identification). If needed, develop additional communications materials or consider revising reports or the order of decision rules if appropriate.
Claim 1 Reflection Questions		Claim 1 Response
Reflecting on your notes above, consider your confidence in responding to the reflection prompts below. If you answer "no" or are not confident in your response, use the notes from your discussion to determine next steps.
We have sufficiently explored the confidence claims above to understand how our decision rules reflect the state's theory of action and stakeholder vision.		Yes / No
We have collected enough evidence to sufficiently address key questions and can confirm that the considerations associated with Claim 1 reflect the state's theory of action and stakeholder vision, as appropriate.		Yes / No

Claim 2: The empirical results of decision-rule implementation reflect the intended sequencing of decision rules to promote valid, fair, and reliable school results.
Consideration 2.1: Indicators can be compared and contrasted based on technical characteristics of the data.
Consideration 2.2: Empirical decision rules reflect intended policy weights and result in accountability signals as designed and intended.
How these indicators are combined in the form of decision rules plays a major role in how schools are differentiated and identified. Because of the variety of indicators and the diversity of data that may be used in systems of AMD, several technical considerations should be made. These might include examining the appropriateness of comparing and contrasting measures, ensuring whether decision rules are appropriately reflected in system operations, or identifying sources of volatility and error within and across indicators. For each set of claims, consider the following statements and explore the suggested evidence for summative rating systems or non-summative systems.
Reflection Prompts	Notes
Key questions: Are the measures that comprise the indicator appropriate for use? Are the measures across indicators functioning appropriately for use as part of a state's system of AMD?
Why is this important? The technical characteristics of measures reflected within and across indicators are important to consider. These technical characteristics include both the process- and outcome-related characteristics of the data. Process-related characteristics may include things like policies related to data collection, data collection processes, ownership of data, corruptibility of data, and cleanliness of data. Outcome-related characteristics may include things like the shape, skew, range, mean, and mode associated with measures after data are collected and cleaned.
Key evidence checks: Identify any issues that may exist around level of data collection burden, points of failure in data entry, and who controls or owns data. Confirm that measures comprising indicators are not susceptible to corruption or fairness issues. Examine indicators to determine whether they have similar ranges, shapes, and variability.
Potential next steps: Identified issues associated with process-related characteristics introduce uncertainty when trying to compare schools in the aggregate. Increase consistency in policy interpretation, establish systematic mechanisms for data collection, or increase training in collaboration with districts to help address data entry and process concerns. In addition to logistical process characteristics, data that are subjective or lead to any unintended negative consequences can result in non-representative data, under-representing the construct or reducing the fairness of comparisons. This may require a revision to the theory of action, inclusion of certain data, or changing the order of decision rules based on indicator priority. The outcome-related characteristics of data are important to ensuring composites of measures and comparisons across indicators can and should be made. If indicators—or the measures that comprise them—exhibit sufficiently different data characteristics, this can lead to unexpected changes in ratings or rankings. Address any identified issues through transformation of measures, indicators, or composites (e.g., standardizing, norming, or indexing data or indicators) and potential revision of decision rules or the ordering of indicators in decisions.
Reflection Prompts	Notes
Key questions: What empirical evidence is available to show that design decisions for decision rules correspond to the predictive power of indicators?
Why is this important? The way in which indicators interact affects how the overall state's system of AMD identifies schools. It is important to have a clear understanding of what indicators are contributing the most influence in the state's system of AMD, how changes within indicators affect differentiation over time, and whether any indicators have an unexpected amount of influence on the system.
Key evidence checks: Examine correlations among indicators to understand interaction among indicators and the stability of interactions year over year. Determine whether the order of decision rules influences the identification of CSI, TSI, and ATSI schools as intended, based on the predictive power of indicators on school identification.
Potential next steps: Correlations are intended to help you understand the overall direction and size of the relationship among indicators. Although a good first step, it should be used as a preliminary technique to determine whether relationships between indicators are too strong or weak. Use the results of correlation analyses to define a baseline for relationships, allow comparisons to historical data, and compare relationships to future operational years of your state's system of AMD. Consider the predictive power of indicators on school identification rates through the use of empirical analyses* that test the amount of influence each indicator has on school results (e.g., logistic regression). After identifying the indicators that have the strongest influence on school identification, determine whether any indicators contribute to the decision rules more than anticipated. This may signal the need to revise the order of rules, transform data, or supplement decisions with additional data. If decision rules cannot be revised, it will be important to have materials and documentation that explain how to interpret how changes in indicator performance affect changes in the state's system of AMD. * For examples, please see State Systems of Identification and Support under ESSA: Evaluating Identification Methods and Results in an Accountability System (link is external) from the Council of Chief State School Officers.
Claim 2 Reflection Questions		Claim 2 Response
Reflecting on your notes above, consider your confidence in responding to the reflection prompts below. If you answer "no" or are not confident in your response, use the notes from your discussion to determine next steps.
We have sufficiently explored the confidence claims above to understand whether the order of decision rules promote valid, fair, and reliable school results.		Yes / No
We have collected enough evidence to sufficiently address key questions and can confirm that the state's system of AMD reflects the intended policy weights or intended sequencing of decision rules to promote valid, fair, and reliable school results.		Yes / No

[If you would like to further explore the state's system of AMD through reflection on specific indicators, please click here to continue on to Module 3A — 3E: Indicators.]

[If you would like to further explore the state's system of AMD through reflection on identified schools, please click here to continue on to Module 4: Comprehensive Support and Improvement (CSI) Schools]

[If you do not wish to further explore specific indicators or identified schools, click here to continue on to Module 6: Reporting.]

¹ For more information, please see Accountability Identification Is Only the Beginning: Monitoring and Evaluating Accountability Results and Implementation (link is external) from the Council of Chief State School Officers. Note: The inclusion of links to resources and examples do not reflect their importance, nor is it intended to represent or be an endorsement by the Department of any views expressed, or materials provided. The U.S. Department of Education does not control or guarantee the accuracy, relevance, timeliness, or completeness of any outside information included in this document.

² Relevant inferential analyses might include variability, commonality, principal components, or discriminant analyses. For examples please see State Systems of Identification and Support under ESSA: Evaluating Identification Methods and Results in an Accountability System (link is external) from the Council of Chief State School Officers. Note: The inclusion of links to resources and examples do not reflect their importance, nor is it intended to represent or be an endorsement by the Department of any views expressed, or materials provided. The U.S. Department of Education does not control or guarantee the accuracy, relevance, timeliness, or completeness of any outside information included in this document.