Program/Policy
The Growth Model Pilot Project (GMPP) was initiated to allow states to experiment with adjustments to the No Child Left Behind Act (NCLB) status accountability system in order to improve the validity of AYP determinations by giving schools credit for students who are making significant growth. The pilot allowed states, districts, and schools to count students who were on track to being proficient (but not yet there). Under NCLB, such students are not counted as proficient for the purpose of AYP determinations.
The pilot was initiated in November 2005 with the goal of approving up to ten states to incorporate growth models in school AYP determinations under NCLB. Eventually, nine states were approved under the pilot: Alaska, Arizona, Arkansas, Delaware, Florida, Iowa, North Carolina, Ohio, and Tennessee. Now no longer a pilot, the project was written into regulation in 2008; any state may apply to use a growth model meeting certain core principles. Currently 15 states are implementing growth models under this authority, including additional states of Colorado, Michigan, Minnesota, Missouri, Pennsylvania, and Texas (the latter six are not within the scope of this report).
Main Study Questions
- How have states in the pilot implemented growth models?
- How does each pilot state's growth model affect the number and kinds of schools that make AYP?
- What are the implications of the pilot experience for strengthening the use of growth within the context of NCLB?
Findings and Implications
- Growth models enabled additional schools to make AYP compared to status and safe-harbor rules alone, but the percentages of schools that made AYP because of the growth models were generally not large. The growth models in the nine pilot states resulted in an additional 1,246 schools making AYP in 2007–08; this represented 9 percent of all schools with grades using the growth models. This was much higher than the 3 percent figure for the 2006–07 school year documented in the interim report, but the increase was primarily due to the addition of Ohio. The majority of the growth schools (983) were in Ohio, where 34 percent of the schools made AYP because of the growth model. The next highest percentages were in Arkansas (6 percent), Florida (5 percent), and Delaware (3 percent). Ohio used a different method than the other states of determining whether students were on-track to reach proficiency and this led to the much higher rates of schools making AYP by growth in that state. Excluding Ohio, the percentage of all schools making AYP in the other eight states because of the growth model was just 2 percent.
- The impact of growth models varied widely across states. he impact of adding the growth models to AYP determinations is measured by the percentage increase in the number of schools making AYP due to use of the growth model; this standardizes for state differences in the percentage of schools making AYP by status and safe harbor. Overall, the number of schools making AYP due to growth models represented a 20 percent increase over the number that made AYP under the NCLB status and safe harbor criteria. The percentage increases ranged from 0 to 4 percent in six states to highs of 10 percent in Arkansas, 24 percent in Florida, and 102% percent in Ohio. Excluding Ohio, the number of schools making AYP because of the growth model increased by 5 percent. Similar to the interim report, high-poverty schools were more likely to benefit from the status-plus-growth framework (a 20 percent increase in the number making AYP) than were schools with low poverty rates (an 18 percent increase).
- Most (but not all) schools that made AYP by status would also have made AYP under the growth model alone. Sixty-two percent of the schools that made AYP under the NCLB status criteria also would have made AYP strictly by using the growth criteria. Results again varied widely among the states, ranging from only 46 percent in Arizona and 47 percent in Arkansas and North Carolina to 75 percent or more in Ohio, Delaware, and Tennessee. The percentage of schools that made AYP by safe-harbor and that also met or exceeded their reading and mathematics AMOs under the growth-only criteria was much lower (28 percent overall). Across the eight states with safe-harbor schools (Delaware had none), the percentages did not exceed 30 percent in any states except Arkansas (64 percent) and Ohio (45 percent).
- Controlled simulations comparing the impacts of different types of growth models on student and school growth results show that the projection model functions in stark contrast with transition and trajectory models. When a generic version of a projection model classified a nonproficient student as on-track, the probability that the generic transition or trajectory model agreed was near 0, and vice-versa. For status-plus-growth models, projection models had the least impact, affecting only 10 to 20 percent of eligible (nonproficient) students, while transition and trajectory models affected over 20 percent. However, in simulations of AYP determinations, the models did not yield large differences in the percentages of schools making AYP when non-proficient students who were on-track to proficiency according to each type of model were added to the numbers of students meeting or exceeding the proficiency cut point (i.e., status-plus-growth).
- Simulations comparing the results of different growth models using the same data show that projection models have the highest correct classification rates for future proficiency: over 80 percent. These rates are 5 to 20 percentage points higher than trajectory and transition-matrix models depending on the grade level and proximity to the growth model time limit. While the projection model is more accurate, it is theoretically more difficult to implement and to explain to practitioners and parents than the other models.
- Although not an option under the Growth Model Pilot guidelines, growth models not tied directly to proficiency standards could identify a broader contingent of students as making adequate growth. One alternative to the GMPP growth-to-proficiency models that can be used with vertical test score scales is the difference between the proficiency cut scores in successive grade levels. Students gaining that amount or more would be considered to make "adequate yearly growth" (as opposed to NCLB's "adequate yearly progress"), regardless of whether they are proficient or on-track to become proficiency. A simulation shows that the overall percentages of students meeting that alternative standard of growth are lower than the percentage of proficient students in both reading and mathematics, but that the percentages of nonproficient students meeting the alternative standard are higher than those meeting the GMPP standard. Consequently, adding non-proficient students who meet the alternative growth standard to the pool of proficient students could increase the overall rates of students who are arguably performing adequately (i.e., proficient or making reasonable progress over the preceding year).
Study Rationale
The purpose of this study was to determine whether there was any effect, and the kind of effect, of application of the growth models; to provide information about how to strengthen the use of growth under the Elementary and Secondary Education Act (ESEA); and to provide information for states that might consider applying to use a growth model under current regulations. The final report analyzes the effects of growth models in the nine states approved under the pilot, for the 2007–08 school year.
Study Design
The study combined a qualitative description of the Growth Model Pilot Project and the unique mechanisms of the growth models in each of the pilot states, together with quantitative analyses of the increase in the numbers and kinds of schools making AYP under the project and the numbers of schools that could have made their AMOs under a variety of hypothetical "growth only" and other regimens compared to the numbers of schools that made AYP under the original status-plus-safe-harbor model. Simulations were also conducted to identify the impact of different types of growth models on the rates of students being identified as on-track to proficiency and schools meeting their AMOs.
Data Sources
- To determine how states implemented growth models, the study used information provided by states both in written documentation to the U.S. Department of Education (model proposals, reviewers' comments, decision letters and reviews, and commentary by external groups on models proposed for the pilot program) as well as states' own documentation of how they used growth models for accountability purposes.
- To determine the effect on the number and kinds of schools that made AYP, the study used data submitted to EDFacts by participating states, as well as additional school- and subgroup-aggregated data supplied directly by the states. It also used school characteristics contained in EDFacts to provide evidence on the extent to which growth models may disproportionately identify certain types of schools as making AYP by growth. In order to calculate and compare status versus growth-only designations, safe harbor versus growth model designations, the effects of generic models on on-track determinations, and the effects of alternativegrowth standards on student on-track determinations, the study used student-level performance data provided by the states.
Study Limitations
The data used in the study had a number of limitations. The extant descriptions of states' growth models did not always provide sufficient information to explain anomalies, such as particularly high rates of schools making safe harbor or very low or very high rates of schools making AYP by growth. Sometimes these could be explained with additional follow-up contact with the states, but sometimes resources did not permit a full examination of outliers or the actual use of growth model results in accountability systems. In another instance, both school AYP and subgroup AMO results were defined in EDFacts with a set of mutually exclusive categories; thus it was not possible to determine from these data which particular subgroups accounted for their making AMOs as a result of the GMPP growth provisions. This limitation should be corrected in upcoming iterations of EDFacts.
Study Budget
$1.25 million
Contractor
National Opinion Research Center/NORC at the University of Chicago
Report Date
January 2011