(C)(4) Validating the effectiveness of the State Tiered Quality Rating and Improvement System in improving school readiness

The extent to which the State has a High-Quality Plan to demonstrate the relationship between quality ratings and the learning outcomes of children served by Early Learning and Development Programs, by validating that the tiers in the State's Tiered Quality Rating and Improvement System accurately reflect differential levels of quality, are related to progress in learning and development, and build toward school readiness.

Comments

Strong state plans should include tools and measures used for licensing systems and licensing inter-rater-reliability within the validating process.

On behalf of the Office of the Governor of the State of Illinois, the Illinois State Board of Education and the Illinois Department of Human Services: We recommend that the Administration set forth some of the key indicators of quality that should be included in a QRIS system, including but not limited to, group size, staff child ratios, and teacher credentials.

On behalf of the Office of the Governor of the State of Illinois, the Illinois State Board of Education and the Illinois Department of Human Services: We recommend that the Administration set forth some of the key indicators of quality that should be included in a QRIS system, including but not limited to, group size, staff child ratios, and teacher credentials.

Some questions and notes of caution from the Early Education Initiative at the New America Foundation:

There is no doubt it would be important and valuable to know how these systems correlate with children’s learning and school readiness. If quality ratings are designed to improve children’s chances for success in school, they should be accompanied by evidence showing that, for example, a child who attended a five-star childcare center for two years is having better success in kindergarten than a child who attended a two-star childcare center.

But how would that success be measured? There are no specific mentions of kindergarten-entry assessments in the section so it is a bit murky as to what the agencies want states to do. Are these guidelines leaving a crack in the door for states to use the entry-assessments to reward early learning programs, a move that could lead to a “high-stakes” situation that other sections of the competition are avoiding? Would these results eventually be used by states to pull back funding from “low-performing” preschool programs? Or is this specific section simply designed to be used for research purposes and to guide the design of more-improved systems over time?

Up to this point, there has been limited research on the effectiveness of quality rating systems as a strategy for improving childcare center quality. One study of Colorado’s QRIS by the RAND Corporation included children’s learning outcomes in its effort to determine the validity of the state’s rating system as a tool for improving childcare center quality. Researchers used a variety of multi-domain measures, pre-and-post tests, along with surveys from parents and teachers to provide a comprehensive picture of learning outcomes for a small sample of children. Is this type of validation study what the Departments want states to undertake? Or do they want something that connects individual children to particular centers?

Given that children’s experiences in preschool programs vary greatly it would be complicated to compare readiness levels across programs. Some programs are full-day while others are half-day. Attendance is not compulsory so children might miss multiple days or even weeks. Some programs are year-round others are not. Some children may attend a given program for two or three years in a row, while other children may only participate in the one-year pre-k program.

And thinking about the theme of continuous improvement woven through the guidelines, how would preschool programs know how their rising kindergartners performed? Would the results inform the centers’ practices and early childhood teachers’ instruction in addition to informing instruction in the early grades? Recommendations for this kind data-sharing and collaboration between elementary schools and pre-kindergarten programs are missing from the guidelines.

If the federal agencies’ intention is for states to hold pre-kindergarten programs accountable for how children perform on its kindergarten-entry assessment, then policymakers will need much more research on valid and reliable assessments of young children in multiple domains and much more development of thoughtful plans for adding learning outcomes as a component of states’ QRIS. We hope the Departments of Education and Health and Human Services will provide a clearer vision in the final guidelines for Race to the Top- Early Learning Challenge.

Linking child outcome data to tiered reimbursement is extremely problemmatic. It would be stronger for states to examine the strength of their respective tiers by linking the design to current research that validates the levels and indicators chosen by a state to support their tier structure. There are internal statistical threats to using child outcome data to validate differential levels. The demographic and participatory elements that come into play while attempting to link outcomes to tiers make any validation process extremely weak. States need to be encouraged to use child level assessment for instrucitonal purposes. Separately states need to be concerned with the validity of their tiered design and to use existing research as the link for validation is more appropriate