TOPIC 12.
Development of Web-Technology Structures to Facilitate
State and Local Education Agency Reporting
Through Performance-Based Accountability Systems
and Comparisons of Schools and Subsets
of Student Populations

• Web-Based Test Score Analysis Database Engine
• Integrated Data Reporting System Initiative
• Distributed Spatial Object Visualization Network
• Web-Based Educational Data Collection, Analysis, and Reporting
• Development of a Web-Based System for Comprehensive Analysis of Public School Performance
• Technology to Strengthen School-level Analytical/Reporting Capability
• Using the World Wide Web to Fuel Expansion of Education's Best-Performance Frontier

SPONSORING OFFICE:
Office of Educational Research and Improvement

Abstract:
Most schools across the nation administer achievement tests to measure progress of their student population. There are many firms/organizations that specialize in the preparation, testing, and scoring of these tests. These testing firms/organizations generally provide the school, school district, and state with hard and soft copies of the test results. Each school receives a large volume of information making it impossible to digest and retrieve meaningful statistics except at a very high level. Cimarron's Web-Based Test Score Analysis Database Engine will allow the school, school district, or state to load the soft data, provided by the testing organization, into a central data repository. Once the data has been loaded, anyone with proper authority can view the data in graphic format. The database engine can display historical trend data for a specific test given to a grade across multiple years. It will also be possible to chart the progress of a group of students on a specific test across many years. Additional graphs will allow users to compare test scores of a group of school districts or a group of schools. Authorized personnel will also be able to analyze and chart individual student data.

Summary of Anticipated Results and Implications:
As stated above, there are many firms or organizations that give achievement tests and in almost all cases the test results are available hard and soft copy. For Cimarron's Web-Based Test Score Analysis Database Engine to be successful it must be able to accept the soft data from a variety of sources in a variety of different formats. Cimarron has already worked with soft data from several of these tests. The Phase I Work Plan discusses a comprehensive list of tests we plan to support. The anticipated results can be found below.

For Phase I:

• Evaluate soft data format of all tests. Design a database capable of holding information from tests.
• Obtain actual test data for two or three of the tests and build an import function to load data into the data repository.
• Work with a test state and identify a set of graphs and charts that would be helpful to the state, school districts, and schools.
• Develop a prototype of a web-based data analysis engine that will display the information in several different graphs and charts.
• Design a security system, which can be customized to fit the varying needs of each state.

For Phase II:

• Develop data import functions for most of the remaining tests.
• Develop a comprehensive set of graphs and charts.
• Implement security system designed in Phase I.
• Develop a web-based data import function to allow schools and school districts to import their data over the Internet.
• Develop a web-based data modification function that will allow schools to update their individual student data, logging all updates for audit purposes.
• Develop a web-based export function that will allow a school to extract all data for archive purposes and/or importing to other applications.

Abstract:
The electronic reporting process of student assessment results in New York State requires immediate attention. Data integrity is questionable as many data inputs are manually "derived". For example, under New York State's school accountability program, schools are accountable for students who have been enrolled between the first Wednesday in October and the date of the assessment. Although there are only a few variables necessary to make such a determination, districts are not prompted for the low-level variables, instead a "yes/no" response is solicited. Manual interpretation of such basic fact is unnecessary and increases the likelihood of error. Inaccuracies in core data structures are compounded rapidly as new variables are introduced. Before further development of web technology structures can be considered, the core issue of data integrity must be addressed. During Phase I, CCSI will conduct a pilot program designed to explore alternative options for capturing, managing, and ultimately reporting school data.

Summary of Anticipated Results and Implications:
It is anticipated that at the conclusion of the Phase I pilot, an accurate assessment of workflow similarities and differences between pilot districts and will be made. Whether data is indeed being captured at the lowest level of detail will be determined. Common workflow patterns between and among districts will be will be identified and common database design structures will be further defined. CCSI will determine whether IDRS meets to needs of school districts in its present form, whether it requires intense or moderate modification, and whether a uniform IDRS will inspire more practical workflow in local school districts.

Instead of finding a way to link existing, redundant systems in a logical manner, we are reassessing current workflow thus enabling school leadership to address system issues. We anticipate this approach to be well received by local district superintendents who recognize the need for strategic change. Rather than introducing yet another "add on" or "Band-Aid" system, we are offering a solution that requires total system analysis, and yields a new organization design capable of responding quickly to change.

The Integrated Data Reporting System Initiative is intended for Public and Charter school use.

Abstract:
ScienceSoft has developed several distributed object spatial systems for DOD. These advanced systems use object oriented technology to provide remote access to diverse data sources across the Internet. This distributed spatial object technology (DSOT) provides an ideal mechanism for comparing performance and standards based testing at the school, district, and state level. This spatial technology will allow educators, administrators, parents, and business leaders to compare student performance between schools, districts, states, and to national and even international standards. The system will be able to compare not only performance data spatially (i.e., district to district) but also across time to identify trends in education performance.

Summary of Anticipated Results and Implications:
Successful completion of Phases I and II of this project would have far reaching implications. For the first time, administrators and educators will be able to see areas in their districts and state where students are exceeding or failing in a geographical sense. Performance data coupled with other socioeconomic data would allow leaders to focus resources into specific areas to address deficiencies and to identify successes and duplicate their conditions across the state.

Abstract:
Recent interests in utilizing the Web for instructional purposes have been mainly concentrated in the development of web-based instruction. Web-based instructional technology has already demonstrated its ability to effectively carry out numerous educational and training functions. One obvious advantage of using the web for education is its ease to gather massive amounts of educational data from students, instructors, and parents, analyze the data, and report the results. By applying, data warehousing and data mining technologies to analyze educational data one can gain various insights and understandings of instructional processes. For example, these technologies can help discover the reasons for the low performance of American students and pave a way for improving the quality of education. In this project, we propose to explore the technology of data warehousing and data mining to collect and analyze educational data. The research will develop a model for application of data warehousing and data mining for education, and will implement a web-based educational data collection, analysis, and reporting tool.

Summary of Anticipated Results and Implications:
Successful completion of the proposed project will provide a fundamental framework and environment for improving the quality of Web-based instructions. The final outcome of project will be generic enough to be used across various educational domains.

Potential applications of the proposed technology are numerous. An example application is the analysis of K-12 educational data to discover why students in the US are falling behind their contemporaries in other countries in both content and performance level in mathematics and science.

Abstract:
Policy and resource allocation decisions by school administrators, school boards and legislative bodies are increasingly focusing on performance-based accountability systems to achieve specified goals and standards. ubstantial amounts of data relevant to school and student performance are currently available in separate databases compiled and routinely updated at the school district, state and national levels.

A web-based system developed by SIScom of America, Inc. , and the University of West Florida for measuring performance in higher education institutions could be adapted to assist state and local education agencies in collecting, integrating and analyzing various sets of data, and in preparing timely and "user-friendly" reports to support a comprehensive performance-based accountability and planning system.

This proposed project will extend current R&D efforts of the University of West Florida College of Professional Studies focused on accountability and reform initiatives by providing performance, demographic, and other data along with the protocols for data manipulation to decision makers in schools and districts as they strategically plan for the future for schools, teachers, and students.

In concert with previous related efforts, the SIScom/UWF team has designed the proposed Phase I activity as a cooperative effort with state and local education agency representatives.

Summary of Anticipated Results and Implications:
None provided.

Abstract:
Today's education reforms, focused on standards, performance, accountability and school-based management, have created a significant need for improved information management and analytical capacity at the school level. Principals and other decision-makers in schools are facing increasing pressures to improve student test scores and school quality indicators and are facing heightened accountability consequences. This project will conduct research aimed at improving school level data management, analysis, and reporting capability with software and technical support.

Summary of Anticipated Results and Implications:
A prototype for software that can operate on multiple platforms; help schools monitor, evaluate, and report student academic progress and school performance; and strengthen the data and statistical decision making capability of principals and other decision makers in schools. Our vision is of technology in the service of "total quality management" and "the high-involvement model" for school-based management - practices which research shows lead to improved student achievement and better schools.

The commercial applications are significant since there is a large, recognized need in this area and few successful solutions. We believe the market is particularly significant among schools and school systems currently lagging in their use of technology as identified in recent White House and Department of Education communiqu?s regarding "the digital divide."

Abstract:
Synchronous Performance, LLC (SP) proposes developing a web-based structure to distribute the results of a performance effectiveness and efficiency analysis of the Lancaster, PA and DeKalb, GA schools. Other methodologies are limited to describing performance relative to one objective at a time, apply arbitrary weights, or ignore necessary mathematical assumptions. Complex relations between multiple desired outputs and multiple resource requirements limit analytic methods used in statistics and other operations research methods. The educational process has multiple simultaneous objectives. This fact is highlighted whenever resource choices have to be made to simultaneously optimize results in language arts, math, and science. Frontier Analysis™ (FA™) will produce a single effectiveness and efficiency measure of the educational processes in these two school districts. Several characteristics make the combination of FA™ and web-based performance reporting an ideal methodology to drive performance improvement. FA™ quantifies a measure of efficiency for each entity by describing each resource's contribution towards best performance. By contrasting best performing organizations with others, insights of best performance can guide resource decisions to migrate lesser performing organizations to the best performance frontier. The World Wide Web represents the ideal means to increase visibility of performance comparisons among schools and student populations.

Summary of Anticipated Results and Implications:
At present, the key measure of a school district's performance is student performance on standardized tests. Without question student achievement is an important and reasonable measure of a school district's performance but one has only to read a newspaper or magazine to know it is not enough. Politicians, community leaders, and parents are increasingly vocal in their demands for measures to reassure them that resources are being used effectively. To date no such measure exists and therefore the educational community is left defenseless when accusations of waste and ineffectiveness are made. The performance measurement system we are proposing will end this controversy. Although Phase I represents only a small proof of concept it will unequivocally demonstrate our ability to identify which organizations are producing the best outcomes for the resources they have available. More than that, just as FA™ has guided strategic resourcing decisions in manufacturing and finance, it will help educators chart the most direct path to the best-performance frontier.

We believe Phase I will create a compelling dialogue that further contributes to growing interest in improved educational performance measurement. We hold this belief because the demand for outstanding educational results is ubiquitous. Our approach adds dimensions to the accountability equation, which have never before existed in education. As communities become aware of its presence through the WWW, they are going to want their schools to join those who have already been assessed. Some of these schools are going to redefine the best-performance frontier; others are not. When communities realize their schools are using more resources to produce the same or less student achievement, they are going to want changes. Since this measure can provide performance comparisons from school through national levels, a large market potential exists. As a business, we provide a full range of services to help our clients become more efficient. FA™ will help describe, on an individual basis, which schools, school districts, or states are consuming a disproportionate level of resources to produce their student's achievement. But improvements will not come from our FA™M alone, but educational organizations will require assistance in creating the changes to produce the improved performance that has been identified. Synchronous Performance provides the full range of services they will require to identify best practices from schools on the best performance frontier, translate them into actionable projects, and establish the measurement systems to monitor their progress and outcomes.

[Topic 11] [Topic 13]