![[Contents]](/icons/up.gif){kind=icon}
[Chapter 3: Case Study Summaries Part 4 of 5] Assessment of Student Performance April 1997
In 1991, the Legislative Assembly in Oregon enacted the Oregon Educational Act for the 21st Century, the primary purpose of which was to usher in a new, high-standards education system. As part of establishing the new education system, the state began defining outcome criteria for the Certificate of Initial Mastery (CIM), to be awarded at about the end of grade 10, and the Certificate of Advanced Mastery (CAM), to be awarded at about the end of grade 12. Students were to earn the CIM certificate by demonstrating proficiency in 11 outcome areas (as opposed to simply spending a predetermined amount of time in the classroom). CIM outcomes included: think, self-direct learning, communicate, [use] technology, quantify, collaborate, deliberate on public issues, understand diversity, interpret human experience, apply math and science, and understand positive health habits.
Part of the state's overall reform agenda was to develop an assessment system that consisted of a system of assessment tasks for each CIM and CAM outcome area, scoring rubrics, portfolios for gathering evidence of student work, and a set of criteria for determining whether a student's performance satisfies the requirements of CIM. The state initially hoped to award the first CIM in 1997.
Oregon's Educational Act for the 21st Century proved to be controversial, and, in June of 1995, Oregon's Governor signed into law a bill to amend the Act. Some of the major provisions of the bill included the development of an assessment system composed primarily of reliable and valid multiple-choice tests and performance assessments keyed to traditional content standards and the elimination of the CIM and CAM outcomes.
Crandall High School (the school through which this study investigated the development and implementation of some of Oregon's assessments) was involved with developing performance assessments under the 21st Century Act. However, even before Oregon's legislature amended the Act, Crandall High withdrew from full-scale participation in the reform efforts and from this study in the 1994-95 school year. Like many individuals across the state, Crandall's principal came to oppose the outcomes incorporated in the CIM. In addition, Crandall teachers became overwhelmed and disenchanted by the pace and workload of the reform efforts they had undertaken. However, the school district, District A, of which Crandall High is a part continued to support the reform in 1994-95.
The remainder of this case-study summary focuses on the assessment reform activities which took place at Crandall High School during the 1993-94 school year and in District A during the 1993-94 and 1994-95 school years.
Assessment Reform in District A
In 1992-93, District A volunteered to define and field test assessment tasks for the CIM requirement that students demonstrate proficiency in applying math and science concepts and show an understanding of how these concepts affect the world.
Most assessment development and piloting work in the district was funded by state grants. By the Spring of 1994, the District Student Assessment Team (DSAT), composed of representatives from all schools in the district, had developed and field-tested several performance tasks at each CIM benchmark level (grades 3, 5, 8, and 10). Each task had to reflect the skills and competencies articulated in the state-developed "apply math and science" scoring rubric.
In 1994-95, the district pilot effort had expanded to include a total of 150 teachers and all students at the first three benchmark levels. A new team, the Certificate of Initial Mastery Implementation Team (which replaced the DSAT), coordinated all reform activities and a new sub-committee, Student Performance Assessment Network, worked on developing the CIM assessments. Another team, Science and Math (SCAMA), worked on establishing CIM math and science content standards for grades K through 8.
According to district officials, the state-initiated reform efforts had generated considerable involvement, enthusiasm, and controversy on the part of teachers. Crandall High is a case in point.
Crandall High School
In 1993-94, Crandall High School served approximately 1250 students in grades 9 through 12, most of whom were white. About 35 percent of Crandall graduates go on to attend college.
Work in developing applied math and science assessment tasks was being conducted primarily by two of Crandall's 15 math and science teachers. These two teachers were the first to develop and to pilot assessment tasks in their classes, and they shared the results at District Student Assessment Team meetings.
| TOOLS OF OUR TIME: Applied Technology and Performance Assessment at Crandall High In "Tools of Our Time," an applied technology class, teachers often use performance assessments to evaluate their students' learning and progress. Students work in teams of two or three to complete technology-based assignments in subject areas such as economics and social studies. For example, a student demonstrated his understanding of economics by weaving economics terminology into a story entitled What it Felt Like to Be All Alone for Three Days after a Plane Crash. Using an overhead projector connected to a computer, the student projected the text of his story (which was stored on a diskette) onto a large screen and gave an oral explanation of his story. The story wove in concepts and terms such as capital, labor, land, barter, value, and so on. A question and answer session between the student and his teacher followed the demonstration. |
Some diffusion of the design of performance assessments from math and science into other courses, such as Tools of our Time (an applied technology class), was occurring through informal sharing of ideas among Crandall teachers. However, although the proposed CIM assessment system was forcing most Crandall teachers to take a look at performance assessments, only some — 19 out of 75 — were beginning to incorporate performance assessments into their classrooms. According to one teacher, "most had not accepted the [reform] concept."
These teachers were experimenting with performance assessments primarily within the schools-within-schools programs that enrolled about 23 percent of Crandall's students. (Crandall's high-achieving students typically did not enroll in the school-within-school programs.)
Impact of Performance Assessment at Crandall High School
Teachers who were designing and using performance assessments believed that subject-integration and performance-based assessments offer students a better chance at internalizing what they are learning and at developing higher-order thinking skills.
However, math and science teachers also expressed concerns regarding the connections between general performance criteria articulated in the Apply Math and Science scoring rubric and student work; they were unsure of how to translate the criteria into what could concretely be expected from student work. In fact, the majority of the faculty was not "buying into the concept [of reform]" largely because of a similar issue: a perceived lack of connection between the CIM outcomes and traditional content area outcomes. They viewed this lack of criteria definition (i.e., concrete content standards and related student performance standards) as a hindrance to effective teaching.
In addition, most teachers were concerned about the substantial amount of time it would take to assign and assess performance-based work.
Indeed, these concerns portended Crandall's withdrawal from its CIM development and piloting responsibilities. At the end of the 1993-94 academic year, at the request of the school's math and science teachers, Crandall's site-based committee voted down Crandall's membership in the pilot CIM assessment development program. Individual teachers, nonetheless, continued to develop and score performance tasks.
Crandall students included in this study said they enjoyed performance-based work, mostly because, in many cases, it allowed them to integrate different subject areas. In some cases, performance-based assignments gave students the opportunity to use technology to demonstrate their work. Teachers, in turn, said that their students were better able to learn and develop critical thinking skills as a result of being involved in inter-disciplinary, performance-based work. Teachers also noted that the use of scoring-rubrics to assess their own and their peers' work compelled students to accept more responsibility for, and take more ownership of, their grades.
Future Plans
With the amendment of Oregon's Educational Act for the 21st Century, the state eliminated the outcomes for the Certificate of Initial Mastery. The current state plan is to develop content standards in several subject areas, including mathematics, science, history, and English. The new assessment system will consist of multiple-choice and open-ended assessments that are keyed to specific subject area standards.
District A, however, hopes to continue to develop its own performance assessment system.
By the late 1980s, Kentucky's education system was in crisis, with statistics showing the state ranking near the bottom among U.S. states in (a) per pupil expenditures on education, (b) high school graduation rates, and (c) adult literacy. On June 8, 1989, Kentucky's Supreme Court took official steps to remedy an educational system it perceived as inadequate and inequitable. Essentially, the court directed Kentucky's General Assembly to establish an education system that complied with Kentucky's Constitution.
In 1990, the legislature responded to the court's mandate by enacting the Kentucky Education Reform Act (KERA), which adopted six broad learning goals for all Kentucky students. This act also required that the State Board for Elementary and Secondary Education develop and implement a statewide, assessment program — the Kentucky Instructional Results Information System (KIRIS) — to measure student performance with respect to four of the six goals.
Kentucky Instructional Results Information System
The new KIRIS assessment system has three parts:
The assessments are administered in the 4th, 8th, and 12th grades.5 All schools in Kentucky are held accountable for their students' performance on these assessments; students' scores are figured into the schools' "accountability index." In order to avoid sanctions, schools must improve their accountability index by 18 percent of the difference between their baseline index and 100, over a two-year period.
This study focuses on the portfolio and performance task components of the KIRIS assessments.
KIRIS Performance Assessments
Students in the assessed grade levels spend a year developing their portfolios. A language arts portfolio typically contains entries such as: a personal narrative; a piece of writing that (a) predicts an outcome, (b) defends a position, (c) solves a problem, (d) analyzes or evaluates a situation, person, place, or thing, (e) explains a process or concept, (f) draws a conclusion, or (g) creates a model; a piece of original fiction; and a letter to the Reviewer discussing what the student has learned from keeping a portfolio, which entry is the best piece and why, and from which entry the most was learned. A minimum of one piece of writing must come from a subject area other than English or language arts.
For their mathematics portfolios, students must select their seven "best pieces" that ". . . represent various types of mathematics; employ a variety of mathematics tools such as calculators, computers, or manipulatives; and integrate core concepts within mathematics and the world." In addition, students must provide a table of contents and a letter to the Reviewer.
Each portfolio is scored using a holistic scoring rubric developed by the Kentucky Department of Education (KDE).Classroom teachers complete the initial scoring of portfolios, a sample of which is rescored by Advanced Systems, a private contractor to KDE, to determine scoring reliability.
Performance events assess students in arts and humanities, mathematics, social studies, science, and practical living and vocational studies, but can incorporate knowledge and skills from more than one discipline at a time. Students work in small groups to conduct a short experiment or to solve a puzzle. Performance events are administered by Advanced Systems.
Breckenridge Middle School
Breckenridge Middle School in Lexington was chosen as a site from which to study the impact of Kentucky's performance assessments. In 1994-95 Breckenridge served about 860 students in grades 6 through 8, 80 percent of whom were white, the rest being primarily African American.
Impact of Performance Assessments
At Breckenridge, every 8th-grade teacher included in the study agreed that the new system has benefited students but also has created a great deal of stress and extra work for the teachers. Virtually every teacher in both years of the study said that KIRIS assessments promoted among students the use of higher-level thinking skills to solve open-ended problems. Teachers also felt that students are learning to express themselves better in a variety of written genres.
| Observation of Performance Events Administration Breckenridge 8th-grade students were assigned to eight groups of four for the performance event, which required approximately 45 minutes for group discussion and 15 minutes for individual written responses. The performance tasks administered involved a short experiment or puzzle that students first investigated in groups. For instance, one group was asked to brainstorm four kinds of jobs teenagers could fill to assist the elderly. Students were then asked to construct individual responses that discussed the process their group followed and the reasoning behind the conclusions they drew. They also were asked to describe the task's application to real life, and to discuss in their responses whether and why they agreed or disagreed with their group's conclusions. |
At the same time, 8th-grade teachers felt that their personal and professional reputations are attached to the school scores published in the newspaper and that they receive no recognition when the scores are good, only when they are poor.
The complaint most often voiced by teachers in 1993-94 was the lack of adequate training in scoring, in the design of the portfolio assessments, and in ways to improve their instructional methods to be better aligned with the new assessments.
In 1994-95 Breckenridge teachers raised concerns about scoring reliability. One said she didn't agree with the scores assigned to the Portfolio exemplars by KDE, while two teachers noted that the holistic scoring guides were too complicated to be of much utility. Another felt that it was difficult to assign a reliable score to a portfolio comprised of individual pieces that may be of differing quality.
Teachers also complained vociferously about the amount of lost curriculum time the KIRIS assessments exact. In both 1993-94 and 1994-95, the 8th-grade language arts and mathematics teachers interviewed said they had dropped units from their curriculum in order to focus on elements of the KIRIS program (e.g., portfolio writing or cooperative problem-solving exercises). The language arts teachers said they had stopped teaching important units on grammar, sentence mechanics, and literature in order to do more creative writing in class.
Nonetheless, in 1994-95 both the school principal and teachers themselves felt that 8th-grade instructional strategies had improved as a result of the KIRIS program. One mathematics teacher noted that, "a lot of KIRIS is just what we in the math department and math community have been pushing for years." Another language arts teacher said that, as a result of KIRIS, she was learning to demonstrate to her students the real life application and utility of the principles she taught. Another language arts teacher said that although "KIRIS implementation is very difficult, the end result is worth the effort."
In 1993-94, opinion was divided among Breckenridge teachers about whether instructional changes had spread to 6th- and 7th-grade classrooms. Teachers also disagreed about whether the KIRIS system had affected instruction in mathematics and language arts classrooms only, or in other subject areas as well. In 1994-95, the principal and some teachers indicated that they felt teachers in other grades and disciplines were beginning to align their instructional methods with the KIRIS philosophy, although acceptance of responsibility for KIRIS among the full faculty would take time.
In both years of the study, school and district administrators, as well as parents themselves, felt that parents had not received enough information about the KIRIS assessments. Parents receive their children's performance level in the mail, but no explanation of the performance levels or of their child's weaknesses or strengths is provided. Due to an overall lack of information, therefore, parents do not feel that the KIRIS system has improved their involvement or understanding of their child's education.
Parents interviewed in both years of the study expressed other concerns including:
Students, on the other hand, enjoyed the creative nature of the assessments, noting that they are "allowed to express [themselves]" and that "personal opinions matter." Teachers at Breckenridge felt that students enjoy portfolio writing and the opportunities for group work and problem-solving that KIRIS performance events allowed them. Parents concurred, suggesting that students were more invested in and concerned about their performance on the KIRIS component than they would be on a multiple-choice minimum competency exams.
Future Plans
Despite its relatively promising infancy, KIRIS' future seems to be in question. Conservative groups have criticized the assessments as focusing too heavily on "liberal values" and ignoring fundamental academic skills. Although KDE has invested a considerable amount of energy and time into developing outcomes and associated assessments, there is speculation that the Kentucky legislature may abolish KIRIS before a firm judgement about its utility can be made.
Thoreau High School in Racine, Wisconsin, may be considered an example of what may be achieved in performance assessments, rather than as an example of how to achieve its goals. Thoreau was founded in 1972, setting as its goal to serve students who were not well served in traditional high schools — students who were, in the phrase of the school's founders, "knowledge heavy, but credit light." Thoreau was to be a school in which students were given ". . . control over their academic progress and responsibility for demonstrating their knowledge."
Thoreau today comprises both a middle school and a high school and is characterized by small classes, courses designed for students of all ages, and a wide variety of course selections. Students must apply for admission, and the majority of students who enter Thoreau stay until graduation.
During the 1993-94 school year, Thoreau served 173 middle school students and 226 high school students. At the high school, most students (88 percent) were white, while others were African American (6 percent), Hispanic (5 percent), Native American (1 percent), and Asian American (1 percent).
Rite of Passage Experience
In order to graduate, all Thoreau 12th-grade students must complete the culminating Rite of Passage Experience (ROPE). ROPE — often referred to as the defining feature of Thoreau — requires students to construct a portfolio of essays covering a broad range of subject areas, to write an autobiography and a research paper in U.S. history, and to make a series of presentations to demonstrate their proficiency in 17 subject areas.
ROPE was designed to fulfill the school's objective, which is to allow students opportunities to demonstrate their knowledge and achievement through written work and oral presentation, not simply through satisfactory performance on the district's and state's standardized competency tests and the accumulation of credits. Thoreau has received a waiver from its district's testing requirements, and students' ROPE presentations serve to demonstrate their proficiency in reading, English, government, and mathematics (the four areas covered in the district's standardized tests).
ROPE is organized into two phases during a student's senior year. In the first half of the primary phase, students enroll in the "ROPE Class" to learn about the essay writing process, to write an autobiography, and to complete all of the short essays required for their portfolios. During the second half of this primary phase, students write U.S. history research papers and practice oral presentation strategies.
During the second phase of ROPE, students give their oral presentations before their individual ROPE committees. Each ROPE committee is comprised of two faculty members (one of whom is the student's homeroom teacher), an underclassman (chosen by the student), and one or more adults from outside the school (most students select a parent). Committees and students are reasonably free to organize the presentations however they want. For example, some committees grade presentations on a pass-fail basis, while others assign letter grades; and some presentations are conducted by a group of students, while others are individual presentations.
| A ROPE Presentation For his Fine Arts and Personal Proficiency ROPE presentation, Jeremy chose to demonstrate his interest in music by playing a piece written by Duke Ellington and discussing Ellington's style. Witnessing Jeremy's presentation are the two teachers on his ROPE committee (his music teacher — who is also Jeremy's homeroom teacher — and another teacher with limited knowledge of music) and a group of younger students. During his presentation, Jeremy describes the history and defining characteristics of jazz and explains Ellington's contributions to the jazz form, the musical features of the piece he has chosen to play, and the technical abilities he must possess in order to play the piece successfully. After Jeremy plays the Ellington trumpet solo, his ROPE committee gives him feedback on his presentation. There is no question but that he has passed this presentation (his ROPE committee has chosen to assign only pass/fail grades), and his music teacher tells Jeremy how he measures up against a variety of standards for musical performance, noting that Jeremy "has an aesthetic sense beyond most other high school students." |
Impact of ROPE
Teachers and students alike are enthusiastic about the value of ROPE as a learning experience. They are comfortable with the "loose" design of the ROPE process. Indeed, teachers say they wish the first half of ROPE when students are writing the essays, autobiography, and research paper that comprise most of their portfolios could be structured less formally than it is, but they recognize that their students require the structure of the ROPE class to complete the work.
The school community also is comfortable with the subjective application of standards in the grading of ROPE. Teachers acknowledge — and students recognize — that they adjust their expectations for individual student performance based on what they know of a student's aptitude, achievement, and interests. Students known to be high achievers are expected to perform at higher levels than other students.
Teachers join the staff at Thoreau because they personally believe in Thoreau's philosophy of supporting and enhancing students' knowledge, skills, and individual interests. Although teachers quibble over the details of the process, they are uniform in their enthusiasm for ROPE and its educational value. ROPE does, however, require a tremendous amount of teachers' time. Despite this additional burden, however, teachers agree that the experience students receive with ROPE is worth the extra effort.
Thoreau students also are enthusiastic about their experiences with ROPE. ROPE serves as a unifying force for the senior class, and students feel a real sense of accomplishment upon completing it. As one former Thoreau student said, the ROPE ceremony in May, when students receive certificates of completion for ROPE, ". . . is a much more meaningful ceremony than graduation." According to the school's principal, Thoreau graduates have also found themselves ahead of their peers in college in writing and organization skills. A recent graduate concurred, adding that it is ". . . the school as a whole, not just ROPE, that gives us an advantage. . . ."
Teachers say that parents, who are often members of their children's ROPE committees, are frequently astonished by what their children have accomplished.
Future Plans
Thoreau staff do not envision any major revisions to ROPE in upcoming years, though they will almost certainly continue to tinker with the process. Thoreau's principal and teachers believe that ROPE will continue to promote thoughtfulness in all domains of students' lives, academic and personal.
5In the 1994-95 school year, students in both the 11th and 12th grades took the KIRIS assessment.
-###-
[Chapter 3: Case Study Summaries Part 2 of 5] [Chapter 3: Case Study Summaries Part 4 of 5]