A r c h i v e d  I n f o r m a t i o n

Assessment of Student Performance April 1997

CHAPTER 7

Part 1

CROSS-CASE ANALYSIS 4:
IMPACT OF PERFORMANCE ASSESSMENTS
ON TEACHING AND LEARNING

The previous chapter discussed the factors involved in "teacher appropriation" of assessment technologies. In this chapter, we extend the notion of "teacher appropriation" of assessment technology and discuss what we have learned about how teachers appropriate and extend performance assessments to change the pedagogical strategies they employ in their classrooms. We also discuss the impact of performance assessments and changed pedagogical practices on student learning. The discussion presented in this chapter is based primarily on three sources of information — student work, classroom observations, and interviews with students, teachers, parents, and principals. The data are indicative of the potential impact of various types of performance assessment systems on teaching and learning.

Organization of the Chapter

The discussion of the impact of performance assessments on teaching and learning is organized into the following sections: (a) teaching, including curriculum, instruction, and teacher role; (b) learning, including motivation to learn, thinking skills, and writing skills; and (c) barriers to effective classroom use, including lack of time and lack of clearly articulated content and performance standards.

Our data indicate that an important factor in understanding the impact of performance assessment systems on teaching and learning is the type of assessment tasks used to influence pedagogy. In this sample, interviewees involved in assessment systems composed of portfolios or extended performance tasks tended to report the most extensive effects on teaching and learning, while interviewees involved in implementing only on-demand assessment tasks tended to report the smallest number of effects. Therefore, we discuss the impact of assessments on teaching and learning by the types of assessment tasks being implemented at each site. 1

Limitations to the Analyses

Certain limitations must be noted with regard to discussing the impact of performance assessments on teaching and learning. First, teachers and students at most sites reported changes in teaching strategies and learning outcomes due to the influence of performance assessments. The absence of such reports at some sites does not necessarily imply the absence of similar teaching strategies and learning outcomes at those sites; it only means that assessments were not appropriated and, therefore, did not drive pedagogical changes. Second, the impact data for the state- and district-initiated systems are valid only for the school site included in this study; the results obtained for a particular school site are not generalizable to other sites developing or implementing the same assessment system. Thus, for example, impact information collected for the Kentucky performance assessment system is true only for Breckenridge Middle School in Kentucky, not for all Kentucky schools. Third, the impact on teaching and learning is not necessarily schoolwide; the impact is primarily on students and teachers involved in the assessment systems included in this study. Fourth, some teachers involved in the assessment system may have profoundly changed only one aspect of their pedagogical practices, while other teachers may have tried to alter their pedagogical practices completely but not very successfully. Therefore, in this study, it is not possible to evaluate thoroughly the depth, breadth, and quality of pedagogical changes.

Last, in some cases, the effects of assessment systems on teaching and learning may be minimal because the assessment system is at a very early development stage, or the effects may be washed out due to the presence of other competing assessment systems. The PWC Applications Assessments are in the early implementation phase, and, hence, have not been institutionalized at Westgate Middle. In the case of Walters Middle and Manzanita High, the state-level performance assessment systems (composed of on-demand tasks) are administered once a year. However, both of these schools are located in districts that have implemented their own performance assessment systems. Thus, the state-level assessment systems compete with the locally-developed assessment systems in influencing the teaching and learning processes at these schools.

The cross-case information contained in this section is intended to provide a framework for understanding the general relationships between assessment systems and classroom outcomes. It is not meant to describe comprehensively the impact of any one assessment system on any one school or school system; a representative sample of schools and teachers utilizing the assessment system would be required for such analyses.

Teaching

Our discussion of impacts on teaching is divided into the following sections: curriculum, instruction, and teacher role (see Exhibit 7-1).

Curriculum

Our information on the impact of performance assessments on the classroom curriculum is based upon interviews with teachers and principals. One limitation to our analyses is that such interview data may not be complete and reliable. Teachers may not have been willing to talk about how much they might have changed the curriculum they implement in their classrooms, as diverging from established guidelines and curricular frameworks may have negative professional consequences for them. Therefore, our information with regard to curricular changes may not be complete.

Keeping this limitation in mind, we still find some interesting patterns in our data. Some sites using portfolio assessments show some significant shifts in the curriculum teachers use in their classrooms, while most sites using on-demand assessments exhibit only marginal, if any, impact of assessments on the classroom curriculum.

Curricular Changes

At four sites using portfolios, teachers reported two types of changes in the curriculum they implement in their classrooms:

Teachers at Hudson High, Maple Leaf Middle, Cooper Middle, and Breckenridge Middle reported that, as a result of appropriating and using portfolios and extended performance tasks, they covered certain curricular topics in more depth than they had in the past. However, they also had to curtail the coverage of certain curricular areas, because of the time pressures resulting from implementing the assessment system. For example, at Breckenridge Middle, the 8th-grade language arts and mathematics teachers noted that they had to drop units from their curriculum in order to focus on portfolio writing and cooperative problem-solving exercises. The language arts teachers said they had stopped teaching units on grammar, sentence mechanics, and literature in order to focus on creative writing in class.

Such enhancement and curtailment of curriculum has evoked a variety of teacher responses. For example, at Hudson High, because the Regents waiver courses and assessments are locally developed, the curtailment of content coverage is by design and, therefore, is not viewed negatively by teachers. Hudson teachers are pleased about the fact that their students learn and retain in-depth knowledge of certain topics rather than encounter and often forget a wide range of topics in a given area.

At Cooper Middle School, however, full appropriation of locally developed performance assessments (and thematic teaching) led to some problems. Cooper teachers discovered that their focus on thematic units and project-based assignments took time away from certain math topics they would otherwise have covered. These teachers realized that, because they spent large amounts of time on student portfolios and on coordinating thematic lessons with other teachers, their time preparing for and teaching mathematics and science was being compromised. In the 1994-95 school year, therefore, the problem of not covering enough mathematics was rectified by reverting to a traditional mathematics class for all students. This change from the innovative to the traditional exemplifies the difficulties inherent in creating and appropriating a new system without adequate professional support.

At Manzanita High in Arizona, changes in the classroom curriculum were not a result of appropriating and extending the assessment system. Rather, the curriculum was inadvertently affected by the logistics of implementing the on-demand, state- and district-level, external assessment systems. Manzanita teachers did not appropriate the Arizona assessments, but reported that test-preparation and administration time, combined for both the state- and district-level assessment systems, robbed them of instructional time; one English teacher reported that he had to eliminate some books, such as The Grapes of Wrath and The Great Gatsby, from his curriculum.

No Curricular Changes

At the sites where no changes to the curriculum are indicated, teachers offered two types of reasons for why their classroom curriculum had not changed as a result of the performance assessment system being implemented at their schools. These reasons are:

At several sites using portfolios or extended performance tasks — Ann Chester Elementary, Ni?os Bonitos Elementary, Noakes Elementary, Park Elementary, Thoreau High, and Sommerville High (which are participants in national- and school-initiated assessments), and McGary Elementary and Windermere Elementary (which are participants in district-initiated assessment systems) — curricular changes did not follow changes in the assessment strategy. At these sites, assessments and curriculum were reformed, and are continuing to be reformed, in conjunction with one another. For example, at Park Elementary in New York City, teachers use the Primary Learning Record (PLeR) as an instrument to support the child-centered philosophy of teaching espoused at the school. At another school, Ni?os Bonitos Elementary, assessments were reformed in order to support changes in the curriculum. For example, for language arts, Ni?os Bonitos teachers first articulated the learning outcomes and, for each language developmental level, specified in detail what students should know and be able to do with regard to oral language, reading, and writing. Knowing they wanted a portfolio assessment system, teachers then spelled out what the language arts portfolio at each language development and age level should contain.

Thoreau High is unique in this group of reform-minded schools. It is a veteran of school reform whose performance assessment system was thoroughly institutionalized and integrated with its curriculum several years ago. Hence, Thoreau teachers tend not to use the assessments to drive changes in their curriculum.

In the cases of Walters Middle and Westgate Middle — sites with external, on-demand performance tasks — teachers noted no changes in their curriculum driven by the subject performance assessment, as they had not appropriated the assessment system. In both cases, on-demand performance tasks are administered once a year and, thus, have not been incorporated into the classroom on a regular basis. In the case of Walters Middle, the impact of state assessments is confounded by the fact that the district in which the school is located is implementing its own on-going performance-based assessment system. Walters teachers mentioned that they used the curriculum-embedded district-level assessments in their classrooms. (According to district officials, the district assessments are embedded within new district curricular frameworks that are quite similar to the state curricular frameworks.) Several Manzanita teachers, too, mentioned that they had long ago adapted their teaching to the district's performance assessments. However, as noted earlier, some changes in curriculum came about due to the time involved in test preparation and administration.

In the case of Westgate Middle, PWC, the assessments are in the piloting and early implementation phase, and, hence, any impact on teaching and learning is not yet evident.

Instruction

A wide array of teaching methods, practices, and technologies fall under instruction. It is in this broad area of pedagogy that teacher appropriation of performance assessments and the effects of teacher appropriation are the most evident. We discuss the nature of these instructional changes under the following categories:

Notice that all of these instructional methods and tools also may be integral components of the assessment process itself.

Performance-Based Assignments

The extensive use of research-based or performance-based projects that integrate writing, content knowledge, and social or scientific problem-solving marks a noticeable shift away from textbook-based assignments at several schools involved in assessment systems comprising portfolios and extended performance tasks. Performance-based projects are extended tasks that typically require students to research a topic and demonstrate their understanding of the topic through essays, exhibitions, oral presentations, and experiments. Thus, students must actively seek information, exercise some judgment regarding the parameters (e.g., topic, focus, length) of the task, structure the information, and transmit the information in a comprehensible fashion. Exhibit 7-2 provides some examples of such tasks.

To a large extent, this project-based instructional mode is driven by the nature of the assessments themselves — portfolios and extended performance assessments require student products that are based on assignments that extend beyond writing answers to questions from the textbook. Teachers at most schools working with assessment systems that include extended performance-based tasks noted that they had changed their instructional approach as a result of using the assessment system. Through the use of performance-based projects, teachers are attempting to foster in their students analytical thinking and multidisciplinary understanding.

At five of the 16 schools visited — Ni?os Bonitos Elementary, Westgate Middle, Walters Middle, Manzanita High, and Park Elementary — the use of performance-based assignments did not come about as a result of the assessment system. At at least two of these schools — Ni?os Bonitos and Park Elementary — teachers already were following an instructional approach that functioned independently of the assessment system.

For example, at Park Elementary in New York City, teachers observe student behavior during structured play and listen to student conversations to determine what literacy, scientific, or mathematical concepts the student might be ready to learn next. Thus, students are continually engaged in "projects," — making cookies, writing in their journals, creating lego designs, and sewing and painting — and teachers are continually engaged in "assessing," using classroom observational methods. Thus, use of the Primary Learning Record has provided Park Elementary teachers with a powerful, non-intrusive assessment method that works synergistically with "project-based" instruction. Indeed, teachers claim that the Primary Learning Record can work as a substitute for IEPs, leading to individualization of instruction. At both Ni?os Bonitos and Park Elementary, although teachers do not necessarily model their instructional strategies on the assessments, they do value the information they gain from the use of the assessment system.

At the other three sites — Westgate Middle, Walters Middle, and Manzanita High — teachers have neither appropriated the assessments nor devised performance-based projects modeled on the district- or state-level performance assessments. At Westgate Middle, PWC's Applications Assessments are still too new to have affected instruction. Westgate teachers also indicated that they were waiting for the district's new curriculum before investing much time into modifying instruction. On the other hand, at Walters Middle and at Manzanita High, the assessment systems have not permeated the classroom in any meaningful way. (All three of these assessment systems comprise on-demand tasks that are administered once a year, requiring little teacher and student involvement on a regular basis; in fact, one Westgate teacher speculated that the most likely effect of the Applications Assessments is that "there will be another test-taking strategy to be taught.")

Writing

An increased emphasis on writing skills has been especially powerful in several schools as a direct result of two factors:

Not surprisingly, in classrooms where language arts portfolios are in use, both teachers and students mentioned that students are required to engage in an extensive amount of writing. For example, at Maple Leaf Middle, VT, and Breckenridge Middle, KY, several teachers are placing more emphasis on teaching the writing process (i.e., writing, editing, and rewriting), and are including drafts of student work in portfolios. Teachers also mentioned that the current student cohort is required to write much more than were earlier student cohorts.

Teachers at these schools reported, however, that they are struggling with how to strike a balance between what they see as "the mechanics of writing" and the stylistic, communicative, and expressive aspects of writing. For example, at Maple Leaf, the 8th-grade language arts teacher switched from concentrating mostly on the creative aspects of writing in 1993-94 to paying more attention to grammar in 1994-95.

Whether the writing process has actually led to good writing, however, is still open to debate. In fact, according to teachers at Breckenridge Middle, KY, because "mechanics" of writing and grammar receive minimal weighting in overall scoring of the writing portfolio, the current system actually fails to encourage writing that is good in all aspects.

At only two sites where portfolios or extended performance tasks are in use, Crandall High and Windermere Elementary, teachers and students did not mention that they were placing greater emphasis on writing skills than they previously had. It is quite conceivable that at both of these sites, the emphasis on other aspects of research-based projects overshadows the writing part of these projects.

Group Work

Group student work is another instructional strategy and classroom organizational mode commonly employed at schools engaged in assessment systems comprising portfolios or extended performance tasks. According to teachers at these schools, group work blends in nicely with the use of extended performance-based projects that require several procedures or steps to complete. However, variation in the structure (i.e., what students do in groups) and the composition of student groups, as well as in the length of time these groups are sustained, is considerable. Cooperative exercises range from division of labor coupled with individual accountability for a common student product to peer assessments of individual student work. For example, the Pacesetter teachers at Sommerville require students to work together in groups to complete math problems; however, each student must record his or her own answer to the problem. At McGary Elementary, HSD2, students are encouraged to perform peer evaluations based upon scoring rubrics specially designed for students.

Only at two of the schools using portfolios or extended performance tasks, Thoreau High and Windermere Middle, have teachers not placed a premium on requiring students to work together in groups. The assessments at these sites require individual students to research a topic (or, in the case of Thoreau, topics) and to present their research findings. Hence, group work is not a natural extension of the assessment system.

At Westgate Middle, Walters Middle, and Manzanita High, teachers have not consciously decided to use group work as a result of the district- and state-level assessments systems.

Scoring Rubrics

Our data indicate that the use of scoring rubrics for pedagogical purposes is the most visible aspect of teacher appropriation of assessment technologies. It also is the most significant, as performance assessments are the only likely pathway through which the pedagogical changes we discuss below could take place. Teachers use scoring rubrics with students to:

At state- and district-level sites, teachers use the state- and district-generated rubrics and also design their own rubrics; at school-level sites, teachers design and use their own rubrics; and at national-level sites teachers use the NSP and Pacesetter rubrics, but also design their own.

In Vygotskian terms, the scoring rubric has become a "scaffold" teachers use for setting performance expectations and standards for their students. Prior to assigning work for extended projects or portfolios, teachers share scoring rubrics with their students to communicate the important aspects of the work to be assessed and to empower and engage students in the learning process. Thus, prior to beginning their work, students become aware of, and can take into consideration, the criteria to be used in judging the quality of their work. At sites using extended performance tasks and portfolios, the use of scoring rubrics has brought all students into the same "fold" — into the same frame of reference — for judging their own and their peers' work.

At Hudson High, Maple Leaf Middle, Crandall High, and McGary Elementary, teachers require students to assess their own and their peers' work using scoring rubrics. Teachers at these schools mentioned that students are much more engaged with the assigned task and better able to internalize performance standards and task requirements when they use the scoring rubric to judge their peers' and their own work.

In fact, at two elementary schools, Ann Chester and McGary, to truly engage students in the assessment process, teachers ask students to devise their own scoring rubrics to use with the assigned projects. One McGary teacher was pleasantly surprised by how student-empowering scoring rubrics are: "I have never before had a student come to me with a B and ask, 'How do I get an A?'"

Only at Ni?os Bonitos Elementary, Thoreau High, and Sommerville High do teachers not use scoring rubrics with students. Ni?os Bonitos's scoring rubrics may be too complicated for elementary school children, and Thoreau and Sommerville teachers have not devised any written scoring rubrics.

Our sample of assessments shows that there is a tremendous amount of variation in the quality and types of scoring rubrics in use at the classroom level. Some of the sampled assessment systems include generic rubrics that specify the dimensions of knowledge or areas of importance to be assessed in different types of student work. These systems include Kentucky's, Oregon's, Vermont's, South Brunswick's, Ni?os Bonitos Elementary's, and New Standards Project-influenced systems at Ann Chester Elementary and at Noakes Elementary. Our data indicate that generic rubrics function as powerful pedagogical guides, as teachers must design and use assessment tasks that elicit the skills and competencies articulated in these rubrics. Other rubrics are tailored to specific performance tasks and detail the characteristics that must be present in those tasks. Assessment systems in Kentucky, Maryland, Hudson High in New York, Harrison School District 2, Prince William County, Cooper Middle, Ann Chester Elementary, Noakes Elementary, and Sommerville High (for formal Pacesetter assessment) include specific rubrics. (Note that both types of rubrics are in use at some sites.) Still other scoring rubrics (mainly devised by teachers for use at the classroom level and not a part of the formal assessment system) are more like checklists that communicate to the student the "items" that must be present in his or her work. Teachers in this case tend to use "existence proofs" (i.e., presence of such items or characteristics), rather than the quality of the information present in order to assign grades. (Such checklists are helpful to the student, but they cannot be meaningfully referred to as scoring rubrics.)

Technology

In our sample schools, the use of technology for instructional or assessment purposes has been, so far, minimal. Only Cooper and Crandall are well equipped with computers, scanners, and video cameras and regularly use these technologies for teaching and learning. Both schools have staff members who have been recently trained in the use of technology for pedagogical purposes. At Sommerville High, teachers and students in Pacesetter classes regularly use graphing calculators, since its use is a Pacesetter requirement.

At Maple Leaf Middle, Ni?os Bonitos, and Breckenridge Middle, teachers are attempting to integrate technology into daily teaching and learning practices by requiring their students to use a wordprocessor on a regular basis for their portfolio writing assignments. Technology, however, has not been fully exploited and integrated into instruction to pace student work, to provide access to information, or to create a different (e.g. "floating" or "open") classroom organizational structure.

Teacher Role

The initial promise of reform included the transformation of the teacher's role, from one that conceptualized teachers as disseminators of information working in isolated, individual classrooms, into one that envisioned teachers as professionals needing the feedback and support of their fellow professionals in order to competently fulfill their job of teaching and to advance their own learning. As discussed below, evidence indicates that the appropriation and extension of performance assessments, coupled with professional support, has facilitated some changes in that direction. We discuss these changes under:

Teacher Collaboration

The necessity for devising structures and methods to establish common frames of reference for assessing student work and to discuss dilemmas associated with using performance assessments has fostered considerable teacher collaboration at schools using portfolios and extended performance tasks. Such collaboration, in fact, has been greatest in the case of assessment systems that require teachers to coordinate the work that is incorporated into student portfolios or the work that is distributed across several classes and judged by a group of teachers.

At Cooper Middle, the use of thematic units and composite student portfolios (for cross-class assignments) has compelled teachers to plan together and to discuss the types of assignments they would like their students to complete. Similarly, at Maple Leaf, Ni?os Bonitos, Ann Chester, Thoreau, Windermere, and Park Elementary, the necessity of coordinating student assessment has fostered collaborative activity. At Sommerville High, teachers using the Pacesetter mathematics program collaborate in order to learn Pacesetter curricular and instructional strategies from one another.

Breckenridge Middle School in Kentucky is an exception to the finding that portfolios have fostered collaboration among teachers. Although the Kentucky Department of Education encourages teachers to collaborate on scoring portfolios (through various blind scoring methods), only the language arts teachers mentioned doing so. It is unclear whether Breckenridge teachers collaborate on a regular basis for designing the assignments that must be included in student portfolios.

At three elementary schools — Ann Chester, Park Elementary, and Ni?os Bonitos — melding performance assessments with the philosophy of a child-centered approach to education has resulted in bringing teachers together to discuss issues of instruction and curriculum. For example, at Ann Chester, one teacher group interested in literacy read and discussed Windows into Literacy.

Teacher collaboration, it must be noted, also is affected by another factor. It only exists in supportive environments, where time is available to teachers to discuss the implications of assessments. At Ann Chester, Park Elementary, Ni?os Bonitos, and Maple Leaf Middle, such seeds of collaborative efforts have been firmly planted. Teachers meet on a regular basis to discuss assessments, instruction, individual students, and the current literature on teaching and learning. In other places, however, the virtual absence of professional support for modifying and using state- or district-developed assessments and the lack of direct participation in the reform activities has translated into little collaborative effort. One-shot (i.e., performance events administered once a year), external systems in places such as Westgate Middle, Walters Middle, and Manzanita High have not accorded the on-going opportunities needed for collaborative work. (We will return to this issue of time and professional support in the final part of this section.)

Teacher Creativity and Initiative

Related to teacher collaboration discussed above, another finding is that performance assessments, when coupled with organizational and structural changes, have fueled teacher creativity and initiative in planning lessons, assessments, and classroom organization. Teacher creativity is especially evident in the variety and scope of projects these teachers assign and in the structures they devise to learn from one another. Teacher creativity and initiative were evident at Hudson, Maple Leaf, Ni?os Bonitos, McGary, Cooper, Crandall, Ann Chester, Thoreau, Sommerville, Noakes, McGary, and Windermere.

For example, at Ann Chester Elementary, a math teacher uses chess as a medium for teaching problem-solving skills. At another school, Hudson High, the Earth Sciences teacher has devised a year-long "pet rock" assignment: students must keep a detailed scientific journal containing their observations, inferences, and predictions about their rock's characteristics, genesis, metamorphosis, geographical location, relationship to the earth's structure and weather patterns, and commercial value.

An example of teacher initiative comes from Maple Leaf Middle. After she had become comfortable with the language arts portfolios, the 8th-grade language arts teacher took the initiative to help teachers of other subject areas move in the direction of performance-based instruction. For example, she helped the social studies teacher develop social studies performance-based assignments and a checklist for scoring the assignments.

Such effects on teacher creativity and initiative have been particularly noticeable with the use of portfolios. However, when structural changes such as provision of time for planning and collaborating with other teachers is not provided, the result is teacher frustration and exhaustion. (As noted in the previous chapter, one of the powerful facilitators of teacher appropriation of assessments is the presence of school-level support.)

Teacher Observational Skills and Understanding of Students

As a result of collaborative efforts and effective appropriation of performance assessments, some teachers specifically mentioned that they had acquired a better understanding of some of their students through their students' performance on assessments. For example, the use of mathematics portfolios helped the 8th-grade mathematics teacher at Maple Leaf understand a child with limited facility in mathematics. Another example, albeit an atypical one, comes from Westgate, PWC. Although PWC's Applications Assessments are only in the piloting and early implementation phase and have not been adopted by Westgate teachers for instructional purposes, a special education teacher gained a better understanding of the instructional needs of her hearing impaired students when she observed their reactions of frustration to the on-demand assessments. She realized that she had to place more emphasis on teaching her students language skills in conjunction with critical thinking skills, as the Applications Assessments require the simultaneous application of reading (and, therefore, comprehension), writing, and problem-solving skills.

In addition, teachers at three of the six elementary schools included in this study — Ni?os Bonitos, Ann Chester, and Park — noted that they had honed their observational skills through the use of assessments. Note that teachers at all three schools emphasize a child-centered approach to education, and observations of young children's behaviors are an important aspect of their pedagogical repertoire. Thus, the new assessment instruments fit nicely into their pedagogical framework. Pacesetter teachers at Sommerville High also noted that they had improved their observational skills.

1Note again that release time provided for attendance at specific professional development activities is not included in this discussion of school organizational factors; rather the current discussion is limited to release time that is provided to support the reform but for which teachers direct their own activities.


-###-

[Chapter 6: Cross-Case Analysis 3: Part 3 of 3] [Contents] [Chapter 7: Cross-Case Analysis 4: Part 2 of 2]