A r c h i v e d  I n f o r m a t i o n

Number 9

December 1993

Student Portfolios: Administrative Uses

The purpose of the portfolio may be simply to support instruction or it may also be seen to support administrative functions. This Consumer Guide presents information on what has been learned about using portfolios for administrative purposes, some of the problems involved, and some possible solutions to those problems.

HOW ARE PORTFOLIOS USED FOR INSTRUCTION? Many teachers, administrators, and policymakers have learned that portfolios can provide valuable support for quality teaching and improved learning in many ways, including the following:

• Conveying to students, as an extension of instruction and discussion, the features or criteria of quality performance, so that they can apply these criteria to their own work and monitor their own progress;

• Engaging students in activities that are likely to result in products worthy of sharing with others, retaining in a portfolio, and referring back to periodically; and

• Chronicling student work and opening a new channel for substantive communication between students and classroom teachers that is focused on individual student work.

HOW ARE PORTFOLIOS USED FOR ADMINISTRATIVE PURPOSES? While there is a growing understanding of instructional uses for portfolios, they are increasingly being called upon to serve administrative functions as well. Student portfolios are being used for accountability reporting, program evaluations, and a variety of administrative decisions affecting the future of individual students. Both inside and outside of schools, observers are uneasy about what role portfolios, commercial tests, and other assessment tools should play in these administrative activities.

The foremost question being asked is:

• What do we know about the technical adequacy of portfolios for administrative decision making and reporting? How comprehensive are portfolios in covering important cognitive skills? How valid are they for the purposes schools set for them, and for the uses that go beyond these purposes? How reliable are the ratings we assign to a student's portfolio? Would someone else give a different rating? How generalizable are portfolio assessments in a specific curriculum area? Would a different assessment of the same students in that curriculum area yield different results?

  These questions concerning technical quality take on heightened importance because of the potentially enduring effect of various administrative decisions on individual students. A wide variety of administrative decisions (such as retaining some students in grade, providing special services for others, and admitting still others to special programs) affect students' futures with or without assessment information from student portfolios. The issue is whether current portfolio systems are sufficiently informative and technically strong enough for these added functions. If they are, fine. If they are not, teachers and administrators need to understand what would be involved in making them technically adequate.

A second question spins off the first:

• How will using portfolios for administrative decisions and reports affect their utility as instructional tools? Any move to adopt structural and content requirements that may be needed to make portfolios more suitable for accountability, evaluation, and student-level administrative decisions may well have implications for both the attractiveness and utility of portfolios as instructional tools. Here, the crucial question is whether portfolios that have been revamped to satisfy technical requirements can still play a constructive role in teaching for understanding and in motivating students to be active learners? For example, would students work as enthusiastically on assigned projects as they would on projects they were allowed to choose on their own? Would the amount and quality of their work suffer?

Before turning to a discussion of these two concerns, it is appropriate to step back and consider the use of portfolios in administrative decisions and reports.

WHY ARE PORTFOLIOS USED IN ADMINISTRATIVE DECISIONS AND REPORTS? Experience shows that portfolios--as well as any other data source--will be used for any number of administrative matters, with little regard to their original purpose or limitations, simply because they are available at the time information is needed. Moreover, those who have observed how traditional multiple-choice tests narrow curriculum are determined not to tolerate continued dominance of multiple-choice items in any area that would influence curriculum and instruction. Thus, many educators find themselves willing to try portfolios as a way to support reform of both curriculum and assessment.

WHAT DOES THE RESEARCH SAY? Experience with classroom-level portfolio projects shows that many portfolios are currently highly individualized, if not intensely personal. Judged in light of available standards--some district and school policies, court decisions, and professional association standards--many of our existing student portfolios appear to contain too little information for "high-stakes" administrative uses.

Despite the obvious importance of student learning, no single measure of student knowledge--not even richly documented, broad-based portfolios--should be used as a mechanism for meting out rewards and sanctions for students, schools, or programs. Other indicators must be considered for fair and rational decision making. For example, even within the area of student learning, additional information can be gleaned from systematic teacher observations, short-answer quizzes, multiple-choice tests, and other assessment tools.

Practical procedures for addressing technical problems in performance assessments, including portfolios, are discussed below.

WHAT ARE SOME PROBLEMS AND POSSIBLE REMEDIES? Below are some of the problems and possible remedies concerning the use of portfolios for administrative decision making and reporting.

• Students are ill-prepared to carry out work that is a required part of a portfolio. This, in fact, is an ever-present bundle of problems, which extends well beyond portfolios and assessment. Several strategies are needed.
  • If students have not had an adequate opportunity to learn the subject matter and appreciate some of the linkages among various concepts and procedures, any form of assessment--not just portfolios--will be both meaningless and unfair. Addressing this problem will likely involve changes in course offerings, curriculum coverage, and instructional strategies (as well as staff development programs and possibly school finance). Some assessment paradigms (Baker; See Ron Dietel, National Center for Research on Evaluation, Standards, and Student Testing--CRESST, below) mitigate the problem of student differences in background knowledge by providing fairly extensive subject matter material with each task.

  • If students have not been exposed to or had practical experience in applying their subject matter knowledge to novel problems, in solving open-ended problems, or in working on extended projects or projects that require collaboration with other students, they will likely perform less well than students who have had these experiences. This problem needs to be addressed in the same fashion as the previous problem of disparity in opportunities to learn the subject matter.

  • If students have limited proficiency in reading, writing, and speaking English, they will likely perform less well on examinations requiring these skills than students who have a command of the English language. This problem may be particularly severe for writing tasks and open-ended problems that require written or oral explanations. Various approaches to the problem are possible, including bilingual support, translated materials, and opportunities for students to tackle some problems in their primary language. Some researchers (O'Neil et al; See CRESST, below) are also developing and pilot-testing simulated tasks that retain the content and conceptually challenging character of regular performance tasks but place less linguistic demand on limited-English proficient students.

• Different students have worked on different tasks or projects that are not comparable either to each other or across classrooms. Under the worst circumstances, this common situation is like the classical problem ofcomparing apples and oranges. Several things can be done:
  • Develop general criteria that encompass a wide variety of projects and products, and train teachers and other raters to use the criteria;

  • Include "on-demand" tasks that all students complete as part of their portfolio collection;

  • Arrange to have more than one rater for each of the portfolios; and

  • Use more than just portfolios as the source of evidence of student accomplishments.

• Teachers have used different criteria for rating portfolio work or come up with different scores even when they use the same criteria. This problem can be solved with training, planning time, and the involvement of teachers and other people with subject matter and instructional expertise. Teachers need to be involved in the development of a common set of criteria and the selection of rubrics that very specifically define performance. Research shows that under these conditions teachers and other raters can be trained to rate student work consistently (referred to as high inter-rater reliability).

• Teacher guidance and peer review are different in different classrooms. Where comparisons across classrooms are desired, fairness compels educators to develop rules about teacher guidance and peer review. So many changes are required in reforming curriculum, instruction, and assessment that even the most experienced, most knowledgeable teachers need training and an opportunity to work things through with other teachers and subject matter experts. Reaching agreement on how much and what type of guidance should be given on assessed work can be a part of these training sessions. Such staff development is absolutely essential where every teacher is involved and comparisons are to be made.
  • Peer review can also be troublesome, since students placed in higher ability groups would likely receive more helpful peer input. As a consequence, many teachers and assessment experts have argued that peer review has to be given up if student-level comparisons are to be made equitably.

  • Parent, sibling, or other help may also present a problem in assessing student performance based on portfolio projects that extend over a period of time. Sending notes home with guidance for parents has been one approach, and student honor codes have been another.

• Students have worked on only a small number of tasks. Because performance tasks and extended projects take time to plan and carry out, many portfolios contain a small number of products. However, since not all tasks are alike, it is difficult to specify how many should be required. But researchers (Shavelson et al; See CRESST, below) have learned that about 10 tasks are needed to assess a student's understanding of a particular subject area, such as science.

  With fewer than 10 tasks, we can only judge how a student did on the particular tasks undertaken (the student might perform quite differently on a different set of tasks). This problem of limited generalizability of tasks can be addressed by increasing the number of tasks for all students or by not relying solely on portfolio work to judge a student's accomplishments.

  Occasionally, fewer tasks would be needed if each task came with fairly extensive passages of task-related information, such as those used by some researchers (Baker, See CRESST, below) to assess deep understanding of history, social science, and science. On the other hand, more tasks would be needed if the tasks were less carefully structured or less carefully researched, or the content area to be assessed were defined more broadly, for example, mathematics and science combined.

  Increasing the number of tasks in a portfolio may not be a bad idea anyway: It would give additional emphasis to student production of papers and other work products. In terms of administering tasks or assigning work, the 10 might be carried out over an extended period of time as a continuing cycle of instruction, performance, and assessment. At the opposite extreme, 10 tasks that require 15 minutes each might be administered in a single morning at the high school or junior high school level.

In addition to these problem-specific strategies, several general strategies have been used to buttress the technical underpinnings of portfolios--that is, training raters to criterion (a pre-established standard of acceptability), continued in-service training for teachers, periodic sharing of portfolios across classrooms, auditing, and various research and development activities.

WHAT INSTRUCTIONAL UTILITY DO TECHNICALLY STRENGTHENED PORTFOLIOS HAVE? Where the sole purpose of portfolios is to provide instructional support or curriculum reform, they and the rules that govern them can be created and changed by students in collaboration with their own teacher. Adding administrative uses to portfolios results in an increasing standardization and at least a partial shift in ownership. The shift is away from individual students, teachers, and classrooms, and to the education system in general--a broader but less well defined audience.

A student's sense of ownership of his or her portfolio may well be linked with interest, motivation, and actual engagement and learning, but this is no reason to conclude that students must have complete control over their own portfolios to make portfolio systems work. Some compromise between centralized structure and local, classroom-level discretion may work just as well.

Moreover, a variety of other factors may be equally important in fostering student motivation and learning. More experimentation and research may provide an answer to this controversy. Meanwhile, giving priority to staff development and equity issues--which is essential if portfolios are to be used in administrative decisions and reporting--can be an area of agreement and an important step in advancing student performance.

Who is working in this area?

• Winfield Cooper is editor of the quarterly Portfolio News (a publication of the Portfolio Assessment Clearinghouse). It provides 20 to 30 pages of articles, project briefs, and other materials by teachers, project directors, and researchers about local and state portfolio projects and serves as an information exchange for people interested in portfolios.

• PROPEL is a continuation of ARTS PROPEL, a cooperative research project involving the Pittsburgh Public Schools, Harvard Project Zero, and Educational Testing Service (ETS). Throughout both stages of the project, portfolios have been used along with classroom observations and external assessments to assess learning in three content areas: imaginative writing, music, and the visual arts. Information on the PROPEL/ARTS PROPEL approach is now available from ETS in four handbooks: a general overview handbook and one for each of the three content areas. The handbooks describe program and teacher strategies and illustrate student production, perception, and reflection in projects that extend over time. PROPEL has also used an audit procedure to verify portfolio ratings. (See PROPEL/ARTS PROPEL, below.)

• Several states are using student portfolios in combination with other information on student accomplishments in their accountability systems. For example, Vermont is assessing 4th and 8th grade students in writing and mathematics using three methods: a portfolio, a "best piece" from the portfolio, and a set of equivalent performance tasks. California has also launched 21 pilot projects (11 with portfolios) involving the collaborative efforts of school districts for improving alternatives in assessment. Kentucky will be monitoring schools on the changes, over time, in their percentage of successful students and has established an elaborate system that includes portfolio work for measuring success.

• Lauren Resnick and Marc Tucker are co-directors of the New Standards Project. They are developing a new assessment system to support world-class standards of performance for all students. The system employs advanced forms of performance assessment, such as portfolios, exhibitions, projects, and timed performance examinations. Among its partners are the following states: Arizona, Arkansas, California, Colorado, Connecticut, Delaware, Florida, Iowa, Kentucky, Maine, New York, Oregon, South Carolina, Texas, Vermont, Virginia, and Washington.

• Dozens of research projects are investigating new forms of assessment, including portfolios. Most of those cited in this Consumer Guide were carried out by CRESST researchers, with funding from OERI, the National Science Foundation, or both. A listing of all large projects in this area is maintained by CRESST.

Where can I get more information?

Dale Carlson
California Department of Education
721 Capitol Mall
Sacramento, CA 95814
(916) 657-3011

Winfield Cooper
Portfolio News
Portfolio Assessment Clearinghouse
San Dieguito Union High School District
710 Encinitas Boulevard
Encinitas, CA 92024

Ron Dietel
CRESST--National Center for Research on Evaluation, Standards, and
Student Testing
UCLA 145 Moore Hall
405 Hilgard Avenue
Los Angeles, CA 90024-1522
(310) 206-1532

Joe McDonald
Coalition of Essential Schools
Brown University
Box 1969
Providence, RI 02912
(401) 863-3384

Richard P. Mills
Commissioner of Education
Vermont Department of Education
Montpelier, VT 05602
(802) 828-3135

New Standards Project
Learning, Research and Development Center/University of Pittsburgh
3939 O'Hara Street, Room 408
Pittsburgh, PA 15260
(412) 624-8319

PROPEL/ARTS PROPEL
Pittsburgh Public Schools
341 South Bellefield Avenue
Pittsburgh, PA 15213

Ed Reily
Office of Assessment and Accountability
Kentucky Department of Education
19th Floor Capitol Plaza Tower
500 Mero Street
Frankfort, KY 40601
(502) 564-4394

Ed Roeber
Council of Chief State School Officers
1 Massachusetts Avenue, NW
Suite 700
Washington, DC 20001-1431
(202) 336-7045

This is the ninth Education Research CONSUMER GUIDE--a series published for teachers, parents, and others interested in current education themes.

OR 93-3107
ED/OERI 92-38

-###-