Teaching Fellows Ask, “How do we make sense of the PISA results?”

Earlier this month, the OECD (Organization for Economic Cooperation and Development) released the results of the 2009 PISA assessment. Every three years, PISA assesses the reading, mathematics and science literacy of 15-year-old students in 65 countries and education systems around the world. PISA seeks to answer the question that many educators grapple with daily, “How well can students…apply their knowledge to real-life situations?”

Though our average science score is up from 2006, U.S. performance on the PISA has been largely stagnant in reading and math. Currently, the United States ranks 17th among all participating countries and education systems in reading scores, with Shanghai, Korea, Finland, Hong Kong, Canada, New Zealand, Japan and Australia among the top performing. There has not been a change in the average reading scores of American students since 2000.

Though the U.S. spends more per student than almost every other OECD nation, we do less to target spending on low-income schools and students. There are other major differences between the U.S. and the highest performing systems as well, especially when it comes to teacher recruitment, compensation, evaluation and professional development. We want to hear what educators and students think about these findings:

How much emphasis should we place on these results? How might they help shape reform efforts in the U.S.?

~Antero, Edit, Jemal, Jeff, Katie, Laurie, Leah, Linda, Lisa, Nick, Pam, Patrick, Tracey, Stephanie and Steve
Teaching Ambassador Fellows

To read Secretary Duncan’s speech in response to the PISA results, visit: http://www.ed.gov/news/speeches/secretary-arne-duncans-remarks-oecds-release-program-international-student-assessment-

12 Comments

  1. The PISA results are not reliable. If the instruments used to test are not identical, then the results cannot be compared. PISA and the Dept. of Ed. can rationalize all they want to about the methodology. These results are patently unfair, and this test is being used for political purposes.

  2. These results are important and highlight some serious gaps in American education. However, it is frustrating that some key points are never addressed when discussing the validity of the findings. One major issue is that in the US, we educate all of our students within the same basic format. Most of the other countries in the study separate out the students with college ambitions, and these are the students that are tested for the international comparisons. So we are looking at an entire spectrum of American students and only the top performers in others. In addition, some of these countries do teach to the test in order to perform better in standardized situations. I have been lucky enough to visit schools in Germany, Japan, Turkey and China, and many of these schools admire the US model of more creative and project-based learning that can not easily be tested using A,B,C and D. There is no question that reform is needed, but effective reform must be based on accurate information.

  3. Vic:

    I appreciate your comments and did as you suggested. The criteria for being tested on he PISA according to the website you mentioned? Age.

    According to the comments on the PISA methodology, the structure of the education system and the type of schools do not allow for a viable comparison, which is of course, my point exactly. I have taught at a Gymnasium and a Haputschule in Germany and A KodungHakkyo (College Bound High School) and a Kongopkodonghakkyo (Technical High School) in Korea and can assert without reservation the assumption that using only age as the primary consideration, without ensuring that the type of school system is taken into consideration, is flawed. To better understand some of the concerns take a look at what Oxford noted about the PISA test.

    http://www.beteronderwijsnederland.nl/files/active/0/Prais%20cautions%20PISA.pdf

    The following is the explanation given from the PISA website you recommended.

    The PISA student population
    “In order to ensure the comparability of the results across countries, PISA devoted a great deal of attention to
    assessing comparable target populations. differences between countries in the nature and extent of pre-primary
    education and care, in the age of entry to formal schooling, and in the structure of the education system do not
    allow school grade levels to be defined so that they are internationally comparable. Valid international comparisons
    of educational performance, therefore, need to define their populations with reference to a target age. PISA covers
    students who are aged between 15 years 3 months and 16 years 2 months at the time of the assessment, and who
    have completed at least 6 years of formal schooling, regardless of the type of institution in which they are enrolled,
    whether they are in full-time or part-time education, whether they attend academic or vocational programmes, and
    whether they attend public or private schools or foreign schools within the country. (For an operational definition
    of this target population, see the PISA 2009 Technical Report [oeCd, forthcoming].) the use of this age in PISA,
    across countries and over time, allows the performance of students to be compared in a consistent manner before
    they complete compulsory education. ”

    The Oxford article clearly outlines some of the concerns about the methodology used. I continue to maintain that some schools do not include “special schools” where as some do. In this article, Britain did not include them and naturally scored higher than Germany which DID include students form their Sonderschule (special schools). These practices continue to this day and are not clearly delineated in the website your reference. Interestingly, in the discussion on how schools are selected, it is not at all clear HOW they are selected. If you know where that information is, please share as I am very interested in seeing it. I will continue to look. I am currently completing my doctorate in curriculum and instruction with Texas A&M University and will use the substantial resources to nail down precisely how each country chooses the schools AND students that participate. As I mentioned above, my first hand experience in schools in Korea wherein I watched half the students leave during testing time makes me really doubt that unless there are monitors at all schools at all times, that the Koreans did not employ the same tactics during the administration of the PISA. I have to believe what I saw and until it is proven without a doubt to the contrary, and no amount of desultory comments or insinuations to the contrary will change that simple fact.

    But that’s not all…

    There are other OECD comparisons similar to the PISA that are suspect. Take a look at Higher Education systems and how community colleges are viewed in 27 different countries. According to the OECD process described by the Institute for Higher Education Policy linked below, I would be counted as a failure because I did not graduate from the Community College I attended. Never mind that i transferred to Florida State University and graduated with a B.A in German and History, got an M.A. from Michigan State University, an Ed. S. from Argosy University, plus a teacher’s license from the University of Hawaii and am currently working on finishing up a doctorate from Texas A&M University in Curriculum and Instruction and Educational Leadership. I am still counted as a drop out in this methodology! As are many of my friends who did similar types of transfers.

    http://www.ihep.org/press-room/news_release-detail.cfm?id=172

    Thanks for your post.

    Todd

  4. Check the bottom of the website under “Post a Comment” and a lot of your questions will be cleared up. “Bad Behavior has blocked 12621 access attempts in the last 7 days.” The United States Department of Education does not want to hear criticism. It wants to hear excuses and rationalizations. Lets take the first commentor “Melanie” for example. It is clear that she is parroting the accepted dogma of the Department of Education. Therefore her comment has been cleared. The truth is that memorizing certain facts is essential. If a student cannot give multiplication facts out of memory then they are truly at a disadvantage.

    I am not a teacher. I am a Civil Engineering major attending a major university. I have struggled greatly with Calculus as do many of my peers. The common factor is the inability to “plug and chug” problems. To sit down for hours on end and put out one poblem after the other. You may hold the opinion that using Hershey bars for multiplication is a good idea, but in reality it accomplishes two things: distracting the student from the task at hand, and further crippling their ability to focus on a problem and work through it logically. It is true that students must be able to make sense of a problem. But this comes from the use of math/reading etc.. in every day life. I doubt that a student will break out a bag of Hershey bars when he/she wants to make change at the grocery store.

    Parental involvement as well as our education system are the failures. Knocking the teaching methods of the past doesn’t help either. They helped our nation accomplish numerous things. And in an educationally declining society I don’t think that anyone is justified in doing so. Obviously the new methods aren’t making things better. Just the 2 cents of a college student. Not that it will matter as I am quite sure this comment falls under “Bad Behavior”.

  5. Maybe all of us should read the OECD-PISA 2010 report. It describes the methodology in great detail, as well as the results. Why would so many authorities in the corporate world, in science and at our universities, not challenging it, but people who know nothing about it do?

    The worst thing we could do is not face reality. Why do you think we lost so many industries? In addition, we are losing market share to imports in 111 of 114 key industries. How do you think that happened?

    Talk to our universities. They will tell you what happened to American high school graduates, and how well they are doing in college.

    Then look at what country spends the most dollars per student per year. The US dept of education, or ACT or OECD will tell you that although we are one of the top 4 spenders, our high school output has gone down and down while many other countries improved their methods. Our own companies have been complaining for five years now that the high school grads cannot communicate in English properly and cannot even do basic math.

    It really doesn’t matter what we think, if we have not studied the facts. Arne Duncan is right on the money, and we better sober up and get our kids to work harder, and our schools to learn why the top international performers are successful. And please…forget the selective education argument. Just read the PISA 2010 report first.

    This is not a public opinion topic folks. This is about the future of our country. That requires opinions that are fact based, and that needs a bit of research.

  6. Having experience with the French education system, I agree with both Todd and Elizabeth. Beyond the fact that only select students take these tests, I think that it is also important to consider the implication of groups trying to get Americans to improve their PISA scores. The reality is that we have mainstreamed most of our student populations and are therefore testing individuals who sould be weeded out out by other educations systems. This lowers our scores but speaks directly to the American core belief of living in a meritocracy. As American’s we have the ability to recreate ourselves at any point of our lives. Many other countries are very proud of how they are able to offer inexpensive upper education to their qualified students but they do not explain the price. In many industrialized nations, students start taking aptitude tests early and are pigeon-holed by 9th grade for the rest of their education. To compound this, many of these same systems do not allow for personal change. If a student graduates from medical school, they become a doctor. There is no room to change one’s mind to become a computer programmer instead.
    The American system of education my be out-dated in some respects but it still values the personal direction of the learner and respects the students choice to start over as something new. The PISA scores are more of a justification for a national tracking system of student aptitude than a useful measure of what students are learning.

  7. I find Todd’s comments particularly interesting and would also like to see other information about the schools which participate in the PISA assessment. I am the principal of a school and do not know of any schools in my area which have been assessed so I am not familiar with it. If it is administered similarly to the NAPE, I have grave concerns. In previous teaching positions I have seen the apathy that takes place when these assessments are given to students who are already overly assessed and see no need or reason to perform on yet another assessment. Like Todd, I know many nations to not include all students in their education system assessments as we do in the U. S. schools. Many schools assessed in some of the countries that are touted as being so far ahead of the U. S. do not give equal educational access to all students. In many places, only the highest achieving students have these testing opportunities, so let’s be fair when we draw conclusions.
    I do strongly believe that our schools need to seek to improve and should not be satisfied with our present education system, but I am also skeptical when it comes to these types of international assessments.

  8. Living in a globalized world, we don’t have any other alternatives, but to trust PISA or any other international test. Certainly, there are some minor details that need improvement.

    I think Mr. Duncan, Secretary of Education, made a good synthesis (Dec. 7, 2010):”I have every confidence that America can capitalize on our strengths, adapt where we need to, and learn from others. We’ve done so before”. What is missing is to do our “homework”to provide the missing tools to be able to operate in the 21st Century. And the first requirement is to portray our global reality (based mostly on empirical experiences but with a unified conception and at two levels: A with 20 components or less and B with 200 components or less)and our national reality. This will lead us to identify and position our education system in the real world, and most importantly encourage teachers for additional learning and students for getting involved. Then, “our world is our classroom” could become operative and apply for example,”learning by doing”, only seed money needed, making lesson plans relevant for everybody. Automatically, we will reduce the 636 classrooms full of students that drop out of high school every week!

  9. I have taught at a Korean public school as well as at DoDDS schools (Department of Defense Dependent Schools) in Germany, Korea, and at the Area Office (similar to State Level) in Okinawa, Japan. I find these comparisons to be more problematic than one might imagine. For example, when I taught in Yongwol, South Korea at a public school, students in the lower half of each grade were asked on the intercom not to come to school the day they were testing so the school’s scores would not be low. Furthermore, I really want to know what type of students we are testing when we do the comparisions. If we are taking students from a German Academic High School (known as a Gymnasium) who are hand-picked and are being prepared to attend German Universities and comparing them to American High School students full of all types of students, this would not considered a fair or valid comparision. The other two major types of schools in Germany, Realschule and Hauptschule, graduate students by the 9th or 10th grade so only the cream of the crop are available for comparison with the entirity of the American student population.

    It is for this reason that I am skeptical. I’ve also attended Universität zu Bonn in Germany as a student and taught for a summer at Kangwon National Univeristy in Chuncheon, Korea. To suggest that the students of either of these schools, both major universities in their respective countries, had students vastly ahead of the American students I teach in the Department of Defense Schools in Japan, Germany and Korea is simply not true. I am very interested in this topic and would love to know what is being measured in these comparisons.

  10. One of the most important factors in educational success/failure has nothing to do with all the splendid new programs and methods we have been asked to use in the classrooms. If we truly wish to begin seeing more success in our classrooms, we need to begin influencing what is happening within the walls of the homes. Our president, as well as our national and state educational leaders, must be seen (often) by the public encouraging the parents to provide climates where their children can learn. A place to study quietly, high expectations expressed by parents, rewards from the parents for attaining agreed-upon goals, communication with the teachers and administrators in the schools, parental access to daily-updated grades, lessons, and announcements: these will have a phenomenal effect upon many of our struggling students.
    As a teacher, I have been receiving better results from my students as I communicate more with the parents. Many of them had no idea of what resources are available to them from our school. The better the communication between the teachers and parents, and the better we attempt to work as a team with the parent, the greater will be our success in getting the students to take their studies more seriously.

  11. It has always been my belief that ALL students can learn. Test do not accurately show what a students is capable of doing. Differentiation of instruction is a great help to all. I know that ALL students do not learn the same yet, they are given the same test in the same exact manner.The emphasis should not be placed on the results. In my opinion, test are somewhat bias. For example, a child who has not been exposed to vocabulary and comes in a classroom at the age of 7 without prior background can be quite difficult. This is why there are so many SST cases because the students can learn but they come to school lacking vocabulary.
    I am also an educator and I am tired of hearing about the comparisons of our students to other countries. This seems to me as if we are comparing apples to oranges. I love the fact that we try to educate ALL of our students. But, we are the ONLY country who tries to educate everyone. So, if we are looking at data it seems to be quite invalid. There seems to be a disconnect with education and the people who calling all the shots when it comes to education. Teachers are not involved in any of the decision making.
    I agree NCLB is making some strides to assist with student achievement but I think there is an accountability piece missing when it comes to parents. Parents should be held accountable just as much as teachers. Sometimes it seems as if school is a place to babysit students.

  12. I doubt this news is a surprise to those educators, who have been keeping up with the NAEP. Perhaps, it reinforces the statistics we already have. Certainly, no one wants to know how poorly our children are, “falling through the cracks”. A few years ago (2006 or a few years prior), the statistics for students in 12th grade being able to read at a proficient level was only 36%. Clearly, this disgusts me; and we MUST do something about it.

    The only thing that I know to do is to teach using BEST PRACTICES. Educators must be kept up-to-date on what research states is most effective. In my opinion, all educators must get on the same page, leaving behind the former ways of teaching. NCLB specifically stated to use effective vocabulary instruction.

    Research states that children learn about 2,000- 3,000 new words each school year. In order for these words to stick, children must own each word. Construction of vocabulary helps students to have ownership of these words. Drawing pictures to convey the meaning is an easy way for teachers to check for understanding. In addition, children must hear the words said correctly. Oral fluency can be demonstrated through a simple game of, “I Have, Who Has”.

    To increase children’s reading levels, they must read more. If it is too hard, then students will become frustrated. If it is too easy, then children will become bored. However, a low reader can be encouraged by reading a few easy books. It is very important for children to not have a negative experience in regards to reading. This will cause a child to HATE reading. Once a child reaches this level, it can take years to recover. In some instances, adults do not enjoy reading for this reason. One last thing, students of ALL ages need to have a model reader; one who reads with fluency and models decode strategies when the model reader comes upon a hard word.

    Rote memorization is the lowest level of learning according to Bloom’s Taxonomy. Certainly, it has its place, but it is the lowest form of knowledge. Children must do more than memorize definitions and their multiplication facts.

    Research states children must make sense out of Mathematics. It is no longer thought to be best practice, when students are, “told what to think”. This comes from the Constructivist point of view, where teacher takes on the role of being the facilitator of knowledge. Instead of the classroom being teacher centered, it becomes student centered.

    An example of, “making sense out of Mathematics” would be for a teacher to have hands on manipulatives for students to feel and touch. Perhaps, a teacher is about to teach multiplication to her students. Instead of pulling out the flash cards or make students write off their multiplication tables, she might give each student a Ziploc bag filled with square plastic tiles.

    The teacher may tell the students that those plastic tiles are called Hershey Bars. Then, the teacher would ask the students to put three groups of Hershey Bars on their desks, with four Hershey Bars in each group. Clearly, the teacher would not say, this is: 3 X 4 = 12

    Rather, she would ask them to show her the various construction of Hershey Bars into groups. Division can occur the same way. The teacher would ask the students to take out fifteen Hershey Bars. Next, the teacher would ask the students to construct how many Hershey Bars three friends can share equally. Again, division or the division facts are not connected at this level.

    The sequencing usually goes from hands-on to pictorial representations. Once, they get a knack for constructing multiplication and division facts, the light bulb pops on. The last step in the process is the actual pencil and paper method. Students must make a personal connection to knowledge, in order for it to stick.

Comments are closed.