![[Table of Contents]](/icons/up.gif){kind=icon}
A primary goal of international comparative studies of achievement in mathematics and science is to evaluate the level of performance of students in different countries. An equally important goal is to attempt to understand the bases of differences that emerge. In prior international comparative studies conducted by the International Association for the Evaluation of Educational Achievement (IEA), knowledge of mathematics and science was measured by paper-and-pencil tests, and efforts to understand cross-national differences in achievement were made through questionnaires given to teachers. These methods also constituted the primary means for gathering data for IEA’s Third International Mathematics and Science Study (TIMSS). Again, questionnaires were given to teachers and additional data were gathered through questionnaires given to students, parents, and experts in mathematics and science education.
As is the case with all methods used for the collection of behavioral and attitudinal data, questionnaires have both strengths and weaknesses. They are the obvious choice when it is necessary to collect large amounts of data on an array of topics at the least expense. But interpretation of data from questionnaires is often more difficult than when there is an opportunity to interact with the respondents and to probe for details or elaboration of answers, as is possible with case studies.
Case studies, which involve observations, long conversations, and interviews, are time-consuming, require highly trained examiners, and are necessarily more expensive than questionnaire studies. Even so, the ethnographic methods employed in case studies help us to understand the context and relationships that lie behind quantitative data. Case studies are especially useful in research involving international comparisons, for it is in this type of research that the need for bridging the perspectives of insider and outsider is greatest.
The advantages of including ethnographic studies as a component of international comparative research in education has been described succinctly in a publication of the Board on International Comparative Studies in Education:
There is a great need for small, in-depth studies of local situations that would permit cross-cultural comparisons capable of identifying the myriad of causal variables that are not recognized in large-scale surveys. In fact, much survey data would remain difficult to interpret and explain without the deep understanding of society that other kinds of studies provide. Given that research in cross-national contexts benefits from increased documentation of related contextual information, it would be useful to combine large-scale surveys and qualitative methods. (Gilford, 1993 p. 22)
In line with these arguments, a Case Study Project was included as part of TIMSS to provide in-depth information about the beliefs, attitudes, and practices of students, parents, and teachers that complements and amplifies the information obtained through the TIMSS questionnaires. The case studies involved interviews, conversations, and classroom observations in three countries: the United States and two of its leading economic competitors, Germany and Japan.
Before embarking on a project as complex and expensive as case studies of the process of education in three countries, we tried to become familiar with relevant background information. Government and lay publications provided helpful descriptions of the school systems, and a review of the relevant German, Japanese, and American literature dealing with the correlates of academic achievement yielded additional information. This review is available as one of the TIMSS publications (Stevenson, Lee, & Nerison-Low, Eds. 1998).
Even though we originally planned to have at least four researchers spend a year obtaining the case study data, it was obviously impossible for them to cover all aspects of education, including all grades and the full array of relevant topics. We limited ourselves, therefore, to grades 4, 8, and 12, which approximated the grade levels covered in the TIMSS study, grades 3–4, 7–8, and the end of schooling, and to four topics which were proposed by the U.S. Department of Education because of their importance in current discussions of education policy. The topics were
In order to help the reader understand the structure of the educational systems in the three countries, a brief description of these systems also is included.
When a case study involves only a single individual or a single setting it is difficult to evaluate the degree to which the results constitute a valid indication of what may exist beyond the individual setting. We believed it was necessary, therefore, not only to conduct the study in different locations within each country but also to include different schools within each location. It was also necessary to seek the participation of a reasonably large sample of individuals within each of the locations. Replication across individuals provides the basis for making claims of external validity, for replications demonstrate the resilience of explanations in both similar and dissimilar settings. We realize from the beginning, however, that our descriptions and discussions are derived from the beliefs, attitudes, opinions, and experiences expressed by the persons with whom we interacted or observed.
In order to interpret our findings it is important to understand how the case studies were conducted. The most important elements are described in the following paragraphs, and detailed descriptions are available in the volumes written about each country (Ashwill & Nerison-Low 1998; Hofer 1998; LeTendre 1998).
Because what occurs in schools and families often differs from one region of a country to another, three locations considered to be prototypic were selected after long discussions with representatives from each country. Our goal was to select research sites that were as comparable as possible among the three countries in terms of such factors as size, geographic distribution, and economic base. Because logistical problems make it very difficult to arrange for work in rural communities, the primary sites where most of the data were collected were large metropolitan areas with populations of several million persons that also provided ready access to smaller nearby communities. In addition to the primary site in each country, two secondary sites also were selected. All sites were located in different regions of each country: East, Central, and West in the United States; East, Central, and Southwest in Germany; and North, Central, and South in Japan. The secondary sites were smaller than the primary sites but had populations of at least several hundred thousand.
We selected schools at each site in consultation with local education authorities. Within each site an effort was made to select a range of primary and secondary schools that represented successful, average, and less successful schools in terms of such indices as scores on achievement tests, scores on college or high school entrance tests, and percentage of students entering colleges or universities. To select parents within each classroom we relied on school personnel, mainly the principal and teachers, to advise us about which families to approach in order to obtain representative groups of families to include in our study.
It is useful to indicate the extensiveness of our work. In Germany, we conducted over 366 hours of interviews with 199 parents, teachers, students, principals, counselors, and assistant principals. A total of over 494 hours of interviews were held in Japan with 247 persons. In the United States, we conducted over 542 hours of interviews with 271 persons. In addition, discussions were held with members of the national ministries of education in Bonn and Tokyo, and with staff members of Japan’s National Institute of Educational Research. These discussions helped us to understand the organization and operation of the respective nation’s schools and the kinds of policy issues of special concern to the respective governments. Supplementing the interviews and conversations were 255 hours of observations of science and mathematics classes in the three countries.
In addition to the core information derived from the interviews and observations of classroom practices, researchers were expected to take advantage of special opportunities as they arose. They were instructed, whenever possible, to observe school activities, such as PTA meetings, lunch hours, teachers’ meetings, in-service training sessions, extracurricular activities, and sports day. They were also expected, to the degree possible, to collect samples of in-class tests and daily assignments, report cards, art products, writing samples, magazines for children and teenagers, and other materials relevant to gaining an understanding of background factors that may play a role in students’ education. Descriptions of these events and materials were circulated among the researchers through computer notes in a three-country computer network.
Transcripts of these interviews, conversations, and observations gave us the information needed to describe many of the cultural and contextual factors related to academic achievement in the three societies. Obviously, only a portion of these data can be described in this general overview of the findings. Full descriptions of the findings appear in the volumes written about each of the three countries (Ashwill & Nerison-Low 1998; Hofer 1998; LeTendre 1998).
Identifying the researchers who would be primarily responsible for conducting the case studies was a major task. In addition to being fluent in German, Japanese, or English, they were required to have resided previously in the country where they would be conducting research. They were also required to have a background in education or the social sciences, experience in using ethnographic methods, and the time available to spend several months in residence at the research sites.
We were successful in recruiting an exceptionally able and experienced group of researchers. Most researchers had a Ph.D. and had conducted their dissertation research in the country to which they were assigned for the case study. All of the researchers were born in the United States, with two exceptions. One researcher in Germany and one in Japan were native speakers of German and of Japanese who had attended schools in the United States. In addition to the full-time researchers who spent 2 or 3 months in the field, interviews and observations in the United States also were made by other staff members, including Hispanic American, an African-American, and several other native-born Americans. We also invited a Japanese psychologist to accompany the researchers in the major American site for 1 month in order for us to gain his perspective on what was observed. As a result of these procedures, a total of 19 persons served as researchers in various parts of this study.
The researchers in each country spent between 2 and 3 months at the major research site and were responsible for covering one of the four topic areas. Shorter periods (approximately 2 weeks) were spent at the secondary research sites by two of the full-time researchers in each country. Because of the shortness of the visits it was necessary for the researchers visiting the secondary sites to cover more than a single topic and to have a smaller number of contacts with students, parents, teachers, and education authorities.
To ensure comparability in the three countries, a week-long meeting was held to discuss the structure and procedures of the study before the researchers began their fieldwork.
Our purpose in conducting the case studies was to describe the education processes that exist in a sample of cities in each of the three countries. We did not attempt to test any particular theory or set of hypotheses. Rather, the interviews, observations, and conversations provide needed information about how three industrialized countries cope with several critical issues in education. To be sure that relevant topics would be included, we spent several months compiling lists of questions and topics that each researcher was expected to explore during the fieldwork. Our own earlier research in the three countries provided us with a great deal of information that was helpful in defining specific topics for exploration (e.g., Stevenson & Lee 1990; Stevenson, Chen, & Lee 1993). Rather than rely solely on our own ideas, however, meetings were held with specialists in education from each country to ensure that the coverage would be appropriate, comprehensive, useful, and up to date. Participants in these meetings included policymakers, researchers, educators, and representatives of national groups, such as those concerned with education, science, and mathematics.
Researchers were not expected to follow any particular order in gaining information, but were expected to let the interactions flow naturally. All interviews were tape recorded, and in the case of Japanese and German were translated into English. The transcriptions were then entered into the computer. Verbatim notes from the classroom observations also were stored in the computer.
Compiling computer files that represent the thousands of sheets of paper necessary for recording the observations and interviews is a critical step, but the usefulness of files depends on the ease with which information can be retrieved. By having information readily available in an organized fashion it is possible to establish trends with much greater confidence than if every general statement required a new review of all of the field notes. Moreover, illustrating particular points by reference to especially vivid or characteristic examples can be accomplished much more readily through computer files than through field notes.
Computer programs exist for the analysis of ethnographic data if key words are supplied. To make easy retrieval possible, a list of key words was assigned to each topic prior to the beginning of fieldwork. These lists were based on the items in the questionnaires and on the researchers’ knowledge of the three cultures in which they would be working. Each list was developed by the field researchers assigned to investigate the same research topic. For example, the field researchers dealing with individual differences agreed upon approximately 30 key words for the material related to teachers’ responses to individual differences.
The case studies involved interviews, conversations, and classroom observations in grades 4, 8, and 12 in three metropolitan areas in Germany, Japan, and the United States. The majority of the interviews were held with pupils, teachers, and parents, but also with policymakers, education authorities, and other persons engaged in the education enterprise. A computer network linked all of the researchers and made it possible to enter and retrieve verbatim transcripts, observational records, and other field notes — the data on which the study is based.
-###-
[Chapter 2 - Education Standards (Part 1 of 3)]