|
Chapter Two
The Framework for the NAEP Science Assessment IntroductionIt is customary to collect relevant curriculum guides, frameworks, and other course outlines to get a sense of what students are studying throughout the nation when developing a framework for an educational assessment. The union of all these "learning guides" is used to develop overall test specifications. Such an approach tends to have an unfortunate consequence: it often leads to a broad, trivialized, lowest common denominator approach to assessment. However, these materials can serve another purpose, that of documenting trends and developments in science education throughout the country. In the NAEP Science Assessment Framework development process, science reform reports from states, in addition to several large-city science curriculum guides, were gathered and used to establish consonance between the evolving Framework and the most forwardlooking of the reports and guides. The traditional approach to teaching science tends to emphasize rote memorization of facts without connection or organization. Although it must not lose sight of the need for factual knowledge that is fundamental to science literacy, assessment of science achievement must change to give more emphasis to conceptual understanding and the application of knowledge and skills for several compelling reasons. First, the expansion of scientific information has resulted in far too many facts for a student to memorize. It is more efficient to store them electronically (or in other forms) and access information as needed. Second, isolated science facts that are not organized or used tend to be forgotten quickly and, even when remembered, form a poor basis for learning. Third, it is desirable to encourage science instruction that is used both to deepen understanding and to address challenging problems. Science education is best served when students can understand and discuss ideas rather than simply accumulate unconnected facts. Therefore, it is important that science assessment covers major topics like electricity and magnetism, forces and motions, life cycles, ecosystems, plate tectonics, and climatology. However, even these topics need to be viewed as shorthand for a much richer understanding of what students should attain. Several current reform reports and frameworks, such as Science for All Americans: A Project 2061 Report on Literacy Goals in Science, Mathematics, and Technology (American Association for the Advancement of Science, 1989), innovative state frameworks, and reports from the National Center for Improving Science Education (1990, 1991) describe desired outcomes of science instruction in new ways. They advocate mastery of fundamentals in ways that are more likely to result in students’ learning. The approach advocated in these documents is reinforced by the findings of science educators and cognitive researchers, demonstrating that if students do not completely learn the concepts presented to them, they pass through the K–12 grades without fundamentally changing the conceptual models they learned in their early years. Whereas it is easy to argue for a test that emphasizes conceptual understanding rather than topical listings and recognition of definitions from a list of choices, it is more difficult to agree on how science learning should be assessed. The science courses that were developed in the late 1950s and early 1960s used a more conceptual organization for textbooks and teaching materials. These programs, funded by the National Science Foundation, became major influences on the country’s secondary science textbooks from 1960 to 1970 and helped raise standards of students’ science achievement (Shymansky et al., 1983). Unfortunately, old-style textbooks have since returned, with expanded numbers of pages that include many newly emerging topics. The effect on students’ understanding of science has not been beneficial. The NAEP Science Assessment Framework incorporates a balance of knowledge and skills based on current reform reports, exemplary curriculum guides, and research on the teaching and learning of science. The Framework for the NAEP Science Assessment is organized according to two major dimensions: Fields of Science and Knowing and Doing Science. The Fields of Science are Earth, physical, and life sciences. Knowing and Doing Science includes Conceptual Understanding, Scientific Investigation, and Practical Reasoning. These two dimensions and their subdimensions are explained in greater detail below and in appendix B. The NAEP Science Assessment Framework is structured according to a matrix similar to that used for the 1990 NAEP Science Assessment. The content areas are organized into the same three fields; however, an additional requirement for some interdisciplinary exercises merges technology with the science content areas. The Nature of Science (which also was part of the 1990 Framework) and Themes are categories that should integrate the three fields of science, rather than represent separate content. The Knowing and Doing dimension is a reorganization of the Thinking Skills dimension that was a part of the 1990 NAEP Science Assessment, with a clearer delineation of subcategories, particularly with respect to Practical Reasoning. Each element of the NAEP Science Assessment Framework is addressed briefly below and in greater detail in separate appendixes.
Knowing and Doing dimension is a reorganization of the Thinking Skills dimension that was a part of the 1990 NAEP Science Assessment, with a clearer delineation of subcategories, particularly with respect to Practical Reasoning. Each element of the NAEP Science Assessment Framework is addressed briefly below and in greater detail in separate appendixes. With respect to Fields of Science, the main emphasis of the assessment should be on knowledge in the content areas. Details follow:
More specific guidance on the distribution of content from the three fields is given in the Specifications document, which is used by test developers to create the assessment. A major emphasis with respect to Knowing and Doing Science should be on students’ active expression of conceptual understanding. Details follow:
Every question or task in the assessment should be classifiable by one or more subcategories in each of the two major dimensions of the matrix. In addition to the two major dimensions, the Framework includes two other categories that pertain to a limited subset of items. The first concerns students’ understanding of the Nature of Science:
In specifying these percentages, it is assumed that assessment items can be developed to measure knowledge of content within a field of science or an area of Knowing and Doing Science, in addition to measuring knowledge of the Nature of Science. The second category, Themes, represents big ideas or key organizing concepts that pervade science education. Themes cross traditional science discipline boundaries and make up the inquiry tools that scientists use to better investigate and understand phenomena. These themes include the notion of Systems and their application in the disciplines; Models and their function in the development of scientific understanding and application to practical problems; and Patterns of Change as they are exemplified in natural phenomena. Guidelines follow:
More detailed guidelines on assessing themes are provided in the Specifications document. Understanding, doing, and using science often involve tasks that include more than one category in each dimension. Multiple-duty exercises may present some scoring challenges. Relatively simple exercises will be scorable according to one or two subdimensions that include, for example, a "conceptual understanding in the physical sciences." More complex items may be scorable in several subcategories; for example, an open-ended task involving ecosystems might yield responses scorable according to "conceptual understanding in the Earth sciences," "scientific investigation in the life sciences," and "systems." Such items, which contribute to more than one subcategory, may prove difficult to reproduce through similar items having the same properties in future assessments. It is important, therefore, to specify carefully the several subdimensions or subcategories that each such exercise is intended to probe. Scoring rubrics for each of the subdimensions also must be developed. The descriptions below are summaries of the major topic areas to be probed within each field in the NAEP Science Assessment. This content represents key elements in science that all students should be expected to know and understand. For a complete explanation, including descriptions of subject matter knowledge to be expected at each grade level, see appendix A. The NAEP Science Assessment will probe student understanding of how Earth scientists depict data through maps and other means to interpret objects, their features and structures, and the events and processes that caused them. What do students know about their own position with respect to objects and structures on, below, and above the Earth’s surface? What do students know about the changes in position of objects and environments through time? What do students know about the relative movements of the Earth, the Moon, the Sun, and the planets? The content to be assessed in Earth science centers on objects and features that are relatively accessible or visible: the solid Earth (lithosphere), water (hydrosphere), air (atmosphere), and the Earth in space. With respect to Earth science, the NAEP Science Assessment should focus on the following concepts and topics. Concepts related to solid Earth:
Concepts related to water:
Concepts related to air:
Concepts related to Earth in space:
The physical science component of the NAEP Science Assessment relates to basic knowledge and understanding concerning the structure of the universe as well as the physical principles that operate within it. The assessment should probe the following major topics: matter and its transformations, energy and its transformations, and the motion of things. The NAEP Science Assessment should focus on the following physical science concepts. Matter and its transformations:
Energy and its transformations:
Motion:
The fundamental goal of life science is to attempt to understand and explain the nature and function of living things. During the 20th century, the focus of biological research has changed from descriptive natural history to experimental investigation, with evolution as the central, unifying theory. The following list contains the major concepts to be assessed in life science. Change and evolution:
Cells and their functions:
Organisms:
Ecology:
In the 1990 NAEP Science Assessment, three categories were used: Knowing Science, Solving Problems, and Conducting Inquiries. For the current NAEP Science Assessment, it should be noted that not only has the Knowing Science dimension been changed, but its meaning has also been redefined as well. It has been reformulated as Conceptual Understanding of Science to stress the connections and the organization of factual knowledge in science. The subdimension Scientific Investigation has been substituted for the 1990 category Conducting Inquiries. This new subcategory is intended to probe students’ abilities to use the tools of science, including both cognitive and laboratory tools. Appropriate to their age and grade level, students should be able to acquire new information, plan appropriate investigations, use a variety of scientific tools, and communicate the results of their investigations. The 1990 category Solving Problems proved to be ambiguous. To emphasize the need to assess students’ abilities to use and apply science understanding in new, real-world applications, the category has been redefined as Practical Reasoning. Practical reasoning subsumes competence in analyzing a problem, planning and evaluating appropriate approaches, carrying out the required procedures for the approach(es) selected, and evaluating the result(s). Each of the three categories is described below. Mastery of basic scientific concepts can best be shown by a student’s ability to use information to conduct a scientific investigation or engage in practical reasoning. Optimally, essential scientific concepts involve a variety of information, including:
The goal of school science is to engender conceptual understanding. Students should acquire information in ways that will enable them to apply it efficiently in the design and execution of scientific investigations and in practical reasoning. A challenge in the design of assessment exercises is to capture changes in the characteristics of student performance as children mature. In the primary years, when the goal of school science is to build a rich collection of information derived from examined experiences with the natural environment, the assessment of conceptual understanding will focus on the breadth of information acquired about the natural world and the student’s ability to elaborate on principles by using personal experiences. Does the student know the cyclical changes in the apparent shape of the Moon over time? More importantly, can the student relate how he or she knows about the changes? What evidence does the assessment exercise provide that the student’s information is based on direct experience? Is there a science notebook in which the student recorded observations of the Moon over time? Does the student know that sometimes the Moon is visible during daylight hours? In the primary years, the focus should not be on explanation or prediction, but instead on knowledge obtained from rich experiences in school. Consequently, assessment exercises would not be concerned with having students explain why the Moon appears to change shape but rather with relationships between time of day, apparent positions of the Sun and the Moon, and times of moonrise and sunset. In the middle and high school years, the emphasis should shift from richness of experience to reasonable scientific interpretation of observations. In the elementary years, the primary concern should be with how well reasoned the interpretation is presented by the student, not with whether it reflects the most sophisticated scientific reasoning. However, at grades 8 and 12, the assessment should be increasingly concerned with the congruence of the students’ interpretations with accepted interpretations, as well as with the sophistication of their reasoning in moving from observations of the natural world to explanations and predictions. Of special interest in the 2005 NAEP Science Assessment will be the extent to which students are able to understand and use the notions of models, systems, and patterns of change. It is important to note that many aspects of conceptual understanding as defined for the new NAEP Science Assessment cannot be tested using exclusively multiple-choice items. Items of this kind may be satisfactory for assessing individual parts of the information base, but they are limited in tapping highly valued aspects of conceptual understanding. Scientific investigation represents the activities of science that distinguish it from other ways of knowing about the world. It incorporates such previously used assessment categories as "processes of science" and "scientific problemsolving." This category is not just another name for the scientific method. Indeed, there is great confusion about the scientific method in the teaching of science. Real science is doing what one can in any way one can, often creatively and insightfully and using flashes of insight with little regard for a progression of steps. However, there is a familiar format and context for reporting the results of experiments. It begins with the report of the problem and continues with the hypothesis, the experimental design, the data collected, the analysis of those data, and the conclusions (if any). This convention of science is often mistaken for how scientists actually work. The results must satisfy logical analysis, but logical ordering may appear only when the report is prepared. A great disservice has been done to generations of students because well-meaning people have taught the standard method of reporting science as the standard method of doing science. Scientific investigations must be designed at levels appropriate to the development of the students. This component has important implications for assessment. Young students are limited in their ability to perceive the scale of both very large and very small things. Students’ limitations handicap them when they are forced, either by the textbook or by the curriculum, to deal with developmentally inappropriate concepts such as atoms or even cells. Young students are also developmentally limited in their ability to understand time. The distant past and the future are narrowly perceived by the egocentric student. Instruction, as well as assessment, must recognize where the student is and take developmental levels into account. As students develop and accumulate experiences, their performance in doing scientific investigations should begin to look more and more like "real" science. Central to the ways scientists work is the concern for a fair test, for a controlled experiment. Children seem to have an intuitive sense of what makes a fair test. What they lack is the ability to consider all the variables and the means to control the variables. It might be reasonable to consider a developmental continuum such as the following when thinking about control of variables:
As students are asked to demonstrate their ability to do scientific investigations, it is important to keep in mind this sort of development in understanding and performance, not only with respect to the control of variables, but also regarding the other elements of doing science. The difficulty with the assessment may not be with the content, but with the level of variable embedded in the content. Practical reasoning about matters with scientific content (that is, the ability to apply one’s knowledge, thought, and action to real situations, not textbook problems) is influenced by the ability to (1) abstract and consider hypothetical experiences, (2) consider several factors simultaneously, (3) take a depersonalized view, and (4) realize the importance of practical reasoning and life experience. These factors develop throughout life. One of the characteristics of young children is that they have difficulty dealing with multiple ideas simultaneously. With maturity and experience, they can consider several ideas at once and weigh benefits in relation to costs or risks. Their ability to abstract and consider hypothetical situations develops as students progress in science and learn to deal with more remote phenomena and generalizations. As they mature, students also learn to take depersonalized views of situations and to consider other people’s points of view. Often, real-life problems involve not only theoretical and technical elements, but also personal preferences. What will be the social impact of a new waste disposal system? What will neighbors say if a traffic light is installed? How will other students react if lunchroom noise is diminished by staggering the lunch hour? To consider these questions carefully, it is necessary to understand different perspectives. The ability to understand the viewpoints of others increases with age and experience. Young children also may not realize the need for scientific information in solving problems. For example, children below the age of 12 usually see no need to carry out measurements (Strang, 1990). Also, because young children have little responsibility for decisions affecting their lives, they may not see the need for practical reasoning. However, the more that students have done or seen, the more likely it is that they can solve real-world problems. With age and experience, the possibility increases that a new situation is analogous to a previous one and that the human, technical, and theoretical factors involved in a new situation have already been encountered. All these factors suggest that practical reasoning should become a major factor in science assessment at grades 8 and 12 rather than at grade 4. As students become eager to take control of their lives, wish to try out their understanding of the world, and progress in development, practical situations related to their everyday life, school, and home provide excellent exemplars to demonstrate science-related practical reasoning. Thus, students might be asked to discuss problems such as noise abatement in the lunchroom, to design a simple apparatus such as a flashlight or a burglar alarm, or to plan a school garden. By grade 12, students should be able to discuss larger science and technology-linked problems not directly related to their immediate experiences. Examples of these might be waste disposal, energy uses, air quality, water pollution, noise abatement, and the tradeoffs between the benefits and adverse consequences of various technologies. Knowledge of the nature of science is central to the understanding of scientific enterprise. Yet often this category is relegated to a discussion (or even rote memorization) of some version of the scientific method. There is total agreement that this topic should play a prominent role in the NAEP Science Assessment. A controversy existed within the project committees concerning whether this category is sufficient unto itself, or whether the Framework should include a separate section that deals with the nature of technology. The project committees were split on this issue. However, all acknowledged that technology is integral to the nature of science and ought to be included in the assessment, provided that it clearly does not exist as a separate subcategory within the assessment. Technology, then, will be measured as it relates to science and the scientific enterprise. The following concepts are appropriate for assessment at the given levels:
Students are surrounded by and interact with the manmade world as much as with the natural world. Therefore, they must develop an understanding of what shapes the design and development of the technologies that are a part of that manmade world and their daily lives. Rather than being a content area, technology is embedded within this section because of its close association with science. The following concepts are appropriate for assessment at the given levels:
Themes are the "big ideas" of science that transcend the various scientific disciplines and enable students to consider problems with global implications. To understand the conceptual basis for the themes that have been selected, students must begin to develop an understanding of major ideas by the fourth grade. They should continue to develop their understanding through the 8th grade, and by the 12th grade, they should have the ability to integrate their knowledge and understanding. The review of current state frameworks conducted in the course of developing the new NAEP Science Assessment Framework revealed that many are based in part on crosscutting themes in science. Several national organizations, including the American Association for the Advancement of Science, have issued reports that advocate the importance of common themes. The number of themes defined in these reports and state frameworks varies somewhat, but there is considerable agreement on which common elements or big ideas of science should be understood by students as they complete their high school education. The decision by the NAEP Science Assessment committees to include themes underscores their emerging importance, as well as the necessity of integrating themes (through programmatic threads) into grades K–12. Three of the themes are common to all of the documents: Models, Systems, and Patterns of Change. These three themes were included in the 1996 NAEP Science Assessment because they constitute major, interdisciplinary organizing principles of science. Further, they do not conflict or compete with the factual content of the various fields, but rather augment and help organize that information into a coherent intellectual framework. Models of objects and events in nature can be used to understand complex or abstract phenomena. Models may be first attempts to identify the relevant variables to build evermore useful representations, or they may be highly refined for predictions about the actual phenomenon. Students need to understand the limitations and simplifying assumptions that underlie the many models used in the natural sciences. A model is likely to fit data well only within a limited range of circumstances and to be misleading outside of that range. Systems are complete, predictable cycles, structures, or processes occurring in natural phenomena, but students should understand that the idea of a system is an artificial construction created by people for certain purposes, for example, to gain a better understanding of the natural world or to design an effective technology. The construct of a system entails identifying and defining its boundaries, identifying its component parts and the interrelations and interconnections among those parts, and identifying the inputs and outputs of the system. Regardless of the topic around which the Patterns of Change theme is developed, students should be able to recognize patterns of similarity and difference, to perceive how these patterns change over time, to remember common types of patterns, and to transfer their understanding of a familiar pattern of change to a new and unfamiliar situation. Appendix C contains more detail on these three themes and the developmentally appropriate expectations for students at grades 4, 8, and 12.
|
|||||||||||||||||||||||||||||||||||||||