Introduction

This document contains the design and rationale for assessing science achievement of students throughout the United States in 2005. This same Framework has been used by NAEP for the 1996 and 2000 science assessments. The Framework provides a general overview of NAEP, describes the NAEP Science Framework adopted by NAGB, and reviews the process by which the Framework was developed.

Background

NAEP is authorized by Congress and funded by the federal government; it is the only nationally representative and continuing assessment of what America’s students know and can do. For more than 35 years, NAEP has been charged with collecting and reporting information on student achievement in mathematics, reading, science, U.S. history, writing, and other subjects. Originally, assessments were of students at ages 9, 13, and 17, but beginning in 1983, they also included students in grades 4, 8, and 12.

NAEP reports provide descriptive information about student performance, including basic and higher order skills, in various subjects and comparisons of performance by race/ethnicity, gender, type of community, and geographic region. NAEP data also show relationships between achievement and certain background variables, such as the time spent on homework or the educational level of parents.

In the past, only results from a national sample of students were reported for each NAEP subject assessed, but, in 1987, a national study group chaired by Lamar Alexander, then governor of Tennessee, and H. Thomas James, president emeritus of the Spencer Foundation, recommended to the Secretary of Education that NAEP collect representative data on student achievement at the state level. In 1990, a trial state assessment was conducted for eighth-grade mathematics. State-level assessments since have become a major part of NAEP in reading, writing, mathematics, and science. In 2002, NAEP conducted the first Trial Urban District Assessment (TUDA) in nine urban districts that volunteered to participate. The NAEP TUDA component has continued in subsequent years in reading and mathematics.

Science educators, by and large, do not quarrel with the essential concept of assessment, but there has been no formal agreement on a common framework, outcomes, or goals and objectives to assess. The NAEP Science Assessment attempts to reflect a comprehensive, contemporary view of science so that those affected by the National Assessment are satisfied that it addresses the complex issues in science education without oversimplification. The framers of this document have attempted to tread a fine line between clear communication and technical accuracy in the hope that their efforts represent a step forward in building national consensus on the key outcomes of science education.

Development of the NAEP Science Assessment Framework

The following factors guided the process for developing consensus on the NAEP Science Assessment Framework:

The first factor is the general process of consensus development, both as it is set forth in law and as it evolves over time. The process calls for active participation and broad involvement of curriculum specialists; science teachers; local science supervisors; state supervisors; administrators; parents; representatives of scientific associations, business and industry, government, and unions; cognitive psychologists; and science educators as well as participation from the public and private education sectors.

The second factor is emphasis on the important outcomes of science education. As much as possible, this Framework represents what is considered essential learning in science, and set the stage for the 1996 and subsequent assessments. The Framework also recommends innovative assessment techniques to probe critical abilities and content areas.

The third factor is recognition that the various "players" in education and industry often hold diverse, and sometimes conflicting, views. Further, research and general agreement in the field is lacking. This lack of agreement on a common scope of instruction and sequence, the components of scientific literacy, the important outcomes of learning, and the nature of overarching themes in science hinders clear communication between science educators and the public.

The process of developing the Framework document and accompanying reports occurred between October 1990 and August 1991. Original plans called for a new NAEP Science Assessment to be given in 1994. Due to a budget shortfall, however, both the new science and mathematics assessments were rescheduled for 1996.

For the consensus process, a steering committee of 18 members (see appendix E) recommended by the education community and the private sector was established. Its members developed the principles that guided the creation of the Framework. A smaller planning committee (see appendix E), composed of practitioners, recognized experts in science education, and scientists, was established to identify goals and objectives and to produce the Framework. Together with the steering committee, it was also responsible for suggesting ways to assess important outcomes of science education.

The planning committee met monthly from November 1990 through April 1991 and was joined in its first and final meetings by the steering committee, which reviewed and reacted to all Framework drafts. Staff of the American Institutes for Research (AIR), a subcontractor, also attended the meetings. On the basis of this input and reaction and advice from the committees, AIR formulated specifications for the science assessment.

NAGB Subject Area Committee #2 and technical staff closely monitored the consensus project work and Board members were involved in all key phases. In addition, advice was sought from the organizations and sectors that affect and are affected by science education. For instance:

• Opportunities for public input were provided at two hearings (in Washington, D.C., and in San Francisco). Opinions were expressed by representatives of institutions (for example, public and private education, scientific and science education organizations) likely to be affected by or concerned with the Framework and subsequent assessments.

• The Council of State Science Supervisors was kept abreast of project developments through its electronic communication system, PSInet. Input from the Council of State Science Supervisors was particularly important in helping the NAEP Science Assessment Project define "big ideas" or Themes in science learning (see chapter 2). For example:

  —States with existing frameworks for science instruction were contacted for copies of their frameworks and assessment methods.

  —Planning and steering committee members hosted sessions at both regional and national science education meetings to report on the development of the NAEP Science Assessment Framework.

  —The revised draft of the Framework was widely circulated in June 1991 within the science education and science communities for reaction and comments.

At its early meeting, the steering committee drafted guidelines for the planning committee’s work and recommended that the Framework and ensuing NAEP Science Assessment have the following five characteristics:

• The Framework should reflect the best thinking about the knowledge, skills, and competencies needed for a high degree of scientific understanding among all students in the United States. Accordingly, it should:

  —encompass knowledge and use of organized factual information, relationships among concepts, major ideas unifying the sciences, and thinking and laboratory skills;

  —be based on current understandings from research on teaching, learning, and student performance in science.

• Both the Framework and the NAEP Science Assessment should:

  —address the nature and practices of knowing in science as they differ from other ways of knowing;

  —reflect the quantitative aspects of science as well as the concepts of life, Earth, and physical sciences;

  —deal with issues raised by the role of science and technology in society;

  —include practical problemsolving that involves design, use of materials, and weighing risks in relation to benefits;

  —take into account the developmental levels of students;

  —ensure that students with diverse backgrounds are assessed in ways that provide them with equal and fair opportunities to reflect their knowledge and performance.

• Assessment formats should be used that are consistent with the objectives being assessed. A variety of strategies for assessing student performance are advocated, including:

  —performance tasks that allow students to manipulate physical objects and draw scientific understandings from the materials before them;

  —open-ended items that provide insights into students’ levels of understanding and ability to communicate in the sciences, as well as their ability to generate, rather than simply recognize, information related to scientific concepts and their interconnections;

  —collections of student work over time (such as portfolios) that demonstrate what students can achieve outside the time constraints of a standardized assessment situation;

  —multiple-choice items that probe students’ conceptual understanding and ability to connect ideas in a scientifically sound way.

• The assessment should contain a broad enough range of items at different levels of proficiency to identify three achievement levels for each grade.

• Information on students’ demographic and other background characteristics should be collected. Additional information should be collected from students, teachers, and administrators about instructional programs and delivery systems so that their relationships with student achievement can be ascertained and used to inform program and policy decisions.