|
1996 Science Performance Standards
Introduction
 |
|
||
|
NAEP’s Mission Authorized and funded by Congress, the National Assessment of Educational Progress (NAEP) is the only nationally representative and continuing assessment of what American students know and can do. The National Assessment Governing Board (NAGB), an independent bipartisan body, sets policy for NAEP, and the National Center for Education Statistics (NCES) of the U.S. Department of Education administers the program. For more than 25 years, NAEP has collected and reported information about student achievement in mathematics, science, reading, writing, U.S. history, world geography, and other subjects. From 1969 through 1981, NAEP assessments were conducted annually. After 1981, they became biennial. Originally, NAEP assessed students at ages 9, 13, and 17, but beginning in 1983, the program was expanded to include students at grades 4, 8, and 12. Since 1969, NAEP’s mission has been to survey national samples of school-age children and report on their academic performance in various school subjects. In 1990, Congress expanded NAEP’s mission to include reporting on student achievement in individual states and U.S. territories. Although participation in the program is voluntary, it has grown from 40 jurisdictions in 1990 to 47 in 1996. NAEP has successfully measured performance in various subjects during the past several decades. In the 1990s, it has the added value of tracking progress toward meeting the National Education Goals. Although NAEP has measured science achievement on eight occasions since 1969, the 1996 Science Assessment is the first to use a new framework developed by NAGB. Thus, the results provide baseline information for marking progress toward the fourth National Goal: "American students shall be first in the world in mathematics and science by the year 2000."1 The NAEP 1996 science results are important not only because they provide baseline information for the American public, policymakers, and educators, but also because their release coincides with release of the performance results for the United States on the Third International Mathematics and Science Study (TIMSS).2 It is hoped that the results from these two major surveys will spark a national conversation about how science is taught and learned in the nation’s schools. NAEP 1996 Science Framework The science framework3 for the 1996 NAEP assessment was developed in 1991 through a national consensus process that involved educators, policymakers, science teachers, representatives of the business community, assessment and curriculum experts, and members of the general public. NAGB managed this project through a contract with the Council of Chief State School Officers (CCSSO). Two principles guide the science framework. First, the framework recognizes that scientific knowledge should be organized to connect and create meaning for factual information and that the context in which knowledge is presented influences this organization. Second, the framework assumes that science performance depends on the ability to know and integrate facts into larger constructs and the ability to use scientific tools, procedures, and reasoning processes to develop an increased understanding of the natural world. Based on this framework, the NAEP 1996 Science Assessment includes the following:  Multiple-choice questions that assess students’ knowledge of important facts and concepts and that probe their analytical reasoning
The core of the science framework is organized into three major fields -- earth, physical, and life sciences. It also defines characteristic elements of knowing and doing science -- conceptual understanding, scientific investigation, and practical reasoning. Each question in the assessment measures knowing and doing science within one or more fields of science. Furthermore, two overarching domains integrate the three fields of science -- the nature of science and the organizing themes of science. The nature of science encompasses the historical development of science and technology, the habits of mind that characterize scientists and engineers as well as the methods they employ in their work. It also includes the nature of design and related concepts such as optimization and trade-off. The themes of science include the notions of systems and their application in the scientific disciplines, models and their role in the development of scientific understanding, and patterns of change exemplified in natural phenomena. Following current assessment trends, the science framework includes multiple-choice questions, but emphasizes questions that call for student-constructed responses. Under the new framework up to 80 percent of student assessment time was allocated to answering constructed-response questions. The assessment included two types of constructed-response questions -- short response questions that required students to provide brief one- or two-sentence answers and extended-response questions that required answers one or two paragraphs in length. In addition, students were given hands-on activities that required them to actually "do" a scientific investigation appropriate to their level of development. These structured activities guided students through scientific observation and interpretation, engaging them in an assessment experience that was more closely related to real science than a traditional paper-and-pencil test. (The full text of the Grade 8 Hands-on Tasks can be found in Appendix B.) This approach of "doing" science was similar to many statewide science assessment programs that include nontraditional types of questions such as constructed-response and performance questions.4 A companion NCES report5 on science achievement provides a fuller description of the framework, and the cognitive questions that embody it, while the NAEP Technical Report and the NAGB science framework provide full details. Reports on Science Performance NAEP reports present descriptive information about students’ average performance as well as basic and higher level performance in various subjects across the nation, by region and states, and by selected student background characteristics such as gender, race or ethnicity, and parents’ education. This NAGB report presents information about achievement using the newly adopted science achievement levels. The results are expressed as percentages of students, or percentages of selected subgroups, who have reached the NAEP student performance standards in the nation and states. The companion NCES report focuses on the average achievement for the nation, the states, and various subgroups and on the relationship between achievement and various background variables such as time spent on homework and student motivation to participate in or do well on NAEP. The Achievement Levels Policy The 1988 NAEP legislation6 creating NAGB directed the Board to identify "appropriate achievement goals . . . for each subject area" that NAEP measures. The 1994 NAEP reauthorization7 reaffirmed many of the Board’s statutory responsibilities, including "developing appropriate student performance standards for each age and grade in each subject area to be tested under the National Assessment." Following this directive and striving to achieve a primary mandate of the 1988 statute, "to improve the form and use of NAEP results," the Board has been developing student performance standards (called achievement levels) for NAEP since 1990. The Board has adopted achievement levels in mathematics, reading, U.S. history, world geography, and science. The achievement levels adopted by the Board and used here to report the performance of students on the 1996 NAEP Science Assessment are developmental, and as such, are currently being evaluated by the National Academy of Sciences (NAS). The NAS findings will be available in late 1998. The Board framed the policy for the achievement levels to help answer the question, "How good is good enough?" The goal is to report NAEP results in terms of the quality of student achievement by defining levels of learning linked to a common body of knowledge and skills that all students should attain, regardless of their backgrounds. The Board defined three levels for each grade: Basic, Proficient, and Advanced. These levels are cumulative in nature, that is, it is assumed that students at the Proficient level are likely to be successful at the Basic and Proficient content and students at the Advanced level are likely to be successful at the Basic, Proficient, and Advanced content. Table 1 presents the policy definitions of the achievement levels that apply across grades and subject areas. The specific content descriptions of science achievement levels for grades 4, 8, and 12 can be found in Appendix A. Adopting three levels of achievement for each grade signals the importance of looking at all levels of performance, from the most advanced to the very minimal. The Board believes, however, that all students should reach the Proficient level; the Basic level is not the desired goal, but rather partial mastery, a step toward Proficient. Development of the 1996 NAEP Science Levels In general, NAGB develops achievement levels for NAEP using a method for setting student performance standards that identifies what students should know and be able to do at each level. On behalf of NAGB, ACT assembles panels for each grade level and presents them with the policy definitions and the preliminary descriptions of the content for the achievement levels (crafted during the framework consensus process), the assessment framework, and a selection of questions from the assessment. Using these items, panelists develop and refine the final descriptions of content. The content descriptions continue to be refined throughout the level-setting process and are validated by a supplementary group of panelists after the level-setting meetings. Panelists are also asked to select sample questions for each level. These questions, chosen from the set of released test questions, represent the full range of performance from one achievement level to the next higher level. The goal in creating content definitions and identifying and selecting exemplar questions and student responses is to represent the full range of performance from one level to the next. When developing the science achievement levels in 1996, Board members carefully studied the information generated by the level-setting process designed and implemented by ACT.8 The Board believed that some of the levels derived from the process did not meet its criterion of reasonableness. In several cases, the levels seemed to be set either lower or higher than would be reasonable, resulting in too few or too many students placing at or above the Basic, Proficient, or Advanced levels. In reaching this conclusion, the Board examined the relevant available information, including achievement levels already adopted in other NAEP subjects, 1996 Advanced Placement (AP) results for twelfth-grade students, and information about eighth-grade students gathered from the TIMSS survey. The Board also studied the effect of adopting higher or lower cut scores on the percentages of students at or above the levels as well as the cut scores recommended by individual panelists who participated in the original process. In the final analysis, the Board exercised its judgment about where the levels should be set on the NAEP scale to satisfy the reasonableness criterion. The levels presented in this report reflect the Board’s deliberations and, as such, have been adopted by the Board for reporting the 1996 NAEP science achievement results. Because content descriptions developed by the ACT panelists no longer matched the cut scores adopted by the Board, a broadly representative group of science educators and scientists was asked to develop new descriptions, which would describe what students know and can do at each achievement level based on students’ achievement on the assessment questions.9 Table 2 provides a summary of the NAEP science achievement level descriptions. Since these descriptions were developed dependent on students’ performance on questions in the assessment, they should not be compared either to the preliminary descriptions in the science framework or to the descriptions in other subject areas. Such descriptions are statements of what students should know and be able to do, and as such, may not be comparable to those being reported here for the 1996 Science Assessment. In addition, new exemplar questions were selected to better represent the content of the science achievement levels adopted by the Board. The 1996 Science Achievement Levels The achievement levels adopted by the Board consist of the following: Content descriptions of what students know and can do at each level
The full text of the achievement levels descriptions can be found in Appendix A, and in the exemplar questions. Cautions on Interpretations The averages and percentages presented in this report are estimates because they are based on samples rather than on all members of each population. Consequently, the results are subject to a measure of uncertainty, reflected in the standard errors of the estimates. (The Standard Error Tables can be found in Appendix C.) The comparisons presented in this report are based on statistical tests that consider the magnitude of the difference between the group averages or percentages and the standard errors of those statistics. Throughout this report, differences among reporting groups are defined as significant when they are significant from a statistical perspective. The discussion of a difference as statistically significant means that observed differences in the sample are likely to reflect real differences in the population and are highly unlikely to have resulted from chance factors associated with sampling variability.11 The term "significant," therefore, is not intended to imply a judgment about the educational importance of the absolute magnitude of the differences. It is, rather, intended to identify statistically dependable population differences to help focus subsequent dialogue among policymakers, educators, and the public. The reader is cautioned against interpreting the relationships among subgroup averages or percentages as causal relationships. Average performance differences between two groups of students may result in part from socioeconomic and other factors. For example, differences among racial and ethnic subgroups are almost certainly associated with a broad range of socioeconomic and educational factors not discussed in this report. Similarly, differences in performance between public and nonpublic school students may be better understood by accounting for educational and other factors such as the composition of the student body, parents’ education levels, and parental involvement. Finally, student participation rates and the motivation of students, particularly twelfth graders, to perform on an assessment like NAEP should be considered when interpreting the results. (A further discussion of twelfth graders’ participation rates and motivation is presented in Appendix A of the NCES companion report cited earlier.) The NAEP scales and achievement level cut points were established independently for each grade. As a result, only within-grade comparisons can be made. Comparing the achievement level attainment of males in grade 4 with that of males in grade 12, for example, or making other across-grade comparisons is not meaningful. Finally, a word about the Tables and Figures found in Chapters 2 and 3 of this report. The data in these chapters illustrate the percentage of students at or above each achievement level. Since students at the Proficient and Advanced levels have also satisfied the requirements for the Basic level, the percentage of the students at or above the Basic level includes these students. Similarly, the percentages at or above the Proficient level includes those students who reached the Advanced level. These percentages are cumulative and do not sum to 100 percent. |
