|
Chapter One Overview of Recommendations Since 1973 the National Assessment of Educational Progress (NAEP) has gathered information about levels of student proficiency in mathematics and the related practices of teachers in our nation's schools. These periodic assessments, The Nation's Report Card, are published in an attempt to inform citizens about the nature of students' comprehension of the subject, to inform curriculum specialists about the level and nature of student understanding, and to inform policymakers about factors related to schooling and its relationship to student proficiency in mathematics. Based on these surveys of students at the end of the primary, junior high, and high school levels, The Nation's Report Card has provided comprehensive information about what students in the United States know and can do in the area of mathematics, as well as in a number of other subject areas. These reports present information on strengths and weaknesses in students' understanding and their ability to apply that understanding in problem-solving situations; provide comparative student data according to race/ethnicity, type of community, and geographic region; describe trends in student performance over the years; and report relationships between student proficiency and certain background variables. Context for Planning the 1996 and 2000 Assessments The National Assessment Governing Board (NAGB), created by Congress in 1988, is responsible for formulating policy for NAEP. NAGB is specifically charged with developing assessment objectives and test specifications through a national consensus approach, identifying appropriate achievement goals for each age and grade, and carrying out other NAEP policy responsibilities. In 1990 the U.S. Department of Education conducted the first voluntary state-by-state assessment of mathematics as an adjunct to its periodic NAEP national assessments of mathematics. The 1990 state-level trials were limited to the 8th grade. In 1992 the second voluntary state-level assessments associated with NAEP were carried out at the 4th- and 8th-grade levels in mathematics, and at the 4th-grade level in reading. To prepare for the 1990 trial state assessment, NAGB awarded a contract in 1987 to the Council of Chief State School Officers (CCSSO) to design a framework for the assessment. The CCSSO project gave special attention to the nature of formal state objectives and frameworks for mathematics instruction. In doing so, that panel sampled state-, district-, and school-level objectives; examined the curricular frameworks on which previous NAEP assessments were based; consulted with leaders in mathematics education; and reviewed a draft version of the National Council of Teachers of Mathematics (NCTM) Curriculum and Evaluation Standards for School Mathematics. This project resulted in the "content-by-mathematical-ability" matrix design used to guide both the 1990 and 1992 NAEP mathematics assessments. The design was reported in Mathematics Objectives: 1990 Assessment. To prepare for the next NAEP mathematics assessment, NAGB awarded a contract in fall 1991 to The College Board to develop assessment and item specifications for the 1994 mathematics assessment. The process of developing the recommendations for the planned 1994 NAEP mathematics assessment occurred between September 1991 and March 1992. Due to a budget shortfall, however, both the new NAEP mathematics and science assessments were rescheduled from 1994 to 1996. The NAEP mathematics project conducted by The College Board had two primary purposes. The first was to recommend a framework for the overall design of the mathematics assessment; that is, a structure for describing what students should know and be able to do in mathematics. The second was to develop specifications for the assessment items, with particular attention to a mix of formats, the item distribution for content areas within mathematics, and the conditions under which items are presented to students (e.g., use of manipulatives, use of calculators, and time to complete). The new NAEP Mathematics Framework was considered in light of the three NAEP achievement levels -- basic, proficient, and advanced. Basic denotes partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade. Proficient represents solid academic performance for each grade assessed. Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter. Advanced represents superior performance. These levels are intended to provide descriptions of what students should know and be able to do in mathematics. Established for the 1992 mathematics scale through a broadly inclusive process and adopted by NAGB, the three levels per grade are a major means of reporting NAEP data. The new mathematics assessment was constructed with these levels in mind to ensure congruence between the levels and the test content. See Appendix A for the NAEP Mathematics Achievement Level Descriptions. The College Board convened a steering committee representing national education organizations and agencies to review the direction and scope of the project. A planning committee of mathematics educators met to draft the assessment framework. Both committees considered the status of national reform efforts in mathematics education and assessment evaluations of the NAEP trial state assessment in mathematics (Silver, Kenney, and Salmon-Cox, 1991); and the fit between NAEP assessments and the contemporary teaching of mathematics at grades 4, 8, and 12 in the nation's schools (Romberg, Wilson, Smith, and Smith, 1991). Committee members are listed in Appendix B. The planning committee began by reviewing the Framework used in the 1990 and 1992 NAEP assessments. Committee members also made use of the findings of evaluation studies concerning the NAEP assessments. The findings of these studies were merged with recent suggestions for the assessment of student proficiency in mathematics from the mathematics education and assessment communities. The planning committee also analyzed the 1990 and 1992 NAEP Mathematics Frameworks in light of the recommendations in the NCTM Standards and modifications in state curriculum outlines and assessment programs precipitated by that document. Finally, the committee reviewed information provided by the 1990 assessment, noting features of the Framework and how those features assisted or hindered the clear understanding of what students knew and were able to do in mathematics appropriate to their ages and levels of education. Another important phase in the consensus process involved conducting a national mail review and convening focus groups in six states to gather input on the committee's recommendations. The suggested revisions in the Framework for the new NAEP assessment in mathematics are intended to reflect recent curricular emphases and objectives; to include what various scholars, practitioners, and interested citizens believe should be in the assessment; and to maintain ties to prior assessments to permit the reporting of trends in student achievement across time. Recommendations for the 1996 and Future NAEP Mathematics Assessments As a result of analysis and review, the steering committee and planning committee endorsed the following recommendations for the 1996 and future NAEP mathematics assessments:
The matrix framework employed in the 1990 and 1992 NAEP assessments should be discontinued in favor of a model consisting primarily of the five major content strands used in that matrix model. Evaluation studies of the NAEP trial state assessment and other cognitive science recommendations dealing with assessment suggest that forcing content into a rigidly structured, content-by-ability-level matrix distorts the nature of the discipline. A model that calls for the assessment of knowledge in discrete, content-by-ability-level categories is inappropriate in an era in which other, more progressive recommendations call for attention to a student's ability to connect knowledge in one area of mathematics with knowledge and abilities in other areas of mathematics. Therefore, the recommendation was to use the five major strands: (1) Number Sense, Properties, and Operations; (2) Measurement; (3) Geometry and Spatial Sense; (4) Data Analysis, Statistics, and Probability; and (5) Algebra and Functions. In establishing these five strands, the names of the first and third strands were modified to better reflect the nature of the content relative to recommendations made in the NCTM Standards. The nature of the strands is further discussed in Chapters Two and Three.
The levels of mathematical ability (conceptual understanding, procedural knowledge, and problem solving) should not be used to define specific percentages of items in each of the five content strands as had been done in the 1990 and 1992 assessments. However, these descriptors, along with the more encompassing process goals of reasoning, connections, and communication, should play a central role in defining item descriptors and in achieving a balance across the task sets for each of the grade levels in the NAEP assessment in mathematics. This recommendation is discussed further in Chapters Two and Four.
The percentage of items allotted to each of the five strands should continue the move begun with the 1990 assessment toward a balance among the five strand areas and away from an assessment dominated by number and operations. The recommendations, while still retaining a core of items that reflect traditional goals in the basic skills, provide a continued movement toward a broad algebra- and geometry-oriented program at the 8th- and 12th-grade levels. The specific percentage of items recommended is further discussed in Chapter Two.
To measure the breadth and depth of student knowledge in mathematics, "families" of tasks/items should be created for each grade/age level of the assessment. A family of tasks/items is a related set of assessment tasks that can probe the vertical or horizontal nature of a student's understanding. A vertical family might include items that measure students' abilities to give a concept definition, apply the concept in a familiar setting, use the concept or related principles to solve a new problem, and ultimately generalize knowledge about the concept or related principles to represent a new level of understanding. A vertical family might lie within a single grade level or extend across grade levels. Another family of items might measure students' horizontal understanding of a concept or principle across the various content strands. For example, students' proficiency in solving the proportion 2/3 = 16/x might be measured in a number context, in a measurement setting, in a geometry setting, in a probability setting, and in an algebraic setting. Students' ability to work with the proportion in each of these contexts tells a great deal about the richness of their understanding of the concept and the related procedural skills.
The number of items requiring student construction of the response should be increased as much as possible within the bounds of the statistical design used to carry out the assessment. The evaluation studies associated with the 1990 assessment found that the items requiring student construction of the response were most in keeping with the spirit of the NCTM Standards. Further, these items provide excellent opportunities to see students' abilities to reason, connect, and communicate their knowledge of mathematics. In particular, the number of extended open-ended items should be increased from the number given in the 1992 assessment.
At the 12th-grade level, a special study should be carried out using graphing calculators to establish baseline data for the gradual entry into the curriculum of these tools that assist students in visualizing algebraic relations. In keeping with NCTM recommendations on hand calculator usage, NAGB should consider the unrestricted use of calculators on all phases of the assessment that are not designed to measure trends or students' basic fact, operation, and estimation abilities.
The assessment should continue to utilize reasonable manipulative materials, where possible, in measuring students' ability to represent their understandings and to reason in problem-solving situations. Such manipulative materials and accompanying tasks should be carefully chosen to cause minimal disruption of the test administration process.
While bias analysis is consistently done on NAEP items and performance as mandated by law, these recommendations for shifting the types of items used on the assessment require an especially careful look at potential item bias. Data should be gathered during field testing and during the actual assessment regarding possible types of unforeseen item bias that may arise from incorporating less widely used types of assessment items. The 1996 and future NAEP assessments will incorporate awareness of this critical consideration, especially related to students' previous opportunities to learn and their experience and background both in and outside school. Sensitivity and a sound research base will guide not only test construction but also the reporting of student performance.
|
||||||||