Three Teams Win Top Prizes in Mathematics Automated Scoring Challenge for The Nation's Report Card

Vanderbilt University, UMass Amherst, and University of Oregon teams win challenge to accurately score selection of NAEP mathematics items using advanced natural language processing

WASHINGTON, Aug. 21, 2023 /PRNewswire/ -- The National Center for Education Statistics (NCES), which administers the National Assessment of Educational Progress (NAEP), also known as The Nation's Report Card, has awarded prizes in its second automated scoring challenge to teams from Vanderbilt University (PI: Dr. Scott Crossley; Joon Suh Choi; Langdon Holmes; and Wesley Morris), the University of Massachusetts Amherst (PI: Dr. Andrew Lan; Wanyong Feng; Jaewook Lee; William McNichols; Alex Scarlatos; and Mengxue Zhang) and the University of Oregon (PI: Dr. Cengiz Zopluoglu).

"Recent student performance on mathematics items underscores how important it is for us to provide advanced approaches to scoring," said NCES Commissioner Peggy G. Carr. "Winning teams scored responses accurately and described their results thoroughly. They conducted fairness analyses showing that their algorithms did not score students differently based on their demographic or family background. These results provide encouraging evidence for NAEP to implement automated scoring in several subjects and further explore the potential of automated scoring." 

In this challenge, participants created algorithms to score students' responses to open-ended questions about how they solved a multiple-choice mathematics problem. The winners used advanced natural language processing methods that promise to reduce scoring costs while providing additional insights about student responses. All the winning teams' scores were similar to human scoring on open-ended text questions. Participants were eligible to win up to $60,000 in prizes.

As accurate as human scoring
Because mathematics items include both symbolic information, like arithmetic symbols, and conceptual information, like mathematical operations and terms, open-ended mathematics responses are more difficult to score than writing and reading items. Nonetheless, teams accurately scored almost every item in the challenge.

Like human scorers, the most effective automated approaches evaluated other parts of the question—such as which multiple-choice answer a student selected—when scoring an open-ended response.

Teams show their work
Challenge participants were required to describe their technical approaches to meet defined criteria around transparency to ensure consistency with educational research principles like validity, reliability, and fairness. All winning entries provided technical reports that described their approaches and data processing.

Winners also conducted analyses to ensure their models graded students fairly across different demographic backgrounds. The challenge analyzed multiple student characteristics, including race/ethnicity, gender, Individual Education Plan (IEP) status, and English learner (EL) status.

All challenge awardees were from research institutions. While invited, other sectors did not submit final responses.

Modernizing NCES operations
This challenge is one element of NCES's efforts to incorporate data science and machine learning into its work. It is the second challenge that uses NAEP data. The first challenge used machine learning to score reading results.

Additional details on the challenge are available at https://github.com/NAEP-AS-Challenge/math-prediction.

The National Center for Education Statistics, a principal agency of the U.S. Federal Statistical System, is the statistical center of the U.S. Department of Education and the primary federal entity for collecting and analyzing data related to education in the U.S. and other nations. NCES fulfills a congressional mandate to collect, collate, and report complete statistics on the condition of American education; conduct and publish reports; and review and report on education activities internationally.

The National Assessment of Educational Progress (NAEP) is a congressionally authorized project sponsored by the U.S. Department of Education. The National Center for Education Statistics, within the Institute of Education Sciences, administers NAEP. The commissioner of the National Center for Education Statistics is responsible by law for carrying out the NAEP project. Policy for the NAEP program is set by the National Assessment Governing Board (NAGB), an independent, bipartisan board whose members include governors, state legislators, local and state school officials, educators, business representatives, and members of the general public.

CONTACT:
Eunice Greer, NCES, [email protected] 

SOURCE National Center for Education Statistics