来源简介
arXiv:2511.01650v4 Announce Type: replace Abstract: Most engineering problems involve complex multi-step derivations through checkable intermediate quantities. Grading only the final answer misjudges LLMs, crediting correct answers reached through flawed steps and penalizing wrong answers built on mostly sound ones. Yet, existing benchmarks such as SuperGPQA and EEE-Bench score only final answers, and step-level ones such as TPS-CalcBench rely on LLM judges or human graders, or cover a single domain. We therefore introduce EngTrace, a symbolic benchmark for verifiable process evaluation of engineering reasoning. Its 150 expert-certified templates, spanning five branches, 15 domains, and 42 areas, generate each problem with a gold reasoning trace, so that every intermediate quantity is known exactly. We design and validate an evaluator that checks final answers, coverage of the gold trace's intermediate quantities (milestones), and displayed arithmetic, mostly without an LLM judge. Across 11 LLMs, final answers barely separate the strongest models, while their derivations differ. Wrong answers still state about half of the gold milestones, and domain experts trace most wrong answers to calculation slips, with conceptual errors rarer on easy problems than on harder ones. Overall, as LLMs near the final-answer ceiling, checking derivations, as EngTrace does, reveals differences that final answers miss.
本条尚未生成中文 AI 摘要，以上展示来源提供的原始简介。
采集于 2026/10/20 21:06:31（北京时间）。 arXiv 日期为本次公告时间，论文初次投稿日期请查看原文版本记录。