跳到正文
arXiv · AI· Mohsen Larni (Department of Computer Science, University of Nevada, Las Vegas), Sobhan Ebrahimi Azar (Department of Computer Science, University of Nevada, Las Vegas), Pouyan Nahed (Department of Computer Science, University of Nevada, Las Vegas), Kazem Taghva (Department of Computer Science, University of Nevada, Las Vegas)·· 2026-10-05AI 评分23

SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models

SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models

AI 导读

研究团队推出 SyntaxBench,这是一个针对大语言模型字符级推理能力的诊断性基准与统计评估框架。该框架包含字符计数、回文检测等五项核心任务及一项子串提取压力测试,在 2B 至 32B 参数规模的八个开源权重模型上进行了评估。 评估结果显示,分词方式显著影响准确性,且推理模式并非总是有益;其中 index_to_span 任务表现最差,最佳四样本精确匹配准确率仅为 6.75%。

来源:arXiv · AI · arxiv.org