arXiv · NLP· William Philipp, Finn Fassbender, Daniel Fister, Thorsten Langer, Martje G. Pauly, Rebecca Herzog, Markus A. Hobert, Theresa Paulus, Alexander Baumann, Chi Wang Ip, Lukas L. Goede, Johanna Reimer, Sebastian L\"ons, Ronald B\"ock, Sebastian Fudickar·· 2026-10-07AI 评分32
MedQADE:评估德语开放式临床问题中LLM评分者的一致性、偏见与弃权表现
Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention
AI 导读
研究引入包含3,800组问答的德语临床基准MedQADE,由10名医生标注,用于评估LLM-as-a-judge系统。Gemini 3 Flash评分与医生参考标准接近(kappa=0.694),但四个模型对自身回答评价偏高,且医生弃权率随难度增加。结果表明,LLM评估者存在偏见,需医生验证后方可实现自动化评估。
来源:arXiv · NLP · arxiv.org