RareDiseaseBench: Can AI Agents Diagnose Rare Disease?
AI Summary: The authors introduced RareDiseaseBench, a benchmark of 35 clinically challenging rare‑disease cases that requires AI agents to request and integrate heterogeneous data (genome sequencing, RNA‑seq, imaging, etc.) before producing a diagnosis and management plan. Across 31 model‑harness configurations and 3,255 trial runs, Gemini 4 Argon on the Pi harness achieved the highest preliminary pass rate of 43.8 %, outperforming Opus 5.5 / Claude Code and GPT‑6 Astra / Pi, which reached 33.3 % (the latter at half the compute cost). Performance correlated with model strength and token efficiency, while weaker models frequently failed to request appropriate data or to process large sequencing files efficiently. In total, 29 of the 35 tasks were solved by at least one configuration, with 16 tasks passed in all three attempts by some model.