New Benchmark Reveals AI Agents Struggle with Complex Instructions

A recent study has found that advanced AI models follow complex instructions less than 30% of the time. The benchmark, named AGENTIF, was developed by researchers at Tsinghua University and Zhipu AI and published on May 22, 2025. It evaluated 707 human-annotated instructions across 50 task categories, highlighting the challenges faced by AI in real-world applications.

The AGENTIF benchmark tests instructions that average 1,723 words and contain nearly a dozen constraints, which are categorized into formatting, semantic, and tool specifications. The study revealed that models particularly struggled with tool specifications, which require not only language comprehension but also the correct sequencing of actions across various systems. This complexity is compounded when instructions conflict with the models' pre-existing training biases, leading to further failures in execution.

The evaluation methodology employed by AGENTIF combines code-based, LLM-based, and hybrid methods to assess how well AI models satisfy these constraints. The findings indicate that the gap between demonstration and actual deployment of AI remains significant, necessitating human oversight and careful task management in enterprise settings. Furthermore, regulatory dynamics in China add additional constraints for companies like ByteDance and Alibaba, complicating the deployment of AI systems even further.

© 2026 KLEA News. All Rights Reserved. This article is provided for informational purposes only. It is not offered or intended to be used as legal, tax, investment, financial, or other advice.

Source: KLEA News