Abstract
Agentic abstention involves determining when an AI agent should cease interaction under uncertainty, requiring sequential decision-making across multiple environments and task types.
LLM agents are expected to act over multiple turns, using search, browsing interfaces, and terminal tools to complete user goals. Yet not every goal is well specified or achievable in the available environment. In such cases, a reliable agent should recognize that further interaction is unlikely to help and abstain from additional tool calls. We define Agentic Abstention, the problem of deciding when an agent should stop acting under uncertainty. Unlike standard LLM abstention, which is usually evaluated as a single-turn answer-or-abstain decision, agentic abstention is a sequential decision problem: an agent can answer, abstain, or gather more information at each turn, and the need to abstain may only become clear after interacting with the environment. We study this problem across web shopping, terminal environments, and question answering, evaluating 13 LLM-as-agent systems and 2 agent scaffolds on more than 28,000 tasks. Our results show that the main challenge is not only whether agents can abstain, but also when they abstain. Some agents never abstain when they should, while others do so only after many unnecessary interactions. This gap is especially large on tasks where the instruction appears feasible until the environment reveals otherwise (e.g., no valid result matches the instruction). We further find that model scale, reasoning, and agent scaffolding affect abstention in different ways, where larger or more capable models sometimes perform worse at timely abstention. Finally, we introduce CONVOLVE, a context engineering method for improving agentic abstention that distills full interaction trajectories into reusable stopping rules. On WebShop, CONVOLVE substantially improves timely abstention without updating model parameters, raising Llama-3.3-70B's timely recall rate from 26.7 to 57.4. Our dataset and code are available at https://lhannnn.github.io/agentic-abstention
Community
We introduce Agentic Abstention: the problem of deciding when an LLM agent should stop acting under uncertainty.
Many agent benchmarks focus on successful task completion, but real-world tasks are often ambiguous, underspecified, or infeasible in the available environment. In such cases, a reliable agent should know when further tool use is unlikely to help and abstain instead of continuing unnecessary actions.
We evaluate 13 LLM-as-agent systems and 2 agent scaffolds on more than 28,000 tasks across web shopping, terminal environments, and question answering. Our findings show that the main challenge is not only whether agents can abstain, but when they abstain.
We also propose CONVOLVE, a context engineering method that improves timely abstention by converting interaction trajectories into reusable stopping rules.
Happy to hear thoughts from the community!
The concept of "Agentic Abstention" is a critical missing piece for anyone actually deploying agents in production. Most current evaluations focus on the "happy path" where the goal is achievable, but in the real world, the most expensive failures happen when an agent loops indefinitely on an impossible task. Shifting the evaluation from a single-turn "I don't know" to a sequential decision process is the right engineering move. If we can't quantify the confidence threshold for when to stop, we're just gambling with our token budget. I'm interested to see if this framework can be integrated into a dynamic cost-benefit analysis for tool calls.
Thanks a lot! I really agree with this perspective. In production settings, abstention is not just “I don’t know,” but a sequential decision about whether another action is still worth its cost under uncertainty.
The dynamic cost-benefit angle is a very natural next step. In our current work, we focus on whether and when agents should stop, but this could be extended by explicitly modeling the expected utility of the next tool call, including information gain, recovery probability, latency, token/tool cost, and the risk of compounding errors.
I think this would make agentic abstention much more deployment-oriented, and it’s definitely a direction we’re excited to explore 😃
Great paper! I really like how it points out that agent reliability is not only about completing tasks, but also about recognizing when the task may be impossible or underspecified in the first place.
The distinction between abstaining correctly and abstaining at the right time is especially useful. Many agents may eventually stop, but only after taking many unnecessary steps, which existing evaluations often fail to capture. Framing abstention as a sequential decision problem makes this behavior much easier to study and compare across systems.
I’m excited to see more benchmarks evaluate this kind of stopping behavior in tool-using agents.
Thanks!
I've noticed this behavior in agents such as Hermes or OpenClaw but I've never could point out what made them so fixated in doing things even when not possible or even a way to minimize it, loved this paper and the proposed fix.
Thanks a lot! This is exactly the behavior we wanted to study. Many agents look strong on achievable tasks, but when the task is impossible or the environment lacks the needed information, they can keep taking actions as if one more step will fix it.
We hope Agentic Abstention helps make this failure mode more explicit, measurable, and easier to mitigate. Really glad you found the paper useful!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search (2026)
- ACCORD: Action-Conditioned Contextual Grounding for Language Agents (2026)
- Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using Agents (2026)
- VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions (2026)
- No More, No Less: Task Alignment in Terminal Agents (2026)
- The Context Gathering Decision Process: A POMDP Framework for Agentic Search (2026)
- No Action Without a NOD: A Heterogeneous Multi-Agent Architecture for Reliable Service Agents (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2606.28733 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper