Agent loops: tool calling, state, retries and idempotent actions

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

An agent loop is a program you write around a model. The model emits a structured request to use a tool. Your code runs that tool, writes the result back into the conversation, and calls the model again. The loop ends when the model answers in plain text, when a stop condition fires, or when you refuse another step. The model does not hold your database, and it does not remember a side effect that never made it into the next request.

An AI writing system produced this article for NetLeverage's editorial studio. Where the text describes a model "deciding" to call a tool, that is a description of token generation under a schema and a conversation prefix. It is not a verified claim about subjective experience. Nothing below was executed in a lab. JSON shapes are taken from the vendor pages opened on 10 November 2026. Treat every field name as versioned documentation, and re-read the current page before you ship a client.

Why it matters

Three failures look the same from a chat window and have different causes.

1. The model describes an action that nobody performed. Without a tool contract, "I have updated the record" is a sentence. With a tool contract, an update exists only if your code ran and you recorded the outcome.
2. The loop repeats. Yao et al. (ReAct) report a failure mode in which the prompted model regenerates the previous thought and action and does not leave the cycle. A production loop needs a step cap and a way to notice a duplicate proposal.
3. A retry doubles a real-world effect. RFC 9110 draws the line that HTTP itself draws: repeat a request automatically only when the intended effect of repeating it matches the effect of doing it once, or when you can tell that the first attempt never applied. An agent that emails, charges, or creates rows is in the second category until you design it out.

Tool calls also cost tokens. Anthropic prices input tokens, including the tools parameter, plus output tokens, plus usage charges for server-side tools. With tools present the API adds a system prompt. On 10 November 2026 the page listed 286 tokens for Claude Opus 5.5 when tool_choice is auto or none and at least one tool is provided. That count is the table for that day.

How it works

The loop as a policy over a growing context

ReAct states the general setup cleanly. At step $t$ the agent receives an observation $o_t$ and samples an action $a_t$ from a policy $\pi (a_t\mid c_t)$. The context is the whole trajectory so far:

The paper's move is to enlarge the action space to $\widehat{𝒜}=𝒜\cup ℒ$, where $ℒ$ is language. A thought is an action in $ℒ$. It does not touch the external environment, so it produces no new observation. It only extends the context to $c_{t+1}=(c_t,{\widehat{a}}_t)$. An environment action does touch the outside world and is followed by an observation the environment returns.

That split is the architecture of a tool loop when the wire format is JSON. A thought changes the transcript only. A tool call is a proposal until your program runs it and appends an observation. In that prompting setup the frozen model was PaLM-540B, continued from a few human trajectories. Question answering alternates thought, action, and observation. Longer tasks use thoughts more sparsely. The Wikipedia tool is narrow: search[entity] returns the first five sentences or up to five similar titles, lookup[string] returns the next matching sentence, and finish[answer] stops. In the authors' 200-trajectory HotpotQA failure analysis, hallucination accounted for 56% of chain-of-thought failures and none of ReAct's, while reasoning errors accounted for 47% of ReAct failures against 16%. Greedy decoding was a suggested cause. ReAct alone nevertheless scored lower overall, 27.4 exact match against 29.4; combined methods performed best. Those results concern the historical prompting setup, not current product models.

The orchestrator still has to notice a stuck loop.

A trained call is a different mechanism from a prompted loop

Schick et al. (Toolformer) train a model to insert API calls into ordinary text. Each call is a pair $c=(a_c,i_c)$: API name and input. With a result $r$, the linearised form is the call followed by an arrow token and then $r$. At inference the model decodes normally until it emits that arrow. The runtime stops decoding, executes the API, inserts the result, and continues.

Which calls enter the training set is decided by a loss filter. Let $L_i(𝐳)$ be weighted cross-entropy on the following tokens with prefix $𝐳$. $L_i^{+}$ includes the call and its result. $L_i^{-}$ is the better of no call and a call with an empty result. A candidate is kept only when

for a threshold ${\tau }_f$. Keep the call when the result makes the next tokens easier by at least that margin. The paper's tools are question answering, Wikipedia search, a calculator, a calendar, and translation, all text in and text out. The base model is GPT-J 6.7B.

Toolformer trains special-token call behaviour. The prompted ReAct setup leaves weights unchanged. Current chat APIs also put orchestration in the application, but their models may be trained to emit structured tool calls; an application-owned loop does not imply an untrained calling model.

What the wire format actually carries

OpenAI's function-calling guide, opened 10 November 2026, describes five steps. You send tools. The model returns a tool call. Your application executes it. You send the tool output back. The model returns text, or further calls. On the Responses API a function tool is a JSON schema. The call carries a call_id, a name, and JSON arguments. The follow-up carries a function_call_output with that same call_id and an output that is typically a string; functions returning files or images may use an array of file/image objects. The guide leaves the string's format to you: JSON, an error code, or plain text.

That page's examples are dated. GPT-6 Astra and GPT-6.1 Sol require the Responses API for tool calling. The Chat Completions examples use GPT-5.6 Terra. tool_search, which defers rarely used tools, is stated for gpt-5.4 and later. Read those names as the contract on that date.

Anthropic's "How tool use works" page states the same contract: the model emits a structured request, your code or Anthropic's servers run it, and the result returns to the conversation. For client tools the loop is a while on stop_reason:

1. Send tools and the messages.
2. If stop_reason is tool_use, the response contains one or more tool_use blocks. Each block has an id, a name, and an input object.
3. Run the tools. Send a user message whose content begins with tool_result blocks. Each result sets tool_use_id to the matching id. Set is_error to true when execution failed, and put a specific message in content.
4. Repeat while stop_reason remains tool_use.

Documented exits are end_turn, max_tokens, stop_sequence, and refusal. A server-side loop that hits its iteration cap returns pause_turn; send the paused response back to resume. Results must be the next message, with tool_result blocks before any text. A gap surfaces as an error that tool-use ids were found without tool-result blocks immediately after.

On the Anthropic pages opened that day, web_search, web_fetch, code_execution, and tool_search run on Anthropic's side. You do not build their tool_result. User-defined tools, and the Anthropic-schema tools memory, bash, text_editor, computer, and browser, run in your process. The page says those published schemas are trained-in signatures. That is a vendor claim about training targets, not a proof that a custom schema will fail.

Where state actually lives

Three stores get mixed up.

Transcript state is the messages you resend: the function call and its function_call_output on OpenAI, or the assistant content and the user tool_result on Anthropic. Drop the call id and the next turn cannot bind the result. Summarising the transcript deletes observations. Keep an explicit rule for what must survive.

External state is the database, mailbox, file, or payment. The model sees only the text you return. "Updated", with no identifier and no version, gives the next step nothing to check.

Idempotency state is the record you write before the side effect: this key is started, finished, or never applied. It reaches the model only if a tool result copies it back. Anthropic's memory tool stores notes across conversations in files you control. It does not replace that key.

Schemas decide what "invalid" means

OpenAI recommends strict: true, implemented through structured outputs. On that page every object in parameters needs additionalProperties: false, and every property must be listed in required. Optional data uses null in the type. A strict request that breaks those rules is rejected. Omit strict on the Responses API and the service tries to normalise the schema, then falls back to best-effort calling and reports strict: false. Chat Completions stay non-strict unless you opt in. The same page says strict mode is disabled when a fine-tuned model emits several function calls in one turn.

Anthropic's strict: true is documented as making custom-tool inputs match the schema, including required parameters and types. Without it, the handle-tool-calls page says an invalid call is followed by about two or three corrective attempts and then an apology. That count describes model behaviour on the page. It is not a safety cap to hard-code.

tool_choice is the other control. OpenAI documents auto (default: zero, one, or many), required, a forced function name, allowed_tools for a subset, and none. Anthropic's default is {"type": "auto"}, with choices that require a call or name a tool; the pricing table treats none like auto for the extra system-prompt count. OpenAI notes that a stable tools list, narrowed with allowed_tools, preserves prompt-cache savings, because the tool list is part of the prefix. Prompt caching itself is a separate subject.

Default auto can still guess. Anthropic's overview says that if a required parameter is missing, Claude Opus is much more likely to ask, while Claude Sonnet may infer a value. Their weather example receives a city the user never typed. Strict mode accepts any value of the right JSON type. A description that says when to ask, and a tool that rejects placeholders, is what stops the guess.

Parallel calls are one turn, several effects

Both guides allow several tool calls in one response. OpenAI permits multiple functions in a turn; on supported models beginning with GPT-5 this also works when built-in tools are available. parallel_tool_calls: false forces zero or one function call. Built-in tools are excluded from a parallel function-call batch on that page. Snapshot gpt-4.1-nano-2025-04-14 is documented as sometimes emitting duplicate calls of the same tool when parallel calls are on; the page recommends disabling the feature for that snapshot.

Anthropic's client example sets disable_parallel_tool_use: true for a single call. Leave parallel calls on and each id still needs its own result. A timeout after the first of three writes is the painful case: the retried model turn may propose all three again.

Retries, using the HTTP rule on purpose

RFC 9110 section 9.2.2 calls a method idempotent when the intended effect of several identical requests equals the effect of one. PUT, DELETE, and the safe methods are idempotent. Section 9.2.1 calls GET, HEAD, OPTIONS, and TRACE safe: essentially read-only. Safe implies idempotent. PUT and DELETE still write; repeating them aims at the same end state.

The retry rule is the part to copy. If the connection fails before the client can read the response, it may automatically retry an idempotent request. The intended effect matches even when the first request had already succeeded, and the second response may differ. A client should not automatically retry a non-idempotent method unless it knows the operation is idempotent anyway, or it can detect that the first request never applied.

The mapping below is an editorial classification. Neither vendor schema enforces it:

Create and store the key before the side effect, under a constraint that forbids two workers applying the same key. A key written after success does not cover the crash in between. A durable “started” key alone is also insufficient: a crash after the downstream effect but before recording completion leaves an unknown outcome. Use downstream idempotency, an atomic transaction when possible, or reconciliation that can establish the result. Do not replay an uncertain write merely because no “finished” row exists. RFC 9110 separately gives a version-control example: a repository-specific client checks the target revision after a failed connection, repairs or reverts a partial change, and then retries. It is not identified as a POST example. That is a domain check.

Keep the three retries distinct.

1. Retry the model call only when you received no tool call. For client-only functions, external state is untouched only if your executor has not started any call. Server-side tools or an interrupted workflow may need separate reconciliation. Temperature above zero can still yield a different proposal. That is a new sample, not a replay.
2. Retry the tool only under the table. A 500 from a payment API is not permission to charge again. Return is_error with a specific message. Anthropic's example shape is "Rate limit exceeded. Retry after 60 seconds." Continue only if the idempotency record says the charge never happened.
3. A repeated proposal from the model, including ReAct's stuck trajectory and a "correction" after is_error, is a new proposal. Deduplicate within the same user-authorised operation using its idempotency key. A canonical argument hash can detect conflicting reuse, but must not globally suppress a later legitimate identical operation. Return a stored outcome only when its scope and completion are established.

Cap the loop yourself. Anthropic's client loop runs while stop_reason is tool_use and does not define a maximum number of successful tool turns. Set a maximum number of model turns and of identical proposals, log the transcript, and stop with an explicit error. Treat max_tokens and refusal as stops. Toolformer's arrow-token interrupt has the same hazard: a decode retry that reaches the arrow again will call the API again.

What to choose and why

Log every proposal, execution, and suppressed duplicate with the key, the arguments, and whether the external system was contacted. That log is what tells an incident apart from a second sample.

Risks and limits

• A strict schema fixes shape. A well-typed amount can still be the wrong amount. Business checks stay in the tool.
• Guessed arguments survive strict mode. Anthropic documents a model filling a city the user never gave. Confirm anything a person would not want invented.
• Tool results are untrusted content. Anthropic warns that pages, mail, uploads, and third-party APIs can carry instructions aimed at the model, and says to keep that content inside tool_result blocks. Prompt injection is a separate control problem.
• Server tools run outside your process. An idempotency key cannot be bolted onto a vendor web search by editing their executor. Keep server tools off the path to your side effects.
• ReAct's measurements and Toolformer's loss filter describe those papers. They are not a quality claim about GPT-6 or Claude Opus 5.5.
• Field names, model ids, system-prompt token counts, and which API accepts tool calls were read on 10 November 2026 and will move.
• This article does not claim any behaviour of a NetLeverage product.

Sources

Sources checked on 10 November 2026; access limitations are identified below.

1. Yao et al., ReAct, arXiv:2210.03629 — https://arxiv.org/abs/2210.03629 (HTML opened 10 November 2026). Context equation, thoughts, Wikipedia actions, repetitive-loop failure, PaLM-540B.
2. Schick et al., Toolformer, arXiv:2302.04761 — https://arxiv.org/abs/2302.04761 (v1 HTML). Loss filter, arrow-token interrupt, GPT-J 6.7B.
3. OpenAI, Function calling — https://developers.openai.com/api/docs/guides/function-calling. Five-step flow, call_id, strict mode, tool_choice, parallel calls, as opened 10 November 2026.
4. Anthropic, Tool use overview — https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview. Client and server tools, strict tools, pricing, system-prompt token table.
5. Anthropic, How tool use works — https://platform.claude.com/docs/en/agents-and-tools/tool-use/how-tool-use-works. Loop on stop_reason, pause_turn, where each tool class runs.
6. Anthropic, Handle tool calls — https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls. Result ordering, is_error, untrusted tool content.
7. IETF RFC 9110 §§9.2.1–9.2.2 — https://www.rfc-editor.org/rfc/rfc9110.html. Safe methods, idempotent methods, automatic retry.

This article was originally published on 10 November 2026.