AI BENCHNOTES
RAGBENCH NOTE 005Tool callsAgent loops
AI BENCHNOTES
THE COMPLETE WORKBENCHPick a machine part.
12 NOTES / 12 LABS / ZERO INCENSE

Start anywhere. The numbers are a useful route, not homework police.

00Animated bench map
CONCEPT 005civilized sorcery

The model requests.Your app decides.

A custom tool call is a typed cue slip. The model can name a function and propose arguments; your application still validates, authorizes, executes, and reports what happened.

  • PROPOSEDby the model
  • CHECKEDby your application
  • RETURNEDas fresh context
01 / THE SHORT VERSION

A call is a proposal, not permission.

Tool calling gives a probabilistic model a typed way to request deterministic software. The border between those two worlds is yours to police.

  1. 01
    CUE CALLER

    The model proposes.

    Selects a declared tool and produces structured arguments.

  2. 02
    STAGE MANAGER

    Your application decides.

    Parses, validates, authorizes, confirms, and routes the request.

  3. 03
    STAGE CREW

    The tool does.

    Runs ordinary code against an API, database, service, or device.

  4. 04
    SHOW REPORT

    The result informs.

    Returns to the conversation so the model can continue from evidence.

The model gets a call sheet. Your application keeps the keys to the building.

Scope note:hosted or built-in tools may run inside a provider's infrastructure. This bench note focuses on custom, client-side tools, where your application owns the executor and its safety boundary.

02 / INTERACTIVE DISPATCH

One cue. Four possible fates.

Choose what arrives at the application boundary. The model's proposal changes; your switchboard applies the same gates every time.

LIVE CALL SHEETWhere is order 1842?
DISPATCHED
MODEL PROPOSAL

toolget_order_status

call_idcall_7f3

arguments
order_id
"ord_1842"
GENERATED TEXT, EVEN WHEN IT LOOKS LIKE CODE
  1. 01
    Schemarequired string
  2. 02
    Authorizesupport:read
  3. 03
    Confirmread-only
  4. 04
    Executeorders service
EXECUTION SIDEORDERS APICONNECTED
TOOL RESULT · CALL_7F3{"status":"out_for_delivery","eta":"today"}
NEXT MODEL TURN

It’s out for delivery and due today.

The schema can catch a wrong type. Only application policy can decide whether this user may do this thing, to this object, right now.

03 / THE HANDSHAKE

It is a tiny protocol, not a magic method.

APIs use different envelope names, but client-side tools share a durable rhythm: definitions in, proposal out, execution elsewhere, result back in.

  1. 01
    APP → MODEL

    Declare the callable surface

    Send the user’s message plus tool names, descriptions, and parameter schemas.

  2. 02
    MODEL → APP

    Receive a structured proposal

    The model may answer normally—or return one or more calls with names, arguments, and identifiers.

  3. 03
    APP

    Cross the control boundary

    Parse, validate, authorize, ask for confirmation when needed, then invoke known code.

  4. 04
    TOOL → APP

    Capture the actual result

    Success, empty data, timeouts, and failures are all results. Normalize them deliberately.

  5. 05
    APP → MODEL

    Return evidence with its call ID

    Add the tool output to the next model input, paired with the call that requested it.

  6. 06
    MODEL

    Continue, don’t assume finished

    The model can answer, ask a question, repair a bad call, or request another tool.

ABOUT PARALLEL CALLS

Two calls can leave together. Their luggage tags still matter.

Independent work—say, weather for two cities—can run concurrently. Keep every call ID attached to its own result, preserve a deterministic return order, and avoid parallelizing actions that depend on one another.

04 / DESIGN THE CALL SHEET

A schema tells the model what it may ask for.

Good definitions reduce guessing. They do not replace runtime validation, authorization, or judgment about side effects.

TOOL DECLARATIONJSON-SCHEMA-LIKE
{
  name: "get_order_status",
  description:
    "Read delivery status for an order
     the signed-in user may view.",
  parameters: {
    type: "object",
    properties: {
      order_id: {
        type: "string",
        pattern: "^ord_[0-9]+$"
      }
    },
    required: ["order_id"],
    additionalProperties: false
  }
}
  1. 01

    Name it like code

    Specific verbs beat vague magic: get_order_status, not do_customer_thing.

  2. 02

    Describe when, not just what

    Tool descriptions help the model choose. Include boundaries and important distinctions between similar tools.

  3. 03

    Constrain the arguments

    Required fields, enums, formats, and closed objects turn fuzzy intent into a smaller, testable surface.

TOOL CHOICE

The model may choose. The application sets the menu.

  • Automaticanswer or call a useful tool
  • Requiredproduce some tool call
  • Constrainedchoose only a named subset

Exact controls and names vary by provider. Whichever mode you choose, expose the smallest useful set of tools for this turn.

05 / WHEN TO REACH FOR A TOOL

Use tools when software knows better.

The model is good at translating messy intent into a candidate action. Let trusted systems own facts, exact computation, and effects.

01

Fresh or private data

Look up orders, inventory, account state, or documents that are not in the model’s input.

get_order_status(…)
02

Deterministic work

Hand exact arithmetic, date logic, parsing, or domain rules to software built for exactness.

calculate_tax(…)
03

Real actions

Create tickets, send messages, schedule meetings, or issue refunds—with explicit gates.

create_support_ticket(…)
04

Specialized capabilities

Search, render a chart, query a database, run code, or operate a domain-specific service.

search_catalog(…)
DO NOT CALL THE CATERER FOR A GLASS OF WATER

Sometimes plain text is the whole job.

  • The answer is already in the supplied context.
  • The task is purely rewriting, classifying, or formatting.
  • A high-risk action has no meaningful authorization or confirmation path.
  • You exposed a warehouse of overlapping tools and hope the model invents your product architecture.
06 / THE BORING PART THAT SAVES YOU

Keep the model outside the velvet rope.

Treat every call as untrusted input assembled by a clever, probabilistic parser. Useful? Extremely. Pre-authorized? Absolutely not.

  1. 01
    REGISTRY

    Dispatch from an allowlist.

    Map known names to known handlers. Never eval a model-generated name, import path, URL, SQL fragment, or shell command.

  2. 02
    VALIDATE

    Validate meaning, not only shape.

    Check types and enums, then business invariants: object exists, amount is positive, dates are sensible, fields agree.

  3. 03
    AUTHZ

    Authorize against the real principal.

    Use the signed-in user, tenant, resource, and current policy. The model is neither the user nor an authority.

  4. 04
    CONFIRM

    Put friction before consequences.

    Separate reads from writes. Preview or confirm destructive, costly, external, or difficult-to-reverse actions.

  5. 05
    RELIABILITY

    Design for retries and duplicates.

    Use timeouts, idempotency keys, bounded retries, and explicit success states. A network error may hide a completed action.

  6. 06
    ERRORS

    Return useful errors, not secrets.

    Give the model structured, actionable failure context while redacting tokens, stack traces, private rows, and internal policy details.

SIDE-EFFECT LEDGERMore consequence → more application control
READLookup a delivery ETAauthorize + log
WRITECreate a reversible draftvalidate + authorize + preview
IRREVERSIBLESend, delete, charge, publishconfirm + idempotency + audit
07 / IMPLEMENTATION RECEIPT

The glamorous middle is mostly application code.

SDKs may automate pieces of this loop. Keep the ownership model visible anyway: the client decides which handler runs and what context returns.

safe-dispatch.tsPROVIDER-NEUTRAL PSEUDOCODE
// Provider-neutral sketch. Adapt the message envelopes to your SDK.
const response = await model.generate({
  messages,
  tools: publicToolDefinitions
});

for (const call of response.toolCalls ?? []) {
  // 1. Resolve only a server-owned handler.
  const tool = toolRegistry.get(call.name);
  if (!tool) throw toolError(call.id, "unknown_tool");

  // 2. Model output is untrusted input.
  const args = tool.schema.parse(call.arguments);

  // 3. The current user and resource decide permission.
  await tool.authorize({user, args});

  // 4. Put a human gate before consequential effects.
  if (tool.requiresConfirmation(args)) {
    await confirmations.require({user, call, args});
  }

  // 5. Execute ordinary application code.
  const result = await tool.execute(args, {
    idempotencyKey: call.id,
    timeoutMs: 8_000
  });

  // 6. Pair evidence with the call that requested it.
  messages.push(response.message, {
    role: "tool",
    toolCallId: call.id,
    content: JSON.stringify(toModelSafeResult(result))
  });
}

// The model gets a new turn with the tool result in context.
const final = await model.generate({messages, tools: publicToolDefinitions});

The call ID is not decoration. It ties a result to the request that produced it, especially when multiple calls are in flight.

Tool output is new context. The model sees the returned data on a later turn; it is not permanent memory and it should still be treated as potentially untrusted content.

08 / CLEAR THE LINES

Six crossed wires.

Most tool-calling bugs begin when a convenient API abstraction gets mistaken for a security or execution model.

01

“The model ran my function.”

For client-side tools, it emitted a structured request. Your runtime selected and executed the handler.

02

“The arguments matched the schema, so they’re safe.”

Shape is one gate. You still need semantic validation, authorization, confirmation, and resource limits.

03

“A tool result is the final answer.”

It is evidence added to the next model turn. The model may summarize it, call again, ask a question, or make a mistake.

04

“An error means the agent is done.”

A structured error can help the model repair arguments or choose another path. Bound the retries.

05

“Parallel calls are always faster.”

Only independent calls belong in parallel. Dependent writes need ordering, and every result must keep its call ID.

06

“More tools make the model more capable.”

A crowded, overlapping menu can make selection worse. Expose the smallest relevant tool set per turn.

Let the model be imaginative at the interface. Be gloriously boring at the boundary.

AI BENCHNOTES

For builders who prefer understanding the machine to worshipping it.

Back to top ↑