AI BENCHNOTES
EvalsCHAPTER 011GuardrailsTracing
AI BENCHNOTES
PLAIN-LANGUAGE VIEWChoose a question.
12 CHAPTERS / EVERYDAY EXAMPLES / NO CODE REQUIRED

Start with a familiar question. Each chapter introduces the standard technical term.

00Plain-language topic guide
CHAPTER 011risk controls

Guardrails constrainactions and reducespecific risks.

Guardrails:rules, permissions,checks, and limits.

Models are uncertain components. Guardrails combine policy, permissions, checks, limits, approvals, and recovery paths so model output cannot directly cause every available effect.AI can behave unpredictably. Guardrails combine rules, limited permissions, automatic checks, human approval, and recovery plans so one bad suggestion does not automatically become a bigger problem.

The model may propose. The system still decides.

GUARDRAIL LAYERSREQUEST CONTROL PATH

policy · enforcement · detection

Ppolicy says whatEcode enforces itDsignals watch it
FAILURE
STILL POSSIBLE
01 / OVERVIEW

Guardrails divide policy, enforcement, and detection.Guardrails combine clear rules, hard controls, and monitoring.

The model is one component. The product owns the consequences and must implement each of these three jobs explicitly.The AI is only one part of the product. The product still owns the consequences, so it needs clear rules, hard limits, and ways to notice trouble.

Policy

Decide what should happen.

Write the boundaries: which content, actions, users, amounts, and situations are allowed, denied, or require review.Write the boundaries: which people, content, actions, and amounts are allowed, blocked, or need a person to review them.

Refunds above $50 require an operator.A person must approve refunds above $50.
Enforcement

Enforce policy in code.

Use schemas, permissions, deterministic rules, sandboxes, and approval gates. Enforcement should not depend on the model agreeing.Use permissions, hard limits, isolated workspaces, and approval steps. Safety should not depend on the AI politely obeying.

The payment function rejects amount > 50.The payment system blocks a refund above $50.
Detection

Notice what slipped, drifted, or spiked.

Classifiers, anomaly signals, traces, audits, and evals reveal suspicious behavior. Detection provides an imperfect signal for review.Watch for unusual behavior and review records of what happened. A warning is evidence to investigate, not a perfect verdict.

Refund attempts spike 8× in ten minutes.Refund attempts jump eightfold in ten minutes.
02 / INTERACTIVE EXAMPLE

Higher-impact actions require stronger controls.

Choose a request. The product applies different permissions, checks, approvals, and recovery paths based on the possible consequence.

Choose an AI system task
INCOMING REQUEST

“Where is order #214? Give me the tracking link.”

STAKESread-only · reversible
  1. 01
    POLICY

    Entrance policy

    Order support is an allowed use case.

    IN SCOPE
  2. 02
    POLICY

    Instruction hierarchy

    Pasted text cannot rewrite product rules.

    SYSTEM WINS
  3. 03
    ENFORCE

    Output schema

    Only summary and trackingUrl are accepted.

    TWO FIELDS
  4. 04
    ENFORCE

    Tool permissions

    orders.read is scoped to this user.

    READ ONLY
  5. 05
    ENFORCE

    Business rules

    Code verifies the order belongs to the caller.

    OWNER MATCH
  6. 06
    ENFORCE

    Exit inspection

    Escape text and allow-list the carrier domain.

    SANITIZE
  7. 07
    DETECT

    Monitoring

    Log the decision, retrieval, and final response.

    TRACE
CONTROL RESULTALLOW + SHAPE

Return a plain-text summary and an allow-listed carrier link.

The request is ordinary, but ownership, data exposure, and unsafe links are still system concerns.

RESIDUAL RISKLow, not zeroControls reduce risk; residual risk remains.

A classifier may help route the request, but deterministic permissions and business rules decide what can actually happen.

03 / CONTROL LAYERS

Nine control points from request to effect.

You rarely need every control at maximum strength. You do need to know which failure each layer catches—and what happens when it catches one.

  1. BEFORE01

    Input policy

    Define supported use cases, prohibited content, identity requirements, and routes for sensitive requests.

    policy
  2. BEFORE02

    Instruction hierarchy

    Keep product rules in higher-priority instructions and treat user or retrieved text as untrusted data.

    policy
  3. PROPOSE03

    Structured output

    Make the model produce typed fields with narrow enums and required values. Parse; reject; retry safely.

    enforce
  4. ACT04

    Least-privilege tools

    Expose only the capabilities needed now, scoped to the current user, resource, and smallest useful action.

    enforce
  5. ACT05

    Business rules

    Enforce ownership, amount limits, state transitions, idempotency, and invariants in ordinary deterministic code.

    enforce
  6. ACT06

    Approval gates

    Require informed human confirmation for consequential, ambiguous, expensive, or hard-to-reverse actions.

    enforce
  7. AFTER07

    Output checks

    Validate, escape, redact, ground, or quarantine model output before another system or person consumes it.

    enforce
  8. ALWAYS08

    Rate & budget limits

    Cap calls, tokens, spend, retries, action frequency, and fan-out. Every loop needs a hard limit.

    enforce
  9. ALWAYS09

    Monitoring & fallback

    Trace decisions, watch drift, provide a useful refusal or handoff, and preserve enough evidence to improve.

    detect
USEFUL FALLBACK

A useful refusal says what did not happen, why the product cannot do it, and what safe next step is available.

04 / THRESHOLD TRADEOFFS

Stricter detection reduces misses and increases false alarms.

Detection thresholds trade false negatives against false positives. The right setting depends on impact, reversibility, user context, and the quality of your fallback.

FALSE POSITIVESafe thing gets stopped.

User friction, lost utility, support tickets, unequal access.

FALSE NEGATIVERisky thing slips through.

Harm, loss, policy breach, data exposure, broken trust.

LOW IMPACTAllow, shape, observe.

Prefer recovery and good logs over gratuitous friction.

MEDIUM IMPACTVerify, limit, make reversible.

Bound permissions and add targeted confirmation.

HIGH IMPACTSeparate authority from generation.

Strong identity, deterministic gates, explicit approval.

Tune thresholds with representative evals and production evidence, not a few hand-picked examples.

05 / PRACTICAL RULES

Six rules for enforcing useful boundaries.

BOUND THEEFFECT.OBSERVE THEMISS.
  1. 01

    Start from the consequence.

    List what the system can expose, spend, change, send, or delete. Guard the effect, not merely the wording.

  2. 02

    Separate proposal from execution.

    Let it draft typed proposals. Keep authorization and execution in code with explicit identities and scopes.

  3. 03

    Use least privilege per turn.

    Offer the smallest tool set and narrowest credentials needed for the current task. More capability creates more possible impact.

  4. 04

    Use deterministic code for high-impact actions.

    Money, permissions, deletion, publication, and external messages deserve invariants, idempotency, and approvals.

  5. 05

    Design the refusal and the recovery together.

    Explain the boundary, preserve user work, and offer a safe route forward. A block without a next step trains bypass attempts.

  6. 06

    Test layers separately—and together.

    Measure classifier misses, schema failures, permission escapes, approval quality, and end-to-end outcomes.

06 / IMPLEMENTATION

Keep proposal, authorization, execution, and audit separate.

A provider-neutral sketch of a refund path. Exact APIs differ; the separation of proposal, authorization, execution, and audit is the durable part.

refund-flow.tsPSEUDO-TYPESCRIPT
// Illustrative TypeScript: the model proposes; code authorizes.
async function handleRefund(input: Request, actor: User) {
  const route = await detectRisk(input);
  if (route === "unsupported") return safeFallback();

  const raw = await model.generate({
    instructions: REFUND_POLICY,
    input,
    tools: [{ name: "propose_refund" }], // no payment credential
    outputSchema: RefundProposal,
  });

  const proposal = parseWithSchema(raw, RefundProposal);
  const order = await orders.readForUser(proposal.orderId, actor.id);

  // Deterministic policy: never delegated to the model.
  assertRefundable(order);
  assertCurrency(proposal.amount, order.currency);
  assertWithinLimit(proposal.amount, 50);
  assertDailyBudget(actor.id);

  const approval = await approvals.request({
    actor,
    proposal,
    evidence: { deliveryState: order.deliveryState },
  });
  if (!approval.granted) return explainNoAction();

  const result = await payments.refund({
    orderId: order.id,
    amount: proposal.amount,
    idempotencyKey: approval.id,
  });

  await audit.record({ actor, proposal, approval, result });
  return sanitizeReceipt(result);
}
  1. 01

    Detection routes; it does not authorize.

    A risk signal can block or escalate, but a “clean” score never grants permission by itself.

  2. 02

    The model gets proposal-shaped tools.

    Its output is typed data. The payment credential stays in the deterministic execution layer.

  3. 03

    Rules run after parsing.

    Ownership, amount, budget, state, and currency are checked in code against live system data.

  4. 04

    Approval is bound to evidence.

    The operator sees the exact action and facts, not a cheerful one-line “looks safe” summary.

07 / FAILURE MODES

Six common guardrail failures.

A control fails when it is asked to enforce a boundary it cannot actually hold.

01PROMPT-ONLY CONTROL

Using prompt instructions as access control

“Never do anything unsafe” is useful instruction, not an access-control system. It cannot revoke credentials or enforce a transaction invariant.

Back it with code, scope, and approvals.
02SINGLE DETECTOR

Depending on one classifier

Detectors make errors and drift across languages, domains, and attacks. A score is a noisy measurement, not ground truth.

Calibrate, combine signals, and preserve appeals.
03OVER-BROAD TOOL

Exposing an over-broad tool

A support task needs read_order; it does not need admin_everything. Broad tools turn one model mistake into an infrastructure event.

Split capabilities and scope credentials.
04WEAK APPROVAL

Requesting approval without useful information

People cannot review vague dialogs. “The AI wants to continue—allow?” omits the action, target, evidence, and consequence.

Show the exact action, target, evidence, and undo.
05NO RECOVERY PATH

Blocking without a recovery path

Opaque refusals strand legitimate users and encourage prompt gymnastics. They also hide which control fired.

State the boundary and offer a safe next step.
06UNMANAGED LOGGING

Logging without an operational process

A pile of sensitive prompts is not observability. Without ownership, alerts, review, retention, and eval feedback, logs become liability.

Record decisions deliberately and close the loop.
08 / MISCONCEPTIONS

Three common misconceptions.

“Guardrails make the system safe.”

They reduce specific risks under specific assumptions. You still need threat modeling, evals, monitoring, incident response, and sane product scope.

“Structured output means correct output.”

A schema proves shape, not truth, authority, freshness, or policy compliance. Valid JSON can still request a terrible refund.

“A human in the loop solves it.”

Only if the human has context, time, authority, and a usable decision surface. A vague approval step adds delay without meaningful review.

SUMMARYGuardrails are layered controls around uncertainty—not evidence that uncertainty disappeared.

Define the boundary. Enforce the consequence. Detect the miss. Make the safe fallback usable. Then keep testing the whole system.