Same Show Every Night — Building Deterministic Systems from Non-Deterministic AI

Congratulations. You've just cast the most talented performer in showbiz. This actor can do Shakespeare, tax law, Python, and a shockingly good pirate accent — sometimes all in the same sentence, which is exactly the problem. Because the role you've cast them in is "Enterprise Backend Service, a drama in JSON," and the audience — your users, your auditors, your on-call engineer — expects the same performance every single night.

Ask a normal function "what's 2+2" a thousand times, you get "4" a thousand times. Ask your new star the same question and you get "4", "Four!", "Great question — the answer is 4 🎉", and once, memorably, a haiku about arithmetic. The talent is real. The consistency is… a work in progress.

This is the fundamental comedy of building on LLMs: we are casting improvisers in roles written for machines. The punchline — and the entire discipline of AI engineering — is that you don't fix this by begging the actor to behave. You fix it by building a theater around them.

ACT I — MEET THE TALENTWhy the actor never does the same take twice

An LLM doesn't "look up" an answer; it rolls weighted dice over the next token, again and again, at performance speed. Turn temperature up and the dice get loose and jazzy. Turn it to 0 and you get greedy decoding — mostly repeatable, but still not a contract: floating-point quirks, batching effects, and model updates on the provider side mean even your most sober take can drift a syllable between shows.

So let's retire the fantasy up front: you will not make the actor deterministic. Improvisation is not a bug in this performer — it's the engine of the talent. The system around the actor is where determinism lives. The director's mantra, and this article's thesis:

Don't make the model deterministic.
Make the show deterministic.

ACT II — HAND THEM A SCRIPTStructured outputs: freedom inside a format

The first crew member you hire is the scriptwriter. You stop asking the actor open questions ("tell me about this invoice, in your own words, surprise me") and start handing them a script with fixed blanks: your lines are a JSON object, these are the fields, these are the types, improvise the content — never the shape.

Every serious model API now supports this: JSON schemas, structured outputs, function/tool calling. It's the difference between "monologue about the customer's mood" and:

The script — a schema the actor cannot wander out of

{
  "sentiment":  "negative",          // enum: positive | neutral | negative
  "confidence": 0.87,                // number, 0–1
  "refund_eligible": true,           // boolean, no interpretive dance
  "reason_code": "DAMAGED_ITEM"      // enum, not freestyle poetry
}

Notice what happened: the actor still acts — judgment, nuance, reading between the lines of a furious customer email. But the performance now arrives in a costume your downstream code can dress-check in one line. Half of "AI is unreliable" complaints are really "I asked for vibes and received vibes."

ACT III — THE SCRIPT SUPERVISORValidate, retry, and never trust a live performance

Next hire: the script supervisor — the person in the wings with a clipboard who checks every line against the script and, when the actor ad-libs, stops the take and says "again, from the top, and this time the date field is a date."

In code, that's a validation gate and a bounded retry loop. Parse the output against the schema. Check the business rules the schema can't express (refund amount ≤ order total; date not in 1847). If it fails, retry with the error fed back — the actor is genuinely good at taking notes — and after N takes, fall back to a deterministic path: a default, a queue, a human.

ScriptSupervisor.cs — bounded takes, then fallback (.NET)

public async Task<RefundDecision> GetDecisionAsync(string email)
{
    for (var take = 1; take <= 3; take++)          // max three takes
    {
        var raw = await _model.CompleteAsync(_prompt.Render(email));

        if (!RefundDecision.TryParse(raw, out var d, out var errors))
        {
            _prompt.AddNote(errors);              // "again — with the fixes"
            continue;
        }
        if (d.Amount > order.Total)               // business rules: no method acting
        {
            _prompt.AddNote("refund exceeds order total");
            continue;
        }
        return d;                                 // a take we can print
    }
    return RefundDecision.EscalateToHuman();     // the understudy is a person
}

The profound shift hiding in that boring loop: the system's behavior is now deterministic even though the model's isn't. Every possible outcome — valid decision, corrected decision, or escalation — was written by you, in advance, in ordinary code. The actor can surprise you; the show cannot.

🎭 Live Demo — Interrogate the Actor Hire the crew (schema + validator)

You ask: "Is this customer eligible for a refund?"

// press ACTION a few times. then flip the switch and press it again.

Same question every time. Without the crew: a different performance per take. With the crew: the improvisation happens inside the box — the shape that reaches your code never changes.

ACT IV — THE STAGE MANAGERThe deterministic sandwich

Now the biggest hire of all: the stage manager — the person who runs the actual show. Curtain at 8. Scene 2 follows Scene 1. Props are preset. Nobody, however talented, decides mid-performance to do Act III first.

Architecturally: the LLM should be a small, well-lit box in the middle of a large deterministic machine. Ordinary code decides when the model is called, what it's allowed to see, which tools it may touch, and what happens with every possible shape of answer. The workflow is a state machine you wrote; the model fills in one creative slot per state. This is the "deterministic sandwich": plain code on top, plain code on the bottom, improvisation only in the filling.

The deterministic sandwich: improvisation lives in one small box. Everything the outside world sees was decided by code you wrote.

Version your prompts like code. A prompt in a database field edited over lunch is an actor rewriting the script backstage. Prompts live in the repo, go through PR review, and ship with the release — so a behavior change is always traceable to a diff.
Cache aggressively. Same input, same prompt version, same model version? Serve the recorded take. Idempotency keys on AI operations mean a retry replays the performance instead of commissioning a new one.
Pin your model versions. "latest" means the studio can recast your lead overnight. Pin the exact version, upgrade deliberately, re-run the auditions (next act) before opening night.
Give tools, not trust. Function calling with an allowlist is props with fixed handles: the actor may request "look up order status," but your deterministic code executes it, checks permissions, and decides what comes back.

ACT V — DRESS REHEARSALS, FOREVEREvals: how you know the show still works

Final hire, and the one most productions skip until a critic burns them: the rehearsal director. Traditional tests ask "does this function return 4?" — useless for an actor who might say "Four!". Evals ask the grown-up version: across 500 representative scenes, how often is the performance acceptable — and did this week's change make it better or worse?

Build a golden dataset of real inputs with known-good outcomes — your standard audition scenes, including the weird ones (the angry email in three languages, the refund request for an item never ordered).
Score with code where possible (schema validity, exact fields, business-rule compliance) and with a judge model where necessary (tone, faithfulness) — but let the judge grade against a written rubric, not vibes.
Gate deploys on eval scores. New prompt, new model version, new temperature — nothing reaches the stage without rehearsal numbers. A prompt tweak that "reads better" and drops refund-accuracy from 97% to 89% is a bad review you caught in rehearsal instead of in production.
Keep rehearsing in production. Sample live traffic into the eval pipeline. Drift is real: the audience changes, the inputs change, and yesterday's five-star show quietly becomes a matinee nobody claps for.

Evals are the answer to the question every stakeholder eventually asks with narrowed eyes: "but how do you KNOW it works?" The honest answer is never "we tried it a few times and it seemed fine." It's a number, on a chart, per release.

★ CURTAIN ★

Curtain call

So no — you never made the actor deterministic. Nobody ever has. What you built instead is a theater: a script that fixes the shape (schemas), a supervisor that checks every take (validation + bounded retries), a stage manager that runs the show (deterministic orchestration, versioned prompts, pinned models, cached takes, allowlisted props), and a rehearsal director who proves — with numbers — that tonight's show is at least as good as last night's (evals).

Inside that theater, the improviser is not a liability. It's the star. The brilliance stays; the chaos gets a stage door it can't wander out of. That's the whole craft of AI engineering in one sentence:

Let the model improvise the lines. Never let it improvise the show. 🎭