Go into Claude. Or ChatGPT. Or whatever. Ask it for something with real structure, five things in a specific order. This could be a presentation, a word doc, a webpage, a data file.
It will get it right. Most of the time. Let’s give it an optimistic number: nine out of ten times it holds what you asked for. It’s pretty good, but it’s not perfect. Even with the recent drop of OpenAI’s Astra, there’s a lot of back-and-forth and tuning. Ask it for something more complex like an architecture diagram: oof.
Going back to that tenth time, maybe it’s even further down this imaginary scenario. The one that messed up. It merged two sections, dropped another, or quietly reordered things in a way that looks fine until something downstream expects a particular order. I’ve found it rarely looks wrong at the time, so it ships. We did it! Victory lap… or is it?
Here’s a quick made-up example. Nothing confidential, semi-inspired by something real.
| Ask for, in this order | What you sometimes get back |
|---|---|
| 1. Summary | 1. Summary |
| 2. What’s New | 2. What’s New |
| 3. Bug Fixes | 3. Upgrade Steps, moved up |
| 4. Breaking Changes | 4. Breaking Changes, with Bug Fixes folded in |
| 5. Upgrade Steps |
Now someone’s told how to upgrade before they’re told what actually broke.
There’s actual math behind this. I looked it up, and it’s not just a bad feeling. Say a model holds structure right 97% of the time on a single step, better than my “nine out of ten” guess up top. String ten of those steps together and the odds of the whole thing surviving clean fall to about 74%. Twenty steps puts you at 54%, which is a coin flip. That’s not the model being dumb. That’s just what compounding does to anything that’s merely “very good” instead of perfectly reliable.
What’s become standard is structured output. Give the thing a schema, force JSON or YAML, use function calling, constrain how it’s decoded. It helps, for a single request. Generally, if you ask it for five fields you’ll get five fields back nearly every time. Here’s the kicker though: most people are not actually building a flat one-off thing.
They’re building a document or a plan, a sequence of parts or events that may reference each other. You might not know the length of it, and the pieces may come out differently depending on the scenario. If I think about this from the video game perspective, which you’ll find I do frequently at the moment, it’s like narrative content, and I have a separate article coming on that which I think you’ll enjoy. If you’re playing an RPG, you walk up to a character and start talking. They give you four or five options and you might get different ones based on the state of the game. You may have a failed quest, so the character responds differently. Funny enough, those are the same considerations that show up for businesses. These are not unique data challenges.
If you push your schema across something long and tangled up, the discipline is going to start slipping exactly where you don’t want it to. It could be at the very beginning, it could be right in the middle where you’re less likely to look.
What catches you is a later instruction arguing with the structure. If you run the same document through ten rounds of edits it will probably hold up fine. Then someone asks to put the most urgent thing first, or to merge anything redundant, and now there are two orders and something has to give. Both requests are reasonable, and neither one mentions the schema. That’s when the shape of this “thing” moves, and it moves for a good-sounding reason, which is also how it gets through review.
We’ve been living in a world with AI for a few years now and I still see this happening regularly among people I work with. The fix they reach for is still a better prompt, a bigger schema, some sort of strict validator that rejects and retries. We’re still asking a model to do two different jobs: decide the shape of the “thing”, and produce the content that goes inside it.
Only one of those is a good job for a large language model. And it isn’t the shape.
The order of the items, the number of the items, what the format must be, whether one part is allowed to reference another. None of that is a content decision. Those are structural decisions, and they belong in code you can read, not in a probability distribution where you’re hoping it stays on its best behavior for the length of a generation.
What I actually did about it
None of what I’m saying is new, although I’d argue that when it comes to AI, “new” is extremely relative. And I’ll also point out that I’m sure there’s some people out there who are way better at writing than me who’ve argued this better.
But, I’ve watched this exact pattern play out more than once. A system I built started out re-deriving the same structural decisions from scratch on every single call. It worked from a technical perspective, but it had a problem. It was burning money re-solving something it had already gotten right a hundred times before, and I mean that literally, maybe more. This isn’t some quiet internal tool either. It’s in real daily use across a decent sized user base, handling dozens of these every single day.
My first fix was the obvious one: a bigger prompt, more emphasis, trying to align a model to “always do it exactly this way.” It helped a little. But in the long term it just kinda made it harder to debug when the shape didn’t hold up. I started missing the failures because they became rarer, until a month later one of my users reported it again.
So I locked the structure down instead. Costs dropped, obviously. What got people’s attention was the other half, and to me, shockingly, it wasn’t much of a surprise.
The output got better, not just cheaper. That isn’t supposed to happen. Determinism is supposed to cost you something, a bit of range or surprise traded away in exchange for consistency. It wasn’t a trade here. Both moved the same direction at once, because all the effort the model used to burn re-deciding structure was suddenly free to go toward the one thing it should’ve been focused on the whole time: making really, excuse my language, gosh darn good content.
I find that almost no one does this stuff end-to-end, because doing it means giving up the idea that a good enough prompt will eventually get you where you need to be, and admitting the sad fact that a model was never going to reliably do the thing you wanted no matter how many ways you asked.
Then there’s the second cost, the one you miss if you’re only thinking about correctness. And that’s the ACTUAL cost of the thing. I believe we need to be as efficient as we possibly can. There’s a lot of debate about the impact of AI, environmentally and financially. I want my daughter to grow up in a nice green world.
Every time you ask a model to redecide the shape of something it already got right, you’re paying full price to rediscover a thing you already knew: in tokens, in latency, in your electric bill, in the chance it drifts. It makes me think of that old line about the definition of insanity, doing the same thing over and over and expecting a different result. This is the reverse of that, and somehow worse: you do the same thing over and over, get the same answer, and pay full price for it every time.
If you can get the model to shape the thing once, that’s the moment to stop paying for it.
The first time you see a genuinely new request, let the model do its thing. Spend the tokens. Let it reason through what the shape should be, and treat that as the expensive, exploratory part. But the moment you have a result worth keeping, don’t throw it away. Turn it into a schema, into code, into a workflow, into a fixed list of parts and the order they go in.
What this actually buys you
Now you’ve bought something worth having. For every request that arrives shaped like one you’ve already seen, skip the model completely for structure. It runs the template you already locked in, straight through.
Is this caching? Kinda. Sorta. Not really. Caching hands you back one saved output. This saves the shape and fills it in fresh every single time. You’re not storing an artifact, you’re storing the shape it has to come out in.
There’s also a bonus as you continue to enhance your tooling. A template is something you can open, read, diff, and put a test around. A prompt that’s supposed to hold a shape every time isn’t any of those things. Once the shape is code, you write a test that says this thing has five parts in this order, and you mean it.
None of this works if you treat every request like a total one-off. It also doesn’t work if you get lazy and force two genuinely different requests into the same template just to grab the savings. That’s how a template ends up confidently wrong, which is the worst kind of wrong there is. The actual skill is telling those two apart. Is this the same shape with new content, or does this one need to earn the model’s full attention again?
Where else this shows up
You see this same ‘movie’ in at least three places, not just the one I’ve been describing.
- Templates hold the shape of the thing.
- Skills hold the process, instructions for one specific kind of job, written down once instead of re-derived from a blank context window every time.
- RAG holds the knowledge, for the same reason: you don’t want the model reconstructing what your company already knows from whatever it happened to pick up in training. You already have the fact, sitting in a document somewhere. Retrieve it, hand it to the model as context, and let the model do the one part that’s actually generative, turning a retrieved fact into the answer you’re looking for.
If you already have it, whether that’s the shape, the process, or the fact itself, don’t make the model rebuild it from scratch. Go get it, and only spend the model’s attention on the sliver that actually has to be generated fresh.
And if you keep pulling on this thread long enough, you stop writing the template altogether. You just retrain the model. Fine-tuning is the same move taken all the way: instead of handing the model a schema to follow every time, you bake the shape straight into the weights, and you stop paying for even the instructions.
Well… I guess that’s the whole thing, and the end of my first properly technical writeup outside of my demo! We did it! (Insert Dora the Explorer “We Did It” song.)
So to sum it up: the model’s most valuable output was never the artifact, or the doc, or whatever you asked it to build. It was the shape. Get that right once, write it down, and let the code run it for free from there. Everything after the first correct answer should be cheap, and if it isn’t, that’s on you, not the model.
Sources
- The compounding schema-compliance math, 97% per step falling to about 74% over ten steps and 54% over twenty, is from “JSON Mode Won’t Save You: Structured Output Failures in Production LLM Systems” by Tian Pan.