Valuestream Episode 5: Context Is the Job. Why Reliable AI Agents Are a Context Problem, Not a Prompt Problem

Episode 5 of Valuestream is live. This is the context-engineering episode, and it's the one I've been building toward since I started talking about giving an agent a brain. The argument is simple and a little uncomfortable: your agents don't fail because someone phrased the prompt wrong. They fail because the model couldn't see the right thing at the moment it had to act. That's not a prompt problem. It's a context problem, and context is a delivery discipline you already know how to run.

Prompt engineering was the tutorial. Context engineering is the job. This episode walks the context budget, the four failure modes that break agents in production, and the operating model that turns "make the AI smarter" into something your delivery org can actually plan, staff, and own.

Listen on Spotify above, or on Apple Podcasts and the rest once distribution propagates. Companion essay below has the full framework.

The opening

Nineteen. That's how many times the team rewrote the system prompt. Every week, a new draft. More rules, more "you must," more "always remember," a few more capital letters for the parts that really mattered. And every week, the agent gave customers the same confidently wrong answer about a refund policy that had changed back in March.

On the twentieth week, someone finally did the boring thing. They logged what the model actually saw at the moment it answered. Not the prompt. The whole context window. And there it was: the retrieval had pulled a policy document from 2023, a tool had dumped 900 lines of raw output on top of it, and the current policy, the correct one, was in there too, sitting in the middle of about 40,000 tokens, in the exact spot where the model pays the least attention.

The prompt was fine. It had always been fine. The context was garbage.

That's the whole episode in one story. Almost nobody's agent fails on phrasing. It fails on what the model could see when it acted.

Intake: prompt engineering was the tutorial

For two years the industry said the same thing: learn to write better prompts. Prompt engineering became a job title, a course category, a badge. Words matter, I'm not here to say they don't. But for a running agent in production, the prompt is one tile in a much bigger picture, and it's the tile that matters least.

A prompt is a single input. Context is everything the model has in front of it when it acts: the system instructions, yes, but also the documents your retrieval layer injected, the output of every tool the agent called, its memory and state, the running conversation history forty steps deep, and the format and guardrails around the answer. In a single-turn chatbot, the prompt is most of the context, so wording feels decisive. In a multi-turn agent, the prompt is a rounding error next to the documents retrieval pulled in and the history piling up. Polish the system prompt all you want; if the retrieval grabbed the wrong document, the agent is going to be confidently wrong, and no adjective saves it.

This isn't a style opinion. It's where the failures come from. A 2026 industry survey found that 57% of enterprises traced their agent reliability failures to missing or inconsistent business context, not the model. IDC projects companies that fail to build AI-ready data foundations will give up roughly 15% in productivity by 2027, because their agents reason over the wrong material. The model isn't the bottleneck. What the model can see is the bottleneck.

Here's the reframe that changes how you staff this. Every component that assembles context, retrieval pipelines, vector indexes, tool integrations, memory stores, prompt templates in a repo, is a system your delivery teams already build and ship. Those things have owners, release cycles, regressions, and incidents. When answer quality drops the morning after someone reindexed the knowledge base, that's a delivery regression with a commit attached, not a mysterious model mood. And it's the same thread I've pulled all series: when the agent acts, you answer, and ownership was the layer that decided whether anything shipped. You can't answer for what an agent did if you can't see what it knew. The context is invisible by default, and in this show, visibility is the intervention.

Flow: the budget, the failure modes, the operating model

Start with the budget. The seductive mistake is treating the context window like free storage. It isn't. Every token competes for the model's attention, and attention degrades long before the window fills. The Lost in the Middle research showed models retrieve information most reliably when it sits near the beginning or end of the input, and measurably worse when the fact you need is buried in the middle, even on long-context models. More context is not more capability. Past a threshold it's more noise, more latency, and a bigger bill, all at once. Trim each source to what earns its place, and keep real headroom for reasoning.

Then the four failure modes. Once you look through this lens, production failures stop looking random:

Retrieval that pulls the wrong thing. Your agent is only as good as the top few chunks the retriever hands it. Stale embeddings, bad chunking, or a query that doesn't match the source, and the model reasons beautifully over the wrong material. Retrieval quality is the single biggest lever on agent accuracy, and almost nobody puts a metric on it.
Tool outputs that flood the window. A tool returns 900 lines of raw output and you paste all of it back. The tool result is raw material, not a finished ingredient. Summarize, filter, and shape it before it re-enters.
Memory that never forgets. Memory without a forgetting policy is just history sprawl wearing a nicer label. A real memory system decides what to keep, for how long, at what fidelity. If your agent's memory only ever grows, you don't have a memory system, you have a landfill.
History that never compacts. Long sessions push the earliest, most important instructions into that low-attention middle. Compaction, summarizing prior turns into a dense running state, is core to keeping an agent coherent past the first dozen steps, not a later optimization.

One structural move hits two of those at once: split retrieval from reasoning. One step finds and shapes the grounded context and hands over a clean package; a second reasons over that package and nothing else. In BCG deployments, that split cut human-in-the-loop rates 30-45% against a single-agent baseline, because the reasoning step finally works off a clean plate.

Then run it like delivery. Assemble, evaluate, ship, observe, refine, with an owner at every stage:

Version control the context. Prompt templates, retrieval configs, tool schemas, and memory policies are artifacts with diffs, reviews, and rollbacks. If you can't answer "what changed in our context assembly last Tuesday," you can't run this reliably.
Test with evals, not vibes. Evals are the new acceptance criteria, and context changes are exactly the silent regression that net exists to catch. Ship a retrieval change because a graded suite held, not because one demo looked better.
Observe context in production. Log which chunks were retrieved and where the token budget went, because you can't debug an agent whose context you can't see after the fact. When an agent acts on bad context and something breaks, the constraint isn't fixing it, the agent can remediate in seconds. The constraint is mean time to understand. The team that logged its context reads the failure over coffee; the team that didn't spends two days re-enacting a crime with no witnesses.

Outcome: same model, one-fifth the errors

A composite, shape real and names changed. A company ran a customer-facing support agent on a good, current model, and it was confidently wrong about 22% of the time. The whole org had decided this was a model problem, so they swapped in a bigger model (marginal) and rewrote the prompt (the nineteen times from the cold open). Nothing held.

Then a new lead did the boring thing first. Before touching the model or the prompt, she stood up observability on the context and logged what the agent actually saw per answer, with token counts by source. The picture was damning in the most ordinary way: the average interaction carried about 38,000 tokens, more than half of it junk, and the retriever was pulling the right document into the top results only about 60% of the time.

So they ran the discipline. A real metric on retrieval quality, wired into an eval suite. Tool outputs trimmed and shaped before they hit the window. A memory retention policy and history compaction turned on. Retrieval configs and prompt templates moved into version control. No new model.

Average context dropped to about 12,000 tokens, so latency and the bill both came down. Retrieval quality climbed into the low 90s. And the confidently-wrong rate fell from 22% to about 4%, on the same model they'd been blaming. A month in, when the agent handed a batch of customers the wrong entitlement at 2 a.m., the on-call engineer pulled up the exact context it saw and found the cause, one stale document from a bad reindex, in about 90 seconds. Mean time to understand, roughly fifteen minutes, most of it waiting on coffee.

The second-order effect is the one I care about. The conversation changed. Before, every quality problem became a debate about which model to buy next. After, the first question was "what did it see?" Context stopped being invisible and became a named, owned, measured surface. What I'd do differently is the same lesson every time: they spent a quarter swapping models and rewriting prompts before a single human logged the context window once. The first thing to build wasn't a better prompt. It was a window into what the model was actually reading. The agent wasn't dumb. It was blind.

The takeaway

One agent. Not the fleet. Pick one you have in production, take one real interaction that went wrong, and log the entire context window the model saw, with a token count next to each source. Then read what your model actually read. You'll find a stale document, a raw tool dump nobody trimmed, and the one fact that mattered buried in the middle. That's your first finding. Then name one person who owns that agent's context. Not the model. The context. Because context assembly that belongs to everyone belongs to no one, and it will rot.

Coach, don't command. And here's the specific move: praise the engineer who deletes context as loudly as the one who adds a feature. The best context change your team makes this quarter is probably something they removed. Reward the subtraction, because everyone's instinct is to add, and addition is what's drowning your agent.

The boring news is that none of this is new: version control, evals, observability, ownership, the loop your delivery org has run for twenty years. The brave news is you now run it on a surface that's invisible by default, changes every time someone reindexes, and quietly decides whether your agent is brilliant or confidently wrong in front of a customer. Making that surface visible, owned, and measured is the work.

What's next

Once your context is under version control and your agents are live, every retrieval tweak and prompt change is a deployment, and a bad one reaches a customer at machine speed. So next episode is about progressive delivery in the AI era: canary releases, feature flags, and automated rollback wired to an error budget. Speed without a kill switch is just faster chaos. We'll build the kill switch.

Listen

Spotify · episode link populates the moment publishing finishes · follow the show
Apple Podcasts · Valuestream on Apple (episode appears once the feed refreshes)
Amazon Music · Valuestream on Amazon
Pocket Casts / Overcast / Castbox / Podcast Addict · auto-discovered from the RSS feed
RSS · subscribe in any podcast app at the show's RSS feed

Get notified when episode 6 drops

Drop your email on the Valuestream show page and you'll get a single line in your inbox the morning episode 6 publishes. Same list as the blog notifications. No fluff in between.

This is Valuestream. I'm Rick Pollick. Context is the job.