Production-ready agents, Part 15: What breaks in production: answers, loops, cost and model changes
The first half of a field guide to running agents in production. Twenty failure patterns you will meet once real users arrive, grouped into wrong answers, loops and stalls, cost and latency, and model changes. Each one has the symptom, the likely causes, how to confirm which it is from traces, logs and evals, the fix, how to stop it coming back, and the part of the series that explains it.
Every part of this series has ended with a section called “What can go wrong”. Each one lists the mistakes that part’s mechanism invites. That is useful while you are building. Once an agent is live, you meet those mistakes from the other side: a user says an answer is wrong, a bill is higher than it should be, a dashboard turns red at 9am on a Monday. You see the symptom first, and you have to work back to the cause.
This part and the next turn the series into a field guide for that moment. Each pattern starts from what you see, lists the likely causes, says how to confirm which one it is, and then gives the fix, the way to stop it coming back, and a link to the part that explains the mechanism. Nothing here is invented: every pattern comes from documented behaviour or from something the series has already shown, mostly in those “What can go wrong” sections.
This part covers the failures that affect the quality and cost of answers: wrong answers, loops and stalls, cost and latency, and model changes. Part 16 covers identity and access, agents talking to agents, safety, and rollout, and ends with an incident triage flow, a runbook template and a recap of the whole series.
If you prefer to watch first, here is the heart of this part as a short animated explainer.
Video: how to trace wrong answers, loops, cost drift and model changes back to their cause, in three minutes. The narration is a synthetic voice; the script and animation are mine.
Where are we in the architecture?
This part adds no new box. It is a field guide to what goes wrong across the whole picture, and how to fix it. Part 1.25 walks through the whole picture, layer by layer.
the complete Acme architecture: Maria asks in Gemini Enterprise; the router runs on Agent Runtime behind plugins and Model Armor and calls the marketing and supply agents over A2A; they reach BigQuery and Databricks through Agent Gateway; identity and secrets on the left, models, knowledge and the market watch managed agent for public research on the right, traces, logs and analytics below; above it, shoppers reach Gemini Enterprise for CX, whose conversation insights land in BigQuery, and the Legal industry edition plugs into Gemini Enterprise
What do I need to know first?
Debugging an agent is diagnosis, not guesswork: you see a symptom, you confirm the cause from evidence, and only then do you fix it. Three ideas make the rest of this part easier.
Start with a simple example. A patient tells a doctor “I have a cough”. That is the symptom. The cause could be a cold, an allergy or reflux, and the treatments are different: an antihistamine does nothing for reflux. So a good doctor listens to the chest or orders a test before writing a prescription, and books a follow-up so the cough is noticed early if it returns. Symptom, test, treatment, follow-up: the same four steps appear below, with traces and logs as the stethoscope and an eval case as the follow-up.
Where people get tripped up is treating the first plausible cause as the confirmed one. Here is how the same four steps look on an agent.
- A symptom is what you observe (“the answer has a wrong number”). A cause is why it happened (“the writer model invented a figure, and the check that should catch it ran too late”). The same symptom often has several causes with different fixes, so confirm before you fix.
- You have four sources of evidence, all built earlier in the series. Traces in Cloud Trace show
each step of one question, with ADK’s span names:
invoke_agentfor an agent run,generate_contentfor a model call andexecute_toolfor a tool call (Part 11). Logs in Cloud Logging hold structured, redacted events, including every tool call from the audit plugin (Part 9). The usage ledger records tokens per model call, and the cost report turns them into cost per resolved question (Part 11). Evals tell you whether a fix made things better or worse (Part 10). - A fix is not finished until something would catch the failure next time: a code check in the eval set, a unit test, or a limit in code. That is the Prevent it next time step in every pattern.
How do I use this field guide?
Look things up by what you see, not by what you think is broken. Find the group that matches what you see, then the pattern. Every pattern has the same six parts.
Think of the troubleshooting page in a washing machine manual. It is indexed by what you notice (“water left in the drum”, “the door will not open”), not by part names, because when something goes wrong you know the symptom, not the faulty pump. Each entry lists the likely causes, how to check each one and what to do. This guide works the same way: the H2 groups are what you notice (a wrong answer, a loop, a rising bill, a model change), and each H3 is one entry.
the format of every pattern in the field guide: the symptom you see, the likely causes, how to confirm which one it is from traces, logs or evals, the fix, how to prevent it next time, and the part of the series that explains the mechanism
The architecture diagram at the top of this post is the map. When you have confirmed the cause, you know which box it lives in, and the part that added that box explains how it works.
What does a wrong answer look like, and where does it start?
A wrong answer is usually made at one step and only noticed at the end, so walk the trace from the top and fix the first step that went wrong, not the last one. Wrong answers are the failures users notice first.
A simple example first. A café takes an order for “a flat white with oat milk”. The till operator types “flat white”, the barista makes it with dairy milk, and the server hands it over. The customer complains to the server, but the server did nothing wrong: the order was lost at the till. Telling the server to “be more careful” fixes nothing. You read the ticket at each station, and the first station where it differs from what the customer said is where the fix goes.
How does this apply to Acme? The stations are the extract model, the route step, the specialist tools and the writer model, and the ticket at each one is a span in the trace. In a system like Acme’s, almost every wrong answer starts in one of those four places, and each one has its own span.
where wrong answers show up in one trace: the extract model call can change a word, the route step can pick the wrong branch or none, a specialist tool call can return the wrong rows, and the writer model call can add a number that no tool returned; each has its own span to inspect
Why did the answer contain a number no tool returned?
Symptom. An answer states a figure, such as a spend total or a penalty, that does not appear in any tool result in the trace.
Likely causes. The writer model filled a gap with a plausible number; the number check ran in the wrong callback and could not change the reply; or someone “fixed” a failed answer by deleting the one wrong figure and left a misleading sentence.
How to confirm. Open the trace and compare every number in the final generate_content span of the writer with the outputs of the execute_tool spans. Check the logs for the number check’s result on that turn.
Fix. Make sure the check runs as an after-model callback, which ADK documents as the place to change a model’s reply, and replace the whole answer when it fails rather than patching it.
Prevent it next time. Keep “every number comes from a tool” as a code metric in the eval set, so a regression fails the gate.
Example. Maria asks the q3_delegation question from Part 10, “What did the stockouts behind the Acme Pure Refill Launch cost us?”. The second figure below is illustrative:
1. execute_tool supply returns penalties totalling 51,331.20 (EUR)
2. generate_content writer: "The stockouts cost EUR 51,331 in penalties, plus about EUR 4,200
in lost sales."
3. number check 51,331 found in the findings; 4,200 found in no tool result -> fail
4. reply the whole answer is replaced with a safe reply, not trimmed to one number
5. eval set a case with expected_numbers [51331] and the code metric keeps it fixedThe writer model invented the second number; code caught it, because the check runs as an after-model callback and can change the reply.
Covered in Part 5: Step 4: how do we stop the writer making up a number.
Why did the wrong team answer, or nobody at all?
Symptom. A supply question gets a marketing answer, or the user gets no answer and no error.
Likely causes. The extract step changed a word that routing depends on (for example “Western Europe” became “Europe”), or the route function returned a value with no matching edge, in which case ADK logs a warning and the branch stops.
How to confirm. In the trace, read the output of the extract generate_content span, then the route value in the logs. A warning about a missing edge confirms the second cause.
Fix. Tell the extract model to keep names exactly and test it with real follow-up questions; add a DEFAULT_ROUTE branch so no value is left without an edge.
Prevent it next time. Keep route correctness at 100 per cent on the golden set and unit-test every route value.
Example. Both causes, on one illustrative question that names a region:
Cause 1, a changed word
1. Question "What was Verde's OTIF in Western Europe?"
2. extract "What was Verde's OTIF in Europe?" <- the region name was shortened
3. route the routing rule no longer matches the region it routes on, so the question
does not go where it should
Cause 2, a value with no edge
1. Suppose someone adds a new target to the routing rule but not to the dict edge
2. route returns that new value; ADK logs a warning about a missing edge
3. user gets nothing: no answer and no error
After the fix
extract keeps "Western Europe" word for word, DEFAULT_ROUTE catches any unknown value,
and a unit test calls route once for every value it can returnCovered in Part 5: Step 2: how does route decide, with no tokens at all.
Why did the agent call the wrong tool, or the right tool with the wrong arguments?
Symptom. The trace shows a tool call that does not fit the question, or a call rejected for invalid arguments.
Likely causes. A vague docstring gives the model nothing to choose by; two tools overlap, so the model picks between them more or less at random; or there are simply too many tools.
How to confirm. Look at the execute_tool span’s name and arguments for the failing questions, and count how often each tool is chosen across the golden set.
Fix. Rewrite the docstring with valid values, formats and when not to use the tool; merge or remove overlapping tools; keep the set small.
Prevent it next time. Add the failing questions to the golden set with the expected tool and arguments as code checks.
Example. The inner loop from Part 1.5 is this pattern caught and fixed inside one turn:
1. Act query_marketing_measures(measures=["ctr"], brand=["Verde"], region=["Western Europe"], ...)
2. Observe {"status": "error", "message": "unknown measure 'ctr'; use one of
media_spend_eur, ctr_pct, cpm_eur"}
3. Act query_marketing_measures(measures=["ctr_pct"], brand=["Verde"], region=["Western Europe"], ...)
4. Answer Verde's CTR, with the number taken from the tool resultIf that error shows up on many questions in the trace, the docstring is the fix, not the retry. A vague line such as measures: what to measure. gives the model nothing to choose by; measures: one or more of "media_spend_eur", "ctr_pct", "cpm_eur". lists the valid values, so the first call is right.
Covered in Part 2: What does the tool tell the model.
Why did the answer use a different region or period from the one the user asked about?
Symptom. The user names a region, and the answer is about another one, often the one they asked about last week.
Likely causes. A remembered preference from Memory Bank overruled what the user typed, or the extract step filled in a region from memory when the question already had one.
How to confirm. Check the memories preloaded for that user and the rewritten question in the extract span.
Fix. State in the instruction that a remembered value is only used when the question names none, as Acme’s extract step does.
Prevent it next time. Add eval questions that name a region different from the remembered one.
Example. Memory holds “Maria works on Western Europe”. She asks “What was Verde’s CTR in the UK last month?”, and the extract span shows “What was Verde’s CTR in Western Europe last month?”: the remembered region replaced the one she typed. With the instruction “use a remembered region only when the question names none”, the extract keeps “UK”, and the new eval case that names the UK keeps it that way.
Covered in Part 8: How does the router use memory.
Why did one user’s details appear in another user’s answer?
Symptom. A user sees a preference, region or finding that belongs to someone else.
Likely causes. Memories and user: state are keyed by user ID, and some local tools, such as ADK Web, use "user" for every caller by default; or the user ID is not taken from the signed-in identity in production.
How to confirm. Look at the user ID on the session and on the memories in the logs. If many people share one ID, that is the cause.
Fix. Take the user ID from the signed-in identity, as Part 7 does, and clear the shared memories.
Prevent it next time. Add a test that two different users never see each other’s state.
Example. Picture a cloakroom that hands every guest ticket number 1. The first coat is hung on hook
1, the second guest’s coat goes on the same hook, and whoever comes back first takes whichever coat is
on top. Nothing is wrong with the hooks; the tickets are. In an agent, the ticket is the user ID and
the hook is the memory store. If Maria and Sam both arrive as "user" (an illustrative local setup,
not Acme’s production one), Maria’s “I work on Western Europe” is stored under "user", and Sam’s
next question is answered with her region. Giving each person their own ticket, the signed-in
identity, fixes it.
Covered in Part 8: What can go wrong.
Why did a follow-up question lose track of what “those” meant?
Symptom. A first answer is fine, and a follow-up such as “what did those stockouts cost us?” gets a confused or generic reply.
Likely causes. The extract step was not given the earlier turns, or compaction removed them without leaving a summary. For a root agent that is a Workflow, compaction needs a summarizer set explicitly.
How to confirm. Read the input of the extract span on the follow-up turn: is the earlier answer, or a summary of it, there?
Fix. Pass the conversation to the extract step and set the summariser on compaction.
Prevent it next time. Keep multi-turn cases in the eval set, and use a user simulator for longer conversations.
Example. Someone walks into a meeting halfway through and hears “so what did those cost us?”. Without the earlier discussion, or at least the minutes, “those” means nothing. The extract step is that late arrival: the earlier turns are the discussion, and a compaction summary is the minutes. On Acme (an illustrative trace), Maria first asks about stockouts behind the Acme Pure Refill Launch, then asks “what did those stockouts cost us?”. If the extract span’s input holds only the second question, it cannot name the launch; with the earlier turn, or a summary of it, in the input, it rewrites the question with the launch named and routes it to supply.
Covered in Part 8: What happens in a long conversation? Compaction.
Why did a retrieved document answer the wrong question?
Symptom. An answer cites a passage that is out of date, is about something else, or should not have been visible to the user.
Likely causes. The index has not synced since the source changed; chunking split a table or a procedure; or retrieval ranked all documents before filtering by permission.
How to confirm. Look at the retrieved passages in the tool span, their document dates, and the user’s role.
Fix. Re-sync the index, fix the chunking, and filter by permission before results reach the model.
Prevent it next time. Keep a retrieval eval set of questions with the passages that should come back, and evaluate citations.
Covered in Part 14: What can go wrong.
Why do agents get stuck in loops or stall?
An agent loops or stalls when nothing in code tells it to stop; the fix is a limit in code, not a politer instruction. An agent is a loop: the model decides, a tool runs, the model looks at the result and decides again. Part 1.5 called the code around that loop the harness. When the harness has no limits, a loop that should take three steps can take three hundred.
A simple example. You ring a shop, and an automated voice says “invalid option” with no menu. You press 1 again, hear the same message, and press 1 again. Nothing changes between attempts, so nothing changes in the result. A sensible caller has two limits: “I will try three times and then hang up” (a cap), and “if the message tells me which options exist, I will press a different one” (a useful error). Without either, the call goes on until the phone dies.
How does this apply to Acme? The caller is the model, pressing 1 is a tool call with the same
arguments, and the unhelpful message is a tool that raises an exception instead of returning a status
the model can act on. The cap is RunConfig.max_llm_calls or max_iterations, and the useful error is
{"status": "error", "message": ...} listing the valid values, as in the wrong-arguments pattern above.
The common misconception is that a stronger model will notice it is looping and stop by itself; some
do, sometimes, but a limit you cannot rely on is not a limit.
the signature of a loop in a trace: the same tool called again and again with the same arguments, each call followed by another model call; three limits in code end it: max_llm_calls in RunConfig, max_iterations on a LoopAgent, and a budget plugin that stops the next model call
Why did one question make dozens of model calls?
Symptom. A single turn takes minutes, and the trace shows the same execute_tool span with the same arguments again and again.
Likely causes. The tool raises an exception or returns an unhelpful error, so the model tries the same call again, and nothing in code stops it. In ADK, RunConfig.max_llm_calls defaults to 500.
How to confirm. Count the generate_content spans in the turn and look for repeated tool calls with identical arguments.
Fix. Make the tool return a status and a message the model can act on, and set max_llm_calls to a limit that fits the job.
Prevent it next time. Keep a budget plugin that checks tokens, cost and time before each model call, and a test that the cap holds.
Covered in Part 11.5: A hard cap on model calls: RunConfig.max_llm_calls.
Why does a reviewer loop never finish?
Symptom. A writer and a critic go round and round, and the turn ends only when something external stops it.
Likely causes. A LoopAgent without max_iterations, or a stopping rule written in the critic’s prompt (“stop when it is good”) instead of in code.
How to confirm. Count the iterations in the trace; check the agent definition for max_iterations and an exit signal.
Fix. Set both a maximum and an exit signal, and count rounds in code.
Prevent it next time. Add a test where the critic never approves, and check the loop still ends.
Example. At Acme, this is how it would work with a drafter and a critic for a weekly summary, in a LoopAgent with max_iterations=3. The critic replies “missing source” on rounds 1, 2 and 3, so the loop stops after round 3 and the turn ends with an honest “not approved after 3 rounds” instead of running on. Without max_iterations, nothing in code would stop round 4. The test uses a critic that always rejects and asserts that the trace shows exactly three rounds.
Covered in Part 6.5: 5. Loop and critic.
Why is the approval for a write still waiting, or why did it start again?
Symptom. A user approves the replan, and nothing happens, or the agent asks for approval again.
Likely causes. With a resumable app, the confirmation reply must carry the same invocation ID as the paused run; without it, ADK starts a new invocation instead of resuming the old one.
How to confirm. In the logs, compare the invocation ID of the paused run with the one on the confirmation reply.
Fix. Have the client send the confirmation with the paused run’s invocation ID.
Prevent it next time. Keep the test from Part 10 that removes the confirmation on purpose, and add one that resumes with the wrong ID.
Example. The replan pauses in invocation inv-A (an illustrative ID). Maria approves, but the client sends the confirmation with no invocation ID, so ADK starts a new invocation, inv-B, which reaches the same write and asks for approval again. Resending the confirmation with inv-A resumes the paused run, and the replan starts once.
Covered in Part 9: How does a person approve the one action that writes data.
Why do cost and latency drift after launch?
Cost and latency rarely fail all at once. They drift: a little more history on every call, one more retry layer, a specialist that slows down when its data grows. The cost report from Part 11 is the instrument that shows the drift, as long as you divide by resolved questions and not by requests.
A simple example. An email thread where everyone hits reply all and keeps the whole quoted history. The first message is one paragraph. By the tenth reply, every new message carries nine earlier ones underneath it, so the thread is far heavier even though each person only typed a line or two. No single reply looks expensive; the total does.
How does this apply to Acme? Each turn of a conversation with the router is one reply, and a model call that resends the whole session is the quoted history. The new text per turn (Maria’s question) stays small, while the input tokens per call keep growing, which is the orange part of the bars below. The misconception is reading cost per request: if the agent now needs two attempts to resolve a question, cost per request can look flat while cost per resolved question has doubled.
why cost per question creeps up: input tokens grow on every turn as history, memories and tool results accumulate, thinking tokens are billed as output, and retries at several layers multiply calls; compaction, capped tool results, cache-friendly prompt order and one retry layer bring it down
Why does cost per question keep creeping up?
Symptom. The cost report shows more tokens per resolved question week after week, with no change in traffic.
Likely causes. Long conversations send their whole history on every call; tool results are not capped; thinking tokens are left out of the cost model; or Memory Bank is given the whole session after every answer.
How to confirm. Compare input, output and thinking tokens per turn in the usage ledger across a long conversation.
Fix. Turn on compaction with a summariser, cap the rows tools return, set the thinking level per step, and send only new events to memory.
Prevent it next time. Watch tokens per resolved question in the cost report, and fail the gate if it jumps.
Covered in Part 11: What does a question really cost.
Why did one failure turn into a flood of requests?
Symptom. A short outage of a model or a specialist is followed by a spike in requests and 429s that outlasts the outage.
Likely causes. Retries at every layer multiply: an SDK retry inside a tool retry inside an A2A retry. Permanent errors such as a 400 or a safety block are retried as if they were temporary.
How to confirm. Count attempts per user question in the logs, and check which status codes are being retried.
Fix. Pick one layer for each kind of retry, retry only the codes on your list, and add a circuit breaker.
Prevent it next time. Test the fallback ladder with injected failures, as Part 11.5 does.
Example. The arithmetic, with illustrative numbers. Each layer retries up to 3 attempts in total: the SDK inside the tool, the tool inside the A2A call, and the A2A call from the router. One question can therefore make 3 x 3 x 3 = 27 model attempts. If 200 questions arrive in the minute the model is recovering, that is 5,400 requests landing on it at once. With retries in one layer only, the same 200 questions make at most 600 attempts.
Covered in Part 11.5: Rung 1: a bounded retry with backoff and jitter.
Why does every answer take as long as the slowest specialist?
Symptom. Latency at p95 is high, and the trace shows the marketing branch finished long before the supply branch.
Likely causes. The router waits for both teams by design, and the slow branch has no deadline; or a fixed 300 second A2A timeout outlives a 60 second turn.
How to confirm. In the trace, compare the two invoke_agent spans for the specialists.
Fix. Propagate the turn’s deadline to tools and remote agents, and return a partial but honest answer when one team runs out of time.
Prevent it next time. Track p95 latency per specialist, not just for the whole question.
Example. Illustrative timings for one question that goes to both teams, with a 60 second turn deadline: the marketing invoke_agent span ends after 4 seconds, and the supply span is still running at 58 seconds because its A2A timeout is 300 seconds. With the deadline propagated, the supply call gets only the time left in the turn, stops cleanly, and the answer gives marketing’s finding and says plainly that supply did not respond in time.
Covered in Part 11.5: Deadline propagation to tools and remote agents.
Why is the cache never used?
Symptom. The cached_tokens field in the usage ledger stays at zero, and input cost is higher than planned.
Likely causes. The prompt puts changing content (the question, a timestamp) before the stable content, so no two requests share a prefix; or prompts are below the minimum size for caching, which Part 11 gives as 4,096 tokens for most Gemini 3 models.
How to confirm. Look at cached_tokens per call, and at the order of the prompt’s parts.
Fix. Put instructions, skill text and tool descriptions first and the question last.
Prevent it next time. Keep the cache hit rate on the cost dashboard.
Example. A before and after on prompt order, for a prompt above the 4,096 token minimum:
Before [question, timestamp, instructions, skill text, tool descriptions]
every request starts differently -> cached_tokens = 0 on every call
After [instructions, skill text, tool descriptions, question]
the second and later calls share the same long prefix -> cached_tokens covers itCovered in Part 11: How do I bring the cost down without hurting quality.
Why did one query cost more than a month of model calls?
Symptom. A spike on the BigQuery bill, traced to a single agent query.
Likely causes. A query scanned a whole table, and no cost limit was set.
How to confirm. Find the job in BigQuery’s job history and match it to the tool call in the trace.
Fix. Set maximum_bytes_billed on every query the agent runs.
Prevent it next time. Keep the limit in the tool code, not in the instruction, and test it.
Example. A query with no filter, a plain SELECT over all of campaign_performance, would scan the whole table. With maximum_bytes_billed set to 2 GB as in Part 3, BigQuery estimates the scan before it runs, rejects the job at no charge if the estimate is over the limit, and the tool returns a status error the model can act on, such as adding a brand and a date range.
Covered in Part 3: What limits does every query get.
What breaks when the model changes?
A model is a dependency that can change without a code change on your side, so treat it like any other supplier: name exactly what you buy, and test before you switch. Part 13 explained the mechanisms; these are the symptoms.
A simple example. A bakery orders “the usual flour” every week. One week the supplier quietly substitutes a different mill, and the bread comes out denser, although the bakers changed nothing. A bakery that orders “mill X, type 550, 25 kg” would have noticed the substitution on the delivery note. It also needs a note in the diary for the day mill X stops making that flour, and a check that a new supplier can deliver enough on a busy Saturday.
How does this apply to Acme? “The usual flour” is an alias such as gemini-flash-latest; the exact
order is a pinned model ID in MODEL and WRITER_MODEL; the diary note is the retirement date; and
the busy Saturday is quota at peak traffic. The four patterns below are those four cases.
four ways a model change breaks an agent that nobody redeployed or reviewed: an alias moves to a new model, the same ID means a different model on another API, a model reaches its retirement date, or a new model has a different quota; each is caught by pinning IDs and running the eval gate
Why did answers change when nobody deployed anything?
Symptom. Eval scores or user feedback shift on a day with no release.
Likely causes. A model alias such as gemini-flash-latest moved to a new model; a scaffold default changed in an upgrade; or a vendor now serves a different model under the same name, as DeepSeek documents for deepseek-flash.
How to confirm. Compare the model IDs in the deployed settings with the ones your last eval run used, and look for an alias or a scaffold upgrade since then.
Fix. Pin exact model IDs in MODEL and WRITER_MODEL, and roll back to the last good ID.
Prevent it next time. Treat every model change as a release: eval sets, cost report, shadow, then rollout.
Covered in Part 12: How do I upgrade the model safely.
Why does the same model ID behave differently in two places?
Symptom. An agent passes its evals in one environment and behaves differently in another.
Likely causes. The two environments call different APIs. As of October 2026, gemini-3.5-flash is a stable model on Agent Platform, while the Gemini Developer API routes the same ID to gemini-3.6-flash.
How to confirm. Check which endpoint each environment uses, and read that API’s deprecation page.
Fix. Run evals against the exact endpoint you deploy to.
Prevent it next time. Keep the endpoint in the same settings file as the model ID, so they change together.
Covered in Part 13: Gemini (Google).
Why did every call start failing on one day?
Symptom. All requests to one step fail with errors about the model, starting on a particular date.
Likely causes. The model reached its retirement date. Requests to a retired model ID fail.
How to confirm. Look up the model on the model versions page (or the partner or open model deprecations page).
Fix. Switch to the recommended replacement, test the critical paths, and deploy.
Prevent it next time. Put every retirement date in a calendar, with a test that fails a month before.
Covered in Part 13: How do versions, aliases and deprecation work.
Why did 429 errors appear right after switching models?
Symptom. A model change passes its evals, and then production shows 429s at busy times.
Likely causes. Quota and capacity are per model and per consumption option. Standard PayGo shares capacity, and a new model may not have the usage tier or reserved throughput the old one had.
How to confirm. Look at 429s by model ID in the logs, and at the quota pages for that model.
Fix. Keep the fallback model from Part 11.5, and consider Priority PayGo or Provisioned Throughput for business-critical traffic.
Prevent it next time. Include a load test at peak traffic in the model upgrade checklist.
Covered in Part 13: What matters for an agent, beyond leaderboards.
Why did the gate start passing releases it should block?
Symptom. A release that users clearly find worse passed the eval gate.
Likely causes. The judge model or its rubric changed, and the judge was not calibrated again; or a held-out failure was used to tune a prompt, turning the held-out set into a second golden set.
How to confirm. Check the judge model ID and rubric version in the gate’s output, and when the judge was last compared with human labels.
Fix. Calibrate the judge again against the human-labelled set before it can block anything.
Prevent it next time. Record the judge version with every gate result.
Covered in Part 10: Why must a judge be checked against people before it can block a release.
What can go wrong with the fixes themselves?
The most common way a fix fails is that it hides the symptom instead of removing the cause. Fixes have failure modes too. These are the ones I see most often.
- Fixing without confirming. Raising a timeout when the real cause is a retry storm makes the storm last longer. Confirm the cause in a trace first.
- Fixing in the prompt. “Never invent numbers” in an instruction is a request, not a control. The series puts every rule that must hold in code (Parts 3, 5 and 9).
- Tuning on the held-out set. A failure from the held-out set, fixed by editing a prompt, stops being a fair test (Part 10). Add a new case to the golden set instead.
- Removing a limit to make an error go away. A loop that hits
max_llm_callsis telling you something. Raising the cap hides it and costs money. - Hot-fixing around the gate. A change that skips the eval gate because it is “only a prompt” is exactly the change the gate exists for.
Before shipping any fix, I run it through the same short test, in this order:
1. Have I confirmed the cause in a trace, a log or an eval? no -> confirm first
2. Is the fix in code (a check, a limit, a test), not a prompt? no -> move it into code
3. Is there a new golden set case that fails without the fix? no -> add one
4. Does the change go through the eval gate like any release? no -> send it through
Only then: raise a limit or a timeout, and only with a reason written next to it.My view: if I could watch only three numbers for a new agent, they would be the share of answers that pass the number check, tokens per resolved question, and p95 latency per specialist. Between them they catch most of the patterns in this part before a user reports them.
One honest gap: these patterns come from building and testing the series’ system and from the documentation behind each mechanism, not from months of production traffic on Acme. Real traffic will add patterns of its own. When it does, write each one up in this same format and add a case to the golden set.
Try it: spot a loop and runaway history in trace records
This needs nothing but Python. It reads a few trace-like records, the kind you can export from Cloud Trace or build from your logs, and flags two patterns from this part: the same tool called three or more times with the same arguments in one turn, and input tokens that grow quickly from turn to turn.
# loop_and_growth.py: scan trace-like records for a loop and for token growth. No cloud calls.
from collections import Counter
turns = [ # one record per model or tool call, as you might export them from a trace (illustrative)
{"turn": 1, "kind": "generate_content", "input_tokens": 3_100},
{"turn": 1, "kind": "execute_tool", "tool": "get_supply_service", "args": "region=WE"},
{"turn": 2, "kind": "generate_content", "input_tokens": 5_900},
{"turn": 2, "kind": "execute_tool", "tool": "get_supply_service", "args": "region=WE"},
{"turn": 2, "kind": "execute_tool", "tool": "get_supply_service", "args": "region=WE"},
{"turn": 2, "kind": "execute_tool", "tool": "get_supply_service", "args": "region=WE"},
{"turn": 3, "kind": "generate_content", "input_tokens": 9_400},
]
calls = Counter((r["turn"], r["tool"], r["args"]) for r in turns if r["kind"] == "execute_tool")
for (turn, tool, args), n in calls.items():
if n >= 3:
print(f"loop? turn {turn}: {tool}({args}) called {n} times")
inputs = [r["input_tokens"] for r in turns if r["kind"] == "generate_content"]
growth = [b / a for a, b in zip(inputs, inputs[1:])]
if all(g > 1.4 for g in growth):
print(f"history growth: input tokens {inputs}, check compaction")Running it prints:
loop? turn 2: get_supply_service(region=WE) called 3 times
history growth: input tokens [3100, 5900, 9400], check compactionThe records are illustrative. Point the same two checks at an export of your own traces, and you have the start of an alert for two of the most common production failures.
What code did we write in this part?
Every part ends with the same map of Acme’s build: which files exist so far, and the code this part added, explained line by line. The map grows with each part.
This part adds no code to Acme’s build. It is a field guide to failures in wrong answers, loops, cost and model changes, and every fix points back to code built in an earlier part; the only program is loop_and_growth.py, a standalone exercise that scans illustrative trace records.
Where does it live?
router/ the router agent: extract, route, write
app/agent.py P5,8 workflow; extract reads memory; App with compaction and resumability
app/app_utils/services.py P8 picks the session, artifact and memory services
app/fast_api_app.py P8 Runner with session, artifact and memory services
app/remote.py P6,7 A2A client over mTLS; metadata carries Sam's token
app/writer.py P5,8 writer reads memories and saves the conversation to Memory Bank
scripts/create_memory_bank.py P8 creates the Memory Bank and its topics, once
tests/eval/acb_metrics.py P10 code metrics: route_correct, grounded, golden_numbers
tests/eval/build_datasets.py P10 splits source cases into golden and held-out, seed 42
tests/eval/calibrate.py P10 compares judge scores with human labels (kappa)
tests/eval/datasets/golden_marketing.json P10 golden cases with expected route, target and numbers
tests/eval/eval_config.yaml P10 eval metrics, including the faithfulness judge
tests/eval/gate.py P10 thresholds per metric; exits 1 when one fails
tests/unit/test_router.py P10 unit tests; routing checked with one row per question type
marketing/ the marketing specialist, on BigQuery
app/agent.py P1,2,3,4 marketing agent with its skill in a SkillToolset
app/app_utils/a2a.py P6 A2A endpoint and agent card (1.0 and 0.3) for marketing
app/skills/marketing-kpis/SKILL.md P4 marketing know-how: which tool, definitions, story checks
app/skills/marketing-kpis/references/tables.md P4 tables, grain, measures and the join rule, read on demand
app/tools.py P1,2 three narrow tools: measures, unserved demand, SQL fallback
tests/unit/test_marketing.py P2 unit tests for the tools, with a fake context and database
supply/ the supply specialist, on Databricks
app/agent.py P3,7,8,9 supply agent and App with the confirmed replan tool
app/app_utils/a2a.py P6 A2A endpoint and agent card (1.0 and 0.3) for supply
app/skills/supply-catalog/SKILL.md P4 supply know-how, given only to the supply agent
app/tools.py P3,7,9 supply tools; the replan tool asks a human first
shared/ code all three services share
adk_common.py P3,6,9,11 plugins and redacted JSON logs, now with the usage ledger
auth_manager.py P7 auth manager callback: token or consent (off by default)
engines.py P3,7 run_databricks connects as the signed-in user
factcheck.py P5 ungrounded_numbers: finds figures not in the findings
registry.py P2 reads semantic.yaml: measure names, computable measures
routing.py P1 classify: picks the specialist in code, no model
safety.py P9 per-user token bucket and injection pattern scan
semantic.yaml P1,2 registry of measures, owners and published formulas
skill_source.py P4 loads a skill from disk or a pinned registry revision
sqlguard.py P3 guard_sql read-only allow-list; redact for log lines
tests/test_cost_report.py P11 unit tests for the cost maths
ops/ build, deploy and evaluation tooling
.cloudbuild/eval_gate.yaml P10 after staging: eval run, then gate.py
.cloudbuild/pr_checks.yaml P10,11 offline PR checks, cheapest first, ending with the mutants
env.sh.example P11 settings for every engine: ids, models, plugin switches, OTEL; no secrets
infra/gateway/setup_gateway.sh P12 egress gateway, endpoint registry and IAP rules, in dry-run
tools/cost_report.py P11 prices the ledger and reports cost per resolved question
tools/mutants/acb_mutate.py P10 pytest plugin that injects each mutant's fault
tools/pricing.json P11 list prices per model, kept out of the code
tools/run_mutants.py P10 runs the offline suites once per mutant
+ added in this part ~ changed in this part Pn = the parts that touched the fileWhat did we learn?
- Start from the symptom, confirm the cause from traces, logs, the usage ledger and evals, and only then fix. The same symptom often has two causes with different fixes.
- Most wrong answers start in one of four places: the extract step, the route, a tool call or the writer. Each has its own span.
- Loops end only when code ends them:
max_llm_calls,max_iterationsand a budget plugin, with tools that return errors instead of raising them. - Cost and latency drift. Watch tokens per resolved question, cache hits and p95 per specialist, and keep one retry layer.
- Models change without a deploy. Pin IDs, check the endpoint, track retirement dates and treat every model change as a release.
- Every fix ends with something that would catch the failure next time.
What comes next?
Production-ready agents, Part 16: What breaks in production: identity, agents, safety and rollout finishes the field guide with the failures that affect trust: login and permission errors, agents that cannot reach or believe each other, safety controls that block too much or too little, and rollouts that go wrong. It ends with an incident triage flow, a runbook template and a recap of the whole series.
Sources
- ADK: Runtime configuration (RunConfig)
- ADK: Types of callbacks
- ADK: Plugins
- ADK: Loop agents
- ADK: Context compression
- ADK: Memory
- ADK: Traces
- ADK: Tool confirmation
- ADK: Graph routes
- Google Cloud: Model versions and lifecycle
- Google Cloud: Error code 429
- Google Cloud: Retry strategy
- Google Cloud: Context caching overview
- Google AI for Developers: Gemini deprecations
- Google SRE book: Addressing cascading failures
- DeepSeek API documentation
Opinions are my own. Every fact in this post comes from the public documentation linked above.