abhinavbhatia
Sheet 02 / 05 — Blogs← Archive

Forward deployed engineering, Part 8: Debugging in the field

Something breaks in a system you did not write, inside an estate you do not control, in front of the customer. This part covers how a forward deployed engineer reads unfamiliar code, reproduces a failure, reads logs and traces, decides whose problem it is, chooses between a workaround and a fix, and writes a bug report that a product team can act on.

· 15 min read · ~22 min listen· Forward deployed engineering · Debugging · Agents

In Part 7 we took the Acme Consumer Brands prototype to production and handed it to the customer’s team with a runbook. That is the moment real users arrive, and real users find the problems that no test found. The first message usually looks like this one, from Sam, a supply planner at Acme: “The assistant says the supply team has no data for last week’s stockouts. We definitely have that data.”

What happens next is one of the most visible parts of a forward deployed engineer’s job. The code may be yours, the customer’s or a product team’s. The failure happens inside an estate you do not own. And people are watching how quickly and calmly you work out what is going on.

This part covers how to read code you did not write, reproduce a failure, read logs and traces, decide whose problem it is, choose between a workaround and a fix, and write a report that a product team can act on.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: how a forward deployed engineer debugs a failure in a system they did not write, in three minutes. The narration is a synthetic voice; the script and animation are mine.

Where does this show up in an FDE job description?

Debugging is in almost every public FDE posting, usually next to the feedback loop to the product team. Three examples, as of 20 October 2026:

  • Google Cloud’s Forward Deployed Engineer, Gen AI posting describes someone who will “code, debug, and jointly ship bespoke agentic solutions directly within the customer’s environment”, and who will “identify repeatable field patterns and friction points in Google’s AI stack, converting them into reusable modules or formal product feature requests for the Engineering teams.”
  • Microsoft’s Principal Forward Deployed Engineer posting says: “Write the prototypes, read the traces, work in the customer’s repository.” It also asks the engineer to bring product teams the pattern rather than the anecdote, “with reproductions, measurements, architectural analysis and quantified business impact.”
  • OpenAI’s Forward Deployed Engineer posting (London) asks for someone who will “share field feedback that helps Research and Product understand where the models succeed and where they can improve.”

So the job is two things at once: make the customer’s system work today, and turn what you learned into evidence that improves the product for everyone.

What do I need to know first?

Here are the terms this part uses, each with a plain explanation:

  • Symptom: what the user saw, in their words. It is not the cause. Sam’s symptom is “no data”.
  • Reproduction (often shortened to repro): a set of steps that makes the failure happen on demand. A minimal reproduction is the smallest set of steps and code that still fails.
  • Log: a record of one event, written by the code as it runs, such as “tool called” or “request refused”. A structured log is written as data (usually JSON), so you can filter it by field.
  • Trace and span: a trace is the full record of one request from start to finish, made of spans. Each span is one timed step, such as a model call or a call to another agent.
  • Root cause: the earliest thing that, if it had been different, would have prevented the failure. Everything after it is a consequence.
  • Workaround: a change that avoids the problem while the cause is still there.
  • Regression test: a test that fails if this exact problem ever comes back.

Put together, they form a loop that every field investigation follows:

THE FIELD DEBUGGING LOOP Symptom what Sam saw, in his words Reproduce make it fail on demand Observe logs, traces, the code Isolate estate, our code or product Fix or a dated workaround Report evidence the product team can act on close the loop: tell the customer, add a regression test, update the playbook The shaded steps take most of the time. Skip them and the fix lands in the wrong place.

the field debugging loop: start from the symptom in the user's words, make it fail on demand, observe it in logs, traces and code, isolate whose problem it is, fix it or apply a dated workaround, report it with evidence, then close the loop

It is tempting to jump from the symptom straight to a fix, because the customer is waiting. In my work with partners, most of the fixes that did not stick came from skipping the middle steps.

How do I read code I did not write?

A forward deployed engineer often debugs code written by someone else: the customer’s team, a partner, or a colleague who has moved on. The mistake is to start reading files from the top of the repository. A large codebase has far more code than you need, and you will not know which parts matter until you have seen one request go through it.

I use the same reading order every time:

STEP ON THE ACME ROUTER 1 Run it send one of Sam's real questions to the router on a laptop 2 Find the entry point root_agent in agent.py: the Workflow and its edges 3 Follow one request front, extract, route, the supply call, then the writer 4 Read the tests routing rows and transport tests say what the authors meant 5 Search the exact error grep the message, then git log and git blame on that file Follow one request before you read anything else. Breadth comes later, when you know which files matter.

a reading order for code you did not write: run it once, find the entry point, follow one request end to end, read the tests for intent, then search for the exact error text and read the history of that file

The entry point is where a request enters the code. In an ADK project it is the root_agent in agent.py; for the Acme router it is the graph workflow built in Part 5 of the agents series. From there, read only the code that one request touches. The tests tell you what the authors meant the code to do, and git log and git blame on the file that raised the error tell you who changed it last and why. AI coding assistants save hours here, for example by explaining a module or finding every caller of a function. Check what they tell you against the code before you rely on it.

For the Acme router, the entry point is short enough to read in a minute. It is a list of edges:

root_agent = Workflow(
    name="acb_demo_router",
    edges=[
        (START, front),
        (front, {"continue": extract, "reply": reply}),
        (extract, route),
        (route, {
            "marketing": marketing_remote,
            "supply": supply_remote,
            "both": both,
            "decline": decline,
        }),
        (marketing_remote, writer),
        (supply_remote, writer),
        (both, writer),
    ],
)

These few lines tell you which nodes a question can touch, and so which files to open first: for a “no data” answer, the remote specialist calls and the writer. Part 5 of the agents series explains each edge.

How do I make a failure happen on demand?

You cannot fix what you cannot see fail. The first real goal of any investigation is a reproduction. Start where the user saw the problem, then remove one moving part at a time until the failure happens in something small and fast:

FROM SAM'S CHAT TO ONE FAILING TEST Sam in Gemini Enterprise real user and identity; you cannot attach a debugger Staging router, a test user same deployment and identity type; logs you may read A script: one A2A call to supply the router's token, no model in the way A unit test the transport with a fake certificate; under a second Each step down removes one thing that could be to blame and makes each attempt faster.

the reproduction ladder: start where the user saw the problem, then remove one moving part at each step, from Sam in Gemini Enterprise to a staging router, to a script that makes one A2A call, to a unit test that runs in a second

Each step down the ladder removes one thing that could be to blame. When the failure still happens in a script that makes a single call to the supply agent, with no model involved, you have already ruled out the prompt, the model and the user interface. When it happens in a unit test, you can try a fix in seconds instead of redeploying.

Two practical points matter in a customer estate. First, use a test identity, not the user’s. You should never ask Sam for his password or his token so that you can “try it as him”. Second, write down every step as you go, with timestamps. You will need them for the report, and the customer’s security team may ask what you did.

If the failure does not happen every time, it is intermittent: write down how often (“3 of 20 attempts”) and what was different on the failing ones. If it used to work, git bisect finds the change that broke it by testing versions between the last good one and the first bad one.

What do the logs and traces tell me?

Once you can reproduce the failure, you can look at it closely. For agents, the most useful view is usually the trace, because it shows every step of one request on a time line. The Acme agents send their traces to Cloud Trace and their structured logs to Cloud Logging, as set up in Part 11 of the agents series.

This is a simplified version of the trace for Sam’s question:

ONE TRACE, SIMPLIFIED (BAR LENGTHS ILLUSTRATIVE) invoke_agent router front (code) extract: generate_content route (code) A2A call: marketing A2A call: supply writer: generate_content 401 Context-Aware Access requirements are not met 401, same message Sam saw: "The supply team has no data for last week's stockouts." The error was in the trace all along. The answer hid it, because no step treated a 401 as a failure.

the trace for Sam's question: both calls to the specialists end quickly with 401, and the writer still writes an answer, so a login failure reaches Sam as 'the specialists have no data'

The two calls to the specialist agents end almost at once, both with HTTP 401 (the server’s way of saying “I do not accept your credentials”). The logs for those spans contain the full message: “Context-Aware Access requirements are not met.” Then the writer runs anyway, receives no findings, and writes a polite answer saying the specialists have no data.

That last step is why the problem was confusing. The failure hid itself. A login failure reached the user as an empty result. When you read a trace, look for three things: steps that end much faster than usual, steps that end with an error code, and steps that carry on as if nothing happened after an earlier step failed.

Is it the customer’s estate, our code or the product?

This is the most important question in field debugging, because it decides who has to act. The estate is everything the customer runs: their network, their security policies, their identity provider. A failure can come from there, from the code you and the customer wrote, or from the product itself. I ask two questions in order:

Fails outside their estate? same code, in your own project The customer's estate proxy, firewall, VPC Service Controls, org policy or identity provider no yes Product does what docs say? read the page, not a forum post Our code or configuration fix it; if the docs were hard to find, say so A product bug report it with a minimal reproduction yes no THE ACME 401 It failed in our own project too, and the docs say Agent Identity tokens are certificate-bound by default. So: our code. The transport needed mTLS. Ask the first question early. It decides whose time the rest of the investigation uses.

isolating the cause: if it only fails inside the customer's estate, look at their network, policies and identity provider; if it fails in your own project and the product does what its documentation says, it is your code or configuration; if the product does not, it is a product bug

Does it fail outside their estate too? Run the same code against your own test project. If it fails only at the customer, look at their environment: a proxy (a server between their network and the internet) that inspects encrypted traffic, a firewall rule, a VPC Service Controls perimeter (a Google Cloud boundary around a set of projects), an organisation policy or an identity provider setting. Part 3 of this series covered how to ask about these in week one.

Does the product do what its documentation says? Read the official page, not a forum answer or a model’s summary. If the product behaves as documented, the problem is in your code or configuration. If it does not, you may have found a product bug.

For Acme, the answer came quickly. The failure happened in our own test project too, so it was not Acme’s estate. And the Agent Runtime documentation says that Agent Identity tokens are certificate-bound by default: a token is accepted only over a connection that presents the matching client certificate, using mTLS (mutual TLS, where the client proves its identity as well as the server). Our first router used a standard HTTP client that sent the token without the certificate. The product was doing exactly what it said. The bug was in our transport, and the fix was to load the runtime’s certificate and call the mTLS endpoint. The heart of the fix is a small transport, simplified here:

class AgentIdentityMtlsTransport(httpx.AsyncBaseTransport):
    async def handle_async_request(self, request):
        ctx = mtls.get_default_ssl_context()  # None: no cert
        if ctx:
            host = _to_mtls_host(request.url.host)
            request.url = request.url.copy_with(host=host)
        inner = httpx.AsyncHTTPTransport(verify=ctx)
        return await inner.handle_async_request(request)

On the runtime it presents the agent’s certificate and sends Google API calls to the *.mtls.googleapis.com host. On a laptop with no certificate, it behaves as a plain transport. Part 7 of the agents series explains the fix line by line.

One honest point: “our code” does not mean the product team has nothing to learn. We lost time because the error reached the user as missing data, and that is useful feedback. I come back to it below.

Should I fix it or work around it?

Sometimes the fix takes days, and the customer cannot wait. A workaround keeps them moving, but only if it is handled with discipline:

Workaround what it is avoids the problem; the cause remains who owns it a named person, often the FDE when it ends a date, or when the fix ships where it is written the runbook and a code comment Acme pause supply questions, say so clearly Fix what it is removes the cause who owns it the team that owns the code when it ends never; it is the new normal where it is written the change, its test, release notes Acme mTLS transport plus a unit test Never a workaround that weakens security. Switching off token binding would have made the 401 disappear, and made any stolen token work anywhere.

a workaround versus a fix: a workaround avoids the problem while the cause remains, so it needs an owner, an end date and a written record; a fix removes the cause and comes with a test; a workaround must never weaken security

A workaround needs three things written down: who owns it, when it ends and where it is recorded. Without those, it becomes permanent, and the next team inherits a mystery. For Acme, a reasonable workaround while the transport fix was tested would be to pause supply questions and show a clear message that says so, instead of letting the router answer “no data”.

There is one kind of workaround I never accept: one that weakens security. The documentation lets you switch off token binding. That would have made the 401 disappear in minutes. It would also have made any leaked token usable from anywhere. A forward deployed engineer is a guest in the customer’s estate, and lowering their security to make a demo work breaks the trust the whole engagement depends on.

How do I write a report Product and Engineering can act on?

When the cause really is in the product, or the product made a mistake easy to make, your report is what the product team will see. They do not know the customer, they cannot see the estate, and they receive many reports. A good report lets an engineer reproduce the problem without asking you a single question.

Google Cloud’s public guidance for support cases asks for the same core details any product team needs: the exact time in ISO 8601 format, the product and feature, the location, concrete identifiers such as the project ID, useful artefacts, and whether the problem is intermittent or consistent. I add expected behaviour, impact and a minimal reproduction. This is the template I use:

TITLE: <component>: <what happens> when <condition>
       e.g. "Agent Runtime A2A call: 401 'Context-Aware Access requirements
       are not met' when the client sends a bound token over plain TLS"

SUMMARY      One or two sentences. What fails, for whom, since when.
ENVIRONMENT  Product and version, region, project ID (never a token or a
             customer name), SDK and library versions.
TIME         First seen 2026-10-14T09:12:00Z; still happening / ended at ...
STEPS        Numbered, from a clean start. Link the minimal reproduction.
EXPECTED     What the documentation says should happen (link the page).
ACTUAL       What happened. Exact error text, request ID, trace ID.
FREQUENCY    Every time / 3 of 20 attempts / only after ...
IMPACT       Who is blocked and what it costs, in plain words.
WORKAROUND   What we do meanwhile, and why it is not good enough.
ASK          Bug fix, documentation change, or feature request.

Two rules make this template work. First, one report per problem. Google’s support guidance says the same: one case per issue, and duplicates are closed. Second, remove everything confidential. Use your own test project and test identity in the reproduction, never the customer’s tokens, data or name.

If the product worked as documented, the report becomes a feature request or a documentation request. For Acme, it would be something like: “When a certificate-bound token is used without mTLS, return an error that says so, and make the A2A samples use the mTLS transport by default.” That is the pattern the Microsoft posting asks for, and the “formal product feature request” in the Google one. One report from one customer is an anecdote. The same friction seen in three engagements, with three reproductions, is a pattern that a product team can act on.

How do I close the loop?

Filing the report is not the end. The loop only closes when the customer is safe without the workaround and nobody can meet the same problem again:

Field finding the 401 hid as 'no data' Report minimal repro, impact, what you expected Product triage bug, docs fix or feature request Change ships or a clear answer: working as intended Remove workaround on the date you wrote down Make it stick regression test, playbook entry the next engagement starts here Tell the customer what happened at every step. Silence after a report reads as no progress.

closing the loop: a field finding becomes a report, the product team triages it, a change or a clear answer comes back, the workaround is removed on its date, and a regression test and playbook entry make sure nobody meets the same problem twice

  • Tell the customer what you found, what you changed and what is still open, in words their team can repeat to their manager. Part 10 covers how to say it to an executive.
  • Track the report until it is fixed, answered or closed as working as intended. Update the customer when its status changes.
  • Remove the workaround on the date you wrote down, or move the date and say why.
  • Add a regression test. For Acme, two tests came out of this: the transport must use the mTLS host on the runtime, and every 401 or 403 must appear as an error, never as an empty result. Part 10 of the agents series shows how such tests gate a release.
  • Write it in the playbook, so the next engineer on the next engagement recognises the symptom in minutes instead of hours.

Google’s SRE book calls the written version of this a blameless postmortem: a review of causes and fixes that does not look for someone to blame. The same tone works with customers.

Try it: turn a vague symptom into a report

You do not need a cloud project for this exercise. Take this message, which is the kind of thing a forward deployed engineer receives every week:

“The assistant is broken again. It gave Maria a weird answer this morning and now nobody trusts it.”

  1. Write the symptom in the user’s words, then list the five questions you would ask Maria before you touch any code. (Mine start with: what did you ask, at what time, and what did you expect to see?)
  2. Sketch the reproduction ladder for it, as in the diagram above. What is the smallest thing that could still show the problem?
  3. Decide what you would look for in the trace. Which steps would you expect to see, and which would worry you if they were missing, very short or ended with an error?
  4. Fill in the report template with placeholders where you do not know the answer yet. The empty fields show you what to find out next.

If you have a deployed agent, pick one failed request in Cloud Trace and explain it, step by step, in one paragraph.

What can go wrong

  • Fixing the symptom. Changing the writer’s prompt so that it stops saying “no data” would have hidden the 401 even better. Always find the earliest step that went wrong.
  • Debugging in production first. Every attempt is slow and every mistake is visible.
  • Asking for the user’s credentials. Use a test identity. If the bug only happens for one user, compare their permissions with your test user’s instead.
  • Blaming the product too early. A report that turns out to be a configuration mistake costs the product team time and costs you credibility. Check the documentation first.
  • Workarounds with no end date. They become permanent and undocumented.
  • A report full of customer detail. Names, tokens and data in a ticket are a leak. Reproduce in your own project.
  • Going quiet. A customer who hears nothing for three days assumes nothing is happening, even if you are working on it every day.

What I would do now

  1. Write the reproduction ladder for the agent you are building today, before anything breaks.
  2. Make sure every authentication error is logged as an error and fails a test, never as an empty result.
  3. Copy the report template into your team’s tracker as the default for new issues.
  4. Start a playbook page with one entry per field problem: symptom, cause, fix and test.
  5. Agree with the customer, in week one, how they will report problems to you and how you will update them.

What did we learn?

  • A field investigation follows one loop: symptom, reproduction, logs and traces, root cause, fix or workaround, report, and a regression test.
  • Read unfamiliar code from its entry point and follow one request through it, using the tests and the history to learn what the authors meant.
  • Reproduce the failure with a test identity and the smallest set of steps that still fails, and write down every step with a timestamp.
  • In a trace, look for steps that end too fast, steps that end with an error and steps that carry on after an earlier step failed, because a failure can hide itself as an empty result.
  • Decide whose problem it is by asking whether it fails outside the customer’s estate and whether the product does what its documentation says.
  • A workaround needs an owner, an end date and a record and must never weaken security, and a good report lets a product engineer reproduce the problem without asking you anything.

What comes next?

The router now fails loudly and has tests that keep it that way. The next complaint from Acme is a different kind of problem: “It works, but it is slow, and the bill is higher than we expected.” Part 9: LLM-native performance, latency, tokens and cost covers time to first token, tokens per second, context size, caching, streaming, batching and cost per request, and why you measure before you optimise. The traces you learned to read in this part are where it starts.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands and its data are fictional.