Forward deployed engineering, Part 9: LLM-native performance, latency, tokens and cost

In Part 8 we fixed a failure that hid itself: a login error that reached Sam, a supply planner at Acme Consumer Brands, as “no data”. The router now fails loudly and has tests that keep it honest. A week later, a different message arrives from Maria in marketing: “It works, and the answers are good. But it takes ages, and finance wants to know why the bill is higher than the estimate.”

This is a different kind of problem. Nothing is broken. The system is simply slower and more expensive than people expected, and nobody can say exactly why. Language models are slow and expensive in their own ways, and the usual web service measures (requests per second, server time) do not explain them.

This part covers the measures that do. It explains what happens inside one model call, why the length of a prompt matters, what thinking costs, and how caching, streaming, batching and model size each change the picture. It ends with the habit that matters more than any single technique: measure before you optimise.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: where the time and money go in a model-powered system, and which levers move them, in three minutes. The narration is a synthetic voice; the script and animation are mine.

Where does this show up in an FDE job description?

Google’s postings name this skill directly. Three examples, as of 20 October 2026:

• Google Cloud’s Forward Deployed Engineer, Gen AI posting lists, as a preferred qualification: “Knowledge of ‘LLM-native’ metrics (e.g., tokens/sec, cost-per-request) and techniques for optimizing state management and granular tracing.”
• The same posting asks the engineer to “build high-performance evaluation pipelines and observability frameworks to ensure agentic systems meet rigorous requirements for accuracy, safety, and latency.”
• OpenAI’s Forward Deployed Engineer posting (London) looks for someone who has “built or deployed systems powered by LLMs or generative models and understand[s] how model behaviour affects product experience.”

Latency and cost are where model behaviour and product experience meet. A correct answer that takes 20 seconds, or costs more than the analyst it replaces, is not a success.

What do I need to know first?

Here are the terms this part uses, each with a plain explanation:

• Token: the unit a model reads and writes. It is a piece of a word. Google’s documentation says that for Gemini models a token is about 4 characters, and 100 tokens are about 60 to 80 English words. Models are priced and limited in tokens, not words.
• Input (prompt) tokens: everything you send: instructions, tool descriptions, earlier messages, retrieved documents and the question.
• Output tokens: what the model writes back.
• Thinking tokens: tokens a reasoning model generates before its answer, to work the problem through. They are usually not shown to the user, but Google’s pricing page lists output prices as “Text output (response and reasoning)”, so they are billed at the output rate.
• Context window: the maximum number of tokens one request can hold, input and output together.
• Latency: how long someone waits. Throughput: how much work the system finishes in a period, such as questions per minute.
• Percentile: a way to describe a spread of measurements. The p50 (median) is the value half of requests beat. The p95 is the value 95 per cent of requests beat, so it shows what the slowest users experience. Report both, never only the average.

What happens inside one model call?

Every model call goes through the same phases. Seeing them on a time line explains most of what people call “slow”:

anatomy of one model call: the request travels and may queue, the model reads the whole prompt (prefill), may think, then produces the answer one token at a time; time to first token ends when the first visible token arrives, and tokens per second measures the speed after that

1. Network and queue. The request travels to the model’s endpoint and may wait for capacity.
2. Prefill. The model reads the whole prompt before it can write anything. A longer prompt means a longer prefill.
3. Thinking, if the model is allowed to think. Nothing visible appears during this phase.
4. Decode. The model writes the answer one token at a time.

Three measures come out of this picture, and together they describe how a model call feels:

• Time to first token (TTFT): the time from sending the request to receiving the first token of the answer. NVIDIA’s public benchmarking guide defines it this way and notes that it includes queuing, prefill and network time, and that longer prompts increase it. For a person watching a screen, TTFT is the moment the system starts to feel responsive.
• Inter-token latency: the gap between one output token and the next.
• Tokens per second: how fast the answer arrives once it has started, which is the number of output tokens divided by the decode time.

End-to-end latency runs from the request to the last token. An agent’s end-to-end latency is the sum of the model and tool calls that run one after another.

Where does the money go?

The bill for a model call depends on four token counts, each at its own price: fresh input, cached input, output and thinking. Part 11 of the agents series built a cost report for the Acme agents. Its worked example, with token counts made up for illustration, breaks down like this:

where the money goes in one Acme question, using the illustrative token counts from Part 11: the writer on Pro is the most expensive call, thinking tokens are about a third of the total, and cached input is the smallest part

Three things stand out, and they hold for most agents I have seen:

• The largest model is usually the largest cost. The writer runs on Gemini 3.1 Pro Preview, which costs more per token than Flash.
• Thinking is a large share. In this example it is 35 per cent of the cost of the question.
• Cached input is cheap. It is most of the marketing agent’s prompt, yet the smallest part of its cost.

Two pricing rules matter for estimates. Google’s pricing page says that if a request’s input is longer than 200,000 tokens, all of its tokens are charged at the higher long-context rate. It also says you are charged only for requests that return a 200 (success) response; failed requests are not charged. That does not make retries free, because a retry that succeeds is a second paid call.

The number to report is the one from Part 11: cost per resolved question, the total cost divided by the number of questions answered correctly. A cheap answer that is wrong still costs money, and the user asks again.

Why measure before optimising?

When someone says “it is slow”, the temptation is to change something straight away: a smaller model, a shorter prompt, a cache. In my experience, the first guess about where the time goes is wrong more often than it is right. So the first step is always a trace of a real question:

where the time goes in one Acme question, with illustrative timings: the extract call, the slower of the two specialists and the writer make up almost all of the wait; the routing code is too small to see, and without streaming Sam sees nothing until the writer finishes

The numbers in this diagram are made up, but the shape is typical. The routing code, which people often suspect first, takes almost no time. The extract call is short. The two specialists run in parallel, as designed in Part 5 of the agents series, so the slower one sets the middle of the wait. The writer is about half the total, and more than half of the writer’s time is thinking.

With that trace in hand, it is clear that faster routing code would save nothing, and that letting the writer think less, or streaming its answer, would change Maria’s experience the most.

To measure this yourself, read the spans for one slow question in Cloud Trace (ADK records each model call as a generate_content span with its token counts), then repeat for twenty questions and look at the p50 and p95. Do not optimise based on one run.

Why does a long conversation get slower and dearer?

A model keeps no memory between calls. To continue a conversation, the application sends the instructions, the tool descriptions and the whole conversation so far with every new turn. So the input grows every time the user asks a follow-up:

why long conversations get slower and dearer: the model keeps no memory between calls, so every turn sends the instructions and tools again plus the whole conversation so far; the history part grows each turn while the new question stays small

Each turn costs more and takes longer to prefill than the one before. There is also a quality cost. The research paper Lost in the Middle found that language models use information at the start and end of a long context better than information in the middle. A longer prompt is not automatically a better one.

The fixes are simple to state:

• Send only what the step needs. The writer needs the specialists’ findings, not their full tool outputs. The extract step needs the last few turns, not the whole history.
• Trim or summarise old turns once a conversation passes a set length.
• Keep large reference text out of the prompt and retrieve the relevant part when it is needed (Part 4 of this series covers retrieval).
• Keep state in state. Facts the agent must remember, such as the user’s region, belong in session state or memory (Part 8 of the agents series), not in a growing transcript.

How does caching help?

Context caching lets the model reuse the work it already did on the start of a prompt. If many requests begin with the same text, the shared part (the prefix) is read at a much lower price. Google Cloud offers two kinds. Implicit caching is automatic and on by default in every project, with a 90 per cent discount on cached tokens. Explicit caching lets you create a cache yourself and control how long it lives, which also adds a storage charge.

The catch is the word prefix. A cache can only reuse text that is identical from the very first token. So the order of the prompt decides whether caching works at all:

prompt order and caching: a cache can only reuse an identical prefix, so put what never changes first (instructions, skill text, tool declarations) and what changes last (the question); one changing value at the start means nothing after it can be reused

Put what never changes first (instructions, skill text, tool declarations) and what changes last (the question). A single changing value at the start, such as today’s date or the user’s name, means nothing after it can be reused. The Agent Platform documentation sets a minimum of 4,096 tokens for caching on most Gemini 3 models, so a very short prompt is not cached at all. Each response reports how many of its input tokens came from the cache, and the Acme agents log that number as cached_tokens. It tells you whether caching is working in practice.

Two usage entries from the agents series show what to look for. Most of the marketing agent’s prompt came from the cache; none of the writer’s did:

{"event": "model_usage", "agent": "acb_demo_marketing", "invocation": "q1-mkt", "model": "gemini-3.5-flash", "prompt_tokens": 9000, "cached_tokens": 6000, "output_tokens": 150, "thinking_tokens": 400}
{"event": "model_usage", "agent": "writer", "invocation": "q1-router", "model": "gemini-3.1-pro-preview", "prompt_tokens": 5000, "cached_tokens": 0, "output_tokens": 450, "thinking_tokens": 900}

The token counts are made up, as in the cost example above. Part 11 of the agents series shows where these entries are written and how the cost report reads them.

Does streaming make it faster?

Streaming means sending the answer to the user piece by piece as the model writes it, instead of waiting for the whole answer. On Agent Platform the REST method is streamGenerateContent.

streaming versus waiting for the whole answer: both finish at the same moment, but with streaming the user starts reading at the time to first token instead of staring at a spinner until the end

Streaming does not make the model any faster. The total time is the same. What changes is what the user waits for: with streaming, Maria starts reading at the time to first token instead of watching a spinner until the end. For a writer that produces a few paragraphs, that is the difference between a system that feels responsive and one that feels broken.

One honest limit: streaming only helps if every layer between the model and the screen passes the pieces on. In a multi-agent system, a router that collects a specialist’s whole answer before continuing will not stream that part, whatever the model does. Check each hop.

How do I choose a model size, and how much should it think?

Bigger models are usually better at hard reasoning and slower and dearer per token. Smaller models are faster and cheaper, and many steps do not need more. The rule I use is: start each step on the smallest model that passes its evals, and move up only when the evals say so.

For Acme, the extract step only rewrites a question as one line. Part 11 of the agents series suggested trying Gemini 3.1 Flash-Lite for it, at a fraction of Flash’s price, as long as routing accuracy stays at 100 per cent in the eval gate from Part 10 of the agents series. The writer has to combine findings into a clear, correct answer, so it stays on Pro until a smaller model passes the same gate. The gate itself is a short list of minimum scores, simplified here:

THRESHOLDS = {  # metric -> minimum mean score
    "route_correct": 1.0,   # routing is code: < 100 % is a bug
    "grounded": 1.0,        # no invented numbers, ever
    "golden_numbers": 0.9,
    "refusal_rubrics": 0.9, # judge: only after calibration
    "faithfulness": 4.0,    # judge, 1-5 scale
}

A cheaper extract model passes only if route_correct stays at 1.0. Part 10 of the agents series explains each metric.

Thinking is a second dial. For Gemini 3 models, the thinking_level parameter controls how much the model reasons before it answers. The Gemini 3 developer guide says the default is high, and that low “minimizes latency and cost” for simple instruction following and chat. A step like extract is a good candidate for low. The writer may need more. As with model size, let the evals decide.

What if nobody is waiting for the answer?

Some model work has no user watching: nightly eval runs, re-scoring old questions after a prompt change, or summarising a week of feedback. That work does not need the fastest or most available capacity. Google Cloud offers several ways to buy model capacity:

ways to buy model capacity on Gemini Enterprise Agent Platform: Standard PayGo for most interactive traffic, Priority PayGo at a higher rate for business-critical spiky traffic, Flex PayGo in Preview and batch at half price for work that can wait, and Provisioned Throughput for steady, known volume

• Standard PayGo (pay as you go) charges per token on shared capacity.
• Priority PayGo charges a higher rate per token; the documentation suggests it for business-critical workloads with unpredictable traffic, such as customer-facing assistants.
• Flex PayGo, in Preview as of October 2026, is 50 per cent cheaper than Standard, for synchronous work that can tolerate higher latency.
• Batch takes many requests at once and returns results later, also at a 50 per cent discount.
• Provisioned Throughput is a fixed-cost, fixed-term subscription that reserves capacity.

For Acme, the obvious move is to run the nightly golden-set evals on Flex or batch. They are a real share of the model bill during development, and nobody is waiting for them at two in the morning.

Try it: which lever moves the cost most?

This script runs without any cloud resources and only needs Python 3. It takes the Part 11 example question and changes one thing at a time. Save it as levers.py:

PRICES = {  # US$ per million tokens: input, cached input, output (incl. thinking)
    "gemini-3.5-flash": (1.50, 0.15, 9.00),
    "gemini-3.1-flash-lite": (0.25, 0.025, 1.50),
    "gemini-3.1-pro-preview": (2.00, 0.20, 12.00),
}
BASE = [  # step, model, prompt tokens, of which cached, output, thinking
    ("extract", "gemini-3.5-flash", 3200, 0, 120, 300),
    ("marketing", "gemini-3.5-flash", 19500, 15000, 500, 600),
    ("writer", "gemini-3.1-pro-preview", 5000, 0, 450, 900),
]

def cost(calls):
    total = 0.0
    for _, model, prompt, cached, out, think in calls:
        p_in, p_cached, p_out = PRICES[model]
        total += ((prompt - cached) * p_in + cached * p_cached + (out + think) * p_out) / 1e6
    return total

def change(step, **kw):
    keys = ("model", "prompt", "cached", "out", "think")
    rows = []
    for row in BASE:
        if row[0] == step:
            d = dict(zip(keys, row[1:]))
            d.update(kw)
            row = (step, *[d[k] for k in keys])
        rows.append(row)
    return rows

SCENARIOS = {
    "baseline": BASE,
    "extract on Flash-Lite": change("extract", model="gemini-3.1-flash-lite"),
    "writer thinks half as much": change("writer", think=450),
    "writer prompt 84% cached": change("writer", cached=4200),
}
base = cost(BASE)
for name, calls in SCENARIOS.items():
    c = cost(calls)
    print(f"{name:30s} {c:.4f} per question  {c / 0.8:.4f} per resolved (80%)  "
          f"{100 * (c - base) / base:+.0f}%")

Run python3 levers.py. When I ran it on 20 October 2026, the output was:

baseline                       0.0537 per question  0.0671 per resolved (80%)  +0%
extract on Flash-Lite          0.0465 per question  0.0582 per resolved (80%)  -13%
writer thinks half as much     0.0483 per question  0.0604 per resolved (80%)  -10%
writer prompt 84% cached       0.0461 per question  0.0576 per resolved (80%)  -14%

The prices are the list prices checked in Part 11 on 19 October 2026 (global endpoint, prompts under 200,000 tokens); the token counts are made up. Then try two changes. First, combine all three levers and see whether the savings simply add up. Second, change the resolved rate from 80 per cent to 70 per cent for the Flash-Lite scenario: if a cheaper extract step makes routing worse, the cost per resolved question can end up higher than the baseline.

To measure time to first token on a real model, the Gemini API’s documented streaming pattern is enough. This needs a Gemini API key in the GEMINI_API_KEY environment variable:

import time
from google import genai

client = genai.Client()
start, first = time.perf_counter(), None
stream = client.interactions.create(
    model="gemini-3.5-flash",
    input="In three sentences, why do stockouts hurt a marketing campaign?",
    stream=True,
)
for event in stream:
    if event.event_type == "step.delta" and event.delta.type == "text":
        if first is None:
            first = time.perf_counter()
        print(event.delta.text, end="")
end = time.perf_counter()
print(f"\n\nTTFT {first - start:.2f}s, total {end - start:.2f}s")

I have not printed timings here, because they depend on the region, the time of day and the prompt. Run it ten times and look at the spread, not one number.

What can go wrong

• Optimising without a trace. You will speed up the part you suspected, not the part that is slow.
• Reporting only averages. One very slow request in twenty is invisible in the average and very visible to the user who waited. Report p50 and p95.
• Leaving thinking out of the cost model. It is billed as output and can be a third of the cost.
• Breaking the cache by accident. A timestamp or user name at the top of the prompt makes every request look new. Watch cached_tokens after every prompt change.
• A cheaper model that answers worse. Cost per question falls, cost per resolved question rises. Every performance change goes through the same eval gate as any other change.
• Streaming that stops at one hop. If any layer buffers the whole answer, the user sees no benefit.
• Reserving capacity too early. Provisioned Throughput is a commitment. Buy it after usage is measured, not before.

What I would do now

1. Pick twenty real questions and record the p50 and p95 end-to-end latency and the cost per resolved question. That is your baseline.
2. Open the trace of the slowest one and write down which step takes the most time.
3. Reorder every prompt so the unchanging part comes first, then check cached_tokens.
4. Try the smallest model and the lowest thinking level that pass the evals for each step.
5. Move evals and other offline model work to Flex or batch.

What did we learn?

• Every model call goes through the same phases (network and queue, prefill, thinking and decode), and time to first token, tokens per second and end-to-end latency describe how it feels.
• The bill depends on fresh input, cached input, output and thinking tokens, and the number to report is cost per resolved question.
• Measure with a trace, and with p50 and p95 over many questions, before you optimise, because the first guess about where the time goes is often wrong.
• A long conversation gets slower and dearer with every turn, so send each step only what it needs and keep facts in state.
• Caching only reuses an identical prefix, so put what never changes first and watch cached_tokens after every prompt change.
• Streaming lets the user start reading at the time to first token without making the model faster, each step should run on the smallest model and lowest thinking level that pass its evals, and work nobody waits for belongs on Flex or batch.

What comes next?

Maria’s questions now come back faster, and the cost per resolved question has a number and an owner. What is left is the conversation that decides whether the engagement was worth it. The last part, Part 10: Talking to executives and proving the outcome, covers how to turn a technical blocker into business impact, set baselines and outcome metrics, run a steering update, manage scope, and close the engagement with a review. The latency and cost numbers from this part become measures in its metric tree, and it ends with a recap of the whole series.

Sources

• Google Careers: Forward Deployed Engineer, Gen AI, Google Cloud (public posting, checked 20 October 2026)
• OpenAI: Forward Deployed Engineer, London (public posting, checked 20 October 2026)
• Gemini API: Understand and count tokens (about 4 characters per token)
• Generative AI pricing on Agent Platform (reasoning billed as output, long-context rate, batch discount, charged only for 200 responses)
• Context caching overview (implicit caching, 90 per cent discount, minimum token counts)
• Gemini API: Gemini 3 developer guide (thinking_level)
• Gemini API: Text generation, streaming responses
• Gemini API: Batch API
• Agent Platform: Priority PayGo (also shows streamGenerateContent)
• Agent Platform: Flex PayGo (Preview)
• Agent Platform: Provisioned Throughput overview
• NVIDIA: LLM inference benchmarking metrics (time to first token, inter-token latency, tokens per second)
• Liu et al.: Lost in the Middle: How Language Models Use Long Contexts
• ADK: traces
• Google SRE book: Service level objectives (why percentiles beat averages)

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands and its data are fictional, and all timings and token counts are illustrative, not measurements.