Production-ready agents, Part 14: Knowledge: RAG, long context, fine-tuning and the infrastructure behind them

Every few months someone tells me RAG is dead. First it was because context windows grew to a million tokens, then because models got better at searching on their own. And every few months a partner asks me the opposite question: “should we fine-tune a model on our documents?”. Both questions come from the same place. People want the agent to know their business, and they are not sure where that knowledge should go.

This part answers that question from the ground up. An agent’s knowledge can live in three places, each with its own cost, freshness and safety. I explain what each one is, whether retrieval still earns its place, what Google Cloud offers for it, when tuning or training a model is really the right answer, and what you need to run each option. Part 13 covered choosing the model; this part covers what you put around it.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: where an agent's knowledge should live, and when RAG, long context or fine-tuning fits, in three minutes. The narration is a synthetic voice; the script and animation are mine.

Where are we in the architecture?

This part adds knowledge: retrieval, long context and, when it is really needed, a tuned model. Part 1.25 walks through the whole picture, layer by layer.

where Part 14 sits in the Acme architecture: highlighted boxes are the subject of this part, solid boxes come from earlier parts, faded boxes arrive later

What do I need to know first?

Retrieval and tuning come with a lot of vocabulary. These are the terms this part uses.

• An embedding is a list of numbers that represents the meaning of a piece of text (or an image, or audio). Texts with similar meanings get similar numbers. Google’s Gemini Embedding 2 model, for example, accepts up to 8,192 input tokens and returns up to 3,072 numbers.
• A vector is the mathematical name for that list of numbers. A vector database (or vector index) stores many of them and finds the ones closest to a query quickly.
• A chunk is a piece of a document, such as a paragraph or a section, that is embedded and stored on its own. Chunking is how you cut documents into chunks.
• Retrieval is finding the chunks most relevant to a question. Reranking is a second, more careful pass that reorders what retrieval found.
• RAG, or retrieval-augmented generation, means retrieving relevant content and putting it into the model’s context before it answers.
• Grounding is connecting the model’s answer to sources you can check. Google’s grounding overview describes it as the ability “to connect model output to verifiable sources of information”. A citation is the link from a sentence in the answer back to its source.
• Long context means putting a large amount of material straight into the context window instead of retrieving a small part of it.
• Fine-tuning changes a model’s weights by training it on your examples. Supervised fine-tuning uses examples of an input and the output you want. Preference tuning uses pairs of answers marked as preferred and not preferred. Distillation trains a smaller model to copy a larger model’s answers.
• Parameter-efficient tuning (also called adapter tuning) changes only a small part of the model. Google’s introduction to tuning says it “updates a relatively small subset of the model’s parameters”, unlike full fine-tuning, which updates all of them. The best-known method is LoRA (low-rank adaptation): it freezes the weights and trains two small matrices per adapted layer. The section Are we talking about LoRA when we say fine-tuning? explains how it works and how Google Cloud exposes it.
• Continued pre-training trains an existing model further on a large amount of unlabelled text from a new domain. Training from scratch builds a model from nothing.
• An accelerator is a chip built for the maths models do: a GPU, or Google’s TPU (Tensor Processing Unit), which the Cloud TPU introduction describes as “custom-developed, application-specific integrated circuits (ASICs) used to accelerate machine learning workloads”.

Where can an agent’s knowledge live?

There are three homes, and every piece of knowledge your agent needs belongs in one of them: in the weights (learned once, in training), in the context (sent with each call) or behind a tool (fetched from the system that owns it). The common mistake is to assume that “the agent knows our business” means the model itself must hold the knowledge. For most enterprise facts it should not.

A family doctor seeing a patient shows all three homes at once:

• What the doctor learned at medical school (anatomy, how to read a blood test) is in their head. That is the weights: always there, no lookup needed, but slow to change and impossible to point at.
• The patient’s file open on the desk for this appointment is the context: put there for this visit, current, and the doctor can quote the page.
• Ringing the lab for this morning’s test result is a tool: the lab owns the result and decides who may receive it.

No doctor tries to memorise every patient’s latest results, and no doctor looks up anatomy in the file. Each kind of knowledge sits where it is cheapest and safest to keep it.

three places an agent's knowledge can live: in the weights, learned in training; in the context, put into the prompt on each call; or behind a tool the agent calls; with where Acme keeps each kind of knowledge today

In the weights. This is what the model learned in training: language, general knowledge, reasoning. Fine-tuning adds to it. Knowledge in the weights costs nothing to look up, but it is slow and expensive to change, the model cannot tell you where it learned something, and it cannot hide a fact from a user who should not see it. Once a fact is in the weights, everyone who uses the model can get at it.

In the context. This is everything you send with each call: the instructions, the skill that was loaded (Part 4), the memories that were preloaded (Part 8), and any documents you retrieved or pasted. Knowledge in the context changes the moment the data does and can come with a citation. The price is that every token is paid for on every call, and too much context makes answers worse, which Part 1.5 called context rot.

Behind a tool. This is knowledge the agent fetches when it needs it, from a system that owns it: a search service, a database, an API. Acme’s numbers live here. The marketing agent reads spend from BigQuery and the supply agent reads penalties from Databricks, as the user, so Databricks’ own access rules decide what comes back (Parts 2, 3 and 7). Knowledge behind a tool is always current and exact, and the source enforces who may see it.

The question for every new piece of knowledge is therefore not “RAG or fine-tuning?”. It is “which of the three homes fits this kind of knowledge?”. For enterprise data, which changes, has an owner and has access rules, the answer is almost always the context or a tool.

How does this apply to Acme? Here is the same test on three things an Acme agent might need to know. Like the doctor, ask of each one: is it a lab result (owned elsewhere, changing), a page in the file (needed for this call) or medical-school training (stable, general)?

• “Promotions in Western Europe need supply sign-off two weeks before launch.” A policy that changes now and then and should be cited: behind a retrieval tool (it is P1 in the exercise at the end of this part).
• “Verde’s media spend in the UK last month.” An exact number that changes every week and has an owner: behind the marketing agent’s query_marketing_measures tool on campaign_performance.
• “Answer as a short table and name the source of every number.” A small, stable rule about behaviour: in the instructions or a skill, in the context.

None of the three belongs in the weights. The first two are lab results: they have an owner and they change. The third is a page in the file: small, and needed on every call.

Is RAG still relevant?

Yes. What is fading is the fixed pipeline that retrieves for every question; what is winning is retrieval as a tool the agent decides to call. It is worth seeing why both of the arguments against it (long context and agentic search) are half right.

What does classic RAG look like?

Classic RAG is a fixed pipeline: for every question, code looks things up first and hands the results to the model, which then answers from them. The model never chooses whether to look.

Think of a reference library with a strict routine. A visitor hands a question to the desk. The librarian always does the same four things: checks the catalogue, pulls the two books that look closest, picks the page in each that best answers the question, and puts those pages in front of the visitor, who writes the answer. The catalogue was built long before the visitor arrived. Mapped to RAG: the catalogue is the index, pulling the books is retrieval, picking the best page is reranking, and the visitor writing from the open pages is generation.

The classic pipeline has the same steps for every question. Documents are cut into chunks, each chunk is embedded and stored in an index ahead of time. At question time, the question is embedded, the closest chunks are retrieved, a reranker may reorder them, and the best ones are put into the prompt. Google’s RAG Engine overview lists the same stages: data ingestion, data transformation (for example “data is split into chunks”), embedding, data indexing into “an index called a corpus”, retrieval and generation.

How does this apply to Acme? On a toy scale, using the five Acme policy snippets (P1 to P5) from the exercise at the end of this part, one pass through the pipeline looks like this:

Ahead of time
1. Chunk     each snippet is short, so each one becomes one chunk: P1, P2, P3, P4, P5
2. Embed     each chunk becomes a vector and is stored in the index with its ID
At question time (a supply user asks "What are the penalty rates for late deliveries?")
3. Embed     the question becomes a vector
4. Retrieve  top k = 2 closest chunks -> P2 (penalty rates), P5 (penalty disputes)
5. Rerank    the reranker keeps P2 first: it answers the question; P5 is only related
6. Generate  both passages go into the prompt with their IDs, and the answer cites [P2]

Code ran every step, and it would have run them for any question, even one that needed no documents at all, just as the librarian pulls books even for a visitor who only asked where the toilets are. That is the property the next two sections question.

Why not just paste everything into the context?

Long context is the right start for a small, stable set of documents; retrieval wins once the material is large, changing, permissioned or queried for many facts at once. The misconception is that a million-token window means the model reads all of it equally well, for free. It does neither.

Picture two ways to brief a colleague before a meeting. You can hand them the whole 400-page binder, or you can hand them the two pages that matter. If the meeting is about one clause in one contract, the binder works: they will find it. If the meeting needs six figures scattered across the binder, they are more likely to miss one, and if you brief them before every meeting, you print 400 pages each time. The binder is long context; the two pages are retrieval.

Long context is a real option, and sometimes the right one. Google’s own long context guide is direct about it: for many cases, “the default place to start is now just putting all of the tokens into the context window”. If your agent answers questions over one contract, one long report or a small, stable set of documents, long context is simpler than building a retrieval pipeline, and there is nothing to keep in sync.

The same guide also explains the limits. Its “near-perfect retrieval” figure is for finding a single fact; “in cases where you might have multiple ‘needles’ or specific pieces of information you are looking for, the model does not perform with the same accuracy”. And you pay for the whole context on every call. This is what that means in money for one question over a 400,000 token document set:

an illustrative cost comparison for one question over a 400,000 token document set at a list price of USD 0.75 per million input tokens: sending everything costs about USD 0.30 of input per call, the same prompt from a cache about USD 0.03 plus storage, and retrieving 8,000 relevant tokens about USD 0.006

Caching changes the arithmetic. On Agent Platform, the context caching overview says implicit caching is on by default with “a 90% discount on cached tokens compared to standard input tokens”, and explicit caches give the same 90% discount on Gemini 2.5 and later models. That makes repeated questions over the same large document affordable. It does not make them free, it does not fix accuracy on many-fact questions, and it does nothing for permissions.

How does this apply to Acme? The two briefings map directly (both cases are illustrative, not built in the series):

One contract, one question     Sam asks about the Harvest Row contract only. It is one document,
(the binder works)             it changes a few times a year and Sam may read all of it. Paste it,
                               let the cache absorb repeat questions, cite the clause.
The whole library              All retailer contracts and policies: hundreds of documents, updated
(hand over the two pages)      monthly, some readable only by supply or legal. Pasting them would
                               show marketing users pages they may not see and pay for every page
                               on every call. Retrieve, filtered by role.

A quick test, cheapest option first: if the material fits comfortably in the window, is stable, and every user may read all of it, paste it (and cache it). Only if one of those fails, retrieve.

What is agentic retrieval?

In agentic retrieval the model, not your code, decides whether to search, what to search for and whether to search again. The second change is who decides when to search. In classic RAG, code retrieves for every question. In agentic retrieval, search is just another tool. The model decides whether it needs to search, writes the query and any filters, reads the results, and searches again if they are not good enough.

Go back to the library. Classic RAG is the librarian with the fixed routine. Agentic retrieval is a researcher with a library card: asked a question, they first decide whether they already have what they need. If not, they search the catalogue, skim what comes back, notice it is close but not quite right, rephrase the search and go back. Asked something that needs no books, they simply answer. The searches are the tool calls, the rephrasing is the query rewrite, and the decision to stop is the model judging that it has enough.

classic RAG compared with agentic retrieval: classic RAG always runs a fixed pipeline of retrieve, rerank and generate before the model answers; with agentic retrieval the model decides whether to search, writes the query and filters itself, reads the results and can search again

ADK’s grounding guide describes this as “Agentic RAG”: agents “that reason about how to search, constructing queries and filters dynamically”. It is the same pattern as Acme’s specialists calling narrow data tools, applied to documents.

How does this apply to Acme? Here is the researcher’s routine for the policy agent from the design example later in this part (not built in the series), with its search_acme_policies tool over the five snippets from the exercise. Maria, who works in marketing, asks a question that needs a policy:

Question  "Can we run a Verde promotion in Western Europe next month?"
1. Decide   The model judges that this is a policy question, so it searches.
2. Act      search_acme_policies("promotion approval Western Europe")
3. Observe  [P4] "Campaign budgets above EUR 250,000 need finance approval before launch."
4. Decide   P4 is about budgets, not about whether a promotion is allowed. Not enough: refine.
5. Act      search_acme_policies("promotion sign-off Western Europe")
6. Observe  [P1] "Promotions in Western Europe need supply sign-off two weeks before launch."
7. Answer   "Yes, but it needs supply sign-off at least two weeks before launch [P1]. If the
            budget is above EUR 250,000 it also needs finance approval [P4]."

Now a question that needs no documents: “What was Verde’s CTR in the UK last month?” No search runs: the router sends it to the marketing agent, whose data tool holds the number. A classic pipeline would have embedded it and pasted policy passages into the prompt anyway. In the policy agent’s trace the model decided whether to search and what to search for; code only ran the search and applied the permission filter.

Who actually does the search?

One misconception is worth clearing up before going further: a model never runs a search. It reads text and writes text. Whether you build classic RAG or give the model a search tool, your code and a database do all of the chunking, embedding, searching and ranking, and code puts the results into the model’s context. The only difference between the two designs is who starts the search, and when.

A barrister preparing for court is a good picture. The barrister never goes into the archive. A clerk files every document when it arrives, labels who may see it, and keeps the index. When the barrister needs something, the clerk finds it and puts the marked pages on the desk. In one chambers the clerk brings a bundle to every meeting whatever the topic (classic RAG); in another the barrister asks for a bundle only when they need one (agentic RAG). Either way, the clerk does the filing and the finding. The barrister is the model; the clerk and the archive are your code and the index.

The diagram shows the clerk’s work, ahead of time and at question time. Part 13.75 covers how the documents are prepared before any of this: parsing, cleaning, chunking and metadata.

who does the search: ahead of time, code ingests documents with permission metadata, chunks them with context, embeds them with an embedding model and indexes vectors and keywords; at question time, code filters by the user's permissions, runs vector and keyword search, merges and reranks, and puts the best chunks into the prompt; the only difference between classic and agentic RAG is whether code or the model's tool call starts the live steps

How does this apply to Acme? Here are the clerk’s seven steps for one question, using Acme’s retailer contracts (illustrative documents and clause numbers). Sam, in supply, asks: “What does clause LDP-7 in the Harvest Row contract say about late deliveries, and can we dispute last week’s penalty?”

Ahead of time, whenever a document changes:

1. Ingest. Code extracts the text of each contract and attaches metadata: who may read it (roles: ["supply", "legal"]), when it was last updated, and its source. The permission filter in step 5 depends on this, so it is set here, not later.
2. Chunk. Code cuts each contract by section heading rather than every 1,000 characters, so a clause and its table stay together. A chunk on its own can lose its meaning: “The supplier pays 1.5% of invoice value per day late” does not say which contract or which clause. Contextual retrieval, described by Anthropic, fixes this by prepending a short, chunk-specific line of context before embedding and keyword indexing, for example “From the Harvest Row contract, clause LDP-7, late delivery penalties.” Anthropic reports that this reduced failed retrievals by 49%, and by 67% when combined with reranking.
3. Embed. Code sends each chunk to an embedding model, such as Gemini Embedding, which returns a vector: a list of numbers that represents its meaning. This is a different model from the one that writes answers.
4. Index. Code stores the text, the metadata and the vector in a vector index, and also builds a keyword index (the textbook-index idea: which chunks contain which words). BM25 is the classic formula for ranking keyword matches.

At question time, every time a search runs:

5. Filter, then search. Code reads Sam’s identity and filters first: only chunks whose metadata allows supply. Then it runs two searches. Vector search embeds the question and finds chunks with similar meaning, so “can we dispute” finds a chunk about “penalty challenges within 30 days”. Keyword search finds exact strings such as LDP-7, which embeddings handle poorly; Google’s hybrid search documentation makes the same point about product codes and internal code names. Code then merges the two lists, for example the top 25 from each, into one candidate list. Combining both is called hybrid search.
6. Rerank. Code passes the question and each candidate to a reranker, a model that reads the question and the chunk together and scores how well the chunk answers it. This is slower than vector search, which is why it runs only on the candidates. The best few, say five, are kept. RAG Engine on Agent Platform offers reranking as a built-in step.
7. Put them into the prompt. Code places the chunks into the model’s context with their IDs, and the instruction asks for an answer from those sources only, with citations:

Answer only from the sources below and cite their IDs. If they do not answer the question, say so.
[C1  Harvest Row contract, LDP-7, updated 2026-03-02] The supplier pays 1.5% of invoice value per day late...
[C2  Harvest Row contract, DSP-2, updated 2026-03-02] Penalty disputes must be raised within 30 days...
Question: What does clause LDP-7 say about late deliveries, and can we dispute last week's penalty?

So what changes with a search tool? Wrap steps 5 to 7 in a function, such as search_contracts(query), and give it to the agent as a tool. Steps 1 to 4 are exactly the same, and the chunks still end up in the model’s context, as the tool’s response rather than pasted in by your code. What changes is who triggers it:

                      Classic RAG                          Search as a tool (agentic RAG)
Who starts the search your code, for every message         the model, only when it decides to
What is searched      the user's message, or one rewrite   a query the model writes; it can split
                                                           a question into two or three searches
Weak results          the model is stuck with them         the model can rewrite the query and
                                                           search again
"Thanks!"             searched anyway, for nothing         no search runs
Cost and latency      low and predictable: one search,     higher and variable: a model call to
                      one model call                       decide, N searches, a final model call

The permission filter belongs inside the function in both cases, keyed on the signed-in user from tool_context, never on anything the model writes (Part 2.5 and the exercise at the end of this part). In short, agentic retrieval is not an alternative to RAG. It is the same retrieval pipeline, triggered by the model instead of by fixed code.

So, is it still relevant?

My view: RAG is not dead, it has become a tool. For enterprise knowledge, retrieval still wins on four things that long context and tuning cannot give you:

• Permissions. A search service can return only what this user may see. A pasted document or a tuned model cannot.
• Freshness. Re-index a changed document and the next answer uses it.
• Cost. Sending 8,000 relevant tokens is cheaper than sending 400,000, call after call.
• Citations. A retrieved passage has an address; Part 5’s rule that every number must come from a tool result works for documents too.

What is fading is the fixed pipeline that retrieves for every question whether it needs to or not. The pattern that is winning is retrieval as a tool the agent calls, with long context for small, stable material.

When a new knowledge need arrives, I work down this list and stop at the first yes:

1. Is it an exact number or record with an owner?     -> a data tool, as Acme's specialists use
                                                         (Parts 2 and 3). Never a vector index.
2. Is it small, stable and readable by every user?    -> put it in the instructions or a skill
                                                         (Part 4), or paste it as long context.
3. Is it large, changing or permissioned text?        -> a retrieval tool the agent calls, with
                                                         the permission filter inside it.
4. Does every single question need the same lookup,   -> only then a fixed classic pipeline,
   and is predictable latency worth more than           which is cheaper and easier to reason
   skipping searches?                                    about per call.

What does Google Cloud offer for retrieval?

The choice is mostly about how much of the clerk’s work you want to own: a packaged RAG service, a search engine you design retrieval on, or search your users already have in Gemini Enterprise. Several services, and the names have been changing, so I give both the current name and the one you may have seen before.

• RAG Engine. A managed RAG service on Agent Platform: you create a corpus, import files (the overview lists sources such as local files, Cloud Storage and Google Drive), and it handles chunking, embedding, indexing and retrieval. By default it uses a managed vector database, RagManagedDb, and you can bring your own instead (vector database choices). Two things to check before you choose it: as of October 2026, the page says data residency is not supported, and some regions need allowlisting. ADK’s documentation now calls it Knowledge Engine.

• Agent Retrieval (formerly Vector Search 2.0). The Agent Retrieval overview describes “a self-tuning, fully managed, AI-native search engine” where you work with collections of data objects rather than managing indexes yourself. Choose it when you want to build your own retrieval, with your own metadata filters, rather than a packaged RAG service.

• Grounding with Google Search. Connects a Gemini model to “world knowledge” from Google Search (grounding with Google Search). It is for public facts, not your company’s documents.

• Grounding with your own search (ADK’s docs call it Agent Search). ADK’s Grounding with Search guide connects an agent to “your private enterprise documents and data repositories” that have been indexed in a search data store, and it notes that you cannot use a Gemini API key from Google AI Studio with it: the agent must authenticate to a Google Cloud project.

• Gemini Enterprise connectors and data stores. If your users already work in Gemini Enterprise (Part 12), its connectors bring data from Google and third-party sources into data stores. They offer data federation, which queries the source directly (no copy, but “search quality might be lower”), or ingestion, which copies the data into an index for better search at the cost of storage and sync time. The page also warns that with federated search “your query string is sent to all enabled third-party search backends”, which your privacy review will want to know.

  For example, suppose Acme kept its retailer contracts in a third-party document store. With federation, the question “late delivery penalty clause for Harvest Row” is sent to that store at question time and nothing is copied, but ranking is the store’s own. With ingestion, the contracts are copied into an index ahead of time, so ranking is better, but a contract changed this morning is only found after the next sync. For contracts that change about once a month, I would ingest and show the sync date next to each citation.

Here is how I choose between them, as a decision guide built only from the points above:

The question is about public facts (news, prices    -> Grounding with Google Search
on the web)
Users already work in Gemini Enterprise and the     -> Gemini Enterprise connectors and data
documents live in Google or third-party apps            stores (ingest for quality, federate to
                                                        avoid copies)
You want a managed corpus over files with the least -> RAG Engine, after checking the data
code, and residency is not a requirement                residency and region notes
Your company documents are already indexed in a     -> grounding with your own search
search data store                                       (VertexAiSearchTool in ADK)
You need your own retrieval design: metadata        -> Agent Retrieval
filters, your own data objects

In ADK, two of these are ready-made tools. VertexAiRagRetrieval queries a RAG Engine corpus, and VertexAiSearchTool queries a search data store. This is the shape of the first, based on ADK’s Knowledge Engine page:

# Illustrative: a policy-retrieval agent over a RAG Engine corpus.
from google.adk.agents import Agent
from google.adk.tools.retrieval.vertex_ai_rag_retrieval import VertexAiRagRetrieval

policy_search = VertexAiRagRetrieval(
    name="search_acme_policies",
    description="Find Acme policy passages relevant to the question. Returns passages with sources.",
    rag_corpora=[os.environ["POLICY_CORPUS"]],   # projects/.../ragCorpora/...
    similarity_top_k=5,
)
policy_agent = Agent(name="policy", model=MODEL, tools=[policy_search],
                     instruction="Answer only from retrieved passages and cite each one.")

One detail from that page matters for design: the retrieval tool “can only be used by itself within an agent instance”. So in a system like Acme’s, retrieval would be its own small agent (or a separate specialist behind A2A), not one more tool on the marketing agent.

When is fine-tuning or training actually needed?

Much less often than people think. The bottom line: retrieval adds facts; tuning changes behaviour. If the agent needs to know something, put it in the context or behind a tool. If it needs to do something differently (a format, a style, one narrow task done cheaply), only then consider changing the weights.

Most teams mistake a knowledge problem for a training problem. The request usually arrives as “tune a model on our documents so it knows our business”. That is a knowledge problem, and tuning is the wrong tool for it.

An everyday example makes the difference clear. Picture a student in an open-book exam and a doctor who has done years of specialist training as a radiologist:

• The student and the doctor are the base model: the same general ability before anything is added.
• The open textbook on the desk is the retrieved passages in the context. Looking a page up is retrieval at question time. Swap in a new edition overnight and the next answer uses it.
• The specialist training is fine-tuning. It changes how the doctor works: how they read an X-ray, the shorthand they write, the shape of every report. It does not tell them who is on this morning’s list, and changing it takes months, not minutes.

At Acme, the retailer penalty rates that supply changes every quarter are the textbook: retrieval. Writing a reason code from a messy delivery note in exactly supply’s format, millions of times, is the specialist training: a candidate for tuning, once cheaper rungs have failed.

Some people compress the options into four verbs: prompt, ground, act, train. They map onto the ladder below: prompt is rung 1, act is rung 2 (skills and tools), ground is rung 3 (retrieval) and train is rungs 4 to 6. The middle two swap places compared with mine, which matters less than the rule they share: you climb only when an eval shows the rung below has failed. Before any of them comes one more question.

Rung 0: can plain code do it?

Rung 0 is the step below the ladder: if a few lines of ordinary code can do the job, no model, prompted or tuned, should. An LLM should never be a glorified if/else or a formatter.

A simple example: checking that a purchase order number has the right shape. Suppose (illustratively) Acme’s format is PO- followed by six digits:

import re

def is_po_number(s: str) -> bool:          # rung 0: plain code
    return re.fullmatch(r"PO-\d{6}", s) is not None

The function is right every time, costs no tokens and answers in microseconds. Ask a model instead, “Is this a valid PO number? Answer yes or no”, and the sums turn against you. At an illustrative 1,000,000 checks a day, a model that is right 99.9% of the time gets 1,000 wrong every day, and you pay for a model call on all million. Tuning the model does not change the shape of that problem; it only moves the error rate. Dates, currency formats, totals, JSON shapes and lookups in a table all belong here.

At Acme, the clearest case is the router. In Part 5 the router’s classify function picks the specialist in code, no model: the same question always goes to the same place, it costs zero tokens to route, and one unit test proves that a penalties question goes to supply. A model choosing the route, even a tuned one, would add a call to every question and could not give that guarantee.

The model earns its place only where the input is messy human language that no rule can parse: “2 cs Verde short, dmg, refused” is a delivery note, not a pattern a regular expression can read reliably.

What are the rungs above code?

I use a ladder: start at the bottom, and climb a rung only when an eval shows that the rung below cannot fix the problem.

the decision ladder for teaching a model something new, from cheapest to most expensive: instructions, skills and tools, retrieval, fine-tuning, continued pre-training and training from scratch; climb only when an eval shows the rung below cannot fix the problem, and note that no rung above retrieval fixes changing facts, permissions or citations

1. Instructions and examples. Most format, tone and rule problems are fixed with a clearer instruction and a few examples. This costs minutes.
2. Skills and tools. If the model needs know-how, package it as a skill that loads when needed (Part 4). If it needs live data or has to take an action, give it a tool (Parts 2 and 3).
3. Retrieval. If it needs to draw on a large or changing body of documents, retrieve. Everything in the previous two sections applies.
4. Fine-tuning. Google’s tuning introduction recommends “starting with prompting to find the optimal prompt”, then moving to fine-tuning “if required”. It says fine-tuning is most effective when you have labelled data (“think 100 examples or more”) and the task is complex or unusual enough that prompting is not sufficient. The benefits it lists include “lower inference latency and cost due to shorter prompts”, because the instructions and examples you no longer have to send each time are now in the weights.
5. Continued pre-training. Only when the model does not understand the language of your domain at all, and you have a very large amount of text from it.
6. Training from scratch. Only if building models is your business.

Here is one climb, worked through. Suppose Acme builds the email-tagging classifier from the design example later in this part. All the numbers are illustrative, not from a run:

Eval      300 incoming emails, each labelled by a person; target 95% correct tags
Rung 1    Flash-Lite, a 1,500-token prompt with the tag list, rules and 5 examples
          -> 88% correct. Better instructions and examples move it to 89%. Fails the target.
Rung 2    Skills and tools: nothing to add. The task needs no live data and no action.
Rung 3    Retrieval: nothing to retrieve. The answer is in the email itself.
Rung 4    Supervised fine-tuning of Flash-Lite on 500 labelled emails
          -> 96% correct on the same 300, with a prompt of about 100 tokens
          -> prompt tokens per email fall from about 1,500 to about 100
Gate      The tuned model goes through the same eval gate as any model change before it ships.

The eval decided each step, not enthusiasm. Rungs 2 and 3 were skipped because they did not apply, not because they were slow, and the climb stopped at rung 4 because the target was met.

The column on the right of the diagram is the part people forget. No amount of tuning will make a model know facts that changed after you tuned it, decide which user may see which fact, or cite a source. Those are properties of retrieval and tools. Tuning changes behaviour: a format, a style, a narrow task done well and cheaply.

A before and after makes the difference plain. Suppose Acme had tuned a model in the second quarter on its policy documents, including the retailer penalty rates (P2 in the exercise), and supply then changed those rates in the third quarter:

Tuned model, asked in the third quarter by a marketing user
  "The penalty rate for late deliveries is ..."   <- the second-quarter rate, stated confidently,
                                                     no source, and shown to marketing
Retrieval, same question, same user
  marketing: "nothing I may show"                 <- P2 is filtered out before ranking
Retrieval, same question, a supply user
  "Here are the current rates [P2, updated in Q3]" <- re-indexed the day the document changed

The tuned model got all three properties wrong at once: freshness, permissions and citation. Retrieval got them right because code filtered by role and the index followed the document.

My view: if a partner tells me they want to fine-tune “so the model knows our documents”, I suggest retrieval first. Fine-tuning on documents bakes in a copy that starts going out of date on the day the job finishes, and it takes away the one thing an enterprise agent cannot lose: control over who sees what.

Why does tuning fail on facts but win on a narrow task at volume?

Both sides deserve a scenario, because “never tune” is as wrong as “always tune”.

Why it fails: tuning on facts that change. The penalty-rate before and after above is the whole story. The tuned model was wrong on freshness, permissions and citation at once, and the next rate change would make it wrong again. Facts belong in retrieval.

Why it wins: distilling a narrow, high-volume task. Distillation means a large model (the teacher) does a task well, its answers become training examples, and a small model (the student) is tuned on them until it can do that one task alone. The everyday version: a senior colleague marks 5,000 cases with the 40-page manual open, and a new hire studies those marked cases until they can do that one job without the senior and without the manual. The senior is the teacher model, the manual is the long prompt, the new hire is the small model, and studying is tuning.

Here is the mechanism on a hypothetical Acme task (not something the series built). Suppose Acme’s warehouses send 200,000 free-text delivery exception notes a day, such as “2 cs Verde short, dmg, refused”, and supply wants each one tagged with one of 40 reason codes before it lands in supply_service. Rung 0 fails: the notes are too messy for rules. All numbers are illustrative, not from a run:

Eval     400 notes labelled by supply planners; target 95% correct codes
Step 1   Teacher: a large Gemini model with a 6,000-token prompt (40 codes, rules, 25 examples)
         -> 97% correct. About 6,300 input tokens per note.
         200,000 notes x 6,300 tokens = 1.26 billion input tokens a day
Step 2   Student: Flash-Lite with the same prompt -> 82%. Fails the target.
Step 3   The teacher labels 5,000 past notes. Planners check a sample, plus every note
         where two teacher runs disagree. These 5,000 become the training set.
Step 4   Supervised fine-tuning of Flash-Lite on the 5,000, prompt cut to about 150 tokens
         -> 96% on the same 400 notes. About 450 input tokens per note.
         200,000 x 450 = 90 million input tokens a day: 14 times fewer, on a cheaper model
Gate     The tuned student passes the same eval gate as any model change, and is re-checked
         every month on fresh notes.

The 40 codes, the rules and the 25 examples did not disappear. They moved from the prompt, where you pay for them on every call, into the weights, where you paid for them once. Training is billed per training token, which the SFT page defines as the tokens in your dataset multiplied by the number of epochs: 5,000 examples of about 450 tokens over an illustrative 3 epochs is about 6.75 million training tokens, once.

What Google actually offers for this, as of November 2026:

• For Gemini, distillation is something you assemble. I found no managed distillation job for Gemini models in the Agent Platform docs. You run the teacher yourself to label examples, then use supervised fine-tuning on a supported student: Gemini 3.5 Flash-Lite, Gemini 3.5 Flash or Gemini 3.1 Flash-Lite. Google’s introduction to tuning says SFT is best suited for “strict output formatting, classification, sentiment analysis, entity extraction, and domain-specific style or tone”, which is exactly this kind of task.
• For open models, distillation is a managed job. The open-model tuning page describes “distillation fine-tuning”, where “you specify a teacher model to generate responses that are then used to tune a smaller student model”. It lists GLM 5.2 (as a managed API) as the teacher and Gemma 3 (1B to 27B) and Qwen 3 (4B to 32B) as students. It says distillation is “most effective when the teacher model is substantially more capable than the student”, and gives “smaller gains” where the student is already close to the teacher. You then serve the student yourself, which puts you in the right-hand column of the infrastructure diagram below.
• Check the serving price. Google’s pricing page says a tuned Gemini model endpoint costs 1.5 times the base model in a non-global region, and the same as the base model in the global region. At 14 times fewer tokens the distilled student still wins, but put the right multiplier in your sums.

one narrow task run three ways, with illustrative numbers: plain code needs no tokens and always gives the same answer but only handles fixed rules; a prompted large model reads messy text but sends a long prompt on every call; a small model tuned on the large model's answers sends a short prompt but must be tuned again when its base model retires; use the leftmost column that passes the eval

My view: distillation is the strongest case for tuning I see with partners, and it only works when three things are true: the task is narrow, it is stable, and the volume is high enough for prompt tokens to dominate the bill. Take away any one of them and a prompted model is the better deal.

When does prompting fail even with a capable model?

The second honest case is behaviour the model never saw: a proprietary language or a strict house style. Here the problem is not cost but habit.

An everyday example: a touch typist moving from a QWERTY keyboard to a different layout. With a cheat sheet taped to the screen they can type the new layout slowly, but under pressure their fingers go back to QWERTY. The QWERTY habit is the model’s pre-training, which has seen huge amounts of SQL and Python. The cheat sheet is the grammar in the prompt. Practice is tuning: it changes the reflex, not the instructions.

A hypothetical Acme version (Acme has no such language; suppose it did). Supply planners write replenishment alerts in an in-house rule language and want to type English instead:

Request    "Flag Verde in Western Europe if OTIF stays below 92% for two weeks"
Wanted     WHEN otif(brand="Verde", region="WEU") < 0.92 FOR 2w THEN flag("supply_review")
Prompted   SELECT * FROM otif WHERE brand = 'Verde' AND otif < 0.92 ...   <- SQL habits leak

The climb, with illustrative numbers:

Rung 0   The language has a parser. Use it in code: it checks every answer, and it is the eval
         (does the rule parse, and does it match the expected rule?). It cannot write rules.
Rung 1-2 The grammar and 30 worked examples as a skill (Part 4) -> 71% pass. Fails a 95% target.
Rung 4   Supervised fine-tuning on 3,000 English-to-rule pairs -> 94%, then more pairs -> 96%
Always   Keep the parser check after tuning: a tuned model is still a model, and a rule that
         does not parse never reaches the scheduler.

Google’s introduction to tuning names this case among the benefits of tuning over prompting: to “align the model with custom syntax, instructions, domain specific semantic rules”. It also describes reinforcement learning fine-tuning (RLFT), suited to “verifiable tasks (such as code or SQL generation)”, where reward functions score the model’s attempts. A parser that says “valid” or “invalid” is a natural reward. As of November 2026 the RLFT page lists only Gemini 3.5 Flash as supported.

The same logic applies to a strict house style, such as a publisher whose editors reject drafts that sound generic however many adjectives the prompt contains. Tuning on a few thousand published pieces teaches the style the prompt could only describe. Try a skill with real examples first; with a long context window, that rung goes further than it used to.

Are we talking about LoRA when we say fine-tuning?

Usually, yes. The bottom line: LoRA (low-rank adaptation) freezes the base model’s weights and, for each layer it adapts, trains two small matrices whose product is added to the frozen weight. It is the best-known way of doing the parameter-efficient tuning defined at the top of this part, and it is still fine-tuning: it changes how the model behaves, not what it knows.

The trap here is a tempting one. Because LoRA is cheap, people start to treat it as a cheap way to add knowledge: “we can’t afford to fine-tune on our documents, but we can LoRA them”. A LoRA adapter is trained exactly like any other fine-tune, on examples, by nudging numbers until the outputs match. Everything the ladder above says about facts that change, permissions and citations applies to it unchanged. Cheaper training does not turn a behaviour tool into a knowledge tool.

What does LoRA look like in everyday terms?

Picture a good camera with a fixed lens, and a small clip-on filter lens:

• The camera’s built-in lens is the base model’s weights. You do not regrind it. In LoRA terms the weights are frozen: tuning never changes them.
• The clip-on lens is the adapter: two small matrices per adapted layer. It is light, cheap to make, and it changes the look of every photo the camera takes.
• How much optical work the clip-on can do is the rank. A thin filter can only shift the colour; a thicker lens can reshape the picture more. A higher rank means more trainable numbers and more capacity to change behaviour.
• A dial on the clip-on that sets how strongly it bends the light is alpha, the scale applied to the update.
• Gluing the clip-on permanently onto the lens is merging the adapter into the weights. Keeping a pouch of clip-ons and swapping them per shoot is serving many adapters on one base.
• A clip-on made for one camera body does not fit the next model. That is the obsolescence trap below, in one picture.

And the important part: a lens changes how the camera sees, never what is in front of it. A LoRA adapter changes the model’s habits; today’s penalty rates still have to be put in front of it.

How does the mechanism work, with small numbers?

A layer in a Transformer is mostly a big grid of numbers, a matrix, called W. The layer multiplies its input by W. Full fine-tuning changes every number in W. LoRA leaves W alone and learns a change to it, written as the product of two thin matrices, B times A.

Start tiny, so the arithmetic is visible. Suppose W is 4 x 4 (16 numbers) and the rank is 1. Then B is 4 x 1 and A is 1 x 4, eight numbers in total:

B = [1, 0, 2, 0] (a column)     A = [0.5, 0, -0.5, 0] (a row)

B x A =  0.5   0  -0.5   0       <- row 1 is A times 1
         0     0   0     0       <- row 2 is A times 0
         1.0   0  -1.0   0       <- row 3 is A times 2
         0     0   0     0       <- row 4 is A times 0

8 trained numbers produce a full 16-number update for W.

Every row of the update is a multiple of the same row A. That is what low rank means: the update is built from very few independent patterns (here, one). The bet behind LoRA, which the LoRA paper (Hu and colleagues, 2021) tests on several models, is that the change a task needs is like this: it has few real directions, even though W is huge.

Now the real size. Take, for illustration, a layer weight of 4096 x 4096, a common size in mid-sized open models:

Full fine-tuning of one layer     4096 x 4096                  = 16,777,216 trainable numbers
LoRA, rank 8                      B: 4096 x 8  +  A: 8 x 4096  =     65,536 trainable numbers
                                  65,536 / 16,777,216          = about 0.39%
Rank 16                           2 x 4096 x 16                =    131,072  (about 0.78%)
Rank 1                            2 x 4096 x 1                 =      8,192  (about 0.05%)

The saving grows with the size of the matrix, because full tuning costs d x d while LoRA costs 2 x d x r. The paper’s headline is the same arithmetic at the largest scale it tried: compared with fully fine-tuning GPT-3 175B, LoRA “can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times”, and it reports quality “on-par or better than fine-tuning” on the models it tested. Memory falls because the optimiser only has to track gradients for the small matrices, not for the frozen billions.

one LoRA-adapted layer with the numbers from the text: the 4096 x 4096 weight W stays frozen at about 16.8 million numbers, while two thin trainable matrices B (4096 x 8) and A (8 x 4096) hold 65,536 numbers, about 0.4%; the input goes through both paths and the outputs are added, and at serving time the update is either merged into W or kept as a small adapter you can swap on one shared base

Three details from the paper make the diagram work:

1. The update starts at zero. A starts with random numbers and B starts at zero, so B x A is zero on the first step and the model begins exactly as the base model. Training then moves only A and B.
2. Rank and alpha. The update is scaled by alpha / r before it is added. With rank r = 8 and an alpha of 16, every update is multiplied by 2. The paper sets alpha to the first rank it tries and does not tune it, because with its optimiser changing alpha is “roughly the same as tuning the learning rate”. The practical knob is the rank: start small, raise it only if the eval says the adapter lacks capacity.
3. Merge or swap at serving time. Because W and B x A have the same shape, you can add them once and serve an ordinary weight: the paper calls this “no additional inference latency”. Or you keep them apart and swap adapters on one base. With rank 4 on two attention matrices in GPT-3 175B, the paper’s checkpoint shrinks “from 350GB to 35MB”, so 100 task adapters need about 354GB instead of 35TB.

What is QLoRA? QLoRA (Dettmers and colleagues, 2023) goes one step further on memory: it keeps the frozen base in a 4-bit quantised form and trains LoRA adapters on top. The paper says this “reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance”. It matters when you tune an open model on your own GPUs. On a managed tuning job, memory is Google’s problem, and the docs I read do not say which tricks the service uses inside.

When is full fine-tuning the better choice? Google’s open-model tuning page puts the trade in one sentence each way: LoRA is “more cost efficient and require less training data than full fine-tuning”, while “full fine-tuning has higher quality potential by adjusting all parameters”. The introduction to tuning adds that full fine-tuning “demands higher computational resources for both tuning and serving”, and that parameter-efficient tuning “offers a solution for multi-task learning without the need for extensive retraining”. My rule of thumb, cheapest first:

Situation                                              Choose            Why
Narrow task, hundreds to a few thousand examples       LoRA, low rank    small data, cheap, one base for many tasks
Several tasks on one base model                        LoRA adapters     swap per task instead of hosting copies
The eval plateaus at the largest adapter size,         full fine-tuning  more capacity, if the extra data and
and you have a large, clean dataset                                      serving cost are worth it
A small open model that only offers full tuning        full fine-tuning  the platform decides (see below)
The model must know new or changing facts              neither           retrieval (rung 3)

How does Google Cloud expose it? As of November 2026, worded exactly as the docs word it:

• Gemini supervised fine-tuning has an adapter size setting. The supervised fine-tuning page lists “Supported values are 1, 2, 4, 8, and 16” for Gemini 3.5 Flash, 3.5 Flash-Lite and 3.1 Flash-Lite, and calls the output of tuning “Tuned model artifacts (adapters)”. The how-to page says “The adapter size influences the number of trainable parameters for the tuning job. A larger adapter size implies that the model can learn more complex tasks, but it requires a larger training dataset and longer training times.” The SFT pages do not use the word LoRA.
• Gemini reinforcement learning fine-tuning does name it. The introduction says RLFT “updates a parameter-efficient LoRA adapter to maximize the reward”, and the RLFT hyperparameters page describes adapterSize as “The LoRA adapter size”, defaulting to ADAPTER_SIZE_SIXTEEN. My reading is that adapter size plays the role of the LoRA rank on the Gemini side too, but for SFT that is my inference, not something the page states.
• Open models offer both modes by name. The open-model page lists “Full fine-tuning” and “Low-Rank Adaptation (LoRA)” as supervised tuning modes, and its SDK example sets tuning_mode = "FULL" # FULL or PEFT_ADAPTER. Support varies by model: the Gemma 4 models list only parameter-efficient fine-tuning, Gemma 3 27B and Llama 3.1 8B list both, and Gemma 3 1B, 4B and 12B list only full fine-tuning.

How does this apply to Acme? At Acme, this is how it would work, using the two tuning examples from earlier in this part. Both are rung 4, on Flash-Lite, and both produce an adapter, not a new copy of the model:

                     Email tagger (rung 4 above)         Delivery-note coder (distillation)
Base model           Flash-Lite, frozen                  Flash-Lite, frozen
Training examples    500 labelled emails                 5,000 teacher-labelled notes
Adapter size         start small, e.g. 4                 start at 4 or 8; try 16 only if the
                     (small dataset)                     eval stalls below 95%
Eval target          95% on 300 emails (reached 96%)     95% on 400 notes (reached 96%)
What it learned      the tag list and rules as a habit   the 40 codes and rules as a habit
What it did not      today's penalty rates, who may      today's penalty rates and who may
learn                see them, where they came from      see them (still retrieval and tools)

The adapter sizes are illustrative starting points that follow the docs’ trade-off (a larger adapter wants more data); the accuracy figures are the same illustrative numbers used above. The lesson is in the last row. If Acme had LoRA-tuned on the second-quarter penalty rates instead of fully fine-tuning, the third-quarter answer would be just as stale and just as unfiltered for Maria in marketing. LoRA would only have made the same mistake cheaper to train.

To lock it in:

Term                What it is                                         In the camera picture
LoRA                frozen weights + a trained low-rank update B x A    a clip-on lens
Rank (r)            how many patterns the update can hold              how thick the clip-on is
Alpha               scale applied to the update (alpha / r)            the strength dial
Merge               add B x A into W once for serving                  glue the clip-on on
Adapter swapping    one base, many small task adapters                 a pouch of clip-ons
QLoRA               LoRA on a 4-bit quantised frozen base              a lighter camera body
Full fine-tuning    every weight is trainable                          regrinding the lens itself

What is the obsolescence trap?

Tuned weights belong to one base model version. A tuned model is the base model plus the parameters the tuning job learned; Google’s introduction says the output “is a new model that combines the newly learned parameters with the original model”. Those learned parameters fit that base and no other, like a lens ground for one camera body.

Google’s migration guide is direct about what that means when you move to a newer Gemini: “you cannot move your existing tuned model from older Gemini versions. You need to run a new tuning job for the new Gemini version.” It also says not to reuse the old hyperparameters, and to “first test your application with the latest model without fine-tuning”, because a newer model may not need the tuning at all. The model versions page gives every model a retirement date and talks about the errors you get when a model reaches it. My reading of the two together: plan for a tuned model to have the same end date as its base.

The numbers are short. Gemini 2.5 Flash-Lite was released on 22 July 2025 and retired on 20 October 2026. A model tuned on it in January 2026 had about nine months of life. As of November 2026, Gemini 3.5 Flash-Lite shows a retirement date of “July 21, 2027 or later”, and the page says retirement dates may be extended but will not be moved earlier.

Compare what each rung has to do when its base model retires:

When the base model retires    What you do
Code (rung 0)                  nothing
Prompts, skills, tools,        re-run the eval on the new model and fix the prompts
retrieval (rungs 1 to 3)
Tuned model (rung 4)           check whether the new base passes untuned; if not, a new tuning
                               job with fresh hyperparameters, the full eval, and a rollout

The lasting asset is not the tuned model. It is the labelled dataset and the eval set: those carry over to the next base model; the weights do not. This is the same lesson as Part 13’s model lifecycle, with a heavier bill.

What is the litmus test?

When someone says “we want to fine-tune”, I walk through three questions, cheapest first:

1. Can plain code do it? A regular expression, a parser, a SQL query, a lookup. If yes, write the code or wrap it as a tool (Parts 2 and 3). Stop.
2. Does the task need messy language, and does a prompted model with retrieval and tools pass the eval? If yes, stop at prompt, ground and act. You are done.
3. Does the prompted model pass but cost too much at your volume, or keep failing on a language or style it never saw, and is the task narrow and stable? Only then tune, with a dataset you keep, an eval gate, and a date to tune again.

a decision guide from cheapest to most expensive: routing and format checks go to code; changing policies go to retrieval; live numbers go to data tools; tone and format go to instructions and skills; a narrow task at very high volume is a candidate for a distilled small model; an in-house language starts with a skill and a parser and is tuned only if the eval fails; making a model know your business is retrieval, not tuning

What infrastructure does each rung need?

Each rung up the ladder adds something you have to run: retrieval adds a data pipeline, managed tuning adds datasets and a model to retune, and self-managed training adds hardware and a team on call. The lower rungs live in your agent’s code. The upper rungs need infrastructure, and a team to run it. The trap is costing only the first build; the running cost is what decides it, which is the point Part 13.5 makes about cost of ownership.

Getting around a city is a fair picture. Taking a taxi needs nothing from you but the fare. Keeping your own bicycle means a lock, a pump and the odd repair. Leasing a car adds insurance and a service plan, and when the model is discontinued you lease a new one. Running your own fleet adds a depot, mechanics and someone on call at night. Mapped to the rungs: prompting a hosted model is the taxi, retrieval is the bicycle (a pipeline to keep in good order), managed tuning is the leased car (a dataset and a tuned model you redo when the base model retires), and self-managed training or serving is the fleet.

what each option needs to run: retrieval needs a document pipeline, an embedding model, an index or search service and an eval set; managed tuning adds a labelled dataset, tuning jobs and a tuned model endpoint; self-managed training adds accelerators, training clusters, serving and a team to run them

Retrieval needs a pipeline more than a model. You need connectors that keep the index in sync with the source, chunking that respects the structure of your documents (a table cut in half is useless), an embedding model, an index or search service, a way to filter results by the user’s permissions, and a retrieval eval set: questions with the passages that should come back. Part 10’s golden set idea applies directly.

How does this apply to Acme? Two cases from such a set, written for the five policy snippets in the exercise below, show the idea. The second is the same question from a user who may not see the answer:

{"question": "What are the penalty rates for late deliveries?", "role": "supply",    "expected_passages": ["P2", "P5"]}
{"question": "What are the penalty rates for late deliveries?", "role": "marketing", "expected_passages": []}

A retriever that returns P2 to the marketing user fails the second case, even if its answer reads well.

Managed tuning on Agent Platform adds a labelled dataset, tuning jobs and a tuned model endpoint. As of October 2026, Google’s supervised fine-tuning page lists Gemini 3.5 Flash, Gemini 3.5 Flash-Lite and Gemini 3.1 Flash-Lite among the supported models, and recommends setting the thinking level to minimal for tuned Gemini 3 models, because “the tuned model can perform tuned tasks effectively with little to no thinking budget”. The model pages from Part 13 list tuning as not supported for Gemini 3.8 Flash and Gemini 4 Argon. The preference tuning page lists only Gemini 2.5 Flash and Flash-Lite, which the model versions page shows as retiring on 20 October 2026, so check it again before you plan preference tuning. Google also offers continuous tuning, which “lets you continue tuning an already tuned model or model checkpoint”, for some models. Two operational points follow. A tuned model is built on one specific base model, so I would plan to tune and evaluate again when that base model retires (Part 13). And the SFT page notes that supervised fine-tuning “is not a Covered Service and is excluded from the SLO of any Service Level Agreement”.

Self-managed training or serving adds accelerators (GPUs on Compute Engine, or TPUs), clusters to run on, a serving stack such as vLLM on GKE (Part 13), and monitoring, patching and capacity planning. This is where the team cost appears: someone has to be on call for it.

One honest gap: the series has not tuned a model. The facts above come from Google’s documentation, and the judgements from my experience with partners, not from an Acme experiment I can show you. If you do tune, put the tuned model through exactly the same eval gate as any other model change.

What would Acme do?

Almost all of Acme’s knowledge belongs in tools and context; only one narrow, high-volume task is even a candidate for tuning. The series has not built any of this, so treat it as a design example. If Acme wanted its agents to know more, this is where I would put each kind of knowledge.

a design example for Acme, not built in the series: product facts and policies go behind a retrieval tool, marketing and supply numbers stay behind their data tools, tone and format stay in instructions and skills, and a very high volume classifier is a candidate for a small tuned model

• Product facts and policies (promotion rules, approval thresholds, delivery terms) go behind a retrieval tool. They are long, change every month or so, and an answer should cite the policy it relies on. Following the ADK limitation above, this would be a small policy agent the router can call, with results filtered by the user’s role.
• Spend, OTIF and penalties stay behind the data tools, exactly as today. They are exact numbers with owners and access rules. Putting them in a vector index would copy them out of the systems that control who may see them, which is the opposite of the design rule from Part 1.
• Tone and answer format stay in instructions and skills. They are small and stable, and editing a skill is far cheaper than retraining anything.
• A classifier that tags millions of incoming emails a day is the one candidate for a small tuned model: a narrow, stable task at very high volume, where shorter prompts and a smaller model could pay for the tuning. Even then, I would first measure a prompted Flash-Lite model on the same eval.

To lock it in, here is the whole part on one card, with the analogies used along the way:

Home or option       What it is                                Analogy
The weights          what the model learned in training        the doctor's medical school
The context          what you send with each call              the patient's file on the desk
Behind a tool        fetched from the system that owns it      ringing the lab for a result
Long context         paste all the material into the prompt    the whole 400-page binder
Classic RAG          code retrieves for every question         the librarian's fixed routine
Agentic retrieval    the model decides when and what to search the researcher with a library card
Fine-tuning          change the weights to change behaviour    the specialist training

What can go wrong

• A stale index. The source changes but the index does not, and the agent quotes last quarter’s policy. Monitor sync, and show the document date with each citation.
• A permission leak through retrieval. The index returns a passage the user may not see, and the model summarises it. Filter by permission before results reach the model, as the exercise below shows.
• Chunking that breaks structure. A price table or a numbered procedure is split across chunks, and neither half makes sense alone. For example, a three-row penalty-rate table is cut after its first row: chunk 1 holds the header and one row, and chunk 2 holds two rows of bare numbers with no column names, so a question about the second row retrieves numbers nobody can read. The fix is to keep each table in one chunk.
• A citation to the wrong passage. The answer is right but points to a different chunk, which destroys trust when someone checks. Evaluate citations, not just answers. For example, the answer “Disputes must be raised within 30 days [P2]” is correct, but that sentence comes from P5. A citation check that looks up the cited passage and tests whether it contains the claim fails this answer.
• Long context everywhere. Every call carries the whole manual, and cost and latency climb. Measure tokens per question (Part 11).
• A model doing a job code should do. Routing, format checks and lookups sent to a model cost tokens on every call and are sometimes wrong. Run the rung 0 question first.
• Tuning on facts that change. The tuned model answers confidently with last year’s numbers.
• A tuned model left behind by its base. The base model it was built on reaches its retirement date. Put that date in the plan, with time to tune and evaluate again.

Try it: why permissions must come before ranking

This needs nothing but Python. It searches five Acme policy snippets for the question “What are the penalty rates for late deliveries?”, once with an unsafe retriever that ranks everything, and once with a safe one that filters by the user’s role first.

# try_retrieval.py: retrieve-then-answer over five Acme policy snippets, with permissions.
# Pure Python, no model and no cloud. The point: filter by permission BEFORE ranking.
import re

DOCS = [
    {"id": "P1", "roles": {"marketing", "supply"}, "text": "Promotions in Western Europe need supply sign-off two weeks before launch."},
    {"id": "P2", "roles": {"supply"}, "text": "Retailer penalty rates for late or short deliveries are confidential to supply."},
    {"id": "P3", "roles": {"marketing", "supply"}, "text": "OTIF is the share of orders delivered on time and in full."},
    {"id": "P4", "roles": {"marketing"}, "text": "Campaign budgets above EUR 250,000 need finance approval before launch."},
    {"id": "P5", "roles": {"supply"}, "text": "Penalty disputes must be raised within 30 days of the delivery."},
]

STOP = {"what", "are", "the", "for", "of", "is", "and", "in", "a", "to", "be", "must", "need"}

def words(s):
    return set(re.findall(r"[a-z0-9]+", s.lower())) - STOP

def rank(question, docs, k=2):
    q = words(question)
    scored = sorted(docs, key=lambda d: len(q & words(d["text"])), reverse=True)
    return [d for d in scored[:k] if q & words(d["text"])]

def retrieve_unsafe(question, role):
    return rank(question, DOCS)                                  # ranks everything: leaks

def retrieve(question, role):
    allowed = [d for d in DOCS if role in d["roles"]]            # 1. permissions first
    return rank(question, allowed)                               # 2. then rank

question = "What are the penalty rates for late deliveries?"
for role in ("marketing", "supply"):
    print(f"{role:9} unsafe -> {[d['id'] for d in retrieve_unsafe(question, role)]}")
    print(f"{role:9} safe   -> {[d['id'] for d in retrieve(question, role)] or 'nothing I may show'}")

Running it prints:

marketing unsafe -> ['P2', 'P5']
marketing safe   -> nothing I may show
supply    unsafe -> ['P2', 'P5']
supply    safe   -> ['P2', 'P5']

The unsafe version hands the confidential penalty rates to a marketing user, because they are the best match. The safe version returns nothing for marketing and the right passages for supply. Real search services do the same with access control lists or metadata filters, and the rule is the same: the model must never see a passage the user could not have opened themselves.

What code did we write in this part?

Every part ends with the same map of Acme’s build: which files exist so far, and the code this part added, explained line by line. The map grows with each part.

This part adds no code to Acme’s build. It explains where an agent’s knowledge should live; the VertexAiRagRetrieval policy agent is marked illustrative and is a design example the series has not built, and try_retrieval.py is a standalone exercise that shows why permissions must be applied before ranking.

Where does it live?

router/                                             the router agent: extract, route, write
  app/agent.py                                      P5,8        workflow; extract reads memory; App with compaction and resumability
  app/app_utils/services.py                         P8          picks the session, artifact and memory services
  app/fast_api_app.py                               P8          Runner with session, artifact and memory services
  app/remote.py                                     P6,7        A2A client over mTLS; metadata carries Sam's token
  app/writer.py                                     P5,8        writer reads memories and saves the conversation to Memory Bank
  scripts/create_memory_bank.py                     P8          creates the Memory Bank and its topics, once
  tests/eval/acb_metrics.py                         P10         code metrics: route_correct, grounded, golden_numbers
  tests/eval/build_datasets.py                      P10         splits source cases into golden and held-out, seed 42
  tests/eval/calibrate.py                           P10         compares judge scores with human labels (kappa)
  tests/eval/datasets/golden_marketing.json         P10         golden cases with expected route, target and numbers
  tests/eval/eval_config.yaml                       P10         eval metrics, including the faithfulness judge
  tests/eval/gate.py                                P10         thresholds per metric; exits 1 when one fails
  tests/unit/test_router.py                         P10         unit tests; routing checked with one row per question type
marketing/                                          the marketing specialist, on BigQuery
  app/agent.py                                      P1,2,3,4    marketing agent with its skill in a SkillToolset
  app/app_utils/a2a.py                              P6          A2A endpoint and agent card (1.0 and 0.3) for marketing
  app/skills/marketing-kpis/SKILL.md                P4          marketing know-how: which tool, definitions, story checks
  app/skills/marketing-kpis/references/tables.md    P4          tables, grain, measures and the join rule, read on demand
  app/tools.py                                      P1,2        three narrow tools: measures, unserved demand, SQL fallback
  tests/unit/test_marketing.py                      P2          unit tests for the tools, with a fake context and database
supply/                                             the supply specialist, on Databricks
  app/agent.py                                      P3,7,8,9    supply agent and App with the confirmed replan tool
  app/app_utils/a2a.py                              P6          A2A endpoint and agent card (1.0 and 0.3) for supply
  app/skills/supply-catalog/SKILL.md                P4          supply know-how, given only to the supply agent
  app/tools.py                                      P3,7,9      supply tools; the replan tool asks a human first
shared/                                             code all three services share
  adk_common.py                                     P3,6,9,11   plugins and redacted JSON logs, now with the usage ledger
  auth_manager.py                                   P7          auth manager callback: token or consent (off by default)
  engines.py                                        P3,7        run_databricks connects as the signed-in user
  factcheck.py                                      P5          ungrounded_numbers: finds figures not in the findings
  registry.py                                       P2          reads semantic.yaml: measure names, computable measures
  routing.py                                        P1          classify: picks the specialist in code, no model
  safety.py                                         P9          per-user token bucket and injection pattern scan
  semantic.yaml                                     P1,2        registry of measures, owners and published formulas
  skill_source.py                                   P4          loads a skill from disk or a pinned registry revision
  sqlguard.py                                       P3          guard_sql read-only allow-list; redact for log lines
  tests/test_cost_report.py                         P11         unit tests for the cost maths
ops/                                                build, deploy and evaluation tooling
  .cloudbuild/eval_gate.yaml                        P10         after staging: eval run, then gate.py
  .cloudbuild/pr_checks.yaml                        P10,11      offline PR checks, cheapest first, ending with the mutants
  env.sh.example                                    P11         settings for every engine: ids, models, plugin switches, OTEL; no secrets
  infra/gateway/setup_gateway.sh                    P12         egress gateway, endpoint registry and IAP rules, in dry-run
  tools/cost_report.py                              P11         prices the ledger and reports cost per resolved question
  tools/mutants/acb_mutate.py                       P10         pytest plugin that injects each mutant's fault
  tools/pricing.json                                P11         list prices per model, kept out of the code
  tools/run_mutants.py                              P10         runs the offline suites once per mutant

+ added in this part   ~ changed in this part   Pn = the parts that touched the file

No retrieval agent and no tuned model exist in Acme’s build; both are design examples.

What did we learn?

• A model never searches: code ingests, chunks, embeds and indexes ahead of time, then filters by permission, runs hybrid search, reranks and puts the best chunks in the prompt. Agentic retrieval only changes who starts the search.
• An agent’s knowledge can live in the weights, in the context or behind a tool. Enterprise data, which changes, has owners and has access rules, belongs in the context or behind a tool.
• RAG is still relevant. It has turned from a fixed pipeline into a tool the agent calls, and it still wins on permissions, freshness, cost and citations. Long context, with caching, is the simpler choice for small, stable material.
• Google Cloud offers RAG Engine, Agent Retrieval, grounding with Google Search, grounding with your own search data stores, and Gemini Enterprise connectors. ADK has ready-made tools for the first and the fourth.
• Climb the ladder from instructions to tools to retrieval before considering fine-tuning, and only when an eval shows the lower rung has failed. Tuning changes behaviour; it does not add fresh facts, permissions or citations.
• Retrieval adds facts; tuning changes behaviour. An open textbook versus specialist training: tuning changes how the model works, never what it knows today.
• Rung 0 comes first: if plain code can do it, no model should. Acme’s router picks the route in code.
• Tuning fails on facts that change and wins when distilling a narrow, stable, high-volume task from a large teacher into a small student. For Gemini you assemble it with supervised fine-tuning; for some open models it is a managed job.
• Prompting can also fail on a language or house style the model never saw. Keep a parser or checker in code even after tuning.
• LoRA is the usual form of parameter-efficient tuning: the base weights stay frozen and two small matrices per layer learn the change (65,536 numbers instead of 16.8 million for a 4096 x 4096 weight at rank 8). It is cheaper fine-tuning, not a way to add knowledge; facts still belong in retrieval.
• Tuned weights belong to one base model version: they cannot be moved, so plan to tune again, and keep the dataset and eval set as the real asset.
• The litmus test: can code do it? Does prompt, ground and act pass the eval? Only then tune.
• Each rung up needs more infrastructure: retrieval needs a pipeline and an eval set, tuning adds datasets, jobs and re-tuning when the base retires, and self-managed training adds accelerators and a team.

What comes next?

In Production-ready agents, Part 15: What breaks in production: answers, loops, cost and model changes we start the field guide: the failures you will actually meet once the system is live, how to confirm each one from traces, logs and evals, and how to fix it, with a link back to the part that explains the mechanism.

Sources

• Anthropic: Contextual Retrieval
• About hybrid search (Vector Search)
• Reranking for RAG Engine
• Google Cloud: RAG Engine on Gemini Enterprise Agent Platform overview
• Google Cloud: Vector database choices in RAG Engine
• Google Cloud: Agent Retrieval (formerly Vector Search 2.0)
• Google Cloud: Grounding overview
• Google Cloud: Grounding with Google Search
• Google Cloud: Long context
• Google Cloud: Context caching overview
• Google Cloud: Gemini Embedding 2
• Google Cloud: Introduction to tuning
• Google Cloud: About supervised fine-tuning for Gemini models
• Google Cloud: About reinforcement learning fine-tuning for Gemini models
• Google Cloud: Tune Gemini models with supervised fine-tuning
• Google Cloud: Hyperparameters for reinforcement learning fine-tuning
• Hu et al. (2021): LoRA: Low-Rank Adaptation of Large Language Models
• Dettmers et al. (2023): QLoRA: Efficient Finetuning of Quantized LLMs
• Google Cloud: Supervised and distillation fine-tuning for open models
• Google Cloud: Migrate to the latest Gemini models
• Google Cloud: Gemini Enterprise Agent Platform pricing (model tuning)
• Google Cloud: About preference tuning for Gemini models
• Google Cloud: Model versions and lifecycle
• Google Cloud: Introduction to connectors and data stores (Gemini Enterprise)
• Google Cloud: Introduction to Cloud TPU
• Google Cloud: GPU machine types
• Google AI for Developers: Gemini Developer API pricing
• ADK: Grounding agents with data
• ADK: Grounding with Search
• ADK: Knowledge Engine tool

Opinions are my own. Every fact in this post comes from the public documentation linked above.