Sheet 02 / 05 — Blogs← Archive

Production-ready agents, Part 13.75: Preparing data for agents: structured and unstructured

An agent is only as good as the data it reads, and tables and documents need different preparation. This part covers both, from the ground up: curating, describing, defining and checking tables so an agent computes the right number, and inventorying, parsing, cleaning, chunking and enriching documents so retrieval finds the right passage. It then covers data that never stops arriving, from the traditional machine learning side: event time, windows, watermarks, feature stores and training-serving skew. Each step has an Acme example, the Google Cloud and Databricks features that help, and a way to test that the preparation worked.

7 November 2026· 30 min read · ~43 min listen· Agents · Data preparation · RAG · Semantic layer · Data quality · Streaming · Feature store

Most of this series is about the agent: its tools, its identity, its guardrails. But an agent that reads badly prepared data gives confident wrong answers, and no prompt fixes that. Gartner predicted in February 2025 that through 2026, organisations will abandon 60% of AI projects that are not supported by AI-ready data. Prerequisite 0 listed “data that is ready” as one of nine things to have in place before building. This part shows what that preparation actually involves.

Agents read two kinds of data, and they need different preparation:

Structured data lives in tables: Acme’s campaign_performance in BigQuery and supply_service in Databricks. Agents compute numbers from it with tools (Parts 2 and 3).
Unstructured data lives in documents: Acme’s retailer contracts and policies. Agents find passages in it with retrieval (Part 14).

Some data also never stops arriving, such as till scans and shipment confirmations. The last preparation section covers it from the traditional machine learning side: streaming features, the clocks and windows they are computed on, and the feature stores that serve them.

Where are we in the architecture?

This part adds no new box. It prepares what the data boxes hold: the tables the specialists read and the documents retrieval will search. Part 1.25 walks through the whole picture, layer by layer.

where Part 13.75 sits in the Acme architecture: highlighted boxes are the subject of this part, solid boxes come from earlier parts, faded boxes arrive later

What do I need to know first?

A schema is the list of a table’s columns and their types. Metadata is data about data: descriptions, owners, tags, update dates.
A semantic layer is where business meaning is defined once: what each metric means and how it is calculated. Acme’s is semantic.yaml (Part 2) plus governed metric views (Prerequisite 2).
Grain is what one row represents, such as one brand, one region and one week.
Parsing turns a file (PDF, Word, a scanned image) into text and structure. OCR (optical character recognition) reads text from images.
A chunk is a piece of a document stored and searched on its own. Embedding turns a chunk into numbers that represent its meaning (Part 14).

Why does preparation decide the answer?

The model can only be as right as what it reads. Preparation decides what it reads, so a strong model over messy data gives a confident wrong answer, and a simple model over well prepared data gives the right one.

Picture a new assistant on their first day in a shop’s stockroom. In one stockroom the boxes are unlabelled, there are two price lists that disagree, and last week’s delivery is still on the lorry. In the other, every shelf is labelled, there is one price list, and a note on the door says when stock was last counted. Ask the same assistant “how many blue kettles do we have?” in each room. They are equally clever in both; only one room lets them be right.

The assistant is the model, the labels are descriptions and metadata, the one price list is the semantic layer, and the note on the door is a freshness check. The trap is to blame the assistant and swap in a cleverer one, when the room is the problem.

How does this apply to Acme? Here is one question, asked over badly prepared and well prepared data:

Maria asks: "What were Acme Pure's service penalties in Western Europe last month?"

Badly prepared                               Well prepared
Table has columns pen_amt, rgn, wk           supply_service has service_penalty_eur, region,
with no descriptions                         week, each described, with sample values
Two views compute penalties differently      One governed definition of service penalties
Last week's rows have not landed yet         A freshness check fails loudly instead
The contract PDF was split mid-table         The clause and its table are one chunk
-> a plausible number, quietly wrong         -> the right number, with its source and date

Nothing about the model changed between the two columns. Here are the two preparation lines, side by side:

two kinds of data, two preparation lines: structured data is curated into a few agent-facing views, described, given one definition per metric, taught with example queries, checked for quality and freshness, and permissioned at the source; unstructured documents are inventoried, parsed with their layout, cleaned, chunked by structure, enriched with metadata and context, embedded and indexed, and kept fresh

How do you prepare structured data?

With tables, the goal is that the agent computes the right number, not that it finds a passage. That means six steps: give it a few clear tables, say what every column means, define every metric once, show it worked examples, check the data before it answers, and let the data platform decide who sees what. Each step below starts with the bottom line, then an everyday picture, then Acme.

1. Curate a few agent-facing views

Give the agent a small menu, not the whole pantry. Do not point an agent at a whole warehouse. Give it a small set of tables or views built for questions, with clear names and a clear grain, and with joins already done where they are always the same.

A restaurant does not hand diners the key to the pantry and say “make what you like”. It prints a menu of a dozen dishes, each prepared the same way every time. The pantry still exists; the menu is the curated view over it. The common mistake is to think that more tables make the agent more capable: every extra table is another wrong choice the model can make. Databricks’ best-practice guide for Genie agents makes the same recommendation: resolve column ambiguities and pre-join or de-normalise tables using views or metric views.

How does this apply to Acme? The menu is one table per agent: the marketing agent sees campaign_performance and the supply agent sees supply_service, each at the grain of brand, region and week (Prerequisite 2). The warehouse behind them is the pantry, and the agents never see it.

2. Describe every table and column

A model reads column names the way a stranger reads a doctor’s shorthand: it guesses. Spell out what each column means, in the words people actually use.

A doctor writes “pt c/o SOB, Hx HTN” in a patient’s notes. A colleague reads it instantly; a locum from another country might not, and a guess about “SOB” could go badly. The fix is not a smarter reader; it is a legend: SOB means shortness of breath, HTN means high blood pressure.

How does this apply to Acme? A column called pen_amt is the shorthand: it means nothing to a person or a model; service_penalty_eur with the description “Service-level penalties charged by the retailer for the week, in euros” is the legend, and means everything. Add descriptions, synonyms (the words people actually use, such as “fines” for penalties) and sample values. BigQuery’s conversational analytics data agents accept exactly this as structured context: a description, synonyms and tags for each table, and a description, synonyms, tags and sample values for each column. In Databricks, the same information lives in Unity Catalog comments:

COMMENT ON COLUMN acme.supply.supply_service.otif_pct IS
  'Share of orders delivered on time and in full that week, as a percentage (0 to 100).';

3. Define every metric once

One metric, one formula, one owner. If there are two definitions, the agent will be right twice, with two different numbers.

Two tills in the same shop both report “sales” at closing time. One counts refunds, the other does not. Till A says 1,000, till B says 940, and both are correct by their own rules. A manager asking “what were sales today?” gets whichever till they happen to ask. The fix is to write down once what “sales” means and make both tills use it.

How does this apply to Acme? OTIF (on time, in full) is Acme’s “sales”: if two teams calculate OTIF differently, an agent will happily quote both. Define each metric once, in a semantic layer, with an owner. Unity Catalog metric views, in Databricks’ words, provide a centralised way to define and manage business metrics. Acme also keeps its own semantic.yaml, which records each measure’s owner and formula and lets code, not the model, decide who may compute what (Part 2).

The rule: the model chooses which metric to ask for; code computes it from the one definition.

4. Teach with examples

Show the agent worked answers to the questions people really ask. A few correct examples teach more than a page of instructions.

A maths textbook does not just state the rule for long division; it shows three worked examples, then asks you to try. You copy the pattern, not the prose. Question-and-SQL pairs are the agent’s worked examples.

How does this apply to Acme? Pair common questions with the SQL that answers them correctly. BigQuery data agents call these example queries (verified queries); Databricks recommends example SQL and SQL expressions for business terms, and text instructions only as a last resort. Acme uses narrow tools instead of model-written SQL for most questions (Part 2), but the same question-and-answer pairs become its eval cases (Part 10).

5. Check quality and freshness

Check the data before the agent answers from it, and fail loudly when it is wrong or late. A wrong number delivered confidently is worse than “I cannot answer yet”.

Before a pharmacist hands over medicine, they glance at the expiry date. If it has passed, they do not hand it over with a smile; they say so and fetch a fresh box. The check is cheap, automatic and happens every time. The misconception is that data quality is a one-off clean-up; like expiry dates, it has to be checked on every use, because data keeps arriving.

How does this apply to Acme? Decide what “good” means for each table and check it automatically: no duplicate weeks, no negative penalties, percentages between 0 and 100, and last week’s rows present by Monday. Google Cloud’s Knowledge Catalog automatic data quality lets you define rules for BigQuery tables, validate data against them on a schedule and raise alerts. An agent should be able to say how fresh its answer is, and a failed check should block the answer rather than let it through quietly.

6. Permission at the source

Access belongs in the data platform, not in the prompt. A rule written in the prompt is a request; a grant on the table is a lock.

A library can put a sign on the rare books room saying “staff only, please”, or it can lock the door and give keys to staff. The sign works until someone ignores it or is talked past it; the lock works whoever asks. Instructions to a model are the sign, which is why Part 9 puts guardrails in code that cannot be talked around: a prompt injection becomes a data leak when permissions live only in the prompt.

How does this apply to Acme? The locks are Unity Catalog grants in Databricks, and IAM and row-level policies in BigQuery. Acme’s supply agent queries as the signed-in user, so Sam’s grants apply to every answer (Part 7): if Sam has no key to a region, no wording of the question opens it.

How do you prepare unstructured data?

With documents, the goal is that retrieval finds the one right passage, whole, current and labelled with where it came from. The model never sees the document, only the chunks retrieval hands it, so every step below protects those chunks.

1. Inventory: which documents, and which version wins

Decide which copy is the truth before anything is indexed. Search cannot tell a draft from a signed original; it only finds words.

A library that shelves the 2019 and 2024 editions of the same timetable side by side will send half its readers to the wrong bus. The librarian’s job is to withdraw the old edition, not to hope readers check the date.

How does this apply to Acme? List the sources, their owners and which version is authoritative. Remove drafts, superseded copies and duplicates before indexing; otherwise retrieval will find the old contract as readily as the new one.

Acme has one folder of signed retailer contracts, owned by legal, with the signed PDF as the only authoritative version; drafts stay out of the index.

2. Parse with the layout, not just the text

Plain text extraction flattens a document: headings, tables and lists become one stream of words. Google’s Document AI layout parser documentation names this problem directly: standard OCR flattens documents and destroys the structure that adds meaning, such as headings, tables and lists. The layout parser identifies those elements, keeps their relationships, and creates context-aware chunks in which all text comes from the same layout element. RAG Engine can use it at import, and it also offers an LLM parser for documents with charts, diagrams and images.

Here is the difference on one contract page:

the same contract page parsed two ways: flat text extraction and fixed-size chunks split the penalty table away from its clause, so a search for LDP-7 finds a chunk without the rate; layout-aware parsing keeps the heading, the table and its rows in one chunk, with a context line saying which contract and clause it is

3. Clean

Strip repeated headers, footers and boilerplate, fix broken characters, and remove near-duplicate passages. If a document holds personal data the agent does not need, redact it before indexing: data that is never indexed can never be retrieved (Prerequisite 5).

4. Chunk by structure

Cut documents where their meaning breaks, not where a character count runs out.

Imagine cutting a cookbook into cards for a recipe box. Cut every 300 words and one card ends with “add the eggs and” while the next starts “bake for 40 minutes”, with no recipe name on it. Cut at each recipe and every card is whole and titled. Retrieval hands the model cards, never the book.

How does this apply to Acme? The contract’s clauses are the recipes: cut at natural boundaries (sections, clauses, table ends) rather than every N characters, and keep a small overlap so a sentence on a boundary is not lost. RAG Engine lets you set both at import; its quickstart uses a chunk size of 512 and an overlap of 100. Smaller chunks are more precise but lose context; larger ones keep context but add noise. Test both on your own questions.

Here is the order I would try, cheapest first:

1. Does the document have headings or clause numbers?  -> chunk by heading (the Try it below)
2. Are there tables, charts or scans?                   -> layout parser, then chunk by element
3. Still no clear structure?                            -> fixed size with overlap, size tested
                                                           on your own questions
Only then: tune size and overlap further, measured by recall at k (below)

5. Enrich with metadata and context

A chunk on its own is a photocopied page with no book title on it. Label it with everything retrieval and permissions will need.

Photocopy page 212 of a textbook and hand it to someone a month later: “1.5% per day” of what, from which book, valid since when? A library would stamp the source on the copy.

How does this apply to Acme? Attach to every chunk what retrieval and permissions need later: owner, allowed roles, document type, effective date and source. Then add a short context line, the contextual retrieval technique from Part 14, so a chunk that says “1.5% per day” also says which contract and clause it belongs to.

6. Index, then keep it fresh

Embed the chunks, build the keyword index as well (Part 14’s hybrid search), and decide how changes flow through. When a contract is amended, its old chunks must be replaced; when one is withdrawn, its chunks must disappear from the index. Show the sync date next to each citation, so users can see how current a source is.

How do you prepare data that never stops arriving?

Batch data is prepared after it has landed; streaming data is prepared while it is still moving. Everything above assumes a table that is complete when the agent reads it. Some data never finishes arriving: till scans, shipment scans, card payments. Preparing it means computing features continuously, on the right clock, and giving the training job and the live model exactly the same number. This section looks at it from the traditional machine learning side, because that is where streaming features come from: a predictive model like the ones in Prerequisite 2.5 needs fresh inputs, and an agent then reads the model’s score or the features themselves. The ideas are the same whichever tools you use, so this section teaches them on their own and maps them to Google Cloud at the end.

Batch or streaming: what is the difference?

Batch processing collects data for a period and processes it in one run, such as a nightly job. Streaming processes each event as it arrives, so results update continuously.

Picture a supermarket chain with 500 stores (a toy scale, used through this section). Batch is the manager counting the milk on the shelves at closing time and ordering tomorrow’s delivery. Streaming is every till scan updating a running count, so that at 14:00 the system can see Store 42 will run out of milk by 16:00 and send a van today.

The misconception is that real time is always better. A streaming pipeline runs all day, keeps state, and has to cope with data that arrives late, so it costs more to build and to run. It is worth it only when the answer changes faster than someone acts on it. A weekly OTIF review is no better because the number updated at 10:02 instead of at midnight.

Who moves the events, and who computes on them?

Two different jobs are often lumped together as “streaming”:

A message bus (also called an event streaming platform or event log) moves events from the systems that produce them to the systems that use them, and stores them in between. The Apache Kafka introduction describes the model well: an event has “a key, value, timestamp, and optional metadata headers”; producers publish events and consumers subscribe to them; events are “organized and durably stored in topics”; and “producers never need to wait for consumers”. Because the bus keeps events for a set retention period, a consumer that falls behind or restarts can catch up, and you can replay history.
A stream processor computes on those events while they flow: counts, sums and averages over time, kept as running state. The Apache Beam programming guide is a good vendor-neutral reference: in Beam’s open source model a data set “can be bounded, meaning it comes from a fixed source like a file, or unbounded, meaning it comes from a continuously updating source”, and one pipeline model handles both.

Message buses include Pub/Sub and Apache Kafka; stream processors include Dataflow (which runs Apache Beam pipelines), Apache Flink and Spark Structured Streaming. The rest of this section uses the generic terms.

In the supermarket, each till publishes a tiny event and moves on to the next customer:

{"store": 42, "item": "milk", "qty": 1, "event_time": "2026-05-06T10:02:14Z"}

The message bus is the shock absorber. On Christmas Eve at 17:00 the tills publish far more than at 03:00, and if the model behind them is slow or restarting, the events wait on the bus instead of the tills waiting for the model. Raw “1 milk sold” events are too fine-grained for a model, so the stream processor turns them into features such as milk_sold_last_15m for each store.

Why does counting each event exactly once take work?

Networks fail halfway through. A consumer can process an event and crash before recording that it did, and a bus that guarantees nothing is lost will then deliver the event again. Kafka’s design documentation names the three possible guarantees:

At most once: “Messages may be lost but are never redelivered.”
At least once: “Messages are never lost but may be redelivered.”
Exactly once: “Each message is processed once and only once.”

On the supermarket: Store 42’s till sends the 50-carton sale, the acknowledgement is lost, and the till sends it again. At least once gives a count of 100 for that sale; at most once risks 0; exactly once gives 50. Exactly once is built by the processing system on top of redelivery: the stream processor deduplicates on a unique event ID and writes its results so that a retry does not double them. For a sales count, a duplicated scan is a wrong number, so give every event an ID at the source and decide which guarantee each feature needs. It is part of preparing the data.

Event time or processing time: which clock counts?

Event time is when something happened. Processing time is when the pipeline handled it. Apache Flink’s notions of time put it precisely: event time “is the time that each individual event occurred on its producing device”, while processing time “refers to the system time of the machine that is executing the respective operation”. In event time, Flink adds, “the progress of time depends on the data, not on any wall clocks”.

Store 42’s Wi-Fi drops at 10:01. At 10:02 a customer buys 50 cartons of milk for a school fair. The Wi-Fi comes back at 10:19, and the till uploads everything it saved. Event time: 10:02. Processing time: 10:19.

Count by processing time and two things go wrong at once: the 10:00 window shows a quiet morning, and the 10:15 window shows 50 cartons at once, a rush that never happened. A model trained on event-time history now receives an input it never saw in training. This is the second misconception: processing time is good enough. It is good enough only if events never arrive late or out of order, and no network promises that. Flink’s documentation notes that processing time is “susceptible to the speed at which records arrive in the system” and “to outages”.

Windows: how do you total a stream that never ends?

A window cuts an endless stream into finite pieces by event time, so each piece can be totalled. Beam and Flink both provide three common kinds (other tools use the names in brackets):

Fixed windows (tumbling): in Beam’s words, “a consistent duration, non overlapping time interval”, such as 10:00 to 10:05, then 10:05 to 10:10. Each sale lands in exactly one window. Use them for “milk sold per five minutes”.
Sliding windows (hopping): a fixed length, with a new window starting every period, so they “can overlap”. “Milk sold in the last 15 minutes, recomputed every minute” puts a 10:02 sale into 15 windows. Use them for smooth running features.
Session windows: one window per key that stays open while activity continues and closes after a minimum gap duration with no events. A shopper browsing the website, with a 30-minute gap, gets one session per visit. Use them for clickstreams.

Watermarks, allowed lateness and triggers: when is a window finished?

If data can arrive late, when do you publish the total for 10:00 to 10:05? Four ideas answer it:

A watermark is the processor’s estimate of how far event time has got. In Flink’s words, “A Watermark(t) declares that event time has reached time t in that stream, meaning that there should be no more elements from the stream with a timestamp t’ <= t.” It is an estimate, so it can be wrong.
Late data is what proves it wrong: Flink defines late elements as those that “arrive after the system’s event time clock (as signaled by the watermarks) has already passed the time of the late element’s timestamp”.
Allowed lateness is how long after the watermark passes the end of a window you still accept late data and update the result. Beam’s default is to discard late data, and withAllowedLateness (or allowed_lateness in Python) changes that. Spark Structured Streaming expresses the same idea as a watermark with “the threshold on how late the data is expected to be in terms of event time”.
A trigger decides when results are emitted. The common default is once, when the watermark passes the end of the window; the Beam guide adds that triggers can also “emit early results, before all the data in a given window has arrived”, and fire again when late data arrives. The accumulation mode decides what each firing contains: accumulating sends the full total so far, discarding sends only what is new since the last firing.

Here is the mechanism on Store 42, with a window of 10:00 to 10:05, an early firing every few minutes, 30 minutes of allowed lateness and accumulating mode:

10:00:30  scan, 2 cartons            arrives 10:00:31        inside the window
10:00:50  scan, 1 carton             arrives 10:00:51        inside the window
10:01     Store 42 Wi-Fi drops
10:02:14  scan, 50 cartons           saved on the till
10:03     early firing               -> total 3   (provisional)
10:06     watermark passes 10:05     -> total 3   (on-time result)
10:19     Wi-Fi back: the 10:02 scan arrives after the watermark, so it is late
          late firing                -> total 53  (discarding mode would send +50)
Once the watermark passes 10:35 (window end + 30 minutes), the window is closed and a
scan for it is dropped.

The feature for 10:00 to 10:05 was 3 at 10:06 and 53 at 10:19. Whoever reads it has to cope with a value that can still change, which is why an answer built on streaming data should say what time it is “as of” and whether it is provisional.

event time against processing time for one window: Store 42 sells 3 cartons before its Wi-Fi drops at 10:01, sells 50 at 10:02 while offline, and uploads that sale at 10:19; the event-time window 10:00 to 10:05 fires an early result of 3, an on-time result of 3 when the watermark passes 10:05, and a late, corrected result of 53 at 10:19, within 30 minutes of allowed lateness; a processing-time window would have shown a fake rush of 50 at 10:19

What is a feature store, and why does it have two stores?

A feature store keeps one definition of each feature and serves it two ways: the full history for training, and the latest value for live predictions.

Back at the supermarket, the model that predicts milk stock-outs reads its features twice, in very different ways. To train, it reads two years of history: for every store and hour, what the features were and whether the shelf emptied. To predict, it reads one row: Store 42’s milk, right now.

The offline store holds every historical value, appended and read in bulk, for training and batch scoring. It is usually a warehouse table or files.
The online store holds only the latest value for each key (each entity, here a store and item), in a database built for fast single-row reads. Store 42’s milk has one row, overwritten as new values arrive: store_42_milk -> [stock 8, milk_sold_last_60m 42, is_weekend 1] (toy values). Live inference passes that one row to the trained model.

Training needs one more piece of care, point-in-time correctness. To build the training example “did Store 42 run out of milk at 12:00 on 3 March?”, join the feature values as they were just before 12:00 on 3 March, never later ones. The open source feature store Feast describes its point-in-time join as scanning “backward in time” from each row’s timestamp. Joining later values is the leakage that Prerequisite 2.5 warns about: the model learns from information it would not have had.

The reason for one shared definition is training-serving skew, which Prerequisite 2.75 explains. On the supermarket, step by step:

Training (SQL, written by a data scientist):
  milk_sold_last_60m = 44 sold - 4 returned = 40
  The model learns: around 40 an hour means this shelf is close to empty.
Serving (separate code, written months later by another team):
  milk_sold_last_60m = 44 sold                = 44   (returns forgotten)
Result: the live model sees 44 where training would have seen 40, and raises
early restock alerts all day. Nothing in the world changed; the two pieces of code differ.
Fix: define the feature once, compute it once, and serve that one number to both.

This is feature engineering from Prerequisite 2.5 (time aggregates such as “sold in the last 60 minutes”), done continuously and done once.

A third misconception is that a feature store is a third database. It is mostly a registry of feature definitions plus the plumbing that keeps two stores consistent, and the stores are often ones you already run. Feature stores differ in how a value reaches both stores. Some copy the latest rows from the offline store to the online store, on a schedule or continuously. Others accept one write and send it to both: Feast’s push sources let you push feature values to the offline store, the online store or both, and Redis is one of its supported online stores. The mechanics differ; the principle does not: one definition, computed once, read two ways.

The loop can also close with an event. When the stream processor updates Store 42’s milk, the live model endpoint reads the latest row, predicts a stock-out within two hours, and publishes a RESTOCK_ALERT event to another topic on the bus, which the warehouse system consumes. Events in, features in the middle, events out. And because the tills are busy at 17:00 and nearly silent at 03:00, size the stream processor for the load you have: managed processors can add and remove workers as the load and the backlog change, so the night costs less without starving the morning rush.

one streaming feature pipeline: tills or scanners publish events to a message bus, which stores and buffers them; a stream processor computes each feature once on event time and writes it to the offline store, a table with the full history; the feature store copies the latest row per key to the online store; training reads history with point-in-time joins, while the live model and the agent read the one latest row, and the model can publish an alert back onto the message bus

On Google Cloud

Here is how the generic pieces map to Google Cloud services, as of November 2026:

Message bus: Pub/Sub, “an asynchronous and scalable messaging service that decouples services producing messages from services processing those messages”. It can redeliver a message, so consumers must cope with duplicates.
Stream processor: Dataflow, which runs Apache Beam pipelines and uses the Beam windowing, watermark and trigger model described above (streaming pipelines calls fixed and sliding windows tumbling and hopping). When reading from Pub/Sub, Dataflow’s exactly-once mode deduplicates redelivered messages, and an at-least-once mode can lower latency and cost if duplicates are acceptable (reading from Pub/Sub). Horizontal Autoscaling adds and removes workers based on CPU utilisation and pipeline parallelism.
Feature store: Feature Store on Gemini Enterprise Agent Platform (formerly Vertex AI Feature Store; the Vertex AI release notes say Vertex AI’s services are now part of Agent Platform). BigQuery tables or views are the offline store, and Feature Store acts as “a metadata layer interfacing with the BigQuery data sources”. It copies the latest values to an online store through feature views. With Bigtable online serving, a feature view syncs on a schedule or with continuous data sync, which refreshes the online copy whenever new rows land in BigQuery; continuous sync covers only new records (not updates or deletes) and needs the BigQuery source in eu, us or us-central1. The other option, Optimized online serving, is deprecated, with no new features from 17 May 2026 and sunset on 17 February 2027 (online serving types). So here, “write once” means the stream processor writes each feature row to BigQuery once, and Feature Store copies the latest row per key to the online store.
Lighter options: a Pub/Sub BigQuery subscription “writes messages to an existing BigQuery table as they are received”, with no pipeline code; BigQuery continuous queries are “SQL statements that run continuously” over rows as they arrive, though their stateful operations (joins, aggregations and windowing) are in Preview.

How does this apply to Acme?

At Acme on Google Cloud, this is how it would work. Supply’s data today is weekly: supply_service has one row per brand, region and week, and the service-risk model in Prerequisite 2.5 predicts next week’s OTIF from it. Suppose Sam wants to know on a Wednesday whether this week is already slipping.

Events. The warehouse systems publish each shipment confirmation to a Pub/Sub topic, supply-events (toy data):
```
{"event": "shipment_confirmed", "event_id": "WE-58812-3-c1", "brand": "Acme Pure",
 "region": "Western Europe", "order_line": "WE-58812-3", "units": 120,
 "promised_by": "2026-05-06T08:00:00Z", "event_time": "2026-05-06T10:02:00Z"}
```
It was confirmed two hours after its promise, so it counts as a late line. A handheld scanner in a warehouse with a weak signal is Store 42’s till: its events can arrive late, and the event_id lets the pipeline drop a resent copy.
Features. A Dataflow pipeline keyed by brand and region computes, on event time, late_lines_last_24h (a sliding window of 24 hours, recomputed every 15 minutes) and otif_week_to_date_pct (a fixed window per week, with early firings every 15 minutes and allowed lateness for scanners that reconnect).
One definition. The pipeline writes each row once to a BigQuery feature table, the offline store; Feature Store syncs the latest row per brand and region to the online store.
Model. A retrained version of the service-risk model that uses late_lines_last_24h, trained with point-in-time joins on that table’s history, scores the latest row.
Agent. The supply agent reads the row and the score through a narrow tool (Part 2), and says how fresh they are.

1. Sam: "Is Acme Pure on track for OTIF in Western Europe this week?"
2. Router (Part 5): a supply question -> delegate to the supply agent.
3. Supply agent calls get_live_service_risk("Acme Pure", "Western Europe")
   -> {"otif_week_to_date_pct": 93.4, "late_lines_last_24h": 37, "p_miss_95": 0.71,
       "zone": "review", "as_of_event_time": "2026-05-06T10:15:00Z", "provisional": true}
4. Answer: "As of 10:15 today, Acme Pure's OTIF in Western Europe this week is 93.4%,
   below the 95% line, with 37 late lines in the last 24 hours. Supply's service-risk model
   puts the chance of missing 95% this week at 71%. These figures are provisional: scans
   that arrive late can still change them."

The numbers are toy data. Who decided what? The model chose the tool and wrote the sentence. The figures came from the pipeline and the predictive model. The “as of” time and the word “provisional” came from fields the tool returned, so code, not the model, decided how fresh the answer claims to be.

Most of Acme’s agent questions do not need any of this. Maria’s ROAS by campaign last week, service penalties last month, and the weekly supply review are all answered correctly, and more cheaply, from weekly batch tables. Only the in-week operational question gains from streaming. (Acme’s build reads only the weekly tables; nothing in it streams.)

Why does streaming fail, and why does it win?

Why it fails. Acme streams campaign_performance so the marketing agent can quote ROAS “live”. Spend and sales are reconciled weekly, returns and late invoices revise them afterwards, and Maria plans on Mondays. The pipeline runs all week, most of what the agent quotes is provisional, and Maria makes the same decision she would have made from the weekly batch.
Why it wins. Store 42 sells milk fast on a hot Saturday. The nightly batch notices the empty shelf after it has been empty all afternoon. The streaming feature crosses the model’s threshold at 14:00, the model publishes a RESTOCK_ALERT, and a van arrives at 15:30. The answer changes within hours, and someone acts within hours.

Do you need streaming? A litmus test

Ask these in order, and stop at the first yes:

Is the answer stable for as long as a person takes to act on it? If Sam replans weekly, a weekly batch is enough. Most agent questions stop here.
Would a scheduled refresh do? Many “real time” requests turn out to mean “fresher than weekly”, and a daily or hourly batch job meets them.
Do you only need the events in a table quickly? Land them straight from the message bus into a warehouse table and let the agent query it with SQL as usual.
Do you need simple SQL over rows as they arrive? Use a continuous SQL query in the warehouse, if yours offers one (the Google Cloud options are listed above).
Only then build a stream processor with event-time windows, watermarks and allowed lateness, plus a feature store, for a model that must score live on running features.

Lock it in

Term                What it is                                        Supermarket
Message bus         Moves, stores and buffers events                  tills publish scans
Stream processor    Computes on events in flight, keeps state         running milk count
At least once       Nothing lost, duplicates possible                 50 counted as 100
Exactly once        Each event counted once (dedup by event ID)       50 counted as 50
Event time          When it happened                                  sale at 10:02
Processing time     When the pipeline saw it                          upload at 10:19
Window              A finite slice of the stream, by event time       10:00 to 10:05
Watermark           Estimate of how far event time has got            passes 10:05 at 10:06
Allowed lateness    How long late data may still update a window      30 minutes
Trigger             When results are emitted: early, on time, late    3, then 3, then 53
Offline store       Every historical value, for training              two years of counts
Online store        Latest row per key, for live scoring              one row: Store 42 milk
Point-in-time join  Training joins values as they were at the time    values at 11:59, 3 March
Skew                Training and serving compute a feature apart      40 versus 44

My view: in streaming, the clock matters more than the speed. Get event time, lateness and one shared feature definition right, and a pipeline that runs a few minutes behind is fine. Get them wrong, and a pipeline that answers in milliseconds is quickly wrong.

How do you know the preparation worked?

Mark it like an exam, with the answer key written before the agent sits it. If you only judge the answers by eye afterwards, you will grade a fluent wrong answer as right.

A teacher writes the mark scheme before reading any scripts, so a confident, well-written essay does not earn marks it has not got. The questions with known answers are the mark scheme; the agent is the student.

How does this apply to Acme? Test it with questions whose answers you know, before the agent sees it:

Structured: golden questions with the exact numbers the owning team expects, checked by code (Part 10). Whatever the owning team’s own query returns for Acme Pure’s Western Europe penalties last month, the agent must return the same number, to the euro.
Unstructured: questions with the chunk IDs that should be retrieved. Measure how often the right chunk is in the top results (often called recall at k) before you look at answer quality at all. If the right passage is never retrieved, no model can answer from it.
Streaming: two tests catch most problems. A replay test feeds recorded events into the pipeline in a shuffled order, with one batch held back and delivered late (Store 42’s 10:02 scan arriving at 10:19); every window’s final total must equal a batch recount by event time (53, not 3). A skew test takes a sample of keys and timestamps and computes each feature through the training path and the serving path; the two must match (40 and 40, never 40 and 44). Add an alert when the newest event time the agent can read falls too far behind the clock, so a stalled pipeline shows up as a freshness failure instead of an old number.

Recall at k is simple arithmetic. With illustrative numbers: write 10 contract questions, and for each note the chunk that holds the answer. Run retrieval and look at the top 5 results. If the right chunk is there for 8 of the 10, recall at 5 is 8 / 10 = 0.8, and the 2 misses tell you exactly which documents to re-parse or re-chunk.

When a golden question fails, the symptom usually points to one step:

Symptom                                   Step to fix
Agent picked the wrong table              Structured 1: curate fewer views
Right table, wrong column or unit         Structured 2: describe columns
Two runs, two different totals            Structured 3: define the metric once
Number is right but a week old            Structured 5: freshness check
Right document, rate missing from chunk   Unstructured 2 and 4: parse and chunk by structure
Old contract cited                        Unstructured 1 and 6: inventory and re-index
Total jumps when a store reconnects       Streaming: window by event time, allow lateness
Model fine in tests, odd scores live      Streaming: one feature definition (skew)

My view: spend the first week of any agent project on the data, not the agent. A described, curated, checked table and a well-parsed document set make even a simple agent good; no agent design rescues data that is ambiguous, stale or split in the wrong places.

A note on the build: Acme’s build prepares its two tables carefully and has a semantic layer, but it has no document index yet, so the contract examples in this part and in Part 14 show how it would work at Acme. The chunking exercise below is the one part that has been run.

Try it: chunk by size or by heading

This needs only Python. It chunks one contract section two ways and checks which chunk holds the penalty rate for clause LDP-7.

# try_chunking.py: fixed-size chunks versus chunks by heading, on one contract section.
import re

CONTRACT = """## LDP-7 Late delivery penalties
The supplier pays a penalty for each day a delivery is late.
| Days late | Penalty (share of invoice) |
| 1 to 3    | 1.0% per day               |
| 4 or more | 1.5% per day               |
## DSP-2 Penalty disputes
Disputes must be raised within 30 days of the delivery date."""

def fixed_size(text, size=120):
    return [text[i:i + size] for i in range(0, len(text), size)]

def by_heading(text, doc="Harvest Row contract"):
    parts = re.split(r"(?m)^## ", text)
    return [f"[{doc}, clause {p.split()[0]}] ## {p.strip()}" for p in parts if p.strip()]

def find(chunks, needle):
    return [i for i, c in enumerate(chunks) if needle in c]

for name, chunks in [("fixed size", fixed_size(CONTRACT)), ("by heading", by_heading(CONTRACT))]:
    print(f"{name}: {len(chunks)} chunks")
    for needle in ["LDP-7", "4 or more", "1.5% per day"]:
        print(f"  {needle!r} is in chunk(s) {find(chunks, needle)}")
    hit = [c for c in chunks if "1.5% per day" in c][0]
    print("  the chunk holding the 1.5% rate starts:", repr(hit[:60]))

This is the real output:

fixed size: 3 chunks
  'LDP-7' is in chunk(s) [0]
  '4 or more' is in chunk(s) [1]
  '1.5% per day' is in chunk(s) [1]
  the chunk holding the 1.5% rate starts: 're of invoice) |\n| 1 to 3    | 1.0% per day               |\n'
by heading: 2 chunks
  'LDP-7' is in chunk(s) [0]
  '4 or more' is in chunk(s) [0]
  '1.5% per day' is in chunk(s) [0]
  the chunk holding the 1.5% rate starts: '[Harvest Row contract, clause LDP-7] ## LDP-7 Late delivery '

With fixed-size chunks, the clause name and its rate land in different chunks, and the chunk with the rate starts mid-word with no clause name at all. By heading, one chunk holds the clause, the table and a context line. Change size to 60 or 300 and watch where the table breaks.

What can go wrong

Pointing the agent at everything. Hundreds of raw tables and every file share, with no curation, gives the model too many wrong choices.
Undescribed columns. The model guesses what pen_amt means, and sometimes guesses wrong.
Two definitions of one metric. The agent is right twice, with different numbers.
Stale data, silently. Last week has not landed, and the answer does not say so.
Flattened documents. Tables and headings are lost in parsing, so the right fact is in no single chunk.
Deleted documents that live on. A withdrawn policy is still in the index and still cited.
Counting on the wrong clock. Windows by processing time turn a reconnecting store into a fake rush, and late events with no allowed lateness are silently dropped.
Two feature definitions. Training and serving compute the same feature in separate code, and the live model sees 44 where it learned on 40.
Streaming what batch would do. A pipeline runs all week for a number someone reads on Monday, and most of what the agent quotes is provisional.
Fresh-looking answers with no time on them. A streaming feature can still change; an answer that does not say “as of” and “provisional” overstates what it knows.

What code did we write in this part?

Every part ends with the same map of Acme’s build: which files exist so far, and the code this part added, explained line by line. The map grows with each part.

This part adds no code to Acme’s build. It explains how to prepare the tables the specialists already read and the documents a retrieval agent would search; the only code is a standalone chunking exercise, which is not part of the build.

Where does it live?

router/                                             the router agent: extract, route, write
  app/agent.py                                      P5,8        workflow; extract reads memory; App with compaction and resumability
  app/app_utils/services.py                         P8          picks the session, artifact and memory services
  app/fast_api_app.py                               P8          Runner with session, artifact and memory services
  app/remote.py                                     P6,7        A2A client over mTLS; metadata carries Sam's token
  app/writer.py                                     P5,8        writer reads memories and saves the conversation to Memory Bank
  scripts/create_memory_bank.py                     P8          creates the Memory Bank and its topics, once
  tests/eval/acb_metrics.py                         P10         code metrics: route_correct, grounded, golden_numbers
  tests/eval/build_datasets.py                      P10         splits source cases into golden and held-out, seed 42
  tests/eval/calibrate.py                           P10         compares judge scores with human labels (kappa)
  tests/eval/datasets/golden_marketing.json         P10         golden cases with expected route, target and numbers
  tests/eval/eval_config.yaml                       P10         eval metrics, including the faithfulness judge
  tests/eval/gate.py                                P10         thresholds per metric; exits 1 when one fails
  tests/unit/test_router.py                         P10         unit tests; routing checked with one row per question type
marketing/                                          the marketing specialist, on BigQuery
  app/agent.py                                      P1,2,3,4    marketing agent with its skill in a SkillToolset
  app/app_utils/a2a.py                              P6          A2A endpoint and agent card (1.0 and 0.3) for marketing
  app/skills/marketing-kpis/SKILL.md                P4          marketing know-how: which tool, definitions, story checks
  app/skills/marketing-kpis/references/tables.md    P4          tables, grain, measures and the join rule, read on demand
  app/tools.py                                      P1,2        three narrow tools: measures, unserved demand, SQL fallback
  tests/unit/test_marketing.py                      P2          unit tests for the tools, with a fake context and database
supply/                                             the supply specialist, on Databricks
  app/agent.py                                      P3,7,8,9    supply agent and App with the confirmed replan tool
  app/app_utils/a2a.py                              P6          A2A endpoint and agent card (1.0 and 0.3) for supply
  app/skills/supply-catalog/SKILL.md                P4          supply know-how, given only to the supply agent
  app/tools.py                                      P3,7,9      supply tools; the replan tool asks a human first
shared/                                             code all three services share
  adk_common.py                                     P3,6,9,11   plugins and redacted JSON logs, now with the usage ledger
  auth_manager.py                                   P7          auth manager callback: token or consent (off by default)
  engines.py                                        P3,7        run_databricks connects as the signed-in user
  factcheck.py                                      P5          ungrounded_numbers: finds figures not in the findings
  registry.py                                       P2          reads semantic.yaml: measure names, computable measures
  routing.py                                        P1          classify: picks the specialist in code, no model
  safety.py                                         P9          per-user token bucket and injection pattern scan
  semantic.yaml                                     P1,2        registry of measures, owners and published formulas
  skill_source.py                                   P4          loads a skill from disk or a pinned registry revision
  sqlguard.py                                       P3          guard_sql read-only allow-list; redact for log lines
  tests/test_cost_report.py                         P11         unit tests for the cost maths
ops/                                                build, deploy and evaluation tooling
  .cloudbuild/eval_gate.yaml                        P10         after staging: eval run, then gate.py
  .cloudbuild/pr_checks.yaml                        P10,11      offline PR checks, cheapest first, ending with the mutants
  env.sh.example                                    P11         settings for every engine: ids, models, plugin switches, OTEL; no secrets
  infra/gateway/setup_gateway.sh                    P12         egress gateway, endpoint registry and IAP rules, in dry-run
  tools/cost_report.py                              P11         prices the ledger and reports cost per resolved question
  tools/mutants/acb_mutate.py                       P10         pytest plugin that injects each mutant's fault
  tools/pricing.json                                P11         list prices per model, kept out of the code
  tools/run_mutants.py                              P10         runs the offline suites once per mutant

+ added in this part   ~ changed in this part   Pn = the parts that touched the file

What did we learn?

Structured and unstructured data need different preparation, and both matter more than the model.
For tables: curate a few views, describe every column, define each metric once, teach with examples, check quality and freshness, and permission at the source.
For documents: inventory and pick the authoritative version, parse with the layout, clean, chunk by structure, enrich with metadata and context, and keep the index fresh.
Test preparation directly: golden numbers for tables, expected chunk IDs for documents.
Streaming data is prepared while it moves: a message bus stores and buffers events, a stream processor computes features in event-time windows, and watermarks, allowed lateness and triggers decide when a total is final and how late data corrects it.
Buses redeliver, so decide per feature between at-least-once and exactly-once, and give every event an ID so duplicates can be dropped.
A feature store defines each feature once and serves it two ways, history for training with point-in-time joins and the latest row for live scoring, which is how it prevents training-serving skew.
Most agent questions are fine with batch; stream only when the answer changes faster than a person acts on it, and try a scheduled refresh, landing events straight into a table, or a continuous query first.

What comes next?

Production-ready agents, Part 14: Knowledge: RAG, long context, fine-tuning and the infrastructure behind them builds on the unstructured half: who runs the search, hybrid search, reranking and when tuning is worth it.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands and its data are fictional.