Databricks on Google Cloud, Part 7: The platform on Google Cloud, runtime, Photon, SQL, Lakeflow and AI

Parts 3 to 6 built the ground floor for Acme Consumer Brands: the network, serverless egress, the buckets and the governance around them. None of that answers the question the supply team actually asks, which is “how do I get from delivery files landing in a bucket to an on-time-in-full number that a planner can trust, and maybe ask a question about it in plain English?”

That is the job of the platform itself: the Databricks Runtime and its Photon engine, SQL warehouses, the Lakeflow family that ingests and transforms data, the dashboards, Genie, apps and agents that sit on top, and Lakebase, a Postgres database that lives next to the lakehouse. Several of these products have been renamed in the last two years, some are Beta on Google Cloud, and a few are not offered in Acme’s region at all.

This post walks through each one from the basics, shows it on Acme’s supply pipeline, and is honest about status as of October 2026. It builds on Part 1, which introduced the products and the classic and serverless compute planes, and it leaves BigQuery, Gemini models, Pub/Sub and Looker to Part 8.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: what runs above the infrastructure, what it is called today and what to check, in three minutes. The narration is a synthetic voice; the script and animation are mine.

What are the key ideas?

Part 1 introduced the workspace, classic compute (Compute Engine virtual machines in your project), serverless compute and SQL warehouses, and drew every product by its October 2026 name. This post goes into the products that run on that compute. These are the terms that carry it:

Databricks Runtime and Photon: the versioned software image every classic cluster boots, and the C++ engine inside it that runs supported operations faster. The next section explains both.
Lakeflow: the data engineering family. Lakeflow Connect ingests data, Lakeflow pipelines (built on the open-source Apache Spark Declarative Pipelines) turn it into tables, and Lakeflow Jobs schedule the work. Lakeflow pipelines is the product most people still know as Delta Live Tables (DLT). Databricks renamed it in 2025 (for a while it was called Lakeflow Declarative Pipelines), and the documentation page What happened to Delta Live Tables (DLT)? says that “there is no migration required”: old DLT code still works, the Python import dlt can become from pyspark import pipelines as dp, and the classic price-list SKUs still begin with DLT. Lakeflow Jobs was previously called Databricks Workflows. “Declarative” means you describe the tables you want and the engine works out the order of steps, retries and incremental updates.
Unity Catalog: the governance layer that names every table as catalog.schema.table and controls who may use it; Part 6 covers it in depth. Acme’s raw files sit in a volume, governed storage for files that are not tables, at /Volumes/acme_supply/supply_bronze/landing/ (Part 5).
Genie: natural-language questions over governed data. Genie Agents are domain-specific environments that data teams set up with trusted tables, metrics and business rules.
Agent Bricks: the Databricks agent developer platform, with Unity Gateway governing which models, MCP servers (services that expose tools to agents through the open Model Context Protocol) and tools agents may use.
Lakebase: a managed Postgres (PostgreSQL, the open-source relational database built for many small, fast reads and writes from applications, rather than large analytical scans) database for applications and agents, in Beta on Google Cloud as of October 2026.

The stack below shows how those pieces fit together, from files in a bucket at the bottom to workloads at the top.

the key ideas as one stack: files in Cloud Storage, an open table format that turns files into tables, Unity Catalog that names and guards them, engines that compute on them, and the workloads on top; the Databricks control plane runs the whole thing from Databricks' own account

One table runs through the whole post: supply’s gold table, supply_service. Here is what it looks like, so you can picture every query and pipeline that follows.

a quick look at supply_service, the gold table this post keeps returning to: acme_supply.supply_gold.supply_service has one row per brand, region and week, owned by supply and stored as Delta in gs://acme-lake-supply; three real rows

Prerequisite 2 and the series overview explain Acme’s tables and projects in full.

What is the Databricks Runtime?

The bottom line: the Databricks Runtime is the versioned software image every classic cluster boots, and Photon is a faster engine inside it; you choose the version, and long-term support (LTS) versions get fixes for three years.

A simple example first. A company issues laptops to its staff. Every laptop in one team runs the same operating system version, so a script that works on one works on all. IT can choose a long-supported release that only receives security fixes for years, or the newest release that gets new features every few months. When a release reaches end of support it still boots, but nobody patches it, and one day IT stops allowing it on the network. The operating system image is the Databricks Runtime, the version number is the runtime version, the long-supported release is LTS, and the day it is banned is end of life.

Databricks runs your code on Apache Spark, the open-source engine that splits one big job into many small tasks and runs them in parallel across a cluster of virtual machines (the lakehouse prerequisite explains Spark’s driver and executors). A cluster can be classic compute, made of Compute Engine VMs in your project, or serverless compute that Databricks runs for you, or the smaller cluster behind a SQL warehouse.

Databricks builds on Spark with the Databricks Runtime (Spark plus Databricks’ improvements) and Photon, which the documentation calls a “vectorized query engine” written in C++ that takes over execution for supported operations without code changes.

Two words in that sentence need unpacking. A query engine is the part of the system that takes a query that has already been planned and actually runs it: reading the files, filtering rows, joining tables and computing sums. Vectorised describes how it processes the data. A row-at-a-time engine handles one row, then the next, calling the same code again for every row. A vectorised engine processes a batch of values from one column at once (for example a few thousand units_delivered values held next to each other in memory) with one tight loop, and modern CPUs have instructions (SIMD, single instruction, multiple data) that apply one operation to several values in a single step. Photon “processes data in columnar batches”, which suits Parquet because Parquet already stores each column’s values together. C++ matters because it compiles to native machine code and manages memory directly, whereas open-source Spark runs on the Java Virtual Machine (JVM).

The simple definition: the Databricks Runtime is the software image every classic cluster boots. The formal one, from the runtime release notes: each version “includes updates that improve the usability, reliability, performance, and security of the Databricks platform”, and it pins a specific Apache Spark version. Three parts of that are worth unpacking.

A version number. Databricks Runtime 18 LTS ships Apache Spark 4.1.0; Databricks Runtime 19 ships Spark 4.2.0. You pick the version when you create a cluster or a job cluster, and every node in the cluster runs the same image.
Variants. Each version comes as a standard image and a Databricks Runtime for Machine Learning image, which adds libraries such as MLflow (version 3.12.0 in Runtime 19 ML) and the feature engineering client. GPU machine types need the machine learning image.
LTS. Long-term support. The support lifecycle page describes four phases. A version starts as a Beta, becomes generally available (GA) and receives new features “under the same version number” for about six months, then becomes LTS and gets “major stability and security fixes” backported for three years. End of support comes three years after GA: workloads still run, but the version disappears from the UI and gets no fixes. Databricks aims to remove it entirely (end of life) about six months later, after which workloads on it fail.

One change is easy to miss. Before Databricks Runtime 19, Databricks released minor versions such as 18.0, 18.1 and 18.2, each with its own Beta and GA. From 19 onwards there is one version number that gathers features for six months and then turns into an LTS.

the life of one Databricks Runtime version: Beta, then GA with about six months of feature development, then three years of long-term support, then end of support and end of life; and the supported versions on Google Cloud in October 2026, from Databricks Runtime 19 back to 14.3 LTS, with Acme's job compute on 18 LTS

Serverless compute (Part 1 explains it) has no runtime drop-down at all. Databricks upgrades the server side itself, and you choose an environment version instead: a fixed Python version and set of libraries with a three-year lifecycle. Environment version 6, released on 3 September 2026, has Python 3.12.3 and Databricks Connect 19.

How does this apply to Acme? The team’s laptops are the nodes of Acme’s nightly job cluster, and the long-supported release is the newest LTS. My view: pin production jobs on classic compute to the newest LTS (18 LTS for Acme’s nightly job cluster) and plan one runtime upgrade a year, well before the three-year end of support. The lifecycle page includes SQL queries over the system.compute.clusters and system.billing.usage system tables that list every cluster still on an old version.

What is Photon, and does it cost extra?

The bottom line: Photon speeds up the execution step only; Spark still plans the query, anything Photon cannot run falls back to Spark, and the answer is the same either way. The common misconception is that turning on Photon swaps Spark out. It does not.

A restaurant kitchen makes the split clear. The head chef reads the order and writes the plan: which dishes, in which order, on which station. A specialist cook with a high-speed machine handles the jobs the machine knows, such as slicing and frying, far faster than by hand. When a dish needs a step the machine cannot do, a regular cook does that step by hand. The customer gets the same dish. The head chef is Spark’s Catalyst optimiser, the machine is Photon, and the regular cook is the Spark runtime it falls back to.

The speed comes from the batch. Illustratively, to sum 4,096 units_delivered values, a row-at-a-time engine calls its code 4,096 times. A vectorised engine runs one loop over the whole batch, and if a SIMD instruction adds 8 values at once (a 512-bit register holds eight 64-bit numbers), the loop needs about 4,096 / 8 = 512 add steps instead of 4,096.

Photon is not a replacement for Spark. It is an execution engine inside the Databricks Runtime: “The Apache Spark query optimizer (Catalyst) still plans your query, but Photon takes over at the execution layer.” Spark on the driver still parses your code, plans the query and schedules the tasks. On the executors, Photon runs the operators it supports, in native C++ on columnar batches as described above, and “when Photon encounters an unsupported operation during query execution, it transparently falls back to the Spark runtime for the remainder of that operation”. One query can mix both, and the result is the same either way.

What it supports: the Photon documentation lists operators such as scans of Parquet, Delta, CSV and JSON, filters and projections, hash aggregates, joins and shuffles, sorts, window functions and writes to Delta and Parquet, over most common data types.
What it does not support: user-defined functions (UDFs), the RDD API and the Dataset API, and stateful streaming (stateless streaming works). It also does not speed up queries that “normally run in under two seconds”, where planning and scheduling take most of the time.
Where it is on: Photon is enabled on serverless compute, SQL warehouses and serverless Lakeflow pipelines. For classic compute, Databricks enables it by default through the Use Photon Acceleration checkbox; if you create clusters through the Clusters API or Jobs API, you must set runtime_engine to PHOTON yourself.
What it costs: on classic compute, “Photon instance types consume DBUs at a different rate than the same instance type running the non-Photon runtime”, so check the Databricks pricing calculator for your machine type. On SQL warehouses and serverless compute it is always on, so its cost is part of those products’ DBU rates.

A few more details from the same page help when you tune a real workload.

How big a batch is. Photon processes “batches of thousands of rows at a time”, which is what lets the CPU’s SIMD instructions work on several values per cycle. Running native C++ also avoids the JVM’s garbage collection pauses and warm-up time.
What it changes inside a query. Photon replaces sort-merge joins with hash joins, uses a columnar shuffle, and has a native Parquet writer that speeds up MERGE, UPDATE, DELETE, INSERT and CREATE TABLE AS SELECT on Delta and Iceberg tables.
How to see it working. In the Spark UI of a classic cluster, Photon operators are orange in the query graph and Spark operators are blue. On SQL warehouses and serverless compute, the query profile shows the percentage of task time spent in Photon.
What depends on it. Predictive I/O (covered below) and dynamic file pruning in MERGE, UPDATE and DELETE need Photon. Lakeflow pipelines created through the Pipelines API need photon: true.
What Databricks claims. The page says Photon gives “up to 5x better price/performance” compared with other cloud data warehouses on TPC-DS benchmarks. That is Databricks’ own figure, so test it on your workload before you quote it.

Part 1 adds one Google Cloud detail: GPU machine types (A2 and G2 shapes) do not support Photon.

How does this apply to Acme? Building supply_service is scans, filters, a GROUP BY and a write to Delta, all on Photon’s supported list, so the machine does nearly all of it. If someone added a Python UDF to clean carrier names (a hypothetical; Acme’s pipeline has none), that one step would go back to the regular cook, and the rest of the query would stay on Photon.

how Photon fits inside the Databricks Runtime: Spark on the driver still parses the query, plans it with the Catalyst optimiser and schedules tasks; on the executors, Photon runs the operators it supports, here the scans, filter, hash join and hash aggregate, and the Python UDF at the top falls back to the Spark runtime, all within one query

How does Databricks make queries fast?

Photon makes each operation faster. The bigger win is not doing the operation at all, which is what the features in this section are for. The bottom line: the fastest read is the one you skip.

A simple example: you need every May invoice for one region from an archive room of 1,000 boxes. If each box has a label saying which dates and regions it holds, you open perhaps 20 boxes, not 1,000. If someone repacks the boxes so each holds one region’s invoices, the labels get sharper and you open fewer. Photocopies of boxes you opened yesterday sit on your desk. And before you start, you plan the shortest route through the room. The labels are data skipping statistics, the repacking is liquid clustering, the photocopies are the disk cache, and the route is the cost-based optimiser and adaptive query execution. A caretaker who repacks and relabels overnight without being asked is predictive optimization.

Which Databricks features make a query read less?

Prerequisite 1 explains the three warehouse techniques a lakehouse borrows, and draws data skipping on the same supply_service query: indexing (per-file statistics that let an engine skip files), caching (a local copy of files already read) and query optimisation (choosing the cheapest plan, then adjusting it while the query runs). On Databricks each idea has a concrete feature. Data skipping uses the statistics in the Delta log. Liquid clustering “replaces table partitioning and ZORDER”, and Databricks recommends it for all new tables. The disk cache keeps each Parquet file a worker reads on that worker’s local SSD. Spark’s cost-based optimizer and adaptive query execution plan the query, both on by default, and Photon executes the plan. Here is what each feature does on Databricks in October 2026, with the settings that matter.

Data skipping, in detail. The data skipping page says statistics are collected “automatically when you write data to a Delta Lake or managed Apache Iceberg table”. By default that covers the first 32 columns of the schema, so a filter on column 40 skips nothing. For Unity Catalog managed tables, predictive optimization (below) also collects statistics on other columns you often filter on. Otherwise you can change the list with the delta.dataSkippingStatsColumns table property and recompute with ANALYZE TABLE ... COMPUTE DELTA STATISTICS.

Liquid clustering, in detail. You choose up to four clustering keys, the columns the table is organised by, and they must be columns that have statistics. Clustering is incremental: new data is clustered when OPTIMIZE runs, so Databricks recommends running it often or letting predictive optimization do it. CLUSTER BY AUTO goes further and lets Databricks pick and change the keys from your query patterns; it needs predictive optimization and Databricks Runtime 15.4 LTS or above for managed Delta tables. Since 12 June 2026, converting a partitioned table in place with ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY is generally available on Databricks Runtime 18.1 and above.

How does this apply to Acme? Supply’s queries almost always filter on region and week, so those are the labels worth sharpening. For Acme’s silver deliveries table it looks like this (I have not run these commands):

-- let Databricks run OPTIMIZE, VACUUM and ANALYZE on everything in the catalog
ALTER CATALOG acme_supply ENABLE PREDICTIVE OPTIMIZATION;

-- organise the silver table by the two columns supply filters on most
ALTER TABLE acme_supply.supply_silver.deliveries CLUSTER BY (region, week_start);

-- or hand the choice of keys to Databricks
ALTER TABLE acme_supply.supply_silver.deliveries CLUSTER BY AUTO;

The disk cache, in detail. It used to be called the Delta cache (and the DBIO cache), and the disk cache page explains the rename: the cache is a Databricks feature, not part of the Delta Lake protocol. It turns on by itself on any worker type with local SSDs and uses at most half of that space, evicting the least recently used files first. On Google Cloud that makes the machine type matter: local SSDs can only be added to some families such as N2, and Part 1’s cluster settings section shows the local_ssd_count field in Acme’s job cluster.

Predictive optimization, in detail. The predictive optimization page says it “automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables”. OPTIMIZE compacts small files and applies clustering, VACUUM deletes data files no table version still needs, and ANALYZE collects statistics.

On Unity Catalog managed tables, predictive optimization runs VACUUM for you, so nobody has to schedule it.

As of 8 November 2026 the predictive optimization page simply says it “is enabled by default” (earlier versions gave 11 November 2024 as the cut-off for new accounts and August 2026 for older ones). It needs the Premium plan, works only on Unity Catalog managed tables, runs on serverless jobs compute and is billed on a serverless jobs SKU. It is available in 14 of the 15 Google Cloud regions, including europe-west2 (me-central2 is the exception). One warning from the page: VACUUM keeps files for seven days by default, controlled by delta.deletedFileRetentionDuration, so set a longer retention before you switch it on if you rely on longer time travel.

Predictive I/O and deletion vectors. Predictive I/O is a set of optimisations that are “exclusive to the Photon engine”: faster selective reads on serverless and pro SQL warehouses and Photon clusters, and cheaper updates on tables with deletion vectors. A deletion vector is a small file that marks rows as deleted or changed, so an UPDATE of one row no longer rewrites the whole Parquet file it sits in; readers apply the marks at query time. Delta tables need deletion vectors switched on (delta.enableDeletionVectors), while Iceberg v3 tables always have them. Acme’s gold table has Iceberg reads turned on for BigQuery, and Part 8 explains why that rules deletion vectors out for Iceberg v2 readers.

If a query is slow, check in this order, cheapest first:

1. Is predictive optimization on for the catalog?        -> OPTIMIZE, VACUUM, ANALYZE happen for you
2. Do your filters hit columns that have statistics?     -> first 32 columns, or dataSkippingStatsColumns
3. Is the table clustered on the columns you filter by?  -> CLUSTER BY (keys) or CLUSTER BY AUTO
4. Is it running on Photon, on a machine with local SSD? -> Photon features and the disk cache apply
5. Only then                                              -> rewrite the query or resize the compute

the Databricks performance features that matter on Google Cloud, what each does, whether it is on by default and how Acme uses it: disk cache, data skipping, liquid clustering, predictive optimization and predictive I/O help Acme; deletion vectors and row-level concurrency are off because Iceberg v2 readers need the table without deletion vectors

What does row-level concurrency change when two jobs write at once?

The bottom line: without row-level concurrency, two writers clash if they touch the same file; with it, they clash only if they touch the same rows.

A shared document shows the difference. Two people edit page 4 at once, one fixing paragraph 1 and the other paragraph 6. A strict editor that tracks changes by page rejects whoever saves second, who must reload and redo the edit. A smarter editor that tracks paragraphs accepts both, and only refuses when both changed the same paragraph. The page is a Parquet data file, the paragraph is a row, the strict editor is file-level conflict detection, and the smarter one is row-level concurrency, which needs deletion vectors to record row-level changes.

Prerequisite 1 walks through a write conflict step by step: under Delta’s default WriteSerializable level, writers take no locks, and a commit fails with a conflict exception when another commit has removed a file it read, so the losing job must retry on the newer version. What Databricks adds is how finely that check is made. Row-level concurrency helps when two writers change different rows of the same file: it detects changes row by row and resolves “conflicts that occur when concurrent writes update or delete different rows in the same data file”. It is turned on automatically on Databricks Runtime 14.3 LTS and above for tables that are not partitioned and have deletion vectors enabled (explained in the previous section). Changes to the same rows still conflict. Acme’s gold table has Iceberg reads turned on with Iceberg v2 compatibility, so that BigQuery can read it (Part 8 explains how, and the newer Iceberg v3 option), which rules out deletion vectors, so it gets file-level conflict detection.

How does this apply to Acme? The gold table supply_service has the strict editor: if two jobs ever changed different weeks that happen to sit in the same file, the second to commit would fail and retry. That is why the rules below matter for it.

which concurrent Delta writes conflict under the default WriteSerializable level, with and without row-level concurrency: appends never conflict; updates to different rows of the same file conflict unless row-level concurrency is on; updates to the same rows always conflict; schema changes can fail every concurrent write

The row-level concurrency page lists the exceptions, and Prerequisite 1 explains the common ones. On Databricks the one you will meet most is ConcurrentDeleteReadException, the usual error when two MERGE jobs update rows in the same file. MetadataChangedException means someone changed the schema or a table property while you were writing. Two practical rules follow from the matrix:

Only the operation that read the affected data fails, so put retry logic on the UPDATE, DELETE or MERGE, not on plain appends.
Without deletion vectors, keep two jobs that rewrite the same table from overlapping, for example by running them as consecutive tasks in one Lakeflow job, or give each job a condition on a different set of rows.

What is a SQL warehouse, and which type should you choose?

The bottom line: choose serverless unless the warehouse must sit inside your own network; then choose pro. Classic is the entry-level fallback.

Getting across a city makes the choice concrete. A taxi from the rank arrives in seconds, someone else maintains it, and you pay only for the ride. Your own car takes a few minutes to warm up and sits in your own garage, but it can drive through your private gate. An older car of your own does the same, more slowly. The taxi is a serverless warehouse (2 to 6 seconds to start, in Databricks’ account), your car is a pro warehouse (about 4 minutes, in your VPC), and the older car is classic.

A SQL warehouse is compute that runs only SQL: queries from the SQL editor, AI/BI dashboards, Genie Agents, alerts, and BI tools that connect over JDBC or ODBC. Until 2023 they were called SQL endpoints. Underneath, a warehouse is one or more clusters of a chosen size (2X-Small to 4X-Large), and it adds clusters when queries queue. Databricks creates a small one called Starter Warehouse in every new workspace.

There are three types, and the SQL warehouse types page compares them by three performance features: Photon, predictive I/O, and intelligent workload management (IWM), which uses machine learning models to predict each query’s needs and add or remove clusters quickly instead of waiting on fixed thresholds.

Serverless. Runs in Databricks’ serverless compute plane, starts in “typically between 2 and 6 seconds”, and has all three features. It is the default in the UI where it is available, and Databricks recommends it “for most workloads”. Part 4 covers how its network reaches your data.
Pro. Runs in your project and VPC, takes “approximately 4 minutes” to start, and has Photon and predictive I/O but not IWM. Choose it when serverless is not available, or when the warehouse must sit in your own network to reach a database or an on-premises system through Part 3’s VPC.
Classic. Also in your project, also about 4 minutes, Photon only. The documentation calls it “entry-level performance”.

Classic and pro warehouses scale by rules: each cluster takes about 10 concurrent queries, and the autoscaler adds one cluster when the queued work would take 2 to 6 minutes, more beyond that. Every worker is an n2-highmem-8 VM, so a Medium warehouse is a driver plus 8 workers and 96 vCPUs drawn from your N2 CPU, Persistent Disk SSD and Local SSD quotas in the region. A warehouse with too little quota simply does not start; Part 9 covers quotas. A queue holds at most 1,000 queries on any type.

the three SQL warehouse types on Google Cloud: serverless runs in Databricks' account, starts in 2 to 6 seconds and has Photon, predictive I/O and intelligent workload management; pro and classic run in your project, take about 4 minutes to start, and pro adds predictive I/O to classic; classic and pro sizes are built from n2-highmem-8 workers that draw on your Compute Engine quota

One gotcha deserves code. The UI defaults to serverless, but the SQL warehouses API defaults to classic. To create a serverless warehouse from Terraform, a script or a pipeline, set both fields explicitly. A request body for POST /api/2.0/sql/warehouses looks like this (I have not run it):

{
  "name": "acme-supply-bi",
  "cluster_size": "Small",
  "min_num_clusters": 1,
  "max_num_clusters": 4,
  "auto_stop_mins": 10,
  "warehouse_type": "PRO",
  "enable_serverless_compute": true
}

How does this apply to Acme? Supply’s dashboards and Genie Agent read supply_service from Cloud Storage, which serverless reaches, so the taxi wins. As a litmus test:

1. Is serverless offered in the region and can it reach the data?  -> serverless
2. Must the warehouse reach a private database or on-premises?     -> pro, in Acme's VPC
3. Neither available, or an entry-level legacy need?               -> classic (Photon only)

Two status notes as of October 2026. Lakehouse Real-Time (written Lakehouse//RT) is a new serverless warehouse type in Beta since 10 August 2026, built for sub-second, read-only queries from applications and dashboards with “hundreds to thousands of concurrent users”; it is offered in all 15 Google Cloud regions. And one honest gap: the Google Cloud page Connect to a SQL warehouse still carries a Public Preview label on its “What are Serverless SQL warehouses?” section, even though the rest of the documentation treats serverless as the default. Check the label in your own workspace before you promise a customer a GA service.

What is Lakeflow, and what were its parts called before?

The bottom line: Lakeflow is three jobs under one name: bring data in (Connect), turn it into tables (pipelines), and run that on a schedule (Jobs). The old names, Delta Live Tables and Databricks Workflows, still describe the same products.

A restaurant kitchen maps onto it. Deliveries arrive at the back door from many suppliers. Recipes say what each dish is made of, and the kitchen works out the order: stock before soup, soup before service. A rota and a timer say when each shift starts cooking. The back door is Lakeflow Connect, the recipes are Lakeflow pipelines (declarative: you describe the dish, the kitchen works out the steps), and the rota is Lakeflow Jobs.

How does this apply to Acme? Delivery files and orders come in through the back door, one SQL pipeline turns them into bronze, silver and supply_service, and a job starts it when files land.

Lakeflow is Databricks’ name for its data engineering family: getting data in, turning it into tables, and running that work on a schedule. It was announced in the Introducing Databricks Lakeflow post of 13 June 2024, which described the pipelines as “built on the declarative Delta Live Tables framework” and the jobs as built on “Databricks Workflows”. That is why so many teams still use the old names.

the Lakeflow family in October 2026 and what each part used to be called: Lakeflow Connect ingests data, Lakeflow pipelines (formerly Delta Live Tables, then Lakeflow Declarative Pipelines and Lakeflow Spark Declarative Pipelines) turn it into tables, Lakeflow Jobs (formerly Databricks Workflows) runs the work, and Lakeflow Designer (GA since 16 June 2026) is the no-code canvas; all of it writes Unity Catalog tables

The pipeline product has had four names, and the key ideas list above covers what the rename means for existing DLT code (nothing has to change). Its declarative core is now open source as Apache Spark Declarative Pipelines, part of Apache Spark from version 4.1, and the pipelines documentation says Lakeflow pipelines “extend and are interoperable with” it while running on the Databricks Runtime.

How does data get in?

Lakeflow Connect is the ingestion part. The Lakeflow Connect page groups its managed connectors by source: databases through change data capture (CDC), which reads a database’s change log so that only inserts, updates and deletes travel, not whole tables; SaaS applications such as Salesforce and Workday; file sources such as Google Drive and SharePoint; and message buses. Each connector produces its own ingestion pipeline that runs on serverless compute and writes Unity Catalog streaming tables. Database connectors add an ingestion gateway and staging storage that capture changes continuously.

For Acme, the Google Cloud sources map like this:

Cloud Storage files. Auto Loader reads new files from a volume or external location exactly once. With file events turned on for the external location (on by default for new external locations since February 2026, and explained in Part 5), it learns about new files from notifications instead of listing the bucket.
Cloud SQL for MySQL. The MySQL connector names Google Cloud SQL for MySQL as a supported source; it entered Public Preview in December 2025, and a simpler integrated CDC pipeline for MySQL, without a separate gateway, became generally available on 15 October 2026. A PostgreSQL connector covers Cloud SQL for PostgreSQL.
Pub/Sub. Databricks has a built-in Pub/Sub source (read_pubsub in SQL). Part 8’s Pub/Sub section covers it, including the duplicates you must handle.
Datastream. Google’s CDC service can write change events into Cloud Storage, according to the Datastream overview; a pipeline can then read those files with Auto Loader and apply them with AUTO CDC. My view: use the Lakeflow Connect connector when it supports your database, and Datastream when another team already runs it.
Producers that push. Zerobus Ingest is “a push-based streaming API that writes data straight into Unity Catalog Delta tables”, with no message bus in between. It is offered in 8 of the 15 Google Cloud regions, europe-west2 included.

How does a pipeline turn files into tables?

The bottom line: a streaming table processes each new record once, as it arrives; a materialized view keeps a query result up to date. Bronze and silver are usually the first, gold the second.

A parcel depot shows the difference. A counter on the conveyor belt logs every parcel once as it passes and never looks back at parcels already logged. A whiteboard in the office shows the total per region, and someone updates it when new parcels arrive, ideally by adding the new ones rather than recounting the whole warehouse. The belt log is a streaming table, the whiteboard is a materialized view, and adding only the new parcels is incremental refresh.

A Lakeflow pipeline is a set of table definitions in SQL or Python. The pipeline concepts page lists three kinds of dataset:

A streaming table processes each new record exactly once, assuming the source only appends. It suits ingestion and bronze tables.
A materialized view is a stored query result that the pipeline keeps up to date. You write it as an ordinary SELECT, and the engine reprocesses “only new or changed source data when possible”.
A view is a temporary step that is not published to the catalogue.

Pipelines also give you expectations, data quality rules on each row (the default action, warn, keeps bad rows and counts them; DROP ROW removes them; FAIL UPDATE stops the run), and AUTO CDC, which applies a stream of change records as slowly changing dimension (SCD) type 1 (keep only the latest value) or type 2 (keep every version with validity dates), including records that arrive out of order.

Bronze, silver and gold are not three copies of the same data: Prerequisite 1 explains why. Bronze keeps what arrived, silver cleans and joins it, and gold aggregates it into a different shape with a different grain, as Acme’s pipeline shows.

Acme's supply pipeline with Lakeflow: a Lakeflow Jobs job starts when files land; Auto Loader reads delivery files from the landing volume, a Pub/Sub connector reads shipment events and Lakeflow Connect copies changes from a Cloud SQL for MySQL orders database into bronze streaming tables; silver tables apply expectations and AUTO CDC; the gold materialized view supply_service feeds an AI/BI dashboard and a Genie Agent

How does this apply to Acme? The belt logs are deliveries_raw and deliveries; the whiteboard is supply_service. The whole pipeline is one SQL file. The engine reads the references between tables and works out that bronze must refresh before silver, and silver before gold.

a Lakeflow pipeline in SQL for Acme: a bronze streaming table that reads JSON files from the landing volume with Auto Loader, a silver streaming table with two expectations, one dropping rows without a region and one failing the update on negative units, and the gold materialized view supply_service with automatic liquid clustering

One setting changes the economics. The serverless versus classic compute page says incremental refresh of materialized views “is available only on serverless pipelines”, and that materialized views on classic compute “are always fully recomputed”. Databricks recommends serverless “for almost all pipelines”, and Photon is always on there. Two recent additions help teams: pipeline parameters became generally available on 12 October 2026, so one SQL file can run against acme_supply in production and a dev catalogue, and since 5 October 2026 a pipeline can run as an account group instead of one person.

How do jobs run it, and how do you deploy it?

Lakeflow Jobs is the orchestrator. The Lakeflow Jobs page defines three things. A job is the unit you schedule, drawn as a directed acyclic graph (DAG) of tasks. A task is one step: a notebook, a pipeline refresh, a SQL query, a dashboard refresh, a dbt project and more, with if/else and for each control flow. A trigger decides when the job runs: a schedule, a continuous run, a table update, or a file arrival trigger that watches a volume or external location such as /Volumes/acme_supply/supply_bronze/landing/.

To keep jobs and pipelines in Git and promote them from dev to production, Databricks uses Declarative Automation Bundles, renamed on 16 March 2026 from Databricks Asset Bundles (DABs). The bundles page calls them “an infrastructure-as-code (IaC) approach”: a databricks.yml file declares the resources, and the Databricks CLI deploys them per target. A cut-down version for Acme (I have not run it):

bundle:
  name: acme_supply

resources:
  pipelines:
    supply_pipeline:
      name: supply_pipeline
      catalog: acme_supply
      schema: supply_silver
      serverless: true
      libraries:
        - file:
            path: ./src/supply_pipeline.sql
  jobs:
    acme_supply_nightly:
      name: acme_supply_nightly
      trigger:
        file_arrival:
          url: /Volumes/acme_supply/supply_bronze/landing/
      tasks:
        - task_key: refresh_supply
          pipeline_task:
            pipeline_id: ${resources.pipelines.supply_pipeline.id}

targets:
  dev:
    mode: development
  prod:
    mode: production

databricks bundle deploy -t prod then creates or updates both resources in the production workspace, typically from Cloud Build or GitHub Actions. Terraform remains the tool for the workspace, network, buckets and Unity Catalog objects underneath, as Parts 2, 3 and 5 show.

Lakeflow Designer is the fourth member. It is a drag-and-drop canvas, generally available since 16 June 2026, where analysts build visual data preps that are “backed by production-ready code and fully governed by Unity Catalog”, and move them to production with Git, jobs and bundles.

How do people consume the gold table?

Acme has three kinds of consumer for supply_service: planners who want a weekly dashboard, managers who want to ask a question, and one team that wants a small app. Databricks has a product for each, and each uses identity differently. The bottom line: always ask whose permissions a tool uses when it reads data: the author’s, or the person looking. The common misconception is that a dashboard viewer only ever sees what they themselves may see. By default, they do not.

A simple example. A colleague prints a report from the HR system using her own access and pins it on the noticeboard. Everyone who walks past reads what she could see, not what they could. If instead the noticeboard were a screen that asked each reader to sign in and showed only their own view, two people would see two different reports. The printed report is shared data permissions, the sign-in screen is individual data permissions (or, for apps, user authorization). Here is how each Acme consumer lines up.

AI/BI dashboards are Databricks’ built-in dashboards. They run their queries on a SQL warehouse, and once published they can be shared with anyone registered in your Databricks account, even people without access to the workspace. The setting to understand is the data permission chosen at publish time. The publish and share page says that with shared data permissions, the default, “viewers run queries using the publisher’s data permissions”, which “might expose data to users who have not been granted direct access to it”. With individual data permissions, viewers use their own. Compute access always uses the publisher’s credentials.

Genie is the family name for Databricks’ natural-language features, and it has been renamed twice recently. The Genie page lists three products:

Genie One (previously Databricks One): the simple interface for business users, with dashboards, apps and a chat that answers data questions.
Genie Agents (called Genie Spaces until 8 July 2026): a domain-specific chat that an analyst curates with Unity Catalog tables, example SQL, business definitions and instructions. Genie writes a read-only SQL query, runs it on the agent’s warehouse and returns a table or a chart.
Genie Code: the coding assistant inside notebooks, pipelines and dashboards.

The Genie Agents concepts page explains the identity split: compute uses “the compute credentials embedded by the agent author”, but “Unity Catalog evaluates data access as the end user’s own identity”, so row filters and column masks apply per person. On cost, Genie One and Genie Agents usage by users is free through 31 January 2027; service principals and Genie Code are billed.

whose identity each consumption tool uses: an AI/BI dashboard runs on a SQL warehouse with the publisher's credentials and, by default, the publisher's data permissions, which is the risky case; a Genie Agent runs on its warehouse with the author's compute credentials but Unity Catalog checks data access as the end user; a Databricks App acts either as its own service principal or as the signed-in user

Two Google Cloud links belong to Part 8 and get one sentence each here. Looker connects to Databricks SQL warehouses, and Part 8’s Looker section shows the settings. And Gemini Enterprise agents can call a Genie Agent through the Genie One MCP server, generally available since 25 September 2026; my post wiring Gemini Enterprise to Databricks Genie walks through it.

What are Databricks Apps?

The simple definition: a way to host a small web application next to your data without running a server. The formal one, from the key concepts page: “a web application that runs as a containerized service on the Databricks serverless platform”, built with frameworks such as Streamlit, Dash or Gradio, or with Node.js. Breaking that down:

Containerised service. Databricks packages your code and its dependencies into an isolated container and runs it. Each app gets a fixed URL of the form https://<app-name>-<workspace-id>.<region>.databricksapps.com.
Configuration. An app.yaml file says how to start the app and which environment variables it needs; resources such as a SQL warehouse, a job, a secret, a volume or a Genie Agent are declared once and bound at deployment.
Size and cost. Medium (up to 2 vCPUs, 6 GB) costs 0.5 DBU an hour and is the default; Large is 1 DBU and XLarge 3 DBU. Horizontal scaling across several instances behind one URL became generally available on 29 September 2026.
Identity. Every app has its own service principal. With app authorization, everything the app does uses that principal’s permissions, the same for every user. With user authorization, Databricks forwards the signed-in user’s token, so “if a table includes a row filter that limits visibility by region, the app only returns the rows that the user is allowed to query”.

# app.yaml for Acme's supply exceptions app (Streamlit)
command: ['streamlit', 'run', 'app.py']
env:
  - name: 'DATABRICKS_WAREHOUSE_ID'
    value: '<warehouse-id>'

Apps are available in 10 of the 15 Google Cloud regions, europe-west2 included. My view: use user authorization for anything that shows governed data, so the app’s service principal holds no standing access to acme_supply.

What is Lakebase?

The bottom line: Lakebase is a Postgres database for applications, next to the lakehouse, not a faster way to run analytics. On Google Cloud it is also still Beta and not offered in Acme’s region.

A supermarket shows the two kinds of database. Each till records thousands of small sales a day, one basket at a time, and every sale must either complete or not happen at all. Once a month, head office reads all of those sales at once to see which brand sold best. The tills are a transactional (OLTP) database; the monthly report is an analytical one. Copying the monthly totals back to the tills, so a cashier can look up one product’s sales instantly, is reverse ETL.

Everything so far is analytical: large scans, aggregates, a few writers. Applications need the other kind of database, transactional or OLTP (online transaction processing): many small reads and writes per second, each touching a handful of rows, with every write either fully done or not done at all. Lakebase is Databricks’ answer: “a fully managed Postgres database integrated into the Databricks platform”, according to the Lakebase page.

Its design separates compute from storage, which is what makes three of its features possible:

Autoscaling and scale to zero. Compute is measured in compute units (CU), where 1 CU has 2 GB of memory, and it grows with load and suspends when idle (after 60 seconds at the shortest, seven days at the longest), waking “in hundreds of milliseconds”.
Branches. A project holds one or more branches, each a copy-on-write copy of the database at a point in time, with its own read-write compute, up to six read replicas, roles and databases. A developer can branch production, test a schema change and throw the branch away without copying data.
Synced tables. A Unity Catalog table can be copied into Postgres and kept current by a managed Lakeflow pipeline, in snapshot, triggered or continuous mode, so an application can look up one row in milliseconds. The documentation calls this reverse ETL.

The project page supports Postgres 16, 17 (the default) and 18, up to 500 branches per project and 1,000 projects per workspace. A project is created in the workspace’s region, and that region cannot be changed.

Lakebase on Google Cloud in November 2026: a managed Postgres, in Beta on Google Cloud as of 15 June; a project holds branches, each branch has one read-write compute, optional read replicas, Postgres roles and databases; synced tables copy Unity Catalog tables such as supply_service into Postgres in snapshot, triggered or continuous mode; as of November 2026 it runs in us-east4, us-central1, us-west1, europe-west3 and asia-southeast1, so Acme's europe-west2 workspace cannot create one

Now the status, which matters more than the features for Acme. As of October 2026, the Lakebase pages said it had been “available in Beta on GCP” since 15 June, in us-east4, us-central1 and europe-west3. As of 8 November 2026 the project management page lists five regions: us-east4, us-central1, us-west1, europe-west3 and asia-southeast1. One honest gap: those pages no longer show the Beta wording, but Databricks has not announced general availability in its release notes, so I still treat it as Beta. Acme’s workspace is in europe-west2, which is not on the list, so it still cannot create a Lakebase project.

My view, as a Google Cloud architect: Lakebase is Postgres, and so are Cloud SQL for PostgreSQL and AlloyDB. Choose by where the application and its data live. For an existing Google Cloud application, or one that needs a GA service in London today, Cloud SQL or AlloyDB is the answer. When the app is a Databricks App or an agent whose state belongs next to Unity Catalog, and the region is supported, Lakebase removes a sync job and a separate set of credentials. As a litmus test:

1. Is it analytics (scans, aggregates)?                     -> a SQL warehouse, not a database
2. Existing Google Cloud app, or GA needed in London today? -> Cloud SQL for PostgreSQL or AlloyDB
3. Databricks App or agent state, in a supported region?    -> Lakebase, with synced tables

How do you build agents on Databricks?

An agent is an AI system that runs a loop: it asks a large language model what to do next, calls a tool, looks at the result and repeats until it can answer. The What is Agent Bricks? page lists the parts of a production agent: the agent loop (your code, written with a framework such as LangGraph or the OpenAI Agents SDK), a model, tools, agent state (the current session and long-term memory), an agent server that exposes the loop over HTTP, a runtime that hosts it, observability, and governance.

Agent Bricks is “the Databricks agent developer platform”, and in October 2026 it is aimed at people who write code. Its pieces map onto that list:

Agent Runtime runs your agent server on Databricks Apps. Each deployment is an app named agent-bricks-<name> with its own URL, runs as the app’s service principal, and scales from 1 to 5 instances. DurableAgentServer keeps each run’s state so a run survives a restart.
Agent memory and sessions keep conversation state and what the agent learned (managed memory entered Beta in September 2026).
MLflow Tracing records every step, with evaluation and LLM judges on top.
The Agent Bricks CLI (databricks-agentbricks, Beta since 30 September 2026) scaffolds a project from a framework template, runs it locally and deploys it.

agentbricks dev                          # run the agent locally
agentbricks tools add genie-agent        # give it Acme's supply Genie Agent as a tool
agentbricks --profile acme-prod deploy supply-agent --instances 2

(I have not run these commands; they follow the CLI and Agent Runtime pages.)

For agents without code, Databricks now points to Genie Agents. On 12 October 2026 the two earlier no-code products, Knowledge Assistant (document question answering with citations) and Supervisor Agent (which coordinated several agents and tools), became legacy. The legacy offerings page says not to use them for new agents and maps each to Genie Agents or the CLI. Older material also uses the name Mosaic AI for these capabilities, for example Mosaic AI Model Serving in release notes up to April 2026, and AI Search was called Vector Search until 1 June 2026.

Every model and tool call goes through Unity Gateway, “the Databricks governance solution for enterprise AI, part of Unity Catalog”, generally available since 4 August 2026 (release notes from February to April 2026 call it AI Gateway). The Unity Gateway page describes four jobs: control which AI services teams can use, route traffic across providers with rate limits and fallbacks, govern MCP servers, and monitor usage and cost. Models, MCP services and the functions agents call are Unity Catalog securables, granted like tables. Gemini models are among the models it serves; Part 8’s Gemini section covers how, and the cross-geography routing they need in europe-west2.

agents on Databricks in October 2026: Genie Agents for agents without code and the Agent Bricks CLI for agents in code, deployed to Agent Runtime on Databricks Apps; both go through Unity Gateway, which governs model services, model providers, MCP servers, skills and service policies; underneath sit Unity Catalog, AI Search, Lakebase for agent memory and MLflow Tracing; Knowledge Assistant and Supervisor Agent are legacy since 12 October 2026

Which of these can Acme use in europe-west2?

Databricks on Google Cloud runs in 15 regions, and the features with limited regional availability page is the one to check before any design review. Two terms on it need a definition. A Geo is a group of regions that Databricks uses for data residency. Cross-geography routing is a setting that lets some AI features send requests to models hosted in another Geo; where a feature is marked as needing it, it does not work in that region unless an admin turns the setting on.

feature availability for Acme's region, europe-west2, in October 2026: serverless compute, Databricks Apps, managed connectors, Zerobus Ingest, predictive optimization, AI Search, custom agents, Unity Gateway and Lakehouse Real-Time are available; Foundation Model APIs and the legacy Knowledge Assistant and Supervisor Agent need cross-geography routing; MLflow traces in Unity Catalog and Lakebase are not available

The short version for Acme: serverless compute, SQL warehouses, Lakeflow, predictive optimization, Databricks Apps, custom agents and Unity Gateway are all available in London. The Designated Services page adds that Genie Agents chat and the AI features of AI/BI dashboards are processed in the United Kingdom when cross-geography processing is turned off. Foundation Model APIs need cross-geography routing, which an account admin must allow. Lakebase and MLflow traces in Unity Catalog are not offered there.

What can go wrong?

troubleshooting the platform on Google Cloud: symptom, likely cause and fix for warehouses that do not start or come out classic, Photon not helping, MERGE conflicts, clustering with no effect, dashboards showing too much, Lakebase missing in europe-west2, materialized views that fully recompute, and the legacy Supervisor Agent

A few of these deserve a sentence more.

Trusting the API defaults. Warehouses created through the API are classic unless you say otherwise, and clusters created through the Clusters or Jobs API have Photon off unless runtime_engine is PHOTON. Put both in your Terraform modules and bundles.
Clustering on the wrong columns. A clustering key without statistics does nothing, and statistics stop at column 32 by default. Keep filter columns early in the schema, or use CLUSTER BY AUTO with predictive optimization.
Publishing dashboards with the default permissions. Shared data permissions are convenient and quietly widen access. For anything with row filters, publish with individual data permissions.
Building pipelines on classic compute. Materialized views fully recompute on classic compute, so a “cheap” classic pipeline can cost more than serverless for the same gold table.
Following an old guide. Delta Live Tables, Databricks Workflows, Genie Spaces, Vector Search, AI Gateway, Knowledge Assistant and Supervisor Agent all appear in material from 2025 and early 2026. Map each to its October 2026 name or replacement before you build.
Designing around a Beta in the wrong region. Lakebase and Lakehouse Real-Time are Beta, and Lakebase is not in europe-west2. Do not put either on a production critical path yet.

What would I do now?

Read the Databricks components page (it used to be called the concepts page) once from top to bottom. It is the glossary for everything in this series.
Sketch your own version of the account diagram in Part 1: which Google Cloud billing account, which regions, how many workspaces and which project each one uses. Then check every product you plan to use against the regional availability page for those regions.
Decide, per workload, whether it should run on classic or serverless compute, with Part 1’s billing split in mind. For pipelines with materialized views and for SQL, start from serverless.
Turn on predictive optimization for your main catalogue, set CLUSTER BY AUTO on the gold tables, and pin classic job clusters to the newest LTS runtime.
Move one existing DLT pipeline and its job into a Declarative Automation Bundle with a dev and a prod target, so the next change goes through Git.
If your users already work in Gemini Enterprise, read how a Databricks Genie Agent can answer questions inside Gemini Enterprise.

What did we learn?

The Databricks Runtime is a versioned Spark image with a Beta, a six-month GA phase and three years of LTS, while serverless compute replaces the runtime choice with an environment version.
Photon is a vectorised C++ execution engine inside the runtime that speeds up supported operators, falls back to Spark for the rest, and is always on for SQL warehouses and serverless compute.
Queries get fast mostly by reading less: data skipping uses per-file statistics, liquid clustering and predictive optimization keep those statistics useful, and the disk cache keeps hot files on local SSD.
Serverless SQL warehouses start in seconds and scale with intelligent workload management, while pro and classic warehouses run in your project and need N2 quota.
Lakeflow is Connect, pipelines (formerly Delta Live Tables), Jobs (formerly Databricks Workflows) and Designer, and serverless pipelines are the only ones that refresh materialized views incrementally.
Dashboards, Genie Agents, Apps, Lakebase and Agent Bricks each handle identity and availability differently; in europe-west2, Lakebase is not available and the Knowledge Assistant and Supervisor Agent are legacy.

What comes next?

This post stayed inside Databricks. In Part 8: Better together, BigQuery, Gemini and Google Cloud services, we connect it to the rest of Google Cloud: the four ways Databricks and BigQuery read each other’s tables, turning on Iceberg reads for supply_service, Gemini models inside and outside Databricks, streaming from Pub/Sub, Looker and OpenSharing. The pipeline and gold table built here are the ones BigQuery reads there.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands and its data are fictional.