Databricks on Google Cloud, Part 10: Migration to Google Cloud

Nine parts ago we started with an empty Google Cloud organisation. Since then we have subscribed to Databricks, created projects and a workspace, wired up identity, networking, storage and governance, used the platform, connected it to BigQuery and Gemini, and learnt to pay for it and run it. All of that assumed Acme was starting fresh on Google Cloud.

Many teams are not. They already run Databricks, usually on AWS or Azure, sometimes beside an older on-premises Hadoop cluster, and they want to move it. The Databricks platform is the same on every cloud, so most of their skills and code carry over. The parts that do not carry over are exactly the parts this series has covered: projects, service accounts, networks, buckets and billing.

This last part is about the move itself. I compare the three clouds, sort what moves easily from what does not, walk through a five-step approach with the real tools, show three routes for the data, give a translation cheat sheet for code and configuration, draw two reference architectures, and list the problems I see most often. It ends with a recap of the whole series. All status claims are as of October 2026, and the ones that changed since are updated and dated (last checked 8 November 2026).

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: how to move Databricks from AWS to Google Cloud, in three minutes. The narration is a synthetic voice; the script and animation are mine.

What are the key ideas?

Six terms carry this post.

Migration. Moving a running workload, with its data, code, schedules and permissions, from one platform to another, so that the new platform takes over and the old one can be switched off.
Source and target. The source is where the workload runs today (here, Databricks on AWS); the target is where it will run (Databricks on Google Cloud).
Wave. A group of workloads that move together. Migrating in waves means moving one self-contained group at a time, learning from it, and then moving the next, instead of moving everything in one weekend.
Hydrate. Filling a new, empty Unity Catalog metastore with the catalogs, schemas, tables, volumes and grants that existed in the old one. The data is copied; the metadata is recreated.
Parallel run. A period during which the same pipelines run on both source and target, and their outputs are compared, before anyone relies on the target.
Cut-over. The moment consumers (dashboards, applications, people) switch from the source to the target, which then becomes the system of record: the copy everyone treats as correct.

To make this concrete, suppose that before Part 1, Acme’s supply team had built its lakehouse on Databricks on AWS, in the eu-west-2 (London) region, with a workspace called acme-supply-aws-prod and its data in s3://acme-supply-lake. This post moves that workload to the workspace you already know, acme-prod-euw2-supply in europe-west2, with its data in gs://acme-lake-supply.

the migration at a glance: Acme's supply workspace on Databricks on AWS moves to Databricks on Google Cloud in five steps, assess, design, build, migrate, and validate and cut over; data, code, jobs, models and permissions travel, while storage paths, identities, network and billing change underneath

How is Databricks on Google Cloud different from AWS and Azure?

The Databricks platform is the same on all three clouds; what differs is everything around it: the account, the metastore, the storage and identity plumbing, and the tier rules.

Think of moving to another country with the same model of car. You drive it exactly as before, but it needs new number plates, a new registration, new insurance and, in some countries, a different inspection to be allowed on certain roads. The driving is the Databricks workspace, notebooks and Spark; the plates and registration are the new Databricks account and metastore; the insurance is the service accounts and buckets; and the inspection is the tier a feature needs.

The platform is the same on all three clouds, and so is the split into a control plane and compute planes: compare the Databricks on AWS architecture page with Microsoft’s What is Azure Databricks?, and Part 1 explains both planes on Google Cloud. What changes is the cloud underneath and the commercial wrapper.

On Azure, the product is called Azure Databricks, and Databricks’ pricing page notes that its charges “are billed directly by Microsoft”. On Google Cloud, the product launched in February 2021, when Google Cloud and Databricks announced “a new partnership to deliver Databricks at global scale on Google Cloud”, and you subscribe through Google Cloud Marketplace (Part 1 walks through it). As of October 2026, Databricks lists 15 Google Cloud regions, and its regional table shows serverless compute for notebooks, jobs, pipelines and SQL warehouses in all 15.

Databricks on the three clouds: the same platform and the same control plane and compute plane design; what changes is who bills you, the object storage and the virtual machines underneath

The cloud vocabulary itself (account, subscription, project, VPC, IAM role, service account) is mapped in Prerequisite 2, and where a workspace lands on each cloud is drawn in Part 1. For a migration, four further differences matter more than any name. How does this apply to Acme? Each one below is a new set of plates or papers for acme-supply-aws-prod on its way to acme-prod-euw2-supply.

A new account and a new metastore. Your Google Cloud workspaces live in the Databricks account that the Marketplace subscription creates, not in the account that holds your AWS or Azure workspaces. And a Unity Catalog metastore (the top-level container for catalogs, schemas, tables and grants) is regional: Databricks says “you must have one metastore for each region in which your organization operates”. So the target always starts with an empty metastore in europe-west2, which you hydrate.
A new workspace starts without legacy features. Databricks’ legacy features page says that “starting October 8, 2026, Databricks provisions all new workspaces in all accounts without access to legacy features”, and the account setting that controlled this is now deprecated, “enabled and locked” (checked 8 November 2026). Those features are DBFS root and mounts (DBFS, the Databricks File System, is a file-system view over object storage; a mount maps a bucket to a path such as /mnt/raw), the Hive metastore (the per-workspace table catalog that predates Unity Catalog), no-isolation shared clusters, and Databricks Runtime versions before 13.3 LTS (LTS means long-term support). Every Google Cloud workspace you build for the migration is new, so there is no switch to turn them back on, and a source workspace that depends on them is the biggest single source of rework I see. Part 5 explains DBFS and why it is legacy.
Different storage and identity plumbing. Paths change from s3:// or abfss:// to gs://, and Unity Catalog reaches Cloud Storage through a Google service account that Databricks creates (Part 2), not an AWS IAM role or an Azure managed identity.
Different tier boundaries. On Google Cloud, a customer-managed VPC needs the Premium tier and Private Service Connect needs the Enterprise tier (Part 3 and Part 9 explain both). Check that the tier you buy matches the controls your security team expects from the source.

Can I just move my workspace?

This is the first question I am asked, and the answer is no. You do not move a workspace; you build a new one on Google Cloud and move its contents.

Moving house abroad works the same way. Nobody ships the house. You take the boxes, the books and the furniture to a new home. Some things travel well, because they are self-contained: a box of books arrives as it left. Some do not, because they were built into the old house: a fitted kitchen has nowhere to go, and a bookmark that says “page 212 of the copy on the hall shelf” means nothing once that copy stays behind. The house is the workspace, the boxes are Delta tables and code, the fitted kitchen is the Hive metastore and DBFS mounts, and the bookmark is a streaming checkpoint.

I know of no documented way to move a workspace from one cloud to another; a workspace’s cloud, project and region are fixed when it is created (Part 1). A migration builds a new workspace on Google Cloud and moves its contents. Some contents travel easily, and some need real work. Here is how a workspace’s contents sort into boxes and fitted kitchens:

what moves easily and what does not: Delta data, code and notebooks move easily; Unity Catalog objects, grants, jobs and MLflow assets take some work; secret scopes are re-created by hand; Hive metastore tables and streaming checkpoints are the hard parts

Three things make the easy rows easy. Delta tables are open files in object storage (a Parquet data file per chunk and a _delta_log folder of commits, as Prerequisite 1 shows), so they can be copied like any other files. The Delta protocol stores each data file’s path as “a relative path to a data file from the root of the table” (or an absolute path), so a copied table usually still finds its files at the new location. And notebooks and code live in Git, or can be exported with the Databricks CLI.

The hard rows are hard because they depend on something that does not exist on the target. Tables in the Hive metastore have no Hive metastore to land in. A streaming checkpoint (the folder where Spark Structured Streaming records which input it has already processed) records the source it read; the Apache Spark streaming guide says that changing a query’s input sources between restarts “is not allowed”. A stream that read s3:// yesterday needs a new checkpoint when it reads gs:// today.

How does this apply to Acme? Acme’s boxes are the 46 tables in its Unity Catalog catalog and the notebooks in Git: they travel as they are. Its fitted kitchen is the nine tables still in the Hive metastore and the three DBFS mounts that step 1 below turns up, which are fixed on AWS before anything moves. And any stream it runs, such as Auto Loader ingestion, needs a new bookmark: a fresh checkpoint on the gs:// side.

How do you run a migration, step by step?

Plan once, then move in small waves, each proven before the next. The common mistake is the big-bang weekend: everything copied at once, and every problem found at once, by the business. Moving house room by room is the everyday version: you set up the new place once, then move the kitchen, live with it for a few days, fix what is missing, and only then move the next room. For Acme, I would make the first wave one low-risk pipeline, as the checklist at the end recommends.

I use five steps. The first three happen once; the last two repeat for every wave.

a five-step migration approach: assess the source estate, design the Google Cloud landing zone and Unity Catalog layout, build it with Terraform, migrate data, code, jobs, models and grants in waves, then validate and cut over after a parallel run; the tools for each step sit underneath

Step 1: Assess what you have

You cannot plan a move without an inventory: every workspace, job, cluster policy, table, path, secret scope, model and group, and who depends on each. Three tools do most of the work.

UCX, the Unity Catalog migration assistant from Databricks Labs. Its assessment workflow “retrieves, or crawls, details of workspace assets and securable objects in the Hive metastore” and stores the findings in an inventory database with a dashboard. Note the support position on its README: Databricks Labs projects “are not formally supported by Databricks with Service Level Agreements (SLAs)”. You install it with databricks labs install ucx and start the assessment with databricks labs ucx ensure-assessment-run (I have not run these on Acme’s estate).
System tables, the read-only tables in the system catalog where Databricks records usage, lineage and job history (system tables reference). They answer the two questions UCX does not: which jobs cost the most, and which ones read storage paths directly instead of tables.
The Terraform resource exporter, part of the Databricks Terraform provider, which “generates *.tf files for Databricks resources and import.sh to run import state” (experimental resource exporter). It is labelled experimental, and its guide asks you to review every generated file.

These two queries, run on the source workspace, find the work that the storage-path change will create. The column names come from the lineage and billing system table references.

-- Which notebooks, jobs and pipelines read s3:// paths directly? (last 90 days)
SELECT entity_type, entity_id, source_path, count(*) AS reads
FROM system.access.table_lineage
WHERE source_path LIKE 's3://%'
  AND event_date >= current_date() - INTERVAL 90 DAYS
GROUP BY ALL
ORDER BY reads DESC;

-- Which jobs used the most DBUs in the last 30 days?
SELECT usage_metadata.job_id, sum(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_metadata.job_id IS NOT NULL
  AND usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY ALL
ORDER BY dbus DESC
LIMIT 20;

The first list is your code-change backlog. The second tells you which jobs deserve a careful benchmark on the target, because a DBU (Databricks Unit, the unit Databricks bills compute in) spent on a big job matters more than one spent on a small one. Here is what Acme’s assessment found:

Acme's source estate on AWS before the move, as the assessment found it: two workspaces, one Unity Catalog catalog with 46 tables, nine tables still in the Hive metastore, 31 jobs, three DBFS mounts, two secret scopes and four registered models, with where each one lands on Google Cloud

The two highlighted rows are the blockers. Nine tables still sit in the Hive metastore, and three DBFS mounts are still in use. Neither exists in Acme’s new Google Cloud workspaces, so both are fixed on AWS, before anything moves.

Step 2: Design the target

Design is where the rest of this series pays off, because every design decision has a part of its own. Settle these before you build:

Projects and region: which projects hold the workspace, the network and the data, and in which region (Part 1).
Identity: how people sign in and how groups are provisioned from Microsoft Entra ID through SCIM (System for Cross-domain Identity Management, the standard for syncing users and groups) (Part 2).
Network: subnet sizes, Cloud NAT, Private Service Connect and VPC Service Controls (Part 3, and Part 4 for serverless).
Storage and catalogs: buckets, external locations and managed storage (Part 5), and the catalog and grant design (Part 6).
Tier: Premium or Enterprise, decided by the controls you need (Part 9).

My view: keep the catalog, schema and table names the same as on the source. Code that reads acme_supply.supply_gold.supply_service by name then moves without a single edit, and the storage-path change is confined to the places that still use paths.

Step 3: Build the landing zone

Build the target with Terraform, using two providers side by side: the Google provider for projects, the VPC, Cloud NAT and buckets, and the Databricks provider for the workspace, metastore, storage credential and external locations. Part 1 and Part 2 show the resources and how Terraform authenticates, and Part 2’s step-by-step list maps each step of a workspace deployment to its Terraform resource. The exporter output from step 1 helps here: it is a starting point for jobs, cluster policies and permissions, once the AWS-specific values are translated.

Step 4: Migrate in waves

Each wave moves data, metadata, code, jobs, models, secrets and grants, in roughly that order. The next sections take each one in turn.

Step 5: Validate and cut over

The last step proves that the target produces the same answers as the source, then moves consumers. It has its own section below.

How does the data move?

Copy whole buckets when you are moving a lot at once; share or read table by table when you want governance kept in Unity Catalog on both sides.

Think of a library that is moving across town. It can hire a removal van and move every shelf in one go, sorting the catalogue out at the other end. It can lend books one at a time through an inter-library loan, each one checked out and logged. Or the new library’s readers can be given a visitor’s card for the old building, and copy what they need. The van is Storage Transfer Service, the inter-library loan is OpenSharing, and the visitor’s card is a read-only S3 external location.

There are three documented routes for table data from AWS, and each suits a different case.

three documented routes for moving table data from AWS to Google Cloud: Storage Transfer Service copies the files from S3 to Cloud Storage; OpenSharing shares tables from the AWS metastore to the Google Cloud metastore, where CREATE TABLE AS SELECT copies them; a read-only S3 external location lets the Google Cloud workspace read the source files and copy them; all three end as managed tables in gs://acme-lake-supply

Route A: Storage Transfer Service. Storage Transfer Service is Google Cloud’s managed service for bulk copies, and it supports “Amazon S3, Azure Blob Storage, or Cloud Storage to Cloud Storage”, as well as on-premises file systems and HDFS (the Hadoop Distributed File System). It copies objects, so it moves whole buckets or prefixes at once, Delta tables included, and you register or clone the tables afterwards. Agentless transfers from S3 have no Storage Transfer Service charge, according to its pricing page; an optional Google-managed private network for S3 costs US$0.04 per GiB within the EU, on top of the usual network charges.

Route B: OpenSharing, Databricks to Databricks. OpenSharing, formerly Delta Sharing, is the Databricks sharing platform, and its Databricks-to-Databricks protocol shares tables with any Databricks user “regardless of account or cloud host”, as long as they use a Unity Catalog workspace. Acme creates a share in the AWS metastore and a recipient for the Google Cloud metastore; the Google Cloud workspace then sees the share as a read-only catalog, and CREATE TABLE ... AS SELECT (CTAS) writes a managed copy. This route moves table by table, keeps governance in Unity Catalog on both sides, and needs no bucket-level access. Part 8 covers OpenSharing in depth.

Route C: a read-only S3 external location. Databricks on Google Cloud “supports cross-platform read-only access to S3 buckets”. You create a storage credential from an AWS IAM role and an external location on the S3 bucket in the Google Cloud metastore, following Connect to an AWS S3 external location (read-only). The Google Cloud workspace can then read the source files and CTAS them into managed tables. It suits AWS sources only; there is no equivalent for Azure storage in that list.

All three routes read data out of AWS, and AWS charges for data transfer out. AWS’s own post Free data transfer out to internet when moving out of AWS says you “can contact AWS Support to ask for free DTO rates” for data being migrated, so ask before the first large copy.

A word on consistency. Storage Transfer Service copies objects one at a time, and a Delta commit is only valid if the data files it names are present. My view: copy while writers are paused, or finish with a final incremental transfer after the last write, so that no commit arrives without its files.

Which route? My order of questions, simplest first:

1. Many tables, whole buckets or prefixes, writers can pause?   -> A  Storage Transfer Service
2. Source is Databricks with Unity Catalog, you want table by
   table moves with no bucket access?                           -> B  OpenSharing, then CTAS
3. Source is S3 and you want to read the files in place
   before copying?                                              -> C  read-only S3 location
Azure source? Only A and B apply; C covers S3 only.

How does this apply to Acme? The next section follows Acme through Route A, the removal van: the files from s3://acme-supply-lake land in gs://acme-lake-supply first, and the catalogue is rebuilt behind them.

How do you hydrate the new metastore?

Copied files are not yet tables: the metastore has to be told where they are, who may read them and which ones it should own. The trap is to see the files arrive in the bucket and call the data migrated.

Back at the library, the removal van delivers boxes of books, but no reader can find one until the books are shelved, entered in the catalogue and given lending rules. The boxes are the copied Delta files, the catalogue entries are the external and managed tables in Unity Catalog, and the lending rules are the grants.

How does this apply to Acme? For Route A, the files arrive first and the metadata follows. These are the six steps, with Acme’s names:

hydrating the Google Cloud Unity Catalog metastore in six steps: copy the files, map old paths and names to new ones, create the storage credential and external locations, register external tables on the gs:// paths, deep clone managed tables into managed storage, and validate counts, checksums, grants and lineage

Steps 3 to 5 in SQL look like this. I have not run these against Acme’s buckets; the syntax follows the external locations and CREATE TABLE CLONE references.

-- 3. Credential and external location (the credential is created in Catalog Explorer;
--    Databricks generates its Google service account, which you grant on the bucket)
CREATE EXTERNAL LOCATION acme_lake_supply
  URL 'gs://acme-lake-supply/'
  WITH (STORAGE CREDENTIAL acme_lake_cred);

-- 4. Register a copied table directory as a staging external table
CREATE TABLE acme_supply.stage.shipments
  LOCATION 'gs://acme-lake-supply/stage/silver/shipments';

-- 5. Deep clone it into the catalog's managed storage
CREATE TABLE acme_supply.supply_silver.shipments
  DEEP CLONE acme_supply.stage.shipments;

A deep clone copies a table’s data and metadata into a new, independent table. The clone documentation lists what it does not copy: “the table description, user-defined commit metadata, Delta Lake table history, and Unity Catalog properties, such as tags”. Streaming tables and materialized views cannot be cloned at all; their pipelines are redeployed and rebuild them on the target. Re-running CREATE OR REPLACE TABLE ... DEEP CLONE syncs a clone incrementally, which helps during the parallel run.

Before any of this, the Hive metastore tables must already be in Unity Catalog on the source. The upgrade guide offers the upgrade wizard and the SYNC command for external tables, and says that “you must use CLONE or CREATE TABLE AS SELECT (CTAS) if you want to copy existing Hive tables to Unity Catalog as managed tables”. UCX automates the same steps at scale.

What changes in the code and configuration?

Code that names tables moves untouched; code that names storage paths has to be rewritten. Picture a letter addressed “to Priya Shah, care of the Finance team”: it reaches her wherever the team moves. A letter addressed to “desk 4, third floor, 10 Old Street” goes to an empty desk the day after the office moves. Table names and Unity Catalog volumes are the first kind of address; s3://, abfss:// and /mnt/ paths are the second.

Most code changes are translations: the same idea under a different name. This cheat sheet covers the ones that touch Databricks on all three clouds. For security features, the Google Cloud versions are in Part 6 (keys), Part 3 (private connectivity) and Part 4 (perimeters).

a translation cheat sheet for moving a Databricks workspace from AWS or Azure to Google Cloud: storage paths, bucket layout, Unity Catalog storage credentials, secret scopes, Auto Loader file events, streaming, orchestration, private connectivity, cluster attribute blocks, spot settings and an example node type

A few rows deserve a sentence each.

Paths. abfss is the URI scheme of the Azure Blob File System driver, in the form abfss://<container>@<account>.dfs.core.windows.net/<path> (Azure Databricks storage docs). Both it and s3:// become gs://. Better still, replace paths with table names or Unity Catalog volumes (governed folders of files, addressed as /Volumes/<catalog>/<schema>/<volume>/).
Bucket layout. ADLS Gen2 has a hierarchical namespace (HNS: real directories, with atomic renames). The Google Cloud equivalent exists, but the external locations page says “Google Cloud Storage hierarchical namespace (HNS) is not supported with external locations”. Map ADLS Gen2 to flat buckets (Part 5).
Secrets. On Azure, a secret scope can be backed by Azure Key Vault. The Google Cloud secrets page offers two places to keep secrets: workspace secret scopes, which Databricks stores for you, and secrets in Unity Catalog, “available in every workspace attached to the metastore”. So Key Vault-backed scopes are recreated as workspace secret scopes with the same names, or as Unity Catalog secrets, and the values re-entered. Part 6 covers the other options.
File events. In file notification mode, Auto Loader (the Databricks feature that incrementally ingests new files) subscribes to events from the bucket instead of listing it. The file notification page shows the services it uses: SNS and SQS on S3, Event Grid and Queue Storage on Azure, and Pub/Sub on Cloud Storage.
Orchestration. Lakeflow Jobs (formerly Databricks Workflows) runs Databricks tasks (Lakeflow Jobs); Cloud Composer, Google’s managed Apache Airflow (overview), replaces Step Functions, Amazon MWAA (Managed Workflows for Apache Airflow) or Azure Data Factory when the pipeline spans other systems.
Clusters. Each cloud has its own attributes block in a cluster definition. The Terraform cluster reference lists availability values of SPOT_WITH_FALLBACK on AWS, SPOT_WITH_FALLBACK_AZURE on Azure and PREEMPTIBLE_WITH_FALLBACK_GCP on Google Cloud. instance_profile_arn has no Google Cloud equivalent, because data access goes through Unity Catalog. Part 1 maps node types by shape and explains the gcp_attributes block.

How does this apply to Acme? Here is one of Acme’s ingestion notebooks, before and after. The first version uses the desk address, the second the name. The second version runs on any cloud, because it names tables and volumes instead of buckets.

# Before, on AWS: a DBFS mount and a path
orders = spark.read.format("delta").load("/mnt/raw/erp_orders")
erp_key = dbutils.secrets.get("erp", "api-key")

# After, on Google Cloud: a table name, a volume and the same secret scope name
orders = spark.read.table("acme_supply.supply_bronze.erp_orders")
erp_key = dbutils.secrets.get("erp", "api-key")

landing = "/Volumes/acme_supply/supply_bronze/landing/erp/"
(spark.readStream.format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.useNotifications", "true")   # Pub/Sub on Cloud Storage
    .load(landing))

Workspace code moves with Git. Databricks Git folders connect a workspace folder to a repository, so the target workspace simply clones the same repository. Code that is not in Git can be moved with the Databricks CLI’s workspace export-dir and workspace import-dir commands (CLI commands). Jobs and pipelines are best redeployed from Declarative Automation Bundles (formerly Databricks Asset Bundles), which describe “Databricks resources such as jobs and pipelines as source files”; a bundle needs only a new target block and new node types. If there are no bundles yet, the Terraform exporter’s output from step 1 is the fallback.

How do models, grants and secrets move?

Models are exported and imported; grants and secrets do not travel at all: they are replayed or re-entered on the target. When you move house, the furniture goes in the van, but the keys do not: you change the locks and hand out new keys, and you write a new list of who may let themselves in once you know who your neighbours are. The furniture is the MLflow models and experiments, the new locks and keys are re-entered (ideally rotated) secrets, and the list is the grants, replayed once the groups exist.

How does this apply to Acme? Acme’s four registered models, two secret scopes and its grants each take one of those paths.

Models and experiments. MLflow Export Import is an open-source tool that copies “MLflow objects (runs, experiments or registered models) from one MLflow tracking server (Databricks workspace) to another”. It exports to an intermediate directory, then imports, and supports both the workspace model registry and the Unity Catalog model registry.

moving MLflow assets with mlflow-export-import: the tool exports experiments, runs and registered models with their artifacts from the AWS workspace to an intermediate directory, then imports them into the Google Cloud workspace, where artifacts land in Cloud Storage

Its bulk tools are export-all and import-all. A sketch, not run here:

# On the AWS side
export MLFLOW_TRACKING_URI=databricks
export DATABRICKS_HOST=https://<aws-workspace-url>
export-all --output-dir /tmp/mlflow-export

# On the Google Cloud side
export DATABRICKS_HOST=https://<gcp-workspace-url>
import-all --input-dir /tmp/mlflow-export

A registered model in Unity Catalog has a three-part name such as acme_supply.ml.otif_forecast, so if the target catalog and schema have the same names, the model can keep its name. Grants on it do not travel, and model serving endpoints, which belong to the source workspace, are recreated on the target.

Grants. A grant lives in the source metastore, so it is replayed on the target. The information schema views such as table_privileges list who holds which privilege on which table, which makes them a good input for a script that issues GRANT statements on the target. The groups must exist first: provision them into the new account through SCIM from Microsoft Entra ID (SCIM provisioning), and the UCX assessment needs matching account groups too.

Secrets. Recreate each scope with the same name, and re-enter the values from the system that issued them. This is also a good moment to rotate them.

How do you validate and cut over?

Validation is where migrations earn trust. My view: never cut over on the strength of a successful copy; cut over on the strength of the same answers.

A shop that installs a new till does the same thing. For a week, every sale goes through both the old till and the new one, and at closing time someone compares three numbers: how many sales, the total takings, and a fingerprint of the receipts. If all three match every night, including the busy Saturday, the old till goes. The two tills are the source and target, the week is the parallel run, and the three numbers are a row count, a sum and a sum of row hashes.

cutting over one wave: an initial copy, incremental syncs, a parallel run in which both platforms produce the same tables and are compared every day, a switch of consumers to Google Cloud, a freeze of the source and finally its decommissioning

The parallel run is the heart of it. During it, the source stays the system of record, the target runs the same jobs on the same inputs, and a comparison job checks the outputs every day. Run it long enough to cover every schedule at least once, including weekly and month-end jobs. Then switch consumers, make the source read-only, and decommission it after sign-off.

How does this apply to Acme? Acme compares its gold table, acme_supply.supply_gold.supply_service, which has one row per brand, region and week. A spot check of three rows, and a whole-table check per week:

validating supply_service after the copy: a spot check of three rows (brand Acme Pure, region Western Europe, three weeks) that match on both platforms, and a whole-table check per week that finds one missing row on Google Cloud for the week of 18 May 2026; supply_service has one row per brand, region and week and is owned by supply

The whole-table check is two aggregate queries, one on each side, compared by a small job. On the target it looks like this; the source runs the same query against its own copy.

SELECT week_start,
       count(*)             AS row_count,
       sum(units_delivered) AS sum_delivered,
       sum(hash(brand, region, units_ordered, units_delivered, otif_pct)) AS row_hash
FROM acme_supply.supply_gold.supply_service
WHERE week_start >= '2026-05-04'
GROUP BY week_start
ORDER BY week_start;

Counts catch missing rows, sums catch changed numbers, and a sum of row hashes catches a change in any column. In Acme’s case the week of 18 May was one row short, because a late file landed in S3 after the last sync. A re-sync fixed it, which is exactly the kind of gap a parallel run exists to find.

Performance belongs in validation too. Compare run times and DBUs for the top jobs from step 1 using system.lakeflow job tables on both sides, and expect to retune node types and cluster sizes for a few of them.

What do reference architectures look like?

The first is Acme after the move from AWS. Every name is one you have met in Parts 1 to 9; the AWS side is frozen, then switched off.

reference architecture for Acme after the move from AWS: in Google Cloud, the Shared VPC host project holds the network, Cloud NAT and Private Service Connect endpoints; acme-dbx-prod holds the workspace's classic compute; acme-lake holds the Delta tables in gs://acme-lake-supply; Unity Catalog in europe-west2 governs them; Pub/Sub carries file events; BigQuery and Gemini Enterprise read the gold tables

The gold tables end where the rest of the series put them: BigQuery reads them for marketing analysis, and Gemini Enterprise is where Acme’s people ask questions across both sides (Part 8).

The second starts from an on-premises Hadoop estate instead of another cloud: files in HDFS, an Oracle database that the business runs on, and a Hive metastore with the table definitions.

reference architecture for an on-premises Hadoop estate: HDFS files move to Cloud Storage with Storage Transfer Service agents, an Oracle database streams changes to Cloud Storage through Datastream, and Hive metastore federation exposes the old Hive tables in Unity Catalog during the move; Databricks on Google Cloud ingests with Auto Loader, builds bronze, silver and gold Delta tables with Lakeflow pipelines, and serves them to Databricks SQL, BigQuery and Gemini Enterprise; Cloud Monitoring and Cloud Logging run underneath

Each column uses a documented service.

Files: Storage Transfer Service supports transfers “from HDFS to Cloud Storage” through agents installed next to the cluster (overview).
Database changes: Datastream is Google Cloud’s change data capture (CDC) service, which reads a database’s change log and streams every insert, update and delete. It supports Oracle among its sources and can “write change event streams into Cloud Storage”, where Auto Loader picks it up.
Table definitions during the move: Unity Catalog catalog federation can federate to “external Hive metastores”, which Databricks lists “as a step in the migration path to Unity Catalog”. Users query the old tables through Unity Catalog while they are copied to Delta. Hive metastore federation has the details.
Pipelines: Auto Loader lands files in bronze tables, and Lakeflow pipelines (formerly Delta Live Tables) build silver and gold (Part 7).

For moving existing Spark and SQL code into Databricks in general, the Databricks migration overview is the starting point.

What can go wrong?

These are the problems I see most often, with the usual cause and the fix.

troubleshooting a migration to Databricks on Google Cloud: the symptom, the usual cause and the fix, for mounts, old runtimes, the Hive metastore, hierarchical namespace buckets, Key Vault scopes, streaming checkpoints, half-copied tables, missing grants and quotas

Behind the table, a handful of field lessons:

Legacy features first. Mounts, the Hive metastore and old runtimes are switched off in every new workspace. Fix them on the source, where you can test calmly, rather than discovering them during a wave.
Quotas before waves. Compute Engine CPU quota is per region and often per machine family (Part 9 lists the quotas and sizes them for a peak). A wave that doubles the cluster count needs the quota raised a week before, not on the day.
The network cannot be resized in place. A workspace’s VPC is fixed, and changing its IP range means a new subnet and a new network configuration (Part 1). Size subnets for the peak after all waves, not for the first one.
Regional features. Check the regional table for every feature the source uses. It also notes that “some regions do not support GPU instance types”.
Organisation policies. Domain restricted sharing, resource locations and the ban on service account keys all apply to the new projects (Part 1); the key ban matters for BigQuery federation, which still needs a key (Part 2).
DNS for Private Service Connect. If the target uses Private Service Connect, the workspace URL must resolve to the private endpoint (Part 3; Prerequisite 3 explains DNS itself).
Helpers outside the support contract. UCX’s README says Databricks Labs projects are “not formally supported by Databricks with Service Level Agreements (SLAs)”, and MLflow Export Import is an open-source project on GitHub. Pin the versions you test with, and test on a copy first.

What would I do now?

Run the UCX assessment and the two system table queries on the source, and turn the results into a list of blockers (Hive metastore tables, mounts, old runtimes) and a list of code changes (storage paths).
Fix the blockers on the source: upgrade to Unity Catalog, replace mounts with external locations and volumes, and move every job to Databricks Runtime 13.3 LTS or later.
Build the Google Cloud landing zone with Terraform from Parts 1 to 6 (Part 2 maps each step to its resource), keeping catalog, schema and table names identical to the source.
Pick one low-risk pipeline as the first wave, move it with the data route that suits it, and run it in parallel with the source until the comparison job is clean.
Write down what the first wave taught you, raise quotas for the next one, and only then plan the rest.

What did we learn?

Databricks is the same platform on AWS, Azure and Google Cloud; a migration changes what sits underneath it: the account, the metastore, storage paths, identities, the network and billing.
A workspace cannot be moved between clouds, so a migration builds a new one and moves its contents: Delta data and code move easily, while Hive metastore tables, mounts and streaming checkpoints need real work.
Every Databricks workspace created since 8 October 2026 has no DBFS mounts, Hive metastore or runtimes before 13.3 LTS, so those dependencies are best removed on the source first.
Table data has three documented routes from AWS (Storage Transfer Service, OpenSharing and a read-only S3 external location), and every route pays AWS data transfer charges unless AWS waives them.
A five-step approach (assess, design, build, migrate, validate) run in waves, with a parallel run and a row-level comparison before cut-over, turns a risky move into a series of small, checked ones.

What comes next?

This is the last part, so here is the whole series, one line per post:

Databricks on Google Cloud, from the basics: who the series is for, the running example and the reading order.
Prerequisite 1: What a lakehouse is: storage, table formats and Spark: warehouses, lakes and lakehouses, Parquet, Delta Lake, Iceberg and Spark.
Prerequisite 2: What a cloud is, and what Google Cloud is: regions, the resource hierarchy, billing, IAM and the word map from AWS and Azure.
Prerequisite 3: Identity and security foundations: authentication and authorisation, single sign-on, SAML, OAuth and OpenID Connect, tokens, SCIM, federation, machine identities, RBAC and ABAC, encryption and network basics.
Part 1: Foundations, how Databricks is built on Google Cloud: the control plane and compute planes, Marketplace, projects, and what Databricks creates in your project.
Part 2: Identity and access, IAM, service accounts, single sign-on and automation: keyless service accounts, the identities a workspace uses, console and keyless Terraform deployment paths, single sign-on, SCIM, where roles live and which federation to use when.
Part 3: Networking: VPC, Cloud NAT, Private Google Access and Private Service Connect: the customer-managed VPC and its sizing, egress, private connectivity and VPC Service Controls.
Part 4: Serverless networking and egress: how serverless compute reaches your data, and how to control what it can reach.
Part 5: Storage, Cloud Storage and how Unity Catalog uses it: buckets, storage credentials, external locations, managed storage and DBFS.
Part 6: Governance and security: Unity Catalog in depth, customer-managed keys, secrets and audit.
Part 7: The platform on Google Cloud, runtime, Photon, SQL, Lakeflow and AI: the runtime, Photon, SQL warehouses, Lakeflow and AI.
Part 8: Better together, BigQuery, Gemini and Google Cloud services: the Databricks and BigQuery paths, Gemini models, Gemini Enterprise Agent Platform (formerly Vertex AI), Pub/Sub, Looker and OpenSharing.
Part 9: Commercials, quotas and operations: tiers, DBUs, Marketplace, quotas, cost levers and day-two operations.
Part 10: Migration to Google Cloud, this post: moving Databricks workloads from AWS, Azure or Hadoop to Google Cloud.

If you read one thing twice, make it Part 1: almost every question in a migration comes back to which project, which region and which network. Thank you for reading the series.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands and its data are fictional.