01 Overview · TL;DR 01 ✓ Read Two audiences, one shared architecture How Databricks specialists and Google Cloud architects bridge their knowledge gaps. 02 Overview · TL;DR 02 ✓ Read Same Databricks platform, Google Cloud underneath Why cloud-native infrastructure primitives change everything below the control plane. 03 Overview · TL;DR 03 ✓ Read Acme Consumer Brands: BigQuery meets Databricks One fictional enterprise running marketing in BigQuery and supply in Databricks. 04 Overview · TL;DR 04 ✓ Read The 13-part architecture roadmap Four progressive stages from foundational prerequisites to cloud migration. 05 Overview · TL;DR 05 ✓ Read Series rules, verification, and lab setup Dated claims, public documentation sources, and the IAM role needed to build along.
01 Prereq 1 · TL;DR 01 ✓ Read Warehouses, lakes, objects, and the lakehouse How open files in Cloud Storage gain the reliability of a data warehouse. 02 Prereq 1 · TL;DR 02 ✓ Read ACID transactions and optimistic concurrency How Delta Lake guarantees atomicity and isolation over plain Cloud Storage files. 03 Prereq 1 · TL;DR 03 ✓ Read Data skipping, optimisation, and medallion layers How lakehouses skip irrelevant files and refine data through bronze, silver, and gold. 04 Prereq 1 · TL;DR 04 ✓ Read Parquet, Delta Lake, Iceberg, and UniForm Inside binary Parquet files, JSON transaction logs, Avro manifests, and Universal Format. 05 Prereq 1 · TL;DR 05 ✓ Read Apache Spark and common lakehouse pitfalls How distributed Spark execution works and five failure modes to avoid.
01 Prereq 2 · TL;DR 01 ✓ Read Global VPCs, regional subnets, and zonal VMs How Google Cloud physical locations and resource scopes shape your network and compute layout 02 Prereq 2 · TL;DR 02 ✓ Read The 4-level Google Cloud resource hierarchy Organisations, folders, projects, and resources, and how policies flow downward 03 Prereq 2 · TL;DR 03 ✓ Read A Google Cloud project is not an Azure RG Mapping the six container layers across Google Cloud, AWS, and Azure without false friends 04 Prereq 2 · TL;DR 04 ✓ Read IAM allow policies and dual-sided service accounts How role bindings inherit down the hierarchy and why service accounts are both principals and resources 05 Prereq 2 · TL;DR 05 ✓ Read The 12 Google Cloud services under Databricks Compute, storage, private networking, security, and data services that power the lakehouse
01 Prereq 3 · TL;DR 01 ✓ Read Identity, AuthN, AuthZ, and SAML 2.0 SSO How single sign-on works with SAML assertions and why certificate rotation causes outages 02 Prereq 3 · TL;DR 02 ✓ Read OAuth 2.0, OpenID Connect, and JWT claims Delegated authorisation, OIDC discovery and JWKS key rollover, and inside a signed JWT 03 Prereq 3 · TL;DR 03 ✓ Read SCIM, JIT, and six meanings of federation How users and groups reach Databricks and how to untangle the word federation across vendors 04 Prereq 3 · TL;DR 04 ✓ Read Machine identities, MFA, RBAC, and ABAC Replacing long-lived secrets with short-lived tokens and combining role-based and attribute-based controls 05 Prereq 3 · TL;DR 05 ✓ Read Envelope encryption, CMEK, CIDR, and private DNS How DEKs and KEKs protect data at rest and how CIDR blocks and private DNS zones secure the network
01 Part 1 · TL;DR 01 ✓ Read The control plane and two compute planes Where the web app, classic Compute Engine clusters, and serverless workloads actually run. 02 Part 1 · TL;DR 02 ✓ Read Marketplace, billing and account hierarchy How subscribing in Google Cloud Marketplace creates your account and splits DBU and VM bills. 03 Part 1 · TL;DR 03 ✓ Read Projects, workspaces and landing-zone rules How to map workspaces to projects and Shared VPC subnets without hitting org policy walls. 04 Part 1 · TL;DR 04 ✓ Read What Databricks creates in your project The exact APIs, service accounts, custom IAM roles, and buckets provisioned during setup. 05 Part 1 · TL;DR 05 ✓ Read Cluster anatomy, gcp_attributes and launch flow Inside a Compute Engine Spark node, the 7-step startup sequence, and quiet day-two failures.
01 Part 2 · TL;DR 01 ✓ Read Four permission planes and keyless service accounts Why analysts need zero Google Cloud roles and how machines authenticate without JSON keys. 02 Part 2 · TL;DR 02 ✓ Read The eight Google identities behind a workspace Which service accounts Databricks owns, which ones you own, and how credentials passthrough works. 03 Part 2 · TL;DR 03 ✓ Read Console versus keyless Terraform from CI Comparing Path A (account console) with Path B (Workload Identity Federation and least-privilege roles). 04 Part 2 · TL;DR 04 ✓ Read Single sign-on, SCIM, AIM and OAuth federation Connecting Microsoft Entra ID, syncing nested groups, and untangling the five meanings of federation. 05 Part 2 · TL;DR 05 ✓ Read Reaching Cloud Storage and BigQuery safely Keyless Unity Catalog storage credentials, the BigQuery key exception, and cluster service account risks.
01 Part 3 · TL;DR 01 ✓ Read Customer VPC, Shared VPC and subnet sizing Where classic Compute Engine nodes live, how Shared VPC separates duties and how to size the primary CIDR 02 Part 3 · TL;DR 02 ✓ Read SCC relay, Cloud NAT and Private Google Access How the control plane reaches private cluster VMs without inbound ports, and the five paths every cluster uses 03 Part 3 · TL;DR 03 ✓ Read Back-end Private Service Connect and DNS Replacing internet paths to the control plane with two regional PSC endpoints and private Cloud DNS records 04 Part 3 · TL;DR 04 ✓ Read Front-end PSC, IP access lists and PAS How workforce users and CI runners reach the workspace privately, and what actually blocks public internet logins 05 Part 3 · TL;DR 05 ✓ Read VPC Service Controls and day-two updates Wrapping the workspace and storageprojects in a VPC-SC perimeter and updating subnets or PSC in place
01 Part 4 · TL;DR 01 ✓ Read Where serverless runs and how it reaches Google APIs Why your VPC disappears and how Cloud Storage and BigQuery stay on the Google backbone 02 Part 4 · TL;DR 02 ✓ Read NCC and Private Service Connect to your VPC Reaching private databases like orders-db without exposing public IP addresses 03 Part 4 · TL;DR 03 ✓ Read Outbound IP ranges and firewall allowlists How external partners recognise serverless traffic and why allowlists must be automated 04 Part 4 · TL;DR 04 ✓ Read Serverless egress control and network policies Locking down outbound traffic with restricted mode, dry-run testing and system table logs 05 Part 4 · TL;DR 05 ✓ Read VPC Service Controls, serverless workspaces and costs How serverless meets Google Cloud perimeters, what a serverless workspace drops, and future SKUs
01 Part 5 · TL;DR 01 ✓ Read Cloud Storage settings and HNS vs flat buckets Why lakehouse buckets stay regional and flat, and which settings break Delta 02 Part 5 · TL;DR 02 ✓ Read Four storage types: workspace, default, managed, external Untangling which project owns each bucket and where tables actually land 03 Part 5 · TL;DR 03 ✓ Read Connecting a bucket: credentials and external locations The five-step keyless handshake between Unity Catalog and Cloud Storage 04 Part 5 · TL;DR 04 ✓ Read Managed vs external tables and volumes in practice How Acme lays out gs://acme-lake-supply across bronze, silver and gold 05 Part 5 · TL;DR 05 ✓ Read What DROP really deletes, UNDROP and bucket layout The three-phase deletion timeline, recovery rules and multi-domain design
01 Part 6 · TL;DR 01 ✓ Read Namespace, privileges, ownership and bindings How Unity Catalog gates every query and isolates production from dev 02 Part 6 · TL;DR 02 ✓ Read Row filters, column masks, governed tags and ABAC Hiding rows and personal data dynamically across one table or a whole catalog 03 Part 6 · TL;DR 03 ✓ Read Automatic table and column lineage Tracing every gold metric back to its raw delivery file with system tables 04 Part 6 · TL;DR 04 ✓ Read Customer-managed keys (CMEK) in Cloud KMS Why you need keys for both Databricks workspace services and your lake bucket 05 Part 6 · TL;DR 05 ✓ Read Secrets, dual audit trails and compliance profile Where credentials live, joining system.access.audit with Cloud Audit Logs, and HIPAA/PCI
01 Part 7 · TL;DR 01 ✓ Read Databricks Runtime and the Photon engine How versioned Spark images, LTS lifecycles and native C++ vectorised execution work together 02 Part 7 · TL;DR 02 ✓ Read Fast queries and write concurrency Data skipping, liquid clustering, predictive optimization, disk caching and row-level conflict rules 03 Part 7 · TL;DR 03 ✓ Read SQL warehouses on Google Cloud Serverless, Pro and Classic warehouses, N2 quota planning and the REST API default trap 04 Part 7 · TL;DR 04 ✓ Read Lakeflow: Connect, pipelines, Jobs and Bundles How Delta Live Tables and Workflows became Lakeflow, and how Acme builds supply_service in one SQL file 05 Part 7 · TL;DR 05 ✓ Read Dashboards, Genie, Apps, Lakebase and Agents Consumption identity models, Agent Bricks, Unity Gateway and what is available in europe-west2
01 Part 8 · TL;DR 01 ✓ Read Four ways Databricks and BigQuery share data Catalog federation, query federation and choosing the right bidirectional path without copying tables 02 Part 8 · TL;DR 02 ✓ Read BigQuery reading Unity Catalog and UniForm How Lakehouse cross-cloud access, Iceberg REST credential vending and the three UniForm properties work 03 Part 8 · TL;DR 03 ✓ Read Databricks querying BigQuery: Federation modes Direct Storage API reads versus materialised SQL jobs, pushdown rules and the four required egress endpoints 04 Part 8 · TL;DR 04 ✓ Read Gemini models and Pub/Sub streaming Three patterns for Gemini with Databricks, plus ingesting and deduplicating Pub/Sub streams 05 Part 8 · TL;DR 05 ✓ Read Looker, OpenSharing and the reference architecture Preventing Looker PDT wake-ups, sharing live tables with partners and Acme end-to-end architecture
01 Part 9 · TL;DR 01 ✓ Read Premium vs Enterprise and the one bill How pricing plans gate network security and how DBUs land on your Google Cloud invoice 02 Part 9 · TL;DR 02 ✓ Read Marketplace orders, risk and support What subscribing creates, why one billing permission can delete every workspace, and who supports what 03 Part 9 · TL;DR 03 ✓ Read Sizing quotas and the three ceilings How Compute Engine quotas, Databricks limits and design-time network ceilings cap your clusters 04 Part 9 · TL;DR 04 ✓ Read FinOps levers, tags and budgets Cutting spend on both bills and propagating custom tags to Compute Engine labels and BigQuery 05 Part 9 · TL;DR 05 ✓ Read Day-two operations and launch errors Running the telemetry loop, deploying bundles and decoding cluster termination error codes
01 Part 10 · TL;DR 01 ✓ Read Why workspaces do not move and what travels What changes across AWS, Azure and Google Cloud, and why new accounts disable legacy features 02 Part 10 · TL;DR 02 ✓ Read The five-step wave migration method Assess with UCX and system tables, design and build with Terraform, then migrate and validate in waves 03 Part 10 · TL;DR 03 ✓ Read Three data routes and metastore hydration Moving bytes with Storage Transfer Service, OpenSharing or S3 locations, and hydrating Unity Catalog 04 Part 10 · TL;DR 04 ✓ Read Cloud translation cheat sheet Mapping storage, events, cluster attributes and node types from AWS and Azure to Google Cloud 05 Part 10 · TL;DR 05 ✓ Read Parallel run, row-hash validation and cut-over Proving identical answers with SQL row hashes and avoiding the nine common migration pitfalls