Databricks on Google Cloud, Prerequisite 2: What a cloud is, and what Google Cloud is

In Prerequisite 1 we looked at the lakehouse: object storage, open table formats such as Delta Lake and Apache Iceberg, and Apache Spark. This post looks at the other half of the title. Databricks does not run in a vacuum. On Google Cloud, its clusters are Google virtual machines, its storage is Google buckets and its network is a Google network, all inside a Google Cloud project that you own.

If you know Databricks from AWS or Azure, a few Google Cloud words will trip you up: “project” means something very specific, a VPC network spans the whole planet, and a “service account” is not a shared login. If you already know Google Cloud well, skim this post and go on to Part 1, which shows which parts of your platform Databricks touches.

I cover where Google Cloud runs, how it organises and bills everything, how access works, the Google Cloud services that later parts rely on, and a word map from AWS and Azure. Everything here is about Google Cloud itself; the Databricks specifics start in Part 1. If cloud computing is new to you, my Learning Cloud through GCP series covers the basics, and I link to it rather than repeat it.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: what Google Cloud is and how it organises, bills and secures everything, in three minutes. The narration is a synthetic voice; the script and animation are mine.

What is a cloud, and which ideas do I need first?

A cloud is someone else’s data centres, rented by the minute through an API (an application programming interface, a way for programs to ask for things). You ask for a computer, a disk or a network, and a few seconds later it exists. You pay for what you use and give it back when you are done. Part 1 of Learning Cloud explains this with a utility connection, and Part 2 explains who manages what in IaaS, PaaS and SaaS. Databricks on Google Cloud is a PaaS (platform as a service) that runs on Google’s IaaS (infrastructure as a service).

Six Google Cloud ideas come up in every later part of this series:

A region is a geographic area, such as London, where Google runs data centres. A zone is one isolated deployment area inside a region.
A project is the container that holds your resources. Every virtual machine, bucket and network belongs to exactly one project.
An organisation is the top of the tree for a company, with folders to group projects.
A Cloud Billing account decides who pays for a project.
An allow policy in IAM (Identity and Access Management) says who may do what on which resource.
A service account is an identity for software, such as a virtual machine, so that it can call other Google Cloud services without a password.

the key ideas in one picture: an organisation holds folders and projects, a project holds resources such as a VPC network, virtual machines and buckets, a billing account pays for the project, an allow policy says who may do what, and a service account is the identity a virtual machine uses

Who is Acme, and what will it run here?

The running example is Acme Consumer Brands, the same fictional company as my Production-ready agents series. Acme’s marketing data sits in BigQuery and its supply chain data sits in Databricks, as Prerequisite 2 of that series describes. In this series we build the Databricks side properly, on Google Cloud, in the London region. Every example in this post is a piece of Acme’s estate. Here are the two tables in brief:

the two halves of Acme's data at a glance: campaign_performance has one row per campaign, brand, region and week, owned by marketing in BigQuery; supply_service has one row per brand, region and week, owned by supply in Databricks; three real rows of each

That prerequisite explains every column and how the two tables join.

Where does Google Cloud physically run?

Google Cloud is a set of regions joined by Google’s own network; each region holds several zones, and every resource lives at one of three scopes: global, regional or zonal.

A supermarket chain makes the scopes easy to picture. The chain has branches in many cities (the regions), and a big city has several separate stores, each with its own power supply, so a power cut in one store does not close the others (the zones). A till sits in one store. The home-delivery area is drawn for a whole city. The loyalty card works in every store in every city. A till is zonal like a virtual machine, a delivery area is regional like a subnet, and the loyalty card is global like a VPC network.

Google Cloud runs on Google’s own private global network: fibre between data centres and edge locations, owned by Google. On top of that network sit regions, and inside each region sit zones. A zone is one failure domain (a part of the system that can fail on its own without taking the others down). Regions usually have three zones each, so you spread important things across zones.

Names follow a pattern. The region europe-west2 is London; its zones are europe-west2-a, europe-west2-b and europe-west2-c. As of 13 October 2026, Google Cloud lists 43 regions and 130 zones. Acme uses europe-west2.

Every Google Cloud resource has a scope: global, regional or zonal. This matters more than you might expect, and it is the first trap for AWS people: an Amazon VPC lives in one Region, with each subnet inside one Availability Zone, while a Google Cloud VPC network is global and its subnets are regional. A virtual machine is zonal. A subnet and a Cloud NAT gateway are regional. A VPC network and its firewall rules are global. Traffic between your regions can also stay on Google’s backbone: in the Premium Tier, traffic is carried on Google’s network until the last mile, while the Standard Tier uses regular internet service provider networks for more of the journey.

Google's global network connects regions; each region has several zones; a VPC network spans every region, a subnet belongs to one region, and a virtual machine runs in one zone; Acme's Databricks workspace lives in europe-west2

How does this apply to Acme? Acme’s region is London, europe-west2. A cluster virtual machine is the till: it runs in one zone, such as europe-west2-a. The subnet it uses is the delivery area, covering the whole London region. The VPC network is the loyalty card: the same network could also hold a subnet in another region later, with no extra wiring.

For everything that follows, remember that a subnet is regional, so anything that runs in a subnet lives in that subnet’s region. Part 1 shows how this pins a Databricks workspace, its subnet and its cluster virtual machines to one region.

How does Google Cloud organise everything?

Everything in Google Cloud sits in one tree, and the project is the box that matters most: it is where billing, quota and access meet. Picture a company’s office building. The building is the company. Each floor belongs to a department. Each locked room on a floor has its own electricity meter and its own list of who holds a key. The desks and cabinets are inside the rooms. A rule pinned up at the entrance to a floor, such as “no visitors after 6pm”, applies to every room on that floor. The building is the organisation, floors are folders, rooms are projects, the furniture is the resources, and the rule on the floor is a policy inherited down the tree.

Google Cloud uses a resource hierarchy: a tree that decides who owns a resource and which policies apply to it. There are four levels in the resource hierarchy.

The organisation is the root, and it represents the company. It is tied to a Google Workspace or Cloud Identity domain (Google’s directory of users and groups), such as acme.example. Once an organisation exists, projects belong to it, not to the employee who created them.
Folders are optional groups under the organisation. They can hold projects and other folders, so they usually mirror departments or environments.
Projects hold the resources. A project is where you enable APIs, set quotas, link billing and grant access. It has three names: a project ID that you choose and cannot change (for example acme-dbx-prod), a project number that Google assigns, and a display name that you can change.
Resources are the things themselves: virtual machines, buckets, datasets, keys.

Policies flow down the tree. An allow policy, a deny policy or an organisation policy (a guard rail that restricts how resources can be configured) set on a folder applies to every project inside it. This is why landing-zone teams care so much about the folder layout.

Acme's resource hierarchy: the organisation at the top, folders for shared services and the data platform, and one project per job and environment, such as the Shared VPC host project, the Databricks workspace projects and the BigQuery marketing project; policies set higher up flow down

How does this apply to Acme? The building is the organisation for acme.example. The floors are folders for shared services and the data platform. The rooms are projects such as acme-dbx-prod for the Databricks workspace and a separate project for BigQuery, each with its own bill, quota and access list.

The misconception to drop is that a project is a loose label for “a piece of work”, like a project in a ticketing tool. On Google Cloud it is a hard boundary, closer to a locked room than a folder name.

My view: the Google Cloud habit to learn first is one project per job and per environment. Each project is a clean boundary for billing, quota and access. Acme gives the Databricks workspace its own project, acme-dbx-prod, separate from the BigQuery project and from the network team’s project.

That last one is a Shared VPC: one host project owns the network, and service projects attach to it and use its subnets. Network teams like it because routing, NAT and firewall rules stay in one place.

When I am unsure whether something needs its own project, I ask these questions in order, and stop at the first “yes”:

1. Is it a different environment (dev, test, prod)?          New project.
2. Does a different team own its bill or its access list?     New project.
3. Must its quota or blast radius be kept apart?              New project.
4. Only then: same job, same owners, same environment?        Same project, use labels.

Part 3 shows how a Databricks workspace uses one.

What is the top container on Google Cloud, AWS and Azure?

The level that matters is the unit of isolation: a Google Cloud project, an AWS account or an Azure subscription. Everything else in this section lines the three clouds up around that.

If you come from Azure, the Google Cloud tree can look too simple, and the Azure one can look like too many boxes. They do the same jobs, so it helps to line them up by job rather than by name. There are six jobs.

Where people’s identities live. On Google Cloud this is Cloud Identity or Google Workspace (the directory for a domain such as acme.example). On Azure it is the Microsoft Entra tenant, a dedicated directory for one organisation. On AWS, people usually sign in through IAM Identity Center, connected to a corporate directory.
The root of the policy tree. Google Cloud has the organisation node, one per domain. AWS has the organization root, held by the management account. Azure has the root management group, one per tenant.
Optional grouping. Google Cloud folders, AWS organizational units (OUs) and Azure management groups all group the level below so that one policy can cover many of them. Azure’s documentation says a management group tree “can support up to six levels of depth”, not counting the root or the subscriptions.
The unit of isolation. This is the level that matters most, and the one people map wrongly. It is where resources are created, quotas are counted, access is granted and costs are tracked. On Google Cloud it is the project. On AWS it is an AWS account. On Azure it is a subscription, which trusts exactly one Entra tenant and is the unit Azure bills.
Grouping inside the unit. Only Azure has this level: the resource group, “a container that you use to manage related resources for an Azure solution”, so that you can deploy, update and delete them together. Google Cloud has no level between project and resource; you use labels (key and value pairs on resources) for the same grouping, or a separate project if you need a hard boundary. AWS uses tags in the same way.
The resources themselves: virtual machines, buckets or storage accounts, keys.

Billing sits beside the tree rather than inside it on all three. Google Cloud links projects to a Cloud Billing account (the next section). AWS uses consolidated billing in the management account. Azure, under a Microsoft Customer Agreement, has a billing account, billing profiles and invoice sections that pay for subscriptions.

the top containers of Google Cloud, AWS and Azure lined up by role: where people's identities live (Cloud Identity, IAM Identity Center, a Microsoft Entra tenant), the root of the policy tree (organisation, organization root, root management group), optional grouping (folders, organisational units, management groups), the unit of isolation, quota and billing (project, AWS account, subscription), grouping inside that unit (only Azure has one, the resource group), and the resources; billing sits beside each tree

So the one-line translation is: a Google Cloud project is an Azure subscription or an AWS account, not an Azure resource group. A Google Cloud folder is an Azure management group or an AWS OU, and the organisation node is the root management group or the organization root.

Part 1 shows how this difference decides where a Databricks workspace lands on each of the three clouds.

Who pays for it?

A Cloud Billing account is the wallet; projects are linked to it, and each project is paid from exactly one wallet at a time. Think of a household that pays the electricity for three flats from one bank account. Each flat has its own meter and its own line on the statement, but one account pays them all, and a flat can only be on one account at a time. Stop paying and the power to that flat is cut. Add a film channel to your broadband and it appears on the same bill. The bank account is the Cloud Billing account, the flats are projects, the meters are per-project usage, the power cut is what happens when billing is disabled, and the film channel is a Marketplace subscription.

A Cloud Billing account defines who pays for a set of projects. It is backed by a payments profile (the card or invoicing terms behind it), and it tracks the cost of every project linked to it. Accounts are either self-serve (paid automatically by card) or invoiced (monthly invoices, typical for companies). Access to a billing account is controlled with IAM roles such as roles/billing.admin and roles/billing.user.

The relationship is one to many. One billing account can be linked to one or more projects, and each project is billed to one billing account at a time. If billing is disabled on a project, its paid services stop and some of its resources can be deleted.

how billing works: a Cloud Billing account, backed by a payments profile, pays for every project linked to it; each project links to one billing account at a time; a Databricks subscription bought on Google Cloud Marketplace is billed to a billing account you choose

How does this apply to Acme? Acme’s projects, including acme-dbx-prod and the BigQuery project, are the flats, each linked to Acme’s billing account. Databricks is the film channel, bought through Marketplace and billed to the billing account Acme chooses.

Third-party software can be bought through the same bill. Google Cloud Marketplace is Google’s catalogue of third-party products that you subscribe to and pay for through a Cloud Billing account. During the subscription you choose which billing account pays, and companies with a negotiated contract use a private offer (a custom price and terms for one buyer) instead of the public listing. The purchases then appear on that billing account next to your own Google Cloud usage. Acme buys Databricks this way: Part 1 walks through subscribing, and Part 9 covers the order, the entitlement and the commercial details.

Who is allowed to do what?

IAM grants roles to principals on resources, and grants only ever add up as they flow down the tree. Office key cards are a fair picture. A card belongs to a person. The card type, say “facilities”, opens a fixed set of doors. Each building and floor keeps a list of which people hold which card type there. A card type granted for the whole building opens the matching doors on every floor, and the only way to keep someone out despite that is a “do not admit” list at reception, which is checked first. The person is the principal, the card type is a role, each door is a permission, the list is an allow policy, and the reception list is a deny policy.

IAM answers one question: may this principal do this action on this resource? It has three parts.

A principal is who is asking: a Google Account, a group, a service account or an external identity.
A role is a named bundle of permissions (single actions such as storage.objects.get). There are three kinds: basic roles, which are far too broad for production; predefined roles, managed by Google per service, such as roles/storage.objectViewer; and custom roles, which you build yourself. Since 29 October 2026 the basic roles are Admin, Writer and Reader; the older Owner, Editor and Viewer are now called legacy basic roles and still work. Google recommends the newer three when you must grant a basic role, because they accept IAM conditions and the legacy ones do not.
An allow policy is attached to a resource and holds role bindings: this role for these principals.

how IAM works: an allow policy on a resource binds a role to a principal; a role is a set of permissions; a grant on the organisation, a folder or a project is inherited by everything below it, so the effective access is the sum of grants along the path

How does this apply to Acme? As an illustration, granting Acme’s supply analysts group roles/storage.objectViewer on one bucket in acme-dbx-prod gives that group permissions such as storage.objects.get on that bucket only. Granting the same role on the data platform folder would reach every bucket in every project below it.

Grants are inherited down the hierarchy and they only add up. A role granted on the organisation reaches every project. To take access away despite an allow policy, you need a deny policy, which IAM always checks first. Bindings can also carry IAM conditions (rules such as “only for buckets whose name starts with this prefix”), which Part 2 shows Databricks using on its workspace buckets.

What is a service account?

A service account is an identity for software, not a shared login for people. Think of a company van’s fuel card. It belongs to the van, not to any driver. Whoever is driving the van can use it for that trip, a manager can authorise a colleague to borrow it for an afternoon, and if someone photocopies the card, the copy works until the card is cancelled. The van is a virtual machine, the fuel card is the service account, borrowing it is impersonation, and the photocopy is a service account key.

A service account is an identity for a workload rather than a person. Its name looks like an email address: dbx-jobs@acme-dbx-prod.iam.gserviceaccount.com. The closest relatives are an AWS IAM role with an instance profile and an Azure managed identity.

A service account has two sides. It is a principal: you attach it to a virtual machine, and software on that machine gets short-lived tokens from the metadata server and calls other services with them. It is also a resource: other principals can be granted roles on it. Granting someone the Service Account Token Creator role (roles/iam.serviceAccountTokenCreator) on it lets them impersonate it, which means acting as it for a short time without any key.

a service account has two sides: as a principal, a VM uses its attached service account to get short-lived tokens and call Cloud Storage; as a resource, a person with the Service Account Token Creator role can impersonate it; a downloaded key is a long-lived secret to avoid

How does this apply to Acme? Take dbx-jobs@acme-dbx-prod.iam.gserviceaccount.com, the example name above. Attached to Acme’s cluster virtual machines, it would be the van’s fuel card: software on them gets short-lived tokens for it, and an engineer who needs to test as it could be granted Token Creator to impersonate it for a short time, instead of downloading a copy.

The thing to avoid is a service account key: a JSON file holding a private key that works until somebody revokes it. Google’s own guidance is to choose a keyless alternative whenever possible, and many organisations block key creation with the organisation policy iam.disableServiceAccountKeyCreation. Prerequisite 3 explains why long-lived keys are risky and what replaces them. Two more facts worth knowing: the default Compute Engine service account is granted the broad Editor role when it is created, unless the organisation policy iam.automaticIamGrantsForDefaultServiceAccounts stops it; and when you create a workspace, Databricks creates its own service accounts in your project with a custom role. Part 2 lists every one of them.

Which Google Cloud services will this series use?

Twelve services do almost all of the work under a Databricks workspace. If the list feels long, picture a factory. It needs machines, a warehouse, private roads, a one-way exit gate, a private lane to the post office, a dedicated door for one supplier, a key safe and CCTV:

Factory                          Google Cloud service
machines                         Compute Engine
warehouse                        Cloud Storage
private roads                    VPC network and subnets
one-way exit gate                Cloud NAT
private lane to the post office  Private Google Access
dedicated door for a supplier    Private Service Connect
key safe                         Cloud KMS
CCTV recordings                  Cloud Logging

The remaining four, BigQuery, Gemini, Pub/Sub and Looker, are the neighbours the factory trades with.

Twelve Google Cloud services come up again and again in later parts. Here is each one in plain words, as a Google Cloud service; the parts that follow show what Databricks does with it. Learning Cloud covers the general ideas behind them: compute, storage and networking.

Compute Engine provides virtual machines in machine families such as general-purpose, compute-optimised and memory-optimised. A virtual machine is zonal.
Cloud Storage is object storage. A bucket has a globally unique name and a location that is a region, a dual-region or a multi-region. Prerequisite 1 showed what an object inside a bucket is.
VPC network is your private network. The network is global; its subnets are regional, and each subnet has a range of internal IP addresses.
Cloud NAT (network address translation) lets virtual machines without public IP addresses make outbound connections. It is regional and software-defined, with no proxy machines, and it is configured on a Cloud Router. It never accepts unsolicited inbound traffic.
Private Google Access is a setting on a subnet. It lets virtual machines that only have internal addresses reach Google APIs such as Cloud Storage.
Private Service Connect (PSC) gives a managed service a private endpoint with an internal IP address in your network, so traffic stays inside Google Cloud.
Cloud KMS (Key Management Service) holds encryption keys. Keys live on a key ring in one location and have key versions that rotate. A key you create and control, used to encrypt another service’s data, is called a customer-managed encryption key (CMEK).
Cloud Logging stores logs, including audit logs. Admin Activity audit logs (who changed a configuration) are always written and cannot be disabled. Data Access audit logs (who read data) are off by default for every service except BigQuery.
BigQuery is Google’s data warehouse, where Acme’s marketing data lives.
Gemini models and Gemini Enterprise Agent Platform, the platform formerly called Vertex AI, where Google’s models are served to applications and agents.
Pub/Sub is Google’s messaging service: publishers send messages to a topic, and subscribers receive them.
Looker is Google’s business intelligence platform, for dashboards and reports.

the twelve Google Cloud services this series uses, grouped by area: Compute Engine for compute; Cloud Storage for storage; VPC network, Cloud NAT, Private Google Access and Private Service Connect for networking; Cloud KMS and Cloud Logging for security and operations; and BigQuery, Gemini models, Pub/Sub and Looker for data and AI

Part 1 shows which of these a Databricks workspace uses and what it creates in your project. Part 3 puts the network pieces together, including how the Databricks control plane reaches cluster machines that have no public IP address, and Part 8 shows how Databricks works with BigQuery, Gemini, Pub/Sub and Looker.

How do AWS and Azure words map to Google Cloud?

If you arrive from another cloud, most services simply have a new name. Google publishes a full AWS and Azure to Google Cloud comparison; this is the subset a Databricks deployment touches.

a vocabulary map from AWS and Azure to Google Cloud for the concepts a Databricks deployment touches: containers, networks, NAT, private access, object storage, workload identity, keys, secrets, streaming, warehouse, machine learning, dedicated links and logs

A few words are more dangerous than a new name, because they look familiar and mean something different. English has the same trap: the “first floor” is one storey up in London and the ground floor in New York, and both speakers are sure they are right. These six cause most of the confusion I see when Databricks and Google Cloud people first work together.

words that mean different things: project, VPC, service account, workspace, account and NCC each have a meaning in AWS or Azure, in Databricks, or in Google Cloud that is easy to confuse, with a one-line way to keep each straight

How does this apply to Acme? In one design meeting, Acme’s people say “account” and mean three things: the Databricks account that holds its workspaces and metastore, the Cloud Billing account that pays, and service accounts such as dbx-jobs. Naming which one, every time, saves a lot of confusion.

The two to remember are project and VPC. Because a project is the unit of billing, quota, APIs and access, most “where does this go?” questions in a Databricks design are really “which project?” questions. And because a VPC network is global, one network can hold subnets in London and Iowa with no peering between them.

What can go wrong in the first week?

Every one of these comes from the basics in this post.

No organisation. A personal project at the top of its own hierarchy has no organisation above it, so there is nowhere to set company-wide policies, and some products need an organisation to sign up. Create projects under the company’s organisation from the start.
An organisation policy blocks you. Policies such as domain restricted sharing (iam.allowedPolicyMemberDomains), resource locations and key creation are inherited from folders above your project. Ask for the effective policies before you start, not after the first error.
Missing roles on the right project. With a Shared VPC, work in a service project often needs permissions on the host project too.
No egress path. Virtual machines without public IP addresses cannot reach the internet without Cloud NAT, or Google APIs without Private Google Access.
A subnet that is too small or in the wrong region. A subnet belongs to one region, and you can grow its primary range later but never shrink it. Size it for growth.
Quota. Compute Engine quotas belong to a project, and virtual machine quotas are counted per region. Check CPU quota for your machine family before the first big workload.
Broad grants. A basic role such as Editor or Writer granted at project level reaches every resource in the project. Use predefined roles, narrow scopes and IAM conditions.

What would I do now?

Draw your hierarchy: the organisation, the folders, and one project for each job and environment, plus the network host project if you use a Shared VPC.
Pick a region close to your data and your users, and keep the related resources in it. Part 1 lists the regions Databricks supports.
Decide on the billing account and who holds Billing Administrator on it.
List the organisation policies that apply to your projects (key creation, external IP addresses, domain restricted sharing, resource locations) and agree any exceptions early.
Sketch your network: a custom-mode VPC network, a regional subnet with Private Google Access, and Cloud NAT for outbound traffic. Part 3 turns that sketch into a Databricks design.

What did we learn?

Google Cloud runs on Google’s own global network, with regions made of zones, and every resource is global, regional or zonal; a subnet, and everything that runs in it, lives in one region.
The resource hierarchy of organisation, folders and projects decides who owns a resource and which policies apply, and the project is the unit of billing, quota, APIs and access.
A Google Cloud project maps to an AWS account or an Azure subscription, not to an Azure resource group; folders map to OUs and management groups, and the organisation to the organization root or root management group.
A Cloud Billing account pays for every project linked to it, and products bought on Google Cloud Marketplace are billed to a billing account you choose.
IAM grants roles to principals through allow policies on resources, grants are inherited down the hierarchy, and service accounts give workloads an identity without a key.
Twelve Google Cloud services carry the series, from Compute Engine, Cloud Storage and the VPC network pieces to BigQuery, Gemini, Pub/Sub and Looker, and project and VPC are the two words that mean most different things to people arriving from AWS or Azure.

What comes next?

Prerequisite 3: Identity and security foundations covers the last set of basics: authentication and authorisation, single sign-on, SAML, OAuth 2.0 and OpenID Connect, tokens, SCIM, the many meanings of federation, machine identities, encryption keys and the network terms security teams use. If those are already familiar, go straight to Part 1: Foundations, how Databricks is built on Google Cloud, which puts these pieces to work: what Databricks is, the control plane and the classic and serverless compute planes, accounts, workspaces and metastores, subscribing through Marketplace, how projects map to workspaces, what Databricks creates in your project, and how a cluster launches on Compute Engine.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands is a fictional company used for teaching.