Databricks on Google Cloud, Prerequisite 3: Identity and security foundations

In Prerequisite 1 we learned what a lakehouse is, and in Prerequisite 2 we learned how Google Cloud is organised, including IAM and service accounts. Before Part 1 there is one more set of basics to cover, and it is the set that causes the longest meetings: identity and security.

When a Databricks team, a Google Cloud team and a security team sit down together, they use words such as SAML, OIDC, SCIM, ID token, client credentials and federation as if everyone already agreed on what they mean. Often they do not. “Federation” alone means at least six different things across Google Cloud, Microsoft and Databricks. This post defines every one of those words from the ground up, without anything specific to Databricks on Google Cloud, so that Part 2 and the other parts can link here and spend their words on the specifics.

We start with a map: two big buckets, user to machine and machine to machine, each split once more, with no product names. Then come the protocols that carry them, then how Google Cloud, Microsoft, AWS and Databricks each implement them, and finally the exceptions, where a product’s name says one thing and means another.

The running example is the same as in the rest of the series. Acme Consumer Brands is a fictional company that runs Databricks on Google Cloud for its supply data. Its staff sign in with Microsoft Entra ID. Sam, a supply analyst, is our person. A Terraform pipeline in GitHub Actions, which deploys Acme’s platform as the Google service account tf-deployer@acme-dbx-admin.iam.gserviceaccount.com, is our machine. Every concept below is shown on one of those two.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: the identity and security basics every later part relies on, in three minutes. The narration is a synthetic voice; the script and animation are mine.

What are identity, authentication and authorisation?

These three words are often used as if they were one. They are three different questions, and every system asks them in the same order: who are you, prove it, and what may you do? The common mistake is to treat “signed in” as “allowed”; signing in answers only the first two.

A simple example: checking in to a hotel. The name on your booking is your identity. Showing your passport at the desk is authentication. The keycard that opens room 412 and the gym, but not room 413, is authorisation, and every door checks the card each time you tap it, so the card stops working the moment you check out, however recently you showed your passport.

In the technical terms:

• An identity is the answer to “who are you?”. It is a unique name for a person or a program inside some system, such as sam@acme.example in Acme’s directory or tf-deployer@acme-dbx-admin.iam.gserviceaccount.com in Google Cloud. A principal is the general word for anything that can have an identity: a user, a group, a program.
• Authentication (often shortened to AuthN) is “prove it”. The principal presents something that only the real owner of the identity should have: a password, a code from a phone, a private key, or a token that a trusted system issued earlier. Authentication answers who, not what.
• Authorisation (often AuthZ, and spelled “authorization” in most specifications and product names) is “what may you do?”. Once the system knows who you are, it checks a policy: may this principal do this action on this resource?

Prerequisite 2 showed one authorisation system, Google Cloud IAM, where an allow policy binds principals to roles. Databricks has another, Unity Catalog, where privileges such as SELECT are granted on tables. Both assume that authentication already happened somewhere else.

How does this apply to Acme? The same three steps, for both of our characters. Sam’s booking is sam@acme.example in Entra ID, his passport is his password and a phone prompt, and his keycard is the SELECT grant that Unity Catalog checks on every query. For Acme’s pipeline, the booking is the tf-deployer service account, the passport is a short-lived token, and the keycard is the set of IAM roles that Google Cloud checks on every call:

identity, authentication and authorisation are three different questions asked in order: who are you, prove it, and what may you do; for Sam the identity is sam@acme.example in Entra ID, he proves it with a password and a phone prompt, and Unity Catalog checks his SELECT grant; for Acme's CI the identity is the tf-deployer service account, it proves it with a short-lived token, and Google Cloud IAM checks its roles on every call

The beginner question here is: if authentication already happened, why check again on every request? Because the two answers change at different speeds. Sam proves who he is once in the morning. Whether he may read a table can change at 11:00 when someone removes him from a group, and the next query must see that change. This is why the rest of this post splits so cleanly: sections on identity providers, SAML, OIDC and tokens are about authentication; the section on RBAC and ABAC is about authorisation.

What are the two big buckets, user to machine and machine to machine?

Before any protocol or product name, ask one question about every sign-in: is a person at the other end, or a program? That splits authentication into two buckets, and each bucket splits once more.

User to machine (U2M) is a person, such as Sam, signing in to an application through a browser, usually with MFA. Underneath it is always single sign-on: an identity provider proves who Sam is and the application trusts it, carried by SAML or OpenID Connect. The split inside U2M is about what the application keeps:

• 1A, sync-based. The identity provider pushes a copy of users and groups into the application ahead of time, usually with SCIM or a directory sync tool. When Sam signs in, the application matches him to its copy and applies grants it already holds. This is how most companies have run identity for years, and many still do, because it lets you grant access before anyone’s first sign-in and remove leavers centrally.
• 1B, sync-less. The application keeps no copy at all. It validates the token at the moment Sam arrives and grants access on the subject or on group claims inside the token. Nothing has to be kept in step, but there is nothing to grant to before the first sign-in either.

Machine to machine (M2M) is a program calling an API: a CI job, a scheduled script, a service. There is no browser and no phone for MFA, so the program needs some other way to prove who it is. The split inside M2M is about what the program holds:

• 2A, static secret. The program stores a client ID and client secret (or a key file, or a personal access token) and trades it for a short-lived access token. The client ID names the identity; the secret proves it. The secret is long-lived and works for whoever holds a copy.
• 2B, dynamic token trust. The platform the program runs on, such as GitHub Actions, signs a short-lived token that says which program this is. The target is configured to trust that issuer, and a policy maps the token to an identity: a Google service account or a Databricks service principal. On Google Cloud, becoming a service account this way needs a role on that service account, Workload Identity User or Service Account Token Creator. This is workload identity federation, and nothing long-lived is stored anywhere.

authentication splits into two buckets: user to machine, where a person signs in through a browser with single sign-on, and machine to machine, where a program signs in without a browser; user to machine is either sync-based, where the identity provider copies users and groups into the application ahead of time with SCIM or directory sync, or sync-less, where the application keeps no copy and trusts the token and its group claims at sign-in, as Google Workforce Identity Federation does; machine to machine is either a static client ID and client secret, key file or personal access token, or dynamic token trust through workload identity federation, where a platform-signed token is mapped to a service account or service principal and acting as a Google service account needs Workload Identity User or Service Account Token Creator on it; at the edges sit on-behalf-of flows, attached identities and the Databricks workspace identity federation exception

At the edges. Three patterns sit on the boundaries of the two buckets rather than forming new ones:

• On behalf of. The flow starts as U2M and continues as M2M: a tool signs Sam in and then calls APIs carrying his identity, through OAuth U2M or token exchange.
• Attached identity. The simplest form of 2B: code running inside the cloud asks the platform’s metadata server for a token, so the platform itself is the issuer.
• Databricks workspace identity federation. The exception that is none of these: it decides where users and groups live, not whose tokens are trusted.

The rest of the post follows the same order you would use on a whiteboard. First the protocols that carry these flows: SAML, OAuth 2.0, OpenID Connect and tokens. Then we return to U2M in What does federation mean?, with sync-based and sync-less side by side, and to M2M in What are machine identities, and why are keys risky?, with static secrets, dynamic trust and the roles each needs. Vendor implementations come after the general idea in each section, and the exceptions come last.

What is an identity provider, and what is single sign-on?

An identity provider (IdP) is the system that holds a company’s identities and authenticates them. It stores the users and groups, checks passwords, runs multi-factor authentication, and then tells other applications, in a signed message, who the user is. Acme’s IdP is Microsoft Entra ID, which Microsoft renamed from Azure Active Directory (Azure AD); you will still see the old name in older documentation and in some product screens. Other common IdPs are Okta, Google Cloud Identity (Google’s identity as a service product for managing users and groups) and AWS IAM Identity Center.

The application on the other side has two names depending on the protocol. SAML calls it a service provider (SP): the system that provides the service Sam wants and relies on the IdP for sign-in. OpenID Connect calls the same thing a relying party (RP), because it relies on the IdP’s statement about the user. In this post the Databricks account is an SP or RP, and Entra ID is the IdP.

Single sign-on (SSO) is the result: Sam signs in once, at the IdP, and every application that trusts that IdP lets him in without asking for another password. Microsoft’s single sign-on overview describes it in the same terms. The security benefit is bigger than the convenience. The applications never see Sam’s password, so there is one place to enforce strong authentication and one place to switch Sam off when he leaves.

A simple example: a passport. Your government checks who you are once and issues the passport. Border officers in other countries never ask for your birth certificate; they accept the passport, because they know in advance what a genuine one from your country looks like. The government is the IdP, each border officer is an SP, the passport is the signed statement (a SAML assertion or an ID token), and knowing what a genuine one looks like is the trust an admin sets up in advance.

How does this apply to Acme? Entra ID is Acme’s passport office: Sam proves who he is there once in the morning, with his password and phone. The Databricks account and the Google Cloud console are border officers: neither ever sees his password, and both accept Entra ID’s signed statement about him.

single sign-on with one identity provider: Sam signs in once at Microsoft Entra ID, which holds Acme's users, groups, passwords and MFA settings; each service provider, such as the Databricks account and the Google Cloud console, trusts Entra ID through configuration exchanged in advance (metadata, a signing certificate or a client ID) and accepts a signed assertion or ID token instead of keeping its own password

The two sides trust each other because an admin exchanges identifiers, addresses and keys in advance; the SAML setup diagram shows exactly what moves.

On Databricks on Google Cloud, sign-in with Google Cloud Identity is available by default, and you can bring your own IdP over SAML 2.0 or OpenID Connect, as Configure SSO in Databricks describes (in Public Preview, available to try but not yet generally available, as of October 2026). Part 2 covers that setup for Acme.

What is SAML, and which version matters?

SAML is the Security Assertion Markup Language, an XML-based standard from OASIS (a standards body) for an IdP to make signed statements about a user to an SP. An assertion is one such statement (“this is Sam, he signed in at 09:00, he is in supply-analysts”), and markup language means it is written in XML.

The OASIS Security Services committee page dates the three versions: SAML 1.0 (November 2002), SAML 1.1 (August 2003) and SAML 2.0 (March 2005, with errata last produced in May 2012). SAML 2.0 replaced the 1.x documents (OASIS publishes a separate “Differences Between SAML V2.0 and SAML V1.1” note), and it is what today’s IdPs and SPs implement. When a product page says “SAML”, it means SAML 2.0, which is the only version the Databricks SSO page names.

How do the IdP and the SP set up trust?

Before anyone signs in, an admin exchanges configuration between the two sides. SAML calls this metadata: each party’s entity ID (a unique name, usually a URL), its endpoints, and its public keys. It can travel as an XML file, a metadata URL, or values copied by hand between two admin screens. The fields below are the ones the Databricks SAML page and its Entra ID guide ask for.

setting up SAML single sign-on between Entra ID and the Databricks account, before anyone signs in: from Entra ID, the admin copies the Login URL, the Microsoft Entra Identifier and the Base64 signing certificate into Databricks' Single sign-on URL, Entity ID and x.509 certificate fields; from Databricks, the admin copies the Databricks SAML URL into Entra ID's Entity ID and Reply URL fields, sets the Name ID to user.mail, and chooses to sign the SAML response and assertion with SHA-256; Entra ID keeps the private signing key, and Databricks stores only the public certificate

Two endpoints matter most. The IdP’s SSO URL (Entra ID calls it the Login URL) is where the SP sends the browser to sign in. The SP’s assertion consumer service (ACS) URL (Entra ID calls it the Reply URL) is where the IdP sends the browser back with the assertion.

How does a SAML sign-in work, step by step?

The SAML 2.0 technical overview describes two starts. In an SP-initiated sign-in, the user opens the application first and the SP sends an authentication request (AuthnRequest); the overview calls this the more common case. In an IdP-initiated sign-in, the user starts at an IdP portal and no request exists.

a SAML 2.0 sign-in started at the service provider, step by step: Sam's browser opens the Databricks workspace with no session; Databricks builds an AuthnRequest with ID r1 and redirects the browser to Entra ID's single sign-on URL with it; the browser carries the request to Entra ID; Entra ID authenticates Sam with his password and MFA, then builds an assertion and signs it with its private key; it returns an HTML form whose SAMLResponse field holds the base64-encoded signed XML; the browser posts that form to Databricks' assertion consumer service URL; Databricks verifies the signature with the certificate stored at setup and checks the issuer, audience, time window and that InResponseTo equals r1; it sets a session cookie

The browser carries both messages (the SAML HTTP POST binding), so the browser, or anything running in it, could alter them. That is why the assertion is signed, and why the SP trusts nothing that the signature does not cover.

A simple example: a sealed letter of introduction that you carry yourself. The club you want to enter gives you a note numbered r1 and sends you to someone who knows you. That person writes a letter about you, quotes r1 in it and seals it with their seal. You carry the sealed letter back, and the club opens it only after checking two things: that the seal is genuine, and that the letter quotes the number it gave you. You could read the letter on the way, but you cannot change it without breaking the seal. The note is the AuthnRequest, the letter is the assertion, the quoted number is InResponseTo, the seal is the signature, and you, the carrier, are the browser.

How does this apply to Acme? In the diagram above, Databricks gives Sam’s browser the note r1, Entra ID writes and signs the letter about Sam, and Databricks checks Entra ID’s seal and r1 before it sets his session cookie.

What is inside an assertion?

Decoded from base64, an assertion is readable XML with an issuer, a subject, conditions and statements, as the technical overview lists:

a simplified SAML 2.0 assertion, as if readable: Entra ID's entity ID as the issuer, an XML signature checked with the certificate from the identity provider's metadata, Sam as the subject, conditions that limit when and for which service provider the assertion is valid, an authentication statement, and attributes such as group membership; in transit it is base64-encoded inside an HTML form

What does the SP check?

Step 8 is a list of comparisons, each against something stored at setup or remembered from step 2:

what a SAML service provider checks before it trusts an assertion: the signature against the IdP certificate stored at setup, the issuer against the stored IdP entity ID, the audience against its own entity ID, the destination against its own ACS URL, NotBefore and NotOnOrAfter against its clock, InResponseTo against the ID of the AuthnRequest it sent, and the NameID against its users; each row gives the symptom when the check fails

What is an X.509 certificate?

The SAML signing key is delivered inside an X.509 certificate, the format defined in RFC 5280: a structure that binds a public key to a name and carries a validity period, signed by an issuer. On disk it is binary DER (Distinguished Encoding Rules) or PEM, the same bytes in Base64 between BEGIN CERTIFICATE lines, which is what Databricks asks you to paste “including the markers for the beginning and end of the certificate”.

an X.509 certificate as if readable, in the layout openssl prints, abridged: version 3, a serial number, the signature algorithm, the issuer, a validity period from Not Before to Not After, the subject, the subject's public key, version 3 extensions such as key usage and subject alternative names, and the issuer's signature over all of it; on disk it is binary DER or Base64 PEM; this example is self-signed, as Entra ID's SAML signing certificates are, so the issuer and subject are the same, and the three-year validity matches Entra ID's default

How do signing and verification work?

The IdP hashes the assertion and signs the digest with its private key, which never leaves Entra ID. The SP hashes what it received and checks the signature with the public key from the certificate it stored. Entra ID’s setup, per the Databricks guide, signs both the SAML response and the assertion with SHA-256.

A simple example: a signet ring and a wax seal. Only the sender owns the ring; the receiver keeps a picture of the ring’s pattern on file. To make the seal quick to check, the sender stamps it onto a short fingerprint of the letter rather than the whole letter, and the receiver works out the same fingerprint from the letter it received: change one word and the fingerprints no longer match. The ring is the private key, the picture on file is the public key in the certificate, and the fingerprint is the SHA-256 digest.

How does this apply to Acme? Entra ID owns the ring. Databricks holds only the picture, the public certificate pasted at setup, so even someone who copied Databricks’ configuration could not sign an assertion as Entra ID.

how signing and verification work: at the identity provider, the assertion is hashed with SHA-256 and the digest is signed with the private key, and the signature travels with the assertion; at the service provider, the received assertion is hashed again, the signature is checked with the public key from the stored certificate, and the assertion is accepted only if the two match; a change to one byte of the assertion, or a signature made with any other key, fails the check

Self-signed or issued by a certificate authority?

A SAML signing certificate is usually self-signed: Entra ID generates one for each SAML application, and trust comes only from the admin storing it at the SP. A TLS certificate for a website is issued by a certificate authority (CA), and trust comes from a chain up to a root the client already has.

two ways a certificate is trusted: a SAML signing certificate from Entra ID is self-signed, its issuer is itself, and the service provider trusts it only because an admin stored that exact certificate at setup; a TLS server certificate is issued by a certificate authority, and the client trusts it by following the chain from the leaf certificate, which names the hostname, through an intermediate certificate to a root certificate already in its trust store, checking each signature, the names and the validity dates

Why do certificates expire, and what is the same-morning outage?

Certificates carry a Not Before and Not After date (RFC 5280’s notBefore and notAfter) so that no key is used forever: rotating keys limits how long a stolen or weak key stays useful. Entra ID makes SAML signing certificates valid for three years by default, lets you create a new certificate that stays Inactive, and then Make certificate active when you choose; it can also email a notification address before expiry, as its certificate guide explains.

The outage comes from the SP’s static copy. The IdP switches to the new key; the SP still holds only the old certificate; every signature fails to verify, for every user, at the same second.

A simple example: a bank keeps a specimen signature of a company’s authorised signatory on file. The company appoints a new signatory overnight and nobody sends the bank the new specimen. The next morning every cheque bounces, all at once, although nothing is wrong with any of them. The specimen on file is the certificate stored at the SP, and the new signatory is the new signing key.

How does this apply to Acme? Entra ID is the company and Databricks is the bank. Databricks’ SAML dialog holds one certificate, so the moment Acme makes a new certificate active in Entra ID without pasting it into Databricks, Sam and every colleague are locked out together:

a SAML signing certificate rotation over time: certificate A is valid for three years; Entra ID creates certificate B, which stays inactive; if the service provider is given B before B is made active, sign-ins keep working through the switch and after A expires; if the service provider still holds only A, every sign-in fails at the same second, the moment B becomes active or A expires, until someone uploads B

Microsoft’s guidance for application vendors is for the SP to re-read the IdP’s federation metadata on a schedule and add the new certificate while it is still inactive. The Databricks SAML dialog, as documented, takes one x.509 certificate, so plan the switch as a short change window: make the new certificate active in Entra ID, paste it into Databricks at once, and use Test SSO before you finish.

What breaks when a SAML setting is wrong?

what breaks when a SAML setting is wrong: a wrong IdP entity ID makes assertions fail the issuer check; a wrong SP entity ID fails the audience check; a wrong ACS or Reply URL sends the browser to the wrong place or produces an error at the IdP; a wrong IdP SSO URL means the sign-in never reaches the IdP; an expired or unknown signing certificate blocks every user; a wrong NameID claim finds no user or the wrong one; and a wrong SP clock makes valid assertions look expired

What are OAuth 2.0 and OpenID Connect?

OAuth 2.0 is a framework for delegated authorisation: letting a program get limited access to an API on someone’s behalf, without being given that person’s password. It is defined in RFC 6749, The OAuth 2.0 Authorization Framework (an RFC, Request for Comments, is a numbered Internet standard document published by the IETF). The result of an OAuth flow is an access token, a credential the program sends with each API call.

RFC 6749 names four roles, and they are worth learning by name because every later section uses them:

• The resource owner: the person who owns the data and can grant access, Sam.
• The client: the program that wants access, such as a command-line tool or a web app.
• The authorisation server: the system that authenticates the owner and issues tokens, such as Entra ID, Google’s OAuth server or Databricks’ own OAuth server.
• The resource server: the API that accepts the tokens, such as the Databricks REST API.

A simple example: a valet key. You ask the hotel’s front desk to give the valet a key for your car; it starts the engine and opens the doors, but not the boot or the glovebox. The key says what its holder may do, not who the holder is. You are the resource owner, the valet is the client, the front desk is the authorisation server, the car is the resource server, and the valet key is an access token with a limited scope.

How does this apply to Acme? When Sam runs a Databricks command-line tool, he is the resource owner, the tool is the valet, Databricks’ OAuth server is the front desk, and the Databricks REST API is the car. The access token lets the tool call the API for him and says nothing to the tool about who he is.

OAuth on its own does not tell the client who the user is. An access token says “the holder may call this API”, and the client is often not meant to read it at all. Companies used OAuth for sign-in anyway, each in its own way, which is the gap OpenID Connect fills.

OpenID Connect (OIDC) is, in the words of the OpenID Connect Core 1.0 specification, “a simple identity layer on top of the OAuth 2.0 protocol”. It was finalised by the OpenID Foundation in February 2014. It adds three things to OAuth:

• An ID token: a signed statement for the client about who signed in and when (the next section opens one up).
• A UserInfo endpoint that returns more facts about the user.
• A discovery document: a JSON file that OpenID Connect Discovery requires at the issuer URL plus /.well-known/openid-configuration, listing the endpoints and a jwks_uri, the address of the IdP’s JWKS (JSON Web Key Set, RFC 7517): its public signing keys.

So the plain summary is: OAuth 2.0 is about authorisation (what a program may call), and OpenID Connect adds authentication (who the user is) on top of it. In the valet example, OpenID Connect adds a signed note from the front desk telling the valet whose car this is and when they checked in. For Acme, that note is what Databricks receives when it uses Entra ID for single sign-on over OIDC: an ID token saying the person is sam@acme.example.

the four OAuth 2.0 roles and what OpenID Connect adds: the resource owner (Sam) lets a client (a CLI or app) get an access token from the authorisation server, and the client sends it as a bearer token to the resource server (an API); OpenID Connect adds an ID token about the user, and publishes a discovery document at the issuer URL that lists the endpoints and the JWKS URL of public signing keys; Google's real discovery document is shown

You can open Google’s discovery document at accounts.google.com/.well-known/openid-configuration: its issuer is https://accounts.google.com, and its jwks_uri points at Google’s public keys.

How do a client and an OpenID provider set up trust?

OIDC setup is lighter than SAML on the key side. The client is registered with the provider (a client ID, a client secret or, with OIDC’s private_key_jwt method, a key pair, and a redirect URI); the provider publishes everything else at its issuer URL. The Databricks OIDC page asks for exactly three values: client ID, client secret and issuer URL.

setting up OpenID Connect single sign-on: the OpenID provider publishes, at its issuer URL, a discovery document listing the issuer, the authorize and token endpoints and the jwks_uri, whose JSON Web Key Set holds public keys identified by key ID; for the Databricks client registration, the admin gives Entra ID the Databricks redirect URL as the redirect URI, and Entra ID generates a client ID and a client secret, which the admin types into Databricks with the issuer URL; Databricks then reads endpoints and keys from the discovery document instead of storing a certificate

No certificate is pasted. The client reads the signing keys from the JWKS, and each token’s header names its key with a kid (key ID), so the client knows which one to use.

What are ID tokens, access tokens, refresh tokens and JWTs?

Four token words come up in every identity conversation. Here is each one, with what it is for and who is meant to read it.

• An ID token is OIDC’s proof of authentication. It is always a JWT (below), it is meant for the client, and it says who the user is, who issued the statement, who it is for, and when it expires. Google’s token types page says ID tokens are “signed but not encrypted, so that they can be decoded, verified, and interpreted by the client”, and that Google’s user and service account ID tokens are valid for one hour.
• An access token is OAuth’s proof of authorisation. It is meant for the resource server, not the client. Its format is up to the issuer: Google’s user and service account access tokens are opaque (a string with no readable structure), while other issuers use JWTs.
• A refresh token is a longer-lived credential that a client uses to get a new access token when the old one expires, without asking the user to sign in again. RFC 6749 defines it, and Google calls it a token-granting token: it cannot call an API itself, but it can be redeemed for an access token. That makes it as sensitive as a password.
• A JWT (JSON Web Token, pronounced “jot”) is a format, not a purpose. RFC 7519 defines it as three base64url-encoded parts separated by dots: a header, a payload of claims (named facts) and a signature.

A simple example: registering at a conference. You show your passport once and leave with three things. A name badge for the organisers’ own system, saying who you are: that is the ID token, a statement meant for the client. A wristband that door staff check at every session, without caring about your name: that is the access token, shown to the resource server on every call. And a voucher you can swap at the desk tomorrow for a fresh wristband, without queueing with your passport again: that is the refresh token, and why losing it is nearly as bad as losing the passport. If any of them is printed on the conference’s standard laminated card, with named fields and a hologram, that card format is the JWT: a format, not a purpose.

How does this apply to Acme? The token in the next section is a name badge, a Google ID token saying who Acme’s deployer is. It is also shown at the door: the Databricks Account API accepts Google ID tokens as credentials, as the bearer token section explains.

What does a JWT look like inside?

Here is an ID token that Acme’s deployer would send to the Databricks Account API on Google Cloud, decoded. The shape follows Google’s own example of a service account ID token on the token types page; only the values are Acme’s.

a decoded Google service account ID token for Acme's deployer, as if readable: a header naming the RS256 algorithm and key ID, a payload with the issuer accounts.google.com, the audience accounts.gcp.databricks.com, the subject and email of tf-deployer, issued-at and expiry one hour apart, and a signature checked against Google's published JWKS

Four claims matter in every JWT:

• iss (issuer): who created the token; the receiver trusts only known issuers and checks the signature with that issuer’s JWKS.
• sub (subject): who the token is about; for a service account, a numeric ID, with the readable name in email.
• aud (audience): who the token is for. Every other API must reject it, or a token sent to one service could be replayed at another. Here it is https://accounts.gcp.databricks.com, the audience the Databricks Google ID token guide asks for on the Account API.
• exp (expiry): when it stops working, in Unix time; here one hour after iat (issued at), 09:00 UTC on 22 October 2026.

Anyone can decode a JWT; nobody without the private key can change it unnoticed. So never put a secret in a claim, and never trust a JWT whose signature you have not checked.

What is a bearer token?

Most access tokens, and ID tokens when used as credentials, are bearer tokens. RFC 6750 defines the term precisely: “any party in possession of a bearer token (a ‘bearer’) can use it to get access to the associated resources (without demonstrating possession of a cryptographic key)”. It is sent as Authorization: Bearer <token>, so a bearer token in a log file or a screenshot works until it expires; short lifetimes limit the damage. Google’s token types page also describes bound tokens, which only work together with a second credential such as a client certificate.

A simple example: a cinema ticket or a banknote. Whoever holds it can use it, and nobody at the door asks whether it is yours. That is why Acme’s pipeline should mint its tokens just before each call and never print them in a CI log.

Which OAuth flows exist, and when is each used?

A flow (OAuth’s word is grant) is the sequence of requests a client uses to get a token. Which one you use depends on two questions: is a person present, and can the client keep a secret? The current guidance is RFC 9700, Best Current Practice for OAuth 2.0 Security, published in January 2025, which updates the original advice in RFC 6749.

Authorisation code with PKCE

This is the flow for an application acting for a signed-in person, and it is the flow OIDC sign-in uses. The steps split into two channels. The front channel goes through the browser, using redirects, so anything in it can be seen or altered there. The back channel is a direct server-to-server call from the client to the token endpoint, so the tokens never pass through the browser.

PKCE (Proof Key for Code Exchange, pronounced “pixie”, RFC 7636) protects the one secret that crosses the front channel, the one-time authorisation code. The client sends only the hash of a random code verifier at the start, and must present the verifier itself to redeem the code. RFC 9700 says public clients (apps that cannot keep a secret, such as a CLI on a laptop) “MUST use PKCE”, and the Databricks U2M page generates a verifier and challenge in the same way. The nonce is a random value the client sends in step 3 and expects back inside the ID token.

A simple example: a cloakroom in a crowded venue. When you hand in your coat, you whisper a secret word, and the attendant writes down only a fingerprint of it. The ticket you get is visible to anyone standing nearby, and on its own it would be enough to collect the coat. So at collection the attendant also asks for the word, checks it against the fingerprint, and only then hands the coat over. The ticket is the authorisation code, the secret word is the code verifier, its fingerprint is the code challenge, and the coat is the tokens.

How does this apply to Acme? When Sam signs in to a Databricks command-line tool on his laptop, the tool is a public client: it has no secret of its own, so it makes up a fresh code verifier for each sign-in. Anyone who catches the code on its way back through the browser still cannot redeem it.

OpenID Connect sign-in with the authorisation code flow and PKCE, step by step: the client creates a code verifier, a code challenge and a nonce, and redirects the browser to the authorize endpoint with the challenge and nonce; Sam signs in and consents; the provider redirects the browser back to the redirect URI with a one-time code; these steps are the front channel, through the browser; the client then posts the code, the verifier and its client credentials to the token endpoint directly, the back channel; the provider checks the verifier and returns an ID token, an access token and a refresh token; the client validates the ID token's signature against the JWKS key with the matching kid, and checks iss, aud, exp and nonce, then calls the API with the access token

How does the client validate the ID token?

OpenID Connect Core section 3.1.3.7 lists the checks. They mirror the SAML checks above, with the JWKS in place of a stored certificate:

how a client validates an OpenID Connect ID token: the signature with the JWKS public key whose kid matches the token header, fetching the JWKS again if the kid is unknown; iss against the configured issuer; aud against its own client ID; exp against the current time; and nonce against the value it sent in the authorisation request; any failure means the token is rejected

How do SAML and OIDC compare for the same sign-in?

Both protocols have Entra ID sign a statement about Sam for Databricks. The practical difference is where the verifier gets the key. OIDC Core’s section on rotating signing keys says the verifier “knows to go back to the jwks_uri location to re-retrieve the keys when it sees an unfamiliar kid value”, and Microsoft says its signing keys roll over on a periodic basis and applications “should be able to programmatically handle the key rollover process”. A SAML SP that holds a pasted certificate has no such path.

SAML 2.0 and OpenID Connect for the same Acme sign-in to Databricks: in both, Entra ID signs; SAML carries an XML assertion through a browser POST, OIDC returns a JWT ID token from the token endpoint; with SAML, Databricks checks the signature with a certificate pasted at setup, a static copy that must be replaced by hand when Entra ID rotates it; with OIDC, Databricks fetches public keys from the JWKS URL in the discovery document and fetches again when it sees an unfamiliar key ID, so key rotation needs no manual step and the same-morning outage is rare

Client credentials (machine to machine)

When no person is involved, the client authenticates as itself. In the client credentials flow, the client sends its client ID (a public identifier) and client secret (a password for a program) to the token endpoint and receives an access token. This is often called M2M, machine to machine. Databricks’ OAuth M2M page shows it for a Databricks service principal: a POST with grant_type=client_credentials to https://accounts.gcp.databricks.com/oidc/accounts/<account-id>/v1/token, returning a token with "expires_in": 3600, one hour.

the OAuth client credentials flow, machine to machine: Acme's nightly job, running as a Databricks service principal, posts its client ID and client secret to the account token endpoint with grant_type=client_credentials; it receives an access token valid for one hour; it calls the Databricks API with that bearer token; no person is involved, and the client secret is a long-lived secret to store and rotate

Device authorisation

The device authorisation flow, RFC 8628, is for clients that cannot open a browser or are awkward to type into: the RFC’s examples are smart TVs, media consoles and printers, and in practice it is common for command-line tools on remote servers. The client shows a short code and a URL; Sam opens the URL on his phone or laptop, signs in, types the code, and the client, which has been polling the token endpoint, receives its tokens.

Token exchange

Token exchange, RFC 8693, defines a Security Token Service (STS): an endpoint that takes one token and returns another. The client sends a subject_token it already holds, with grant_type=urn:ietf:params:oauth:grant-type:token-exchange, and gets back a token from a different issuer or for a different audience. This is the building block behind keyless automation across clouds: a CI job’s own token from GitHub becomes a Google token, with no stored secret anywhere. Google’s Workload Identity Federation says plainly that it “follows the OAuth 2.0 token exchange specification”, and Google’s STS token method requires exactly that grant type.

token exchange in practice, Google Cloud Workload Identity Federation from GitHub Actions: the job asks GitHub for an OIDC token whose subject names Acme's repository; it sends that token to Google's Security Token Service as an RFC 8693 token exchange and gets a short-lived federated access token; it uses that token to call the IAM Credentials API and receives an access token for the tf-deployer service account; Terraform then calls Google Cloud APIs; no key exists at any step

Which flows should you avoid?

RFC 6749 also defined two flows that RFC 9700 now warns against. The implicit grant returned the access token directly in the browser’s redirect URL; RFC 9700 says clients “SHOULD NOT use the implicit grant” because tokens in URLs leak. The resource owner password credentials grant had the client collect the user’s password and send it on; RFC 9700 says it “MUST NOT be used”, because it teaches users to type their password into places other than the IdP. If a vendor’s integration still asks for one of these, treat it as a finding.

the OAuth flows and when each is used: authorisation code with PKCE for apps acting for a signed-in person; client credentials for a service acting as itself; device authorisation for CLIs and devices without a browser; token exchange for swapping a token you already hold for another; the refresh token grant to renew access tokens; the implicit and password grants are legacy and should not be used

My view: when you read any integration guide, find the flow first. Almost every security question about the integration (who holds a secret, whether a person is involved, how long the token lives) is answered by that one word.

What are SCIM and automated identity management?

SSO solves sign-in, not the user list. Databricks must know supply-analysts before Sam’s first sign-in, so that access can be granted in advance, and must know when he leaves. Creating identities in an application is provisioning; removing or disabling them is deprovisioning.

SCIM, the System for Cross-domain Identity Management, is the open standard for this. SCIM 2.0 is two RFCs: RFC 7643 defines the schema (what a User and a Group look like in JSON, including an active flag) and RFC 7644 defines the protocol: REST endpoints named /Users and /Groups, and the HTTP methods to create (POST), replace (PUT), update (PATCH) and delete (DELETE) them. In practice the IdP is the SCIM client and pushes changes: when Sam is added to the Databricks app in Entra ID, Entra’s provisioning service calls the application’s SCIM endpoint with a bearer token that the application issued for that purpose. Microsoft’s app provisioning overview describes the same model from Entra’s side.

Two other approaches sit next to SCIM, and the names are worth getting right:

• Just-in-time (JIT) provisioning creates a user at their first sign-in, from the facts in the SSO assertion or ID token. It is simple, but it acts only when someone signs in, so on its own it has no moment at which to remove a leaver, and a group cannot be granted access before any of its members has signed in. Databricks documents it on Automatically provision users (JIT).
• Automatic identity management is a vendor term, not a standard. In Databricks it means the application reads users and groups directly from the IdP when they are needed, instead of waiting for the IdP to push them. The Databricks on Google Cloud automatic identity management page (dated 15 October 2026 when I checked) describes it for Microsoft Entra ID and Okta, says that with Entra ID it also syncs nested groups and service principals (which SCIM provisioning does not, and Okta does not support), says JIT provisioning is always on when it is enabled, and recommends it over SCIM. It requires workspaces that use Databricks “identity federation”, which the next section untangles.

A simple example: the staff gym at a company. HR can send the gym a list of joiners and leavers every week: that is SCIM, where the IdP pushes. The gym can sign people up the first time they turn up with a staff badge: that is JIT, quick, but the gym never hears about leavers. Or the gym can ring HR whenever it needs to check who works there and which teams they are in: that is automatic identity management, where the application reads from the IdP when it needs to.

How does this apply to Acme? Entra ID is HR and the Databricks account is the gym. Part 2 shows Acme letting the gym ring HR, automatic identity management, because Acme’s Entra groups are nested and SCIM cannot sync nested groups.

three ways identities reach an application such as the Databricks account: SCIM, where the identity provider pushes creates, updates and deactivations to the app's /Users and /Groups endpoints with a bearer token; just-in-time provisioning, where the app creates a user from the SSO token at first sign-in; and automatic identity management, Databricks' term for reading users and groups directly from Entra ID or Okta when they are needed; a SCIM user record is shown

Two details from the Databricks documentation are worth carrying into any design. The SCIM page notes that if Google Cloud Identity is your only IdP, there is no built-in SCIM integration. And the automatic identity management page warns that when a user is deleted from the IdP, Databricks deactivates the account but does not automatically revoke their personal access tokens. Deprovisioning is only as good as the credentials it actually removes. Part 2 applies all of this to Acme’s Databricks account.

What does federation mean?

In general, identity federation means one system trusting another system’s identities instead of keeping its own copy. The SP in a SAML sign-in “federates” with the IdP: it accepts Entra ID’s word for who Sam is. Everything else in this section is a variation on that idea, which is why so many products use the word.

What is the general pattern behind every federation?

Strip away the product names and every federation in this post is the same six steps. An issuer signs a token. A holder, Sam’s browser or a CI job, carries it. A receiver checks it and decides who the holder is in its own world. The only thing the two systems share is trust configuration, set up once: the issuer’s URL, the audience the receiver expects, a rule about which subjects are allowed, and where to find the issuer’s public keys.

the pattern under every federation: once at setup, the receiver records which issuer, audience and subjects it trusts; then the issuer signs a short-lived token about a subject, the holder presents it, the receiver verifies the signature with the issuer's published keys, checks issuer, audience, expiry and subject, maps the subject to a local identity and returns its own session or token, so no passwords or private keys are shared

Step 5 is where implementations differ most. A SAML service provider maps the subject to a user it already knows. Google’s Security Token Service maps it to a principal in a workforce or workload identity pool. Databricks maps it to a user or a service principal in the account. Steps 3 and 4 are the same checks the ID token validation section described, which is why the token sections came first.

Sync-based or sync-less?

This is the split inside user to machine. Both start with single sign-on; they differ in whether the receiver keeps its own copy of you.

A simple example: two ways to run the door of a private event. With a guest list, the organiser sends the names in advance and the door checks each guest against it: you can see who is coming before anyone arrives, but someone must keep the list current. With wristbands, a trusted partner hands out coloured wristbands and the door admits anyone wearing a green one: there is no list to maintain, but the door knows nothing about a guest until they arrive, and everything rides on the partner handing out the right colours. The guest list is sync-based; the wristband colour is the group claim in a sync-less token.

user to machine in two ways: sync-based, where Entra ID first pushes users and groups into the application's directory with SCIM, then Sam signs in, the token arrives and the application looks him up in its copy, which allows grants before first sign-in and removes leavers but needs a provisioning connection; and sync-less, where Sam signs in, the token arrives with his groups, and the receiver maps the claims to a principal set without storing any copy, which leaves nothing to keep in step but depends on the identity provider sending the right groups

With sync-based identity, the identity provider pushes users and groups to the receiver ahead of time, as the SCIM section described, and removes them when people leave. The receiver can show Sam in a permissions screen before he has ever signed in, and an administrator can grant his group access on day one. The cost is a provisioning connection that someone has to keep healthy, with its own credentials and its own failure modes.

With sync-less identity, the receiver stores nothing between sign-ins. It validates each token as it arrives and grants on what the token says: the subject, or a group claim. Google describes Workforce Identity Federation this way, and grants in it are made to principal sets, such as “everyone whose token carries the group supply-analysts”. The cost moves to the identity provider, which must put the right groups into every token, and there is no user record to look at between sign-ins.

How the vendors in this series implement each one:

• Sync-based on Google Cloud: Cloud Identity or Google Workspace accounts, filled from Active Directory with Google Cloud Directory Sync or from Entra ID with Microsoft’s provisioning connector. People then sign in with SSO to accounts that already exist.
• Sync-less on Google Cloud: Workforce Identity Federation, where people from Entra ID use Google Cloud with no Google account at all.
• Sync-based on Databricks: SCIM or automatic identity management into the account. JIT provisioning sits half-way: it creates a user at first sign-in but does not remove leavers.
• Token trust on Databricks: account-wide OAuth token federation accepts tokens from Acme’s IdP, but it signs people in as Databricks users, so in practice it sits on top of a synced directory rather than replacing it.

Many companies still run sync-based identity everywhere, and that is a reasonable choice, not a legacy mistake: licences, audits and pre-approved access all want a user record that exists before the first sign-in. Sync-less is newer and lighter, and it fits best where access is decided by group membership alone.

How does this apply to Acme? Acme runs its Databricks door with a guest list: Entra ID sends users and groups into the Databricks account ahead of time, so supply-analysts can be granted a table before Sam’s first sign-in, and he disappears from the list when he leaves. For your own case, a quick test:

Sync-based or sync-less?                                       Ask in order.
1. Must you grant access, count licences or audit users        yes -> sync-based
   before anyone's first sign-in?
2. Is access decided by group membership alone, and does       yes -> sync-less fits
   the IdP put the right groups into every token?
3. Unsure?                                                         -> sync-based, the common default

Workforce or workload, sign-in or token?

Two distinctions sort most of the product names out.

Workforce or workload. A workforce identity is a person: an employee, a contractor, a partner. A workload identity is a program: a CI job, a virtual machine, a container, a scheduled script. Products are often split along this line, because people sign in through browsers with MFA and programs do not.

Sign-in federation or token federation. Federation for sign-in means an application accepting an IdP’s assertion or ID token for a person (SSO). Federation for tokens means an STS accepting a token from an outside issuer and exchanging it for one of its own (the token exchange flow above). The second is what people usually mean by workload identity federation as a pattern: an outside program presents a short-lived OIDC or SAML token from its own platform, and receives a short-lived token for the cloud, so no long-lived key is ever created. Google, Microsoft and Databricks all implement this pattern, each under a similar name.

How do Google Cloud, Microsoft and Databricks implement it?

Here is how the word is used across the three vendors in this series:

the word federation across vendors: in general, identity federation means trusting another party's identity system; SAML or OIDC single sign-on federates people into one app; Google Cloud Workforce Identity Federation brings people from an external identity provider into Google Cloud; Google Cloud Workload Identity Federation exchanges an external workload token for a short-lived Google token; Microsoft Entra workload identity federation does the same for Entra; Databricks OAuth token federation exchanges an identity provider's JWT for a Databricks token, account-wide or as workload identity federation for one service principal; Databricks identity federation for workspaces means workspaces use account-level identities

A few of these deserve one more sentence each, from the vendors’ own pages.

• Google Cloud’s Workforce Identity Federation is “sync-less”: it federates identities from your IdP and does not store user accounts in Google Cloud, so people from Entra ID can use the Google Cloud console without a Google account.
• Google Cloud’s Workload Identity Federation lets “on-premises or multicloud workloads” access Google Cloud “by using federated identities instead of a service account key”.
• Microsoft’s workload identity federation configures an Entra app registration or user-assigned managed identity “to trust tokens from an external identity provider (IdP), such as GitHub or Google”.
• Databricks’ OAuth token federation lets users and service principals “exchange JWT (JSON Web Tokens) tokens from your identity provider for Databricks OAuth tokens”. It comes in two types: account-wide token federation and workload identity federation for one service principal, configured with a federation policy that names the accepted issuer, audience and subject.

Which “federation” is the exception?

Databricks uses identity federation for something unrelated to tokens. A workspace “enabled for identity federation” uses the users, groups and service principals of the Databricks account rather than its own list, as the identity best practices page describes. The name makes sense once you know the history. Databricks began with each workspace as its own island: its own users, its own groups, its own SCIM connection, its own grants. When the account level arrived, identity moved up to it, and workspaces now “federate” to the account’s list instead of keeping a private one. Groups created the old way are now called workspace-local groups, and Databricks recommends account groups instead.

the Databricks exception: before identity federation, each workspace kept its own users, workspace-local groups, SCIM connection and grants; with identity federation, users, groups and service principals live once in the Databricks account, synced by SCIM or automatic identity management, and workspaces receive assignments; no token crosses a trust boundary, which is why it is not the same as OAuth token federation

So when someone at Acme says “we turned on identity federation”, ask which one they mean. If it is the workspace setting, nothing about tokens changed. If it is OAuth token federation, Databricks now trusts tokens from another issuer, and that is sync-less user sign-in (1B) or dynamic token trust for machines (2B).

One more fact matters for later. Google’s token types page lists ID tokens for Google users, service accounts and agent identities. A person who signs in through Workforce Identity Federation receives a federated access token, not a Google account, and that list has no ID token type for them. Which of these federations Databricks on Google Cloud supports where, and what that means for automation, is the subject of Part 2, and it is the reason this table exists.

What are machine identities, and why are keys risky?

A machine identity (Microsoft says workload identity, Google says service account) is an identity for software rather than a person. Every cloud has one, under a different name:

• Google Cloud: a service account, which Prerequisite 2 introduced.
• Microsoft Azure: a managed identity for software running on Azure, or a service principal, the identity that Entra ID creates for a registered application.
• AWS: an IAM role, which a program assumes to receive temporary credentials.
• Databricks: a service principal, a Databricks identity for automation, as the service principals page explains.

“Service principal” names two different objects: one in an Entra tenant and one in a Databricks account. Automatic identity management with Entra ID can sync the first into the second.

Long-lived secrets or short-lived tokens?

A machine can prove its identity in two ways. It can hold a long-lived secret: a service account key file, a client secret, an access key, a personal access token. Or it can obtain a short-lived token from the platform it runs on, at the moment it needs one, with no secret stored anywhere.

A simple example: you can give a cleaner a copy of your house key, or have a smart lock send them a code that works for two hours on Tuesday. The copied key works for whoever holds it until you change the locks; the code is useless by Tuesday evening, so there is nothing worth stealing. The copied key is a service account key or a client secret; the two-hour code is a short-lived token.

Long-lived secrets are risky for plain reasons. They work until someone revokes them. They get copied into laptops, CI variables, logs and repositories, where nobody remembers them. And every copy is a working credential for whoever finds it. Google’s best practices for service account keys says keys “can become a security risk if not managed carefully”, and recommends reducing the number in circulation and offering alternatives before anyone creates one. AWS’s IAM best practices say the same in their own words: “require workloads to use temporary credentials with IAM roles”.

How does this apply to Acme? Acme’s Terraform pipeline uses the smart-lock code. On each run GitHub signs a fresh token for the job, Google exchanges it, and the job acts as tf-deployer with short-lived tokens. No copy of a tf-deployer key exists anywhere to lose:

machine to machine in two ways: with a stored secret, a CI job sends a client ID and secret to a token endpoint, receives an access token valid for about an hour and calls the cloud API, and the secret works for anyone holding a copy; with workload identity federation, the CI platform signs a short-lived OIDC token, the target security token service checks it against a trust policy naming the repository and branch, and exchanges it for a short-lived token, so nothing long-lived exists to leak while the permissions stay the same

How does machine to machine with a stored secret work?

This is the static branch of machine to machine (2A), and the OAuth name for it is the client credentials flow. The program holds a client ID and a secret, sends both to a token endpoint, and receives an access token, usually valid for about an hour. Every platform has its own version:

• Google Cloud: a service account key, a JSON file containing a private key. The program signs a short JWT with it and exchanges that for an access token, so the key file is the secret.
• Microsoft Entra ID: a client secret or a certificate on an app registration, used in the client credentials grant.
• AWS: an access key ID and secret access key for an IAM user. There is no token endpoint here: the program signs each request with the key.
• Databricks: OAuth machine-to-machine for a service principal, using a client ID and an OAuth secret. A personal access token is the older, simpler version of the same thing: one string that works as a bearer credential until it expires or is revoked.

The flow is safe in itself. The weakness is the secret, which has to be created, stored, copied into CI, rotated and eventually revoked, and every copy along the way works for whoever finds it.

How does secretless workload identity federation work?

This is the dynamic branch of machine to machine (2B), and it is the federation pattern from earlier with a machine as the holder. The platform the program runs on signs a short-lived OIDC token about it. For a GitHub Actions job, the subject names the repository and branch, for example repo:acme/platform:ref:refs/heads/main. The target platform holds a trust policy that names that issuer, the expected audience and the allowed subjects. The job presents its token, the target exchanges it for a short-lived token of its own, and no secret is ever created. Every platform implements the same idea:

• Google Cloud: a workload identity pool and provider hold the trust settings. Google’s Security Token Service exchanges the external token for a federated token, which can be granted roles directly or used to impersonate a service account. Acme’s pipeline does the second: it becomes tf-deployer without a key file.
• Microsoft Entra ID: a federated identity credential on an app registration or user-assigned managed identity names the issuer, subject and audience. The external token replaces the client secret in the token request.
• AWS: an IAM OIDC identity provider plus a role trust policy, and the program calls AssumeRoleWithWebIdentity to receive temporary credentials.
• Databricks: OAuth token federation with a federation policy on one service principal, naming the issuer, audiences and subject. The program exchanges its token at the Databricks token endpoint for a Databricks OAuth token. An account-wide federation policy does the same for users.

What trust and which roles does dynamic token trust need?

Static secrets have one trust decision: whoever holds the secret is the identity. Dynamic token trust splits that into three separate decisions, and most failed set-ups are one of the three missing.

A simple example: a visitor arriving at an office. Reception checks the visitor’s ID against today’s expected list: may this person come in? It then hands over a contractor badge rather than an employee badge: who do they become inside? The badge’s door settings decide which floors open: what may that badge do? Three decisions, made by three different settings, and a visit fails if any one is wrong. Here are the same three on Google Cloud and Databricks, with Acme’s pipeline as the visitor:

1. Who may sign in? The receiver checks the token’s issuer, audience and claims against a trust configuration. On Google Cloud that is the workload identity pool provider and its attribute condition. On Databricks it is a federation policy.
2. Which identity do they become? On Google Cloud the federated token is a principal set with no permissions of its own. To act as a service account such as tf-deployer, that principal set needs a role on the service account itself. Google’s federation guide grants Workload Identity User (roles/iam.workloadIdentityUser). Service Account Token Creator (roles/iam.serviceAccountTokenCreator) is the role that lets any principal mint access tokens and ID tokens for a service account, and it is what a person, another service account or Databricks’ Google ID token authentication relies on. Acme grants Token Creator to its pipeline because the same job also mints a Google ID token for Databricks, as Part 2 shows. On Databricks, decisions 1 and 2 live in one place: the federation policy is attached to one service principal, so a token that passes the policy becomes that service principal.
3. What may that identity do? The service account’s IAM roles on projects, or the service principal’s entitlements and Unity Catalog grants. Nothing about federation changes this step.

dynamic token trust involves three separate decisions: on Google Cloud, a GitHub OIDC token is checked by the Security Token Service against a workload identity pool provider and its attribute condition, which decides who may sign in; the resulting federated principal set has no roles of its own and needs Workload Identity User or Service Account Token Creator on the tf-deployer service account to act as it, which decides who may become the service account; then tf-deployer's own project roles decide what it may do; on Databricks, the token is exchanged at the account token endpoint and a federation policy on one service principal decides both who may sign in and which service principal they become, and that service principal's grants decide what it may do; either Google role should be granted on the one service account and scoped to one repository, never on the whole project

Two rules follow. Grant the role in decision 2 on the one service account, never on the project: a project-level grant lets the holder become every service account in that project. And scope it to the narrowest principal set you can, one repository and ideally one branch or environment, so a token from any other repository fails decision 2 even if it passes decision 1.

Sync-less, keyless or secretless: which word is right?

All three are in use, and each describes what is not stored. Google calls Workforce Identity Federation sync-less because no user accounts are copied into Google Cloud. Google’s own guidance for GitHub Actions calls Workload Identity Federation keyless authentication, because no service account key exists. Secretless is the general word for the same idea across vendors. If a design document uses one of them, the useful follow-up question is the same: which issuer is trusted, for which audience, and which subjects does the policy allow?

Long-lived or short-lived: the same comparison on one page

machine identities on each platform and how each gets a token without a stored secret: a Google Cloud service account through an attached service account, impersonation or Workload Identity Federation, with a JSON key as the long-lived secret to avoid; an Azure managed identity or Entra service principal through managed identities or workload identity federation, with a client secret or certificate to avoid; an AWS IAM role through instance profiles or STS AssumeRole, with IAM user access keys to avoid; a Databricks service principal through OAuth token federation, with an OAuth secret or personal access token to avoid

What are impersonation and secret rotation?

Impersonation is a principal asking for a short-lived token as another identity, because it has been granted the right to. On Google Cloud, a principal with the Service Account Token Creator role on a service account can mint tokens for it, as service account impersonation describes. Acme’s pipeline uses this: its federated identity impersonates tf-deployer, so the permissions live on one service account and no key exists. Impersonation also leaves an audit trail of who acted as whom. RFC 8693 draws a related distinction between impersonation (the new token simply identifies the original subject) and delegation (the token records that one party is acting for another).

Secret rotation is replacing a secret with a new one on a schedule, and revoking the old one. It limits how long a leaked secret stays useful. Where a secret cannot be avoided, such as an OAuth client secret for a tool that does not support federation, store it in a secret manager (a service built to hold and version secrets, such as Google’s Secret Manager or a Databricks secret scope), give it an expiry, and rotate it. Part 6 covers where Databricks secrets should live.

My view: the question to ask of every integration is “where is the secret?”. The best answer is “there is none, the token comes from the platform”. The second best is “in a secret manager, with an owner and a rotation date”.

What is Application Default Credentials, and why doesn’t gcloud auth login set it?

One more piece of machine identity shows up in every Google Cloud project: ADC. It explains why the same Python file runs on a laptop, in CI and on a cluster without a key in the code. Application Default Credentials is not a tool you install. Google’s ADC page describes it as a strategy the authentication libraries use to find credentials from the environment. In Python the search runs inside google.auth.default(), which is why storage.Client() works with no key in sight, and Terraform’s Google provider authenticates through ADC too.

ADC checks three places, in this order, and uses the first one that has credentials:

1. The GOOGLE_APPLICATION_CREDENTIALS environment variable, pointing at a credential file: usually a Workload Identity Federation configuration file, or a service account key, which the section on keys argues against.
2. The local ADC file written by gcloud auth application-default login, at ~/.config/gcloud/application_default_credentials.json on Linux and macOS.
3. The metadata server, when the code runs on a Google Cloud resource with an attached service account, such as a Compute Engine VM, a Cloud Run service or a Databricks classic cluster. It returns a token for that service account.

how Application Default Credentials finds credentials: it checks the GOOGLE_APPLICATION_CREDENTIALS environment variable, then the local ADC file written by gcloud auth application-default login, then the metadata server for the attached service account, and uses the first match; the same code finds the local file on a laptop and the metadata server on a cluster

The trap is that there are two separate sign-ins. gcloud auth login authorises the gcloud CLI itself, and the gcloud auth application-default login reference says that command has no effect on the accounts set up by gcloud auth login. The other direction holds as well: the ADC setup page says the gcloud CLI does not use ADC. So you can be signed in to gcloud while your code and Terraform find no credentials at all.

two separate sign-ins: gcloud auth login authorises the gcloud CLI and ADC does not see it unless you add --update-adc; gcloud auth application-default login writes the local ADC file that client libraries and Terraform use, optionally as an impersonated service account

For local client-library code I prefer the impersonated form. With the Service Account Token Creator role on tf-deployer, this command makes code on my laptop act as tf-deployer with short-lived tokens:

gcloud auth application-default login \
  --impersonate-service-account=tf-deployer@acme-dbx-admin.iam.gserviceaccount.com

It is the same impersonation as the impersonation section describes. Two honest caveats. Google notes that not every authentication library supports a local ADC file made this way. And when Terraform’s Databricks provider already impersonates tf-deployer through google_service_account (see Part 2: How does automation authenticate?), keep the plain user login: otherwise tf-deployer would be asked to impersonate itself, the self-impersonation Google warns against.

What are MFA and conditional access?

Multi-factor authentication (MFA) requires two or more different kinds of evidence at sign-in. Microsoft’s MFA overview lists the kinds: something you know (a password), something you have (a phone or a hardware key) and something you are (a fingerprint). The point is that one stolen factor is not enough. Not all second factors are equal: Google’s 2-Step Verification guidance calls security keys “the most secure form of 2SV” because they protect against phishing, while one-time codes can be typed into a fake page.

Conditional access decides, at sign-in, whether the context of a request is acceptable. Microsoft describes Entra Conditional Access as “if-then statements”: if a user wants to access an application, then they must complete an action, such as MFA, or sign in from a compliant device. Google Cloud has a similar idea in Access Context Manager, which defines access levels based on attributes such as IP address and device.

A simple example: a bank card. Paying needs the card (something you have) and the PIN (something you know): that is MFA. The bank declining a payment from an unusual country until you confirm it in its app is conditional access: the same card and PIN, judged by the context of the request.

The reason these matter for this series is where they run. With SSO, MFA and conditional access run at the IdP, before the application sees anything. When Sam signs in to Databricks through Entra ID, Acme’s conditional access policies apply on the way in, and Databricks receives only the result. That is a strong argument for SSO on its own. It also explains why a machine credential, which never passes through that sign-in, needs its own controls: short lifetimes, narrow permissions and federation instead of secrets.

How do role-based and attribute-based access control differ, and what is least privilege?

Once a request is authenticated, an authorisation model decides the answer. Two models cover almost everything you will meet.

Role-based access control (RBAC) grants permissions to roles, and roles to principals. NIST (the US National Institute of Standards and Technology) describes it in its RBAC project: “each user is assigned one or more roles, and each role is assigned one or more privileges”. Google Cloud IAM is RBAC: a role such as roles/storage.objectViewer is a bundle of permissions bound to principals. Unity Catalog grants are RBAC in the same spirit: SELECT on a table granted to the group supply-analysts.

Attribute-based access control (ABAC) decides from attributes instead. NIST SP 800-162 defines it as authorisation “determined by evaluating attributes associated with the subject, object, requested operations, and, in some cases, environment conditions against policy”. The attributes can be a tag on a column, the user’s department, or the time of day. Google Cloud’s IAM Conditions add attribute checks to RBAC bindings, and Unity Catalog has attribute-based access control with row filter and column mask policies driven by governed tags. Part 6 shows both in Databricks.

A simple example: a hospital. RBAC is “every nurse’s badge opens the ward doors”: the badge carries a role, and the role carries the access. ABAC is “a badge opens a ward only if its holder is on shift and assigned to that ward”: the decision reads attributes of the person, the door and the time. The common misconception is that ABAC replaces RBAC; in practice it adds conditions on top of role grants, as IAM Conditions and Unity Catalog policies do.

How does this apply to Acme? RBAC is SELECT on the orders table granted to supply-analysts: Sam can read it because he is in the group. ABAC is a policy that masks any column tagged pii for everyone outside pii-readers, whichever table the column is in:

role-based and attribute-based access control: with RBAC, Sam is a member of supply-analysts, and the group holds a role or privilege such as SELECT on the orders table; with ABAC, a policy evaluates attributes, such as a pii tag on a column and whether the user is in pii-readers, and masks the column for everyone else; least privilege applies to both, granting only what the job needs, at the narrowest scope and for the shortest time

Least privilege is the rule that applies to both models: grant only the permissions a principal needs for its job, on the narrowest resource, for the shortest time. Google’s IAM security guidance puts basic roles first in its least-privilege list for a reason: they “include thousands of permissions across all Google Cloud services”. In practice least privilege means three habits. Grant to groups, so that leaving a group removes access. Grant low in the hierarchy, on a bucket rather than a project. And prefer temporary grants for rare tasks, such as an Owner role held only for the minutes it takes to create a workspace.

What do encryption in transit, encryption at rest and customer-managed keys mean?

Security reviews ask about encryption in two places: while data moves, and while it is stored.

Encryption in transit: TLS and certificates

Encryption in transit protects data on the network. On the web it means TLS, Transport Layer Security, the protocol behind HTTPS; the current version is TLS 1.3, RFC 8446. The server presents an X.509 certificate, and the client checks its name, its validity dates and its chain up to a trusted root, as the CA-issued side of the earlier diagram shows; only then do both sides agree session keys. Google’s encryption in transit paper says requests to Google Cloud services over HTTPS are secured with TLS. Mutual TLS (mTLS) adds a client certificate, so both sides authenticate; it is what makes the bound tokens mentioned earlier possible.

Encryption at rest, envelope encryption and CMEK

Encryption at rest protects stored data, so that a stolen disk or a misdirected backup is unreadable. Google’s default encryption at rest page says that “all data that is stored by Google is encrypted at the storage layer using the Advanced Encryption Standard (AES) algorithm, AES-256”, with no action from you.

Large systems do this with envelope encryption, which Google’s envelope encryption page defines in two terms. Data is encrypted with a data encryption key (DEK). The DEK is then encrypted, or wrapped, by a key encryption key (KEK). The encrypted data and the wrapped DEK are stored together; to read the data, the service asks the key service to unwrap the DEK. The name describes the shape: the data key travels inside an envelope that only the KEK can open. It lets a system encrypt huge amounts of data locally while the one key that matters stays in a key management service.

A simple example: a holiday cottage with a key safe. The front door key hangs in a small key safe by the door, and only the owner knows the safe’s code. Guests get in quickly, but if the owner changes or forgets the code, nobody gets in. The door key is the DEK, the key safe is the wrapping, and the code is the KEK. With a customer-managed key, you hold the code instead of the letting agent.

A customer-managed encryption key (CMEK) is a KEK that you create and control in Cloud KMS, instead of one Google manages for you. Prerequisite 2 introduced the term. What CMEK gives you is control over access to the key: the service reaches it through its service agent, and the CMEK page says that if you revoke that role, or disable or destroy the key, the data cannot be accessed. It also warns that some services can lose data permanently if a key stays unavailable for too long, so disabling a key is a deliberate act, not a test. Part 6 covers CMEK for Databricks workspaces.

How does this apply to Acme? If Acme chooses CMEK for its workspace storage, which Part 6 weighs up, Acme holds the key safe code in Cloud KMS. Disabling that one key would make the data unreadable for everyone, Databricks included: that is the control Acme’s security team wants, and the risk it has to guard.

encryption at rest with envelope encryption: data is encrypted with a data encryption key, which is itself wrapped by a key encryption key held in Cloud KMS; the ciphertext and the wrapped data key are stored together; with a customer-managed key, Acme controls that key encryption key, and disabling it, destroying it or revoking the service's role makes the data unreadable

What are IP ranges, CIDR notation and DNS?

Three network ideas appear in security conversations constantly, and later parts use them freely.

CIDR notation (Classless Inter-Domain Routing, RFC 4632) writes a block of IP addresses as an address and a prefix length: 10.10.0.0/22 means the first 22 of the 32 bits are fixed, so the block has 2^(32−22) = 1,024 addresses, from 10.10.0.0 to 10.10.3.255. A smaller prefix means a bigger block: a /24 has 256 addresses, a /16 has 65,536. Google Cloud keeps four addresses of every subnet’s primary range for itself, the first two and the last two, as the subnets page explains. Private ranges, which are never routed on the public internet, come from RFC 1918: 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. In security, CIDR blocks appear in firewall rules and in IP access lists, which allow sign-in only from approved networks; Databricks IP access lists work this way for the account console and workspaces, on the Premium plan.

DNS, the Domain Name System (RFC 1034), turns names such as a workspace URL into IP addresses. A private DNS zone answers only inside chosen networks: Cloud DNS describes a private zone as “visible only from one or more Virtual Private Cloud (VPC) networks that you specify”, as its overview says. Private zones are how a name keeps working while the traffic behind it moves to a private endpoint, which is why DNS appears in every private connectivity design. Part 3 uses all three ideas for Acme’s network.

What can go wrong?

• The SAML signing certificate rotates or expires and the SP still holds the old one. Every sign-in fails at the same second. Add a notification address in Entra ID, create the new certificate early, and switch both sides in one planned window, then run Test SSO.
• An entity ID or ACS URL is typed wrong. Assertions fail the issuer or audience check, or the browser is sent to the wrong place. Copy values from the other side’s screen or metadata, never retype them.
• The audience is wrong. A token minted for one API is rejected by another, or worse, an API that does not check aud accepts tokens meant for someone else. Always mint for the exact audience, and always check it when you build a receiver.
• A bearer token leaks. It turns up in a CI log or a support ticket and works until it expires. Keep lifetimes short, never print tokens, and prefer refresh-free machine flows that mint a new token each run.
• JIT without deprovisioning. Users are created at sign-in and never removed, and a leaver’s personal access tokens keep working. Pair SSO with SCIM or automatic identity management, and revoke tokens for deactivated users.
• “Federation” means different things in the same meeting. One person means Workforce Identity Federation, another means Databricks token federation, and the design is approved on a misunderstanding. Use the full product name every time.
• A long-lived secret is the easy path. A service account key or client secret is created “just for testing” and is still in use a year later. Use federation or impersonation, and block key creation with policy where you can.
• Legacy OAuth flows in an integration. A tool that uses the implicit or password grant is a finding under RFC 9700. Ask the vendor for authorisation code with PKCE, or client credentials.

What would I do now?

1. Draw your identity map: the IdP, every application that trusts it, and for each one the protocol (SAML 2.0 or OIDC) and how users are provisioned (SCIM, JIT or automatic identity management).
2. List every machine credential you have and mark each as short-lived or long-lived. For every long-lived one, write down where it is stored, who owns it and when it is rotated.
3. For each automation path, name the OAuth flow and the token issuer. Replace client secrets with token exchange wherever the platform supports it.
4. Agree on vocabulary with your security team before the design review: use the table of “federation” meanings above and point at the row you mean.
5. Check the dates that can lock you out: SAML signing certificates, client secret expiries and server certificates.

What did we learn?

• Authentication splits into user to machine and machine to machine. User to machine is sync-based (the app keeps a copy of users, still the most common) or sync-less (it trusts the token alone). Machine to machine is a static client ID and secret, or dynamic token trust that maps a platform token to a service account or service principal, which on Google Cloud needs Workload Identity User or Token Creator on that service account; Databricks workspace identity federation is the exception, because it is about where identities live, not about trusting tokens.
• Identity, authentication and authorisation are three different questions: who you are, how you prove it, and what you may do; authentication happens at sign-in and authorisation on every request.
• An identity provider such as Entra ID authenticates people once and vouches for them to many service providers, through SAML 2.0 assertions or OpenID Connect ID tokens, which is single sign-on.
• A SAML SP verifies each assertion with a certificate pasted at setup, so a certificate rotation that is not copied to the SP blocks every user at once; an OIDC client fetches keys from the JWKS by key ID, so rotation needs no manual step.
• OAuth 2.0 issues access tokens for delegated authorisation, OpenID Connect adds ID tokens for authentication, and a JWT’s issuer, subject, audience and expiry claims decide whether a receiver should accept it.
• The OAuth flow tells you most of an integration’s security story: authorisation code with PKCE for people, client credentials for machines with a secret, and token exchange for machines without one.
• SCIM pushes users and groups from the IdP so that applications can grant access before sign-in and remove it when people leave, and “federation” names at least six different products, so always say which one you mean; short-lived tokens beat long-lived secrets, and least privilege applies to RBAC and ABAC alike.

What comes next?

With the basics of all three prerequisites in place, we meet Databricks itself. Part 1: Foundations, how Databricks is built on Google Cloud covers the control plane and the two compute planes, accounts, workspaces and metastores, subscribing through Marketplace, and what Databricks creates in your project, including who creates its IAM roles and why none of it needs a key. Part 2 then applies every idea from this post to Databricks on Google Cloud.

Sources

• Microsoft Entra ID: what it is
• New name for Azure Active Directory
• Google Cloud Identity overview
• Microsoft Entra: What is single sign-on?
• Databricks: Configure SSO in Databricks
• OASIS: SAML standard and OASIS Security Services (SAML) committee page
• Databricks: Configure SSO using SAML
• Databricks: Configure SSO using OIDC
• Databricks: SSO to Databricks with Microsoft Entra ID
• RFC 5280: Internet X.509 Public Key Infrastructure Certificate and CRL Profile
• Microsoft identity platform: Signing key rollover
• OASIS: SAML V2.0 Technical Overview
• Microsoft Entra: Manage certificates for federated single sign-on
• RFC 6749: The OAuth 2.0 Authorization Framework
• RFC 6750: OAuth 2.0 Bearer Token Usage
• RFC 7519: JSON Web Token (JWT)
• RFC 7517: JSON Web Key (JWK)
• RFC 7636: Proof Key for Code Exchange (PKCE)
• RFC 8628: OAuth 2.0 Device Authorization Grant
• RFC 8693: OAuth 2.0 Token Exchange
• RFC 9700: Best Current Practice for OAuth 2.0 Security
• OpenID Connect Core 1.0 and OpenID Connect Discovery 1.0
• Google: OpenID Connect discovery document
• Google Cloud: Token types
• Databricks: Authenticate with Google ID tokens
• Databricks: Authorize user access with OAuth (U2M)
• Databricks: Authorize service principal access with OAuth (M2M)
• Google Cloud: Workload Identity Federation
• Google Cloud: Security Token Service token method
• Google Cloud: Workforce Identity Federation
• RFC 7643: SCIM Core Schema and RFC 7644: SCIM Protocol
• Microsoft Entra: What is app provisioning?
• Databricks: Sync users and groups using SCIM
• Databricks: Automatically provision users (JIT)
• Databricks: Automatic identity management
• Databricks: Identity best practices
• Databricks: OAuth token federation
• Microsoft Entra: Workload identity federation
• Microsoft Entra: Configure an app to trust an external identity provider
• Google Workspace: Google Cloud Directory Sync
• Databricks: Authenticate with Google ID tokens
• AWS: Create an OpenID Connect (OIDC) identity provider in IAM
• AWS STS: AssumeRoleWithWebIdentity
• Google Cloud blog: Enabling keyless authentication from GitHub Actions
• Google Cloud: Service accounts overview
• Google Cloud: Service account impersonation
• Google Cloud: Best practices for managing service account keys
• Microsoft: Managed identities for Azure resources
• Microsoft Entra: Application and service principal objects
• AWS: Security best practices in IAM
• Databricks: Service principals
• Google Cloud: How Application Default Credentials works
• Google Cloud: Set up ADC for a local development environment
• Google Cloud: Provide credentials for ADC
• Google Cloud SDK reference: gcloud auth application-default login
• google-auth for Python: google.auth.default
• Terraform Registry: Google provider configuration reference
• Google Cloud: Secret Manager overview
• Databricks: Secret management
• Microsoft Entra: How multifactor authentication works
• Google Workspace: Protect your business with 2-Step Verification
• Microsoft Entra: Conditional Access overview
• Google Cloud: Access Context Manager overview
• NIST: Role Based Access Control project
• NIST SP 800-162: Guide to Attribute Based Access Control
• Google Cloud: IAM Conditions overview
• Databricks: Attribute-based access control in Unity Catalog
• Google Cloud: Use IAM securely
• RFC 8446: TLS 1.3
• Google Cloud: Encryption in transit
• Google Cloud: Default encryption at rest
• Cloud KMS: Envelope encryption
• Cloud KMS: Customer-managed encryption keys
• RFC 4632: Classless Inter-domain Routing (CIDR)
• RFC 1918: Address Allocation for Private Internets
• Google Cloud: Subnets
• Databricks: Manage IP access lists
• RFC 1034: Domain Names, Concepts and Facilities
• Cloud DNS overview

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands and its data are fictional.