Keycloak upgrade guide¶

This guide covers every kind of change you might need to ship for Keycloak:

1. Theme changes (CSS, logo, copy)
2. Keycloak version bumps (upstream)
3. Keycloak CR / runtime configuration
4. Realm settings (clients, flows, themes, signup)
5. Rollback
6. Data safety notes

All changes land in the Skatzi/platform repo. Flux on the mgmt-cluster reconciles components/keycloak/ into the keycloak namespace.

1. Theme changes¶

Any file under components/keycloak/theme/kodexet/ (CSS, logo, HTML overrides, theme.properties, translations) is baked into the Docker image at build time. Shipping a theme change is therefore a new image tag + bump on the Keycloak CR.

Workflow¶

1. Edit the theme on a branch, e.g. tweak theme/kodexet/login/resources/css/styles.css.

2. Pick the next image tag. Tags follow vMAJOR.MINOR.PATCH. Check what's currently deployed:

   grep 'image:' components/keycloak/base/keycloak-instance.yaml
   # image: harbor.prod.skatzi.com/skatzi/keycloak-kodexet:v1.0.1
   

   Bump patch for pure theme tweaks, minor for bigger reworks.

3. Bump the image reference in components/keycloak/base/keycloak-instance.yaml:

   spec:
     image: harbor.prod.skatzi.com/skatzi/keycloak-kodexet:v1.0.2
   

4. Commit, push, and tag. The Gitea CI workflow .gitea/workflows/ci-keycloak.yaml triggers on tags matching keycloak-v*.*.*:

   git add components/keycloak/ && git commit -m "fix(keycloak): <what changed>"
   git push origin main
   git tag keycloak-v1.0.2
   git push origin keycloak-v1.0.2
   

   The tag push kicks Kaniko — it builds from components/keycloak/Dockerfile and pushes:

   • harbor.prod.skatzi.com/skatzi/keycloak-kodexet:v1.0.2
   • harbor.prod.skatzi.com/skatzi/keycloak-kodexet:latest

5. Wait for the image to land in Harbor (check the Gitea Actions tab or Harbor UI). Flux only rolls once the image exists.

6. Flux reconciliation picks up the new image: value within a minute and the operator rolls the StatefulSet. Watch the rollout:

   kubectl --context admin@mgmt-cluster -n keycloak get pod -w
   kubectl --context admin@mgmt-cluster -n keycloak logs keycloak-0 -f
   

   The first boot runs kc.sh build again (because startOptimized: false) — expect a minute of startup before Listening on http://0.0.0.0:8080 appears.

Activating the theme in a realm¶

Adding a theme to the image does not automatically assign it to a realm. Do that once in the admin console:

Admin console → select realm → Realm Settings → Themes → Login Theme: kodexet → Save.

The same applies for Account / Email / Admin themes if they ever get their own variants.

2. Keycloak version bump¶

Upstream releases: https://www.keycloak.org/docs/latest/release_notes/index.html. Always read the release notes for breaking changes before bumping.

Workflow¶

1. Bump the base image in components/keycloak/Dockerfile:

   FROM quay.io/keycloak/keycloak:26.5.0
   

2. Bump the Keycloak Operator too if required by the new version — managed separately via Flux under the operator's own component directory.

3. Bump the image tag in keycloak-instance.yaml (use a minor bump when the upstream minor changes, e.g. v1.1.0).

4. Follow the rest of the theme-change workflow from step 4 onwards.

Checklist for major version bumps¶

• Read upstream release notes end-to-end
• Back up the Keycloak Postgres DB (see Data safety notes)
• Check for deprecated server options in additionalOptions of the CR
• Verify theme parent=keycloak.v2 still resolves (the upstream theme tree is version-specific)
• Validate the admin console loads and existing clients still work before closing the change

3. Keycloak CR / runtime configuration¶

Changes to components/keycloak/base/keycloak-instance.yaml do not require a new image — the operator picks up CR changes and rolls the pod on its own. No tag push needed.

Common fields¶

Ship the change the normal way:

git add components/keycloak/base/keycloak-instance.yaml
git commit -m "chore(keycloak): <what changed>"
git push origin main

Flux reconciles within a minute, the operator rolls the pod, done.

4. Realm settings¶

The realm CRDs are disaster-recovery snapshots only — they do not drive live config. Live changes must be made manually in the admin console, and the YAML in base/*-realm.yaml should be updated to match so the DR snapshot stays current.

Workflow for any realm change¶

1. Make the change in the admin console:

   • https://keycloak.prod.skatzi.com → admin login → select the realm (kodexet or skatzi).
   • Change the setting (themes, login flows, identity providers, clients, registration, etc.).
   • Save.

2. Verify end-to-end — log in with an affected client before mirroring to YAML.

3. Mirror the change into the DR snapshot in components/keycloak/base/<realm>-realm.yaml. Example — enabling self-service registration and switching login theme:

   spec:
     realm:
       registrationAllowed: true
       resetPasswordAllowed: true
       loginTheme: kodexet
   

4. Commit as chore(keycloak): update <realm> realm DR snapshot and push.

Why not drive realms from the CRD?¶

The Keycloak Operator's KeycloakRealmImport resource only imports a realm the first time — subsequent CRD edits are silently ignored against a live realm. Attempting to drive live config from YAML therefore produces silent drift. The DR snapshot exists so that if we ever had to recreate the database from scratch, kubectl apply -f kodexet-realm.yaml would rebuild the realm to the documented state.

Common realm-change pitfalls¶

• "Users get sent straight to Google" — check Authentication → Flows → Browser → Identity Provider Redirector → config. If Default Identity Provider is set, the login page is skipped. Clear it to get the normal login form back.
• "Theme changes don't apply" — make sure the theme is both in the image (step 1) and assigned on the realm (Realm Settings → Themes).
• "Google login fails after changing redirect URIs" — mirror the same URIs in the Google Cloud OAuth client, not just in Keycloak.

5. Rollback¶

Rollback an image¶

# In keycloak-instance.yaml
spec:
  image: harbor.prod.skatzi.com/skatzi/keycloak-kodexet:v1.0.1   # previous tag

Commit, push — Flux rolls back to the previous image. All images stay in Harbor; nothing is garbage-collected automatically.

Rollback a CR change¶

git revert <sha> the offending commit, push, Flux reconciles the previous state.

Rollback a realm change¶

There is no automated rollback. Either:

• Undo the change manually in the admin console (preferred for small changes), or
• Restore the Postgres DB from a backup and re-import the DR snapshot (heavy, loses all user activity since the backup).

6. Data safety notes¶

Keycloak's state lives in its Postgres StatefulSet (postgres-db in the keycloak namespace) with its own PVC. It is not touched by:

• Image bumps (theme or upstream version)
• CR changes (keycloak-instance.yaml)
• Realm DR snapshot updates (the YAML only applies on initial import)

It is at risk from:

• Deleting the PVC (persistent-volume-claim.yaml) or the Postgres StatefulSet
• A Postgres version bump without a proper dump/restore
• KeycloakRealmImport re-import if the operator is ever changed to reconcile continuously — check release notes before upgrading the operator

Before any major version bump or storage change, take a logical dump:

kubectl --context admin@mgmt-cluster -n keycloak exec -it postgres-db-0 -- \
  pg_dumpall -U <admin-user> > keycloak-backup-$(date +%F).sql

Store the dump outside the cluster (OpenBao file path, S3, local encrypted backup) — never in the repo.