Skip to main content

Operations

All state of Cryptomator Hub lives in the PostgreSQL database: the hub database holds vaults, keys, and the audit log, the keycloak database holds users, groups, and credentials. Back up both, and always do so before upgrading. For an end-to-end walkthrough from deployment to backups, see the Self-Hosting Guide.

Backup​

The easiest way is a logical dump of the entire PostgreSQL instance with pg_dumpall, run regularly, e.g. from a cron job on the host.

Docker Compose:

docker compose exec -u postgres postgres pg_dumpall > "$(date +%F)-hub-backup.sql"

Kubernetes:

kubectl exec -n cryptomator hub-pg-0 -- pg_dumpall -U postgres > "$(date +%F)-hub-backup.sql"

On Kubernetes you can alternatively snapshot the volume data-hub-pg-0 with your storage provider.

See the PostgreSQL documentation for more information on pg_dumpall. If you also back up your compose.yaml or Helm values, you can restore the entire installation in minutes.

note

Make sure the backup is moved to another secure location.

Restore​

To bring a Hub deployment back to the state of a backup, replace the hub database with the contents of the dump. The following steps use Docker Compose; the Kubernetes equivalents are noted where they differ.

  1. Create a fresh backup of the entire PostgreSQL instance, including the Keycloak database, so that you can return to the current state if something goes wrong.
  2. Stop Hub with docker compose stop hub. Keycloak and PostgreSQL keep running. (Kubernetes: kubectl scale -n cryptomator deployment/hub-hub --replicas=0)
  3. Connect to PostgreSQL with docker compose exec -ti postgres psql -U postgres. (Kubernetes: kubectl exec -ti -n cryptomator hub-pg-0 -- psql -U postgres)
  4. Rename the existing database with ALTER DATABASE hub RENAME TO hub_backup; so that it remains available as an additional safety net.
  5. Create an empty database with CREATE DATABASE hub WITH ENCODING 'UTF8'; GRANT ALL PRIVILEGES ON DATABASE hub TO hub; and leave the shell with exit.
  6. Import the dump with docker compose exec -T postgres psql -U hub -d hub -v ON_ERROR_STOP=1 < backup.sql. (Kubernetes: kubectl exec -i -n cryptomator hub-pg-0 -- psql -U hub -d hub -v ON_ERROR_STOP=1 < backup.sql)
  7. Start Hub again with docker compose start hub. (Kubernetes: kubectl scale -n cryptomator deployment/hub-hub --replicas=1)

Once you have confirmed that Hub works as expected, you can drop the hub_backup database.

warning

Hub and Keycloak reference each other by user ID. If you restore the Hub database from a backup, restore the Keycloak database from the same point in time as well. Otherwise users may exist in one system but not in the other.

Changing the Database Password​

Change the password in Postgres first, then update the deployment. Connect to the Postgres container with docker compose exec -it postgres /bin/sh or kubectl exec -it hub-pg-0 -n cryptomator -- /bin/sh, open the database with psql -h localhost -d hub -U hub, and run \password to set a new password for the Hub database user.

Afterwards, set the new password in your deployment and restart Hub: the environment variable QUARKUS_DATASOURCE_PASSWORD in compose.yaml, or hub.database.password on helm upgrade.

note

Keycloak uses its own database user. Changing the Hub password does not affect it.

Verifying Container Images​

The Hub and Keycloak container images are published together with build provenance attestations, which allow you to confirm that an image was built by the official GitHub Actions workflow and has not been tampered with.

The following example verifies the Keycloak image using regctl and cosign:

KC_VERSION=26.7.2
regctl manifest get --format raw-body ghcr.io/cryptomator/keycloak:${KC_VERSION} > manifest.json
DIGEST="sha256-$(sha256sum manifest.json | awk '{ print $1 }')"
regctl artifact get ghcr.io/cryptomator/keycloak:${DIGEST} > bundle.json
cosign verify-blob-attestation \
    --bundle bundle.json \
    --new-bundle-format \
    --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
    --certificate-identity-regexp="^https://github.com/cryptomator/hub/.github/workflows/keycloak.yml@refs/heads/release/keycloak-${KC_VERSION}" \
    manifest.json

A successful run prints Verified OK.

The Hub image itself is attested by the build.yml workflow of the same repository. To verify it, use the corresponding image name and adjust --certificate-identity-regexp to that workflow and the Git reference the release was built from.

The Helm chart is signed as well. Verify the signature and inspect the provenance attestation with:

cosign verify \
  --certificate-identity-regexp 'https://github.com/cryptomator/hub/.github/workflows/helm-chart.yml@refs/(heads|tags)/.+' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/cryptomator/charts/cryptomator-hub:2.0.0

cosign verify-attestation \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity-regexp 'https://github.com/cryptomator/hub/.github/workflows/helm-chart.yml@refs/(heads|tags)/.+' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/cryptomator/charts/cryptomator-hub:2.0.0

Trusting a Private Certificate Authority​

If Hub connects to a Keycloak instance whose TLS certificate was not issued by a well-known certificate authority, you have to make the issuing CA known to Hub. Hub runs on the JVM, which uses its own trust store and ignores the certificates trusted by the host system.

Start by preparing a file rootWithIntermediates.pem that contains the root certificate and all intermediate certificates that are not publicly available, in PEM format. Then create a PKCS12 trust store from it:

keytool -importcert \
    -alias keycloak-ca-chain \
    -file rootWithIntermediates.pem \
    -keystore keycloak-truststore.p12 \
    -storepass changeit \
    -noprompt

Replace changeit with a password of your own. You can verify the result with keytool -list -v -keystore keycloak-truststore.p12 -storepass changeit.

Hub reads the trust store from the Java system properties javax.net.ssl.trustStore and javax.net.ssl.trustStorePassword, which you pass as arguments to the application command.

In Docker Compose, mount the file into the container and override the command:

services:
  hub:
    image: ghcr.io/cryptomator/hub:2.0.0
    command: >
      ./application
      -Djavax.net.ssl.trustStore=/etc/certs/keycloak-truststore.p12
      -Djavax.net.ssl.trustStorePassword=changeit
    volumes:
      - './certs/keycloak-truststore.p12:/etc/certs/keycloak-truststore.p12:ro'

In Kubernetes, store the trust store in a secret and mount it as a volume. Encode the file with base64 -w0 keycloak-truststore.p12 and add the output to a secret:

apiVersion: v1
kind: Secret
metadata:
  namespace: cryptomator
  name: keycloak-truststore
type: Opaque
data:
  keycloak-truststore.p12: BASE64_ENCODED_TRUSTSTORE

The chart has no value for this, so patch the Hub deployment (hub-hub for a release named hub) after installing:

spec:
  template:
    spec:
      containers:
        - name: hub
          args:
            - '-Djavax.net.ssl.trustStore=/etc/certs/keycloak-truststore.p12'
            - '-Djavax.net.ssl.trustStorePassword=changeit'
          volumeMounts:
            - name: keycloak-truststore
              mountPath: /etc/certs
              readOnly: true
      volumes:
        - name: keycloak-truststore
          secret:
            secretName: keycloak-truststore
kubectl patch deployment hub-hub -n cryptomator --patch-file truststore-patch.yaml
note

Quarkus also offers the configuration options QUARKUS_OIDC_CERTIFICATE_CHAIN_TRUST_STORE_FILE and QUARKUS_OIDC_CERTIFICATE_CHAIN_TRUST_STORE_PASSWORD. These do not work for this purpose, so use the Java system properties shown above.

If the Cryptomator desktop app also needs to talk to that Hub instance, the same applies there. Add java-options=-Djavax.net.ssl.trustStore=/path/to/your/truststore to the Cryptomator.cfg file in the installation directory.