Operations

Day-2 reference for running Dreadnode after the initial install. All examples assume dreadnode as the release name and Helm CLI — Admin Console equivalents are noted where they differ.

Health checks

# All pods
kubectl -n <namespace> get pods -l app.kubernetes.io/instance=dreadnode

# API health (returns {"status":"ok"} when healthy)
curl http(s)://<your-domain>/api/v1/health

# Resource usage (requires metrics-server)
kubectl -n <namespace> top pods -l app.kubernetes.io/instance=dreadnode

The API’s /api/v1/health endpoint checks Postgres connectivity. A 503 with {"status":"unhealthy","detail":"database unreachable"} means the API is running but can’t reach the database.

Restart components

Rolling restart — no downtime if replicas > 1:

# API
kubectl -n <namespace> rollout restart deploy/dreadnode-api

# Frontend
kubectl -n <namespace> rollout restart deploy/dreadnode-frontend

# StatefulSets (use with care — causes brief data-store unavailability)
kubectl -n <namespace> rollout restart sts/dreadnode-postgresql
kubectl -n <namespace> rollout restart sts/dreadnode-clickhouse
kubectl -n <namespace> rollout restart sts/dreadnode-minio

Watch the rollout:

kubectl -n <namespace> rollout status deploy/dreadnode-api

View applied configuration

# ConfigMap (non-secret env vars)
kubectl -n <namespace> get cm dreadnode-api -o yaml

# Current resource state
kubectl -n <namespace> get deploy,sts,ingress -l app.kubernetes.io/instance=dreadnode

Database access

PostgreSQL

# Port-forward
kubectl -n <namespace> port-forward sts/dreadnode-postgresql 5432:5432

# Connect (in another terminal)
PGPASSWORD=$(kubectl -n <namespace> get secret dreadnode-postgresql \
    -o jsonpath='{.data.password}' | base64 -d) \
    psql -h localhost -U admin -d platform

Or exec directly into the pod:

kubectl -n <namespace> exec -it dreadnode-postgresql-0 -- psql -U admin -d platform

ClickHouse

# Port-forward the HTTP interface
kubectl -n <namespace> port-forward sts/dreadnode-clickhouse 8123:8123

# Query
curl 'http://localhost:8123/?query=SELECT+1'

Or use the CLI inside the pod:

kubectl -n <namespace> exec -it dreadnode-clickhouse-0 -- clickhouse-client

MinIO

# Port-forward the console (not the S3 API)
kubectl -n <namespace> port-forward sts/dreadnode-minio 9001:9001

Open http://localhost:9001 in a browser. Log in with the root credentials:

kubectl -n <namespace> get secret dreadnode-minio \
    -o jsonpath='{.data.rootUser}' | base64 -d
kubectl -n <namespace> get secret dreadnode-minio \
    -o jsonpath='{.data.rootPassword}' | base64 -d

Backups

Backup strategy depends on your environment. The chart deploys in-cluster PostgreSQL, ClickHouse, and MinIO by default — back up at the storage layer (PVC snapshots) or export data logically from inside the pods.

PostgreSQL

# Dump to a local file
kubectl -n <namespace> exec dreadnode-postgresql-0 -- \
    pg_dump -U admin platform > dreadnode-pg-$(date +%Y%m%d).sql

Restore (destroys existing data):

# Drop and recreate
kubectl -n <namespace> exec dreadnode-postgresql-0 -- \
    psql -U admin -d postgres -c "DROP DATABASE platform"
kubectl -n <namespace> exec dreadnode-postgresql-0 -- \
    psql -U admin -d postgres -c "CREATE DATABASE platform"

# Restore
cat dreadnode-pg-20260416.sql | \
    kubectl -n <namespace> exec -i dreadnode-postgresql-0 -- \
    psql -U admin -d platform

Caution

After restoring Postgres, restart the API so Alembic detects the current schema state: kubectl -n <namespace> rollout restart deploy/dreadnode-api

PVC snapshots

If your storage class supports CSI snapshots:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: pg-snapshot
  namespace: <namespace>
spec:
  volumeSnapshotClassName: <your-snapshot-class>
  source:
    persistentVolumeClaimName: data-dreadnode-postgresql-0

Repeat for data-dreadnode-clickhouse-0 and data-dreadnode-minio-0.

External data stores

If you pointed Dreadnode at external services (RDS, managed ClickHouse, S3), use those services’ native backup tools. The chart doesn’t manage backups for external stores.

Secret rotation

The chart auto-generates passwords for in-cluster data stores and security keys for the API. Rotating them requires updating the Secret and restarting the affected pods.

Data store passwords

Data store Secrets have helm.sh/resource-policy: keep — Helm won’t overwrite them on upgrade. To rotate:

NEW_PW=$(openssl rand -base64 32)

# Update the Secret
kubectl -n <namespace> create secret generic dreadnode-postgresql \
    --from-literal=password="$NEW_PW" \
    --dry-run=client -o yaml | kubectl apply -f -

# Update the password inside the running database
kubectl -n <namespace> exec dreadnode-postgresql-0 -- \
    psql -U admin -d platform -c "ALTER USER admin PASSWORD '$NEW_PW'"

# Restart the API to pick up the new credential
kubectl -n <namespace> rollout restart deploy/dreadnode-api

Same pattern for ClickHouse (dreadnode-clickhouse, key admin-password) and MinIO (dreadnode-minio, keys rootUser, rootPassword).

API security keys

The dreadnode-api-security Secret holds secretKey, jwtSecretKey, and refreshSecretKey. Rotating these invalidates all active sessions and issued tokens — every logged-in user gets logged out.

The dreadnode-api-encryption Secret holds the Fernet key for encrypting user secrets stored in Postgres. Do not rotate this key unless you’re prepared to lose all encrypted user secrets. There is no re-encryption migration.

Scaling

Resource presets

The simplest way to scale is to change the resource preset. Set global.resourcesPreset in your values overlay and upgrade:

helm upgrade dreadnode oci://registry.replicated.com/dreadnode/dreadnode \
    --version <version> \
    -f values.yaml \
    --set global.resourcesPreset=medium

For Admin Console installs, change Resource Sizing in the config screen and redeploy.

Manual replica scaling

The API and frontend Deployments can be scaled horizontally:

kubectl -n <namespace> scale deploy/dreadnode-api --replicas=3
kubectl -n <namespace> scale deploy/dreadnode-frontend --replicas=2

This doesn’t survive helm upgrade. For persistent scaling, set replica counts in your values overlay under the subchart overrides.

Note

PostgreSQL, ClickHouse, and MinIO are single-replica StatefulSets. Scaling them horizontally requires configuration changes beyond replica count (replication setup, shared storage, etc.) and is not covered here.

Upgrades

Helm CLI

helm upgrade dreadnode oci://registry.replicated.com/dreadnode/dreadnode \
    --version <new-version> \
    -f values.yaml

Admin Console

The Admin Console checks for new versions automatically. When an update appears on the dashboard, review the release notes and click Deploy.

What happens during an upgrade

1. The migrations init container runs alembic upgrade head against Postgres
2. The API pod starts with the new version
3. The frontend pod rolls to the new version

Migrations are forward-only. helm rollback and the Admin Console Rollback button are disabled. If an upgrade fails, see Reinstall from scratch.

Support bundles

Support bundles collect logs, cluster state, and diagnostics into a single archive.

Admin Console: Go to Troubleshoot → Generate a support bundle.

Helm CLI:

kubectl support-bundle --load-cluster-specs -n <namespace>

Requires the troubleshoot kubectl plugin. The bundle spec is built into the chart — the plugin discovers it automatically.

Share the generated archive with us when you need help debugging.