Skip to content

Upgrades and CD

The images

FACE ships two images. Both are built from source by Runink and pushed to the cluster’s node-local registry.

ImageContents
face-backend/app/server (the runink CLI; serve starts the backend), /app/faceops (operator tools), /app/entrypoint (the image’s default command; a deployment starts /app/server serve instead), the built-in map basemap, and a headless Chromium for rendering. It runs as a non-root user (uid 1000) on a pinned Debian slim base.
face-frontendThe release Flutter web bundle and the static sleeve, on a distroless base.

Two properties of these builds matter to you as an operator. Neither image is assembled from prebuilt binaries — every release is compiled from source, and the protobuf stubs are regenerated during the build rather than carried along. And the frontend has no backend address compiled into it, so one image serves every deployment: you point it at your own control plane at run time, never at build time.

Continuous delivery

A release is built, checked against the cluster, rolled, and then confirmed. These four steps are what a rollout does, whether Runink performs it for a managed deployment or you drive the same sequence yourself:

Build

The image is built with tag latest.

Preflight

Check that kubectl reaches the cluster (kubectl get --raw=/readyz). If it does not, stop and roll nothing — a rollout against an unreachable API server leaves you unable to tell a failed deploy from a failed connection.

Roll

Record the running image digest, roll, and wait up to 300 seconds for rollout status. A backend release rolls the control plane (face-control-plane) and the static face-runner by deleting their pods: the operator owns the control-plane Deployment and reconciles a rollout restart away. A frontend release runs rollout restart on face-frontend, which no operator owns.

Confirm

Compare the new digest with the old one. If they match, treat it as a warning, not a success: either the build never reached the registry, or the commit produced an identical image. A rollout that rolled nothing looks exactly like one that worked.

Rollouts through the operator use the Recreate strategy. The old pod stops and releases its storage lock before the new pod starts. Expect a few seconds of unavailability per roll. Do not change the strategy to rolling: two pods would contend for the same single-writer state.

Image references and pull policy

The operator pulls with Always for a tag, and with IfNotPresent for a digest (image@sha256:…). TIDE_IMAGE_PULL_POLICY on the operator overrides both. For production and air-gapped sites, pin the image field of the ControlPlane resource, and the images in the face-runner and face-frontend manifests, to a digest, so that every roll is reproducible and a rollback is exact.

When the tide-mesh-ca Secret changes, the operator writes its hash into the pod template annotation core.runink.org/mesh-ca-hash. The change rolls the pods, so each one picks up the new CA.

Upgrade checklist

Read the release’s new requirements

If a release adds a boot requirement, the deployment side must land first: Prerequisites and ordering explains why. Check face-system for tide-mesh-ca, tide-envelope-kek, face-session and face-install-config, and check that INFERENCE_REMOTE_URL is set on both face-control-plane and face-runner.

Remove names the new image refuses

Take OBJECTSTORE_USE_SSL out of any env list. It is refused at startup. The full list of retired names is at the end of the Configuration reference.

Escrow the KEK

Confirm that the KEK is in escrow before you upgrade. Storage migrations run on first boot. See Persistence and backup.

Roll the control plane, then the runner

The control plane and runners must be able to verify each other’s mesh certificates, and they must agree on the tenant.

Verify

Check that /readyz returns ready, that the boot log shows FACE appfs root online for the expected tenant, and that a ModelService/Generate call answers. See Health and troubleshooting.

Rolling back

Storage migrations leave the old data in place: <file>.migrated files, older object-store tables and the old knowledge index. See Migrations leave a rollback in place. So an older image can usually boot against the same stores, without anything written after the upgrade. Roll back by setting the ControlPlane image field, and the face-runner image, back to the previous digest. Do not roll back the KEK: the same KEK must stay in place.