Upgrades and CD
The images
FACE ships two images. Both are built from source by Runink and pushed to the cluster’s node-local registry.
| Image | Contents |
|---|---|
face-backend | /app/server (the runink CLI; serve starts the backend), /app/faceops (operator tools), /app/entrypoint (the image’s default command; a deployment starts /app/server serve instead), the built-in map basemap, and a headless Chromium for rendering. It runs as a non-root user (uid 1000) on a pinned Debian slim base. |
face-frontend | The release Flutter web bundle and the static sleeve, on a distroless base. |
Two properties of these builds matter to you as an operator. Neither image is assembled from prebuilt binaries — every release is compiled from source, and the protobuf stubs are regenerated during the build rather than carried along. And the frontend has no backend address compiled into it, so one image serves every deployment: you point it at your own control plane at run time, never at build time.
Continuous delivery
A release is built, checked against the cluster, rolled, and then confirmed. These four steps are what a rollout does, whether Runink performs it for a managed deployment or you drive the same sequence yourself:
Build
The image is built with tag latest.
Preflight
Check that kubectl reaches the cluster (kubectl get --raw=/readyz). If it does not,
stop and roll nothing — a rollout against an unreachable API server leaves you unable to
tell a failed deploy from a failed connection.
Roll
Record the running image digest, roll, and wait up to 300 seconds for
rollout status. A backend release rolls the control plane (face-control-plane) and
the static face-runner by deleting their pods: the operator owns the control-plane
Deployment and reconciles a rollout restart away. A frontend release runs
rollout restart on face-frontend, which no operator owns.
Confirm
Compare the new digest with the old one. If they match, treat it as a warning, not a success: either the build never reached the registry, or the commit produced an identical image. A rollout that rolled nothing looks exactly like one that worked.
Rollouts through the operator use the Recreate strategy. The old pod stops and releases
its storage lock before the new pod starts. Expect a few seconds of unavailability per
roll. Do not change the strategy to rolling: two pods would contend for the same
single-writer state.
Image references and pull policy
The operator pulls with Always for a tag, and with IfNotPresent for a digest
(image@sha256:…). TIDE_IMAGE_PULL_POLICY on the operator overrides both. For
production and air-gapped sites, pin the image field of the ControlPlane resource,
and the images in the face-runner and face-frontend manifests, to a digest, so that
every roll is reproducible and a rollback is exact.
When the tide-mesh-ca Secret changes, the operator writes its hash into the pod
template annotation core.runink.org/mesh-ca-hash. The change rolls the pods, so each one
picks up the new CA.
Upgrade checklist
Read the release’s new requirements
If a release adds a boot requirement, the deployment side must land first:
Prerequisites and ordering explains why. Check
face-system for tide-mesh-ca, tide-envelope-kek, face-session and
face-install-config, and check that INFERENCE_REMOTE_URL is set on both
face-control-plane and face-runner.
Remove names the new image refuses
Take OBJECTSTORE_USE_SSL out of any env list. It is refused at startup. The full list
of retired names is at the end of the
Configuration reference.
Escrow the KEK
Confirm that the KEK is in escrow before you upgrade. Storage migrations run on first boot. See Persistence and backup.
Roll the control plane, then the runner
The control plane and runners must be able to verify each other’s mesh certificates, and they must agree on the tenant.
Verify
Check that /readyz returns ready, that the boot log shows
FACE appfs root online for the expected tenant, and that a ModelService/Generate
call answers. See
Health and troubleshooting.
Rolling back
Storage migrations leave the old data in place: <file>.migrated files, older
object-store tables and the old knowledge index. See
Migrations leave a rollback in place.
So an older image can usually boot against the same stores, without anything written
after the upgrade. Roll back by setting the ControlPlane image field, and the
face-runner image, back to the previous digest.
Do not roll back the KEK: the same KEK must stay in place.