Skip to content
Persistence and backup

Persistence and backup

FACE keeps its state on self-hosted storage inside the deployment’s own cluster. There is no managed cloud database. Every store below is sealed at rest under keys derived from one platform key-encryption key, TIDE_ENVELOPE_KEK. That makes the KEK the most important thing to back up (see Key custody).

StoreWhat it holdsWhere it lives
appfs rootSessions, accounts, connections and their credentials, application metadata, fetch results, briefs, rule policies, compute budgetsThe object-store bucket FACE_APPFS_BUCKET (default face), or an encrypted local directory
Control-plane stateThe application-metadata snapshot and boot marker, in an appfs table (kv) — the same table runners useThe appfs root (object store, or the encrypted local directory)
Record storeSession traces (with clustering feature vectors) and feedback, as self-describing AvroThe object store
Knowledge corpusPlain-text passages per tenant, searched by grep (no vector index)The object store, under dci/knowledge/<tenant>/, or <data dir>/knowledge/; a greppable copy is rebuilt in <data dir>/dci-knowledge/ at start-up
Camera evidenceEvidence frames, when EVIDENCE_FRAME_RETENTION is setThe object store
Fetch ledgerThe fetch cascade’s ledgerFETCH_LEDGER_DIR on local disk

The appfs root

At boot, FACE mounts its own application root, and it will not serve without it. When OBJECTSTORE_ENDPOINT is set, the root is the bucket FACE_APPFS_BUCKET (default face) on the object store. Otherwise, it is an encrypted local directory, FACE_APPFS_DIR. The root’s key is derived from the KEK for FACE alone, and every block is bound to its full path, so a sealed block cannot be moved to another path and still open. The boot log says which backend is in use:

🗂️  FACE appfs root online: objectstore bucket face, tenant <tenant>
🗂️  FACE appfs root online: encrypted local directory <dir>/face, tenant <tenant>

The root is split into two scopes:

  • Deployment-wide: sessions (run/sessions), accounts (var/lib/auth_users), the record of deleted accounts (var/lib/erased_accounts: an opaque account number and the pseudonym that stands for it, no email) and compute budgets.
  • Per tenant (tenants/<FACE_TENANT>/, from the face-install-config ConfigMap in a deployment, default-tenant when unset): connections and their credentials, application metadata, the four fetch snapshots, briefs, remediation overlays and routes, rule policies and support calls.

A runner gets its own tables at tenants/<t>/var/lib/runners.<RUNNER_ID>.<table>. The exception is the fetch snapshots, which the runner writes and the control plane reads. That shared location is why a runner and its control plane must use the same tenant.

Set FACE_TENANT before an instance’s first boot. That boot copies any legacy data into the tenant it sees, once. If you set the tenant later, the legacy data stays under default-tenant. On a runner, keep RUNNER_ID stable. The operator uses the Deployment name. If it fell back to the pod name, every restart would start the runner on a fresh, empty set of tables.

When there is no object store, the local root must be on storage that survives a restart. The operator points FACE_APPFS_DIR at appfs/ inside the pod’s persistent connections volume. If the default location next to face.db were used instead, it would be on the /data emptyDir, and sessions and credentials would be lost on every restart.

Sessions are validated on every request against this root, with a 2-second cache. A revocation on the control plane therefore takes effect on a runner within that window.

No SQLite database

FACE keeps no state in SQLite. The control plane’s application-metadata snapshot lives in an appfs table, like a runner’s state, sealed under FACE’s key in the object store. Nothing restores or replicates a database file, and the control plane’s /data volume is scratch only.

An instance upgraded from an earlier build may still have face.db at SQLITE_PATH. On boot, FACE opens it only if it exists (it never creates one), imports its rows once (the snapshot only when the appfs table has none, plus the legacy session revocations and connections), and never writes to it again. The file is left untouched as the rollback.

No deployment path replicates or restores a SQLite file. The operator starts the server directly (/app/server serve); the image’s /app/entrypoint, for a local podman run, starts the same server and nothing else.

Credential store

A connection’s credentials never sit in an environment variable or in the connection record. The connection record has no field a secret would fit in. When a connection is saved, secret-named fields are lifted into a credential bundle. Bundles are stored as rows of the tenant table tenants/<t>/var/lib/connections (credentials/<ref>), sealed under FACE’s derived key and bound to their path. Consensus replicates only the connection metadata snapshot, never the credentials.

Where FACE writes to local disk (the local appfs root, the knowledge corpus without an object store), directories are created 0700 and files 0600. The older layout (FACE_CONNECTIONS_DIR, with connections.json and credentials/*.json) is imported once on a persistent root, and each imported file is renamed <file>.migrated.

Migrations leave a rollback in place

Each move to a new storage layout copies the data once and leaves the old copy behind, so you can go back to an older image:

  • Legacy metadata files are imported once into their table. Each one is then replaced by an envelope-sealed <file>.migrated. To read one back, use the faceops tool that ships in the backend image, with the same KEK in its environment:

    /app/faceops metalog-unseal --table <table> <file>.migrated
  • Older object-store tables (tables/face/<name>/ in the object store’s own bucket) are copied into the appfs root on first open, and only into a table that has never been written. The originals are left in place.

  • The retired vector corpus (comet/ in the object store) is not read or migrated. It is left in place, untouched. Re-run document ingestion to make documents searchable in the knowledge corpus.

Key custody

TIDE_ENVELOPE_KEK is a base64-encoded 32-byte key held in the tide-envelope-kek Secret in face-system. face-operator creates it if it is absent, and never rotates or overwrites it.

Losing the KEK loses the data. The whole appfs root (sessions, accounts, connection credentials, metadata), the sealed object-store objects and the knowledge snapshots cannot be decrypted without it. No recovery path exists, by design. Export the Secret to your offline key escrow as soon as it is created. Back it up separately from the data it protects.

To change keys, keep the old key readable. TIDE_ENVELOPE_KEK_RETIRED takes a list of earlier KEKs (base64, separated by commas or whitespace). FACE and the storage libraries use them for decryption only, and seal new writes under TIDE_ENVELOPE_KEK. If you replace the KEK without listing the old one as retired, everything sealed under the old key becomes unreadable.

What to back up

  1. The tide-envelope-kek Secret, escrowed offline.
  2. The face-install-config ConfigMap, which names the tenant the data is filed under.
  3. The object-store buckets FACE writes to on face-objectd: the appfs bucket (FACE_APPFS_BUCKET, default face), the documents bucket (FACE_DOCUMENTS_BUCKET, default face-documents) and the bucket in OBJECTSTORE_BUCKET (the deployment sets runink-face-db-nane1).
  4. When there is no object store: the node-local connections volume (/var/lib/tide/face-connections on the node), which contains the appfs root, and the knowledge directory.
  5. The node-local raft volume (/var/lib/tide/raft on the node, mounted at /raft), which holds the control plane’s consensus state (RAFT_DATA_DIR).

To restore, put the KEK Secret and the tenant ConfigMap back first, then the object-store data, and only then start the control plane.