Persistence and backup
FACE keeps its state on self-hosted storage inside the deployment’s own cluster. There is no managed
cloud database. Every store below is sealed at rest under keys derived from one platform
key-encryption key, TIDE_ENVELOPE_KEK. That makes the KEK the most important thing to
back up (see Key custody).
| Store | What it holds | Where it lives |
|---|---|---|
| appfs root | Sessions, accounts, connections and their credentials, application metadata, fetch results, briefs, rule policies, compute budgets | The object-store bucket FACE_APPFS_BUCKET (default face), or an encrypted local directory |
| Control-plane state | The application-metadata snapshot and boot marker, in an appfs table (kv) — the same table runners use | The appfs root (object store, or the encrypted local directory) |
| Record store | Session traces (with clustering feature vectors) and feedback, as self-describing Avro | The object store |
| Knowledge corpus | Plain-text passages per tenant, searched by grep (no vector index) | The object store, under dci/knowledge/<tenant>/, or <data dir>/knowledge/; a greppable copy is rebuilt in <data dir>/dci-knowledge/ at start-up |
| Camera evidence | Evidence frames, when EVIDENCE_FRAME_RETENTION is set | The object store |
| Fetch ledger | The fetch cascade’s ledger | FETCH_LEDGER_DIR on local disk |
The appfs root
At boot, FACE mounts its own application root, and it will not serve without it. When
OBJECTSTORE_ENDPOINT is set, the root is the bucket FACE_APPFS_BUCKET (default
face) on the object store.
Otherwise, it is an encrypted local directory, FACE_APPFS_DIR. The root’s key is
derived from the KEK for FACE alone, and every block is bound to its full path, so a
sealed block cannot be moved to another path and still open. The boot log says which
backend is in use:
🗂️ FACE appfs root online: objectstore bucket face, tenant <tenant>
🗂️ FACE appfs root online: encrypted local directory <dir>/face, tenant <tenant>The root is split into two scopes:
- Deployment-wide: sessions (
run/sessions), accounts (var/lib/auth_users), the record of deleted accounts (var/lib/erased_accounts: an opaque account number and the pseudonym that stands for it, no email) and compute budgets. - Per tenant (
tenants/<FACE_TENANT>/, from theface-install-configConfigMap in a deployment,default-tenantwhen unset): connections and their credentials, application metadata, the four fetch snapshots, briefs, remediation overlays and routes, rule policies and support calls.
A runner gets its own tables at tenants/<t>/var/lib/runners.<RUNNER_ID>.<table>. The
exception is the fetch snapshots, which the runner writes and the control plane reads.
That shared location is why a runner and its control plane must use the same tenant.
FACE_TENANT before an instance’s first boot. That boot copies any legacy data
into the tenant it sees, once. If you set the tenant later, the legacy data stays under
default-tenant. On a runner, keep RUNNER_ID stable. The operator uses the Deployment
name. If it fell back to the pod name, every restart would start the runner on a fresh,
empty set of tables.When there is no object store, the local root must be on storage that survives a
restart. The operator points FACE_APPFS_DIR at appfs/ inside the pod’s persistent
connections volume. If the default location next to face.db were used instead, it
would be on the /data emptyDir, and sessions and credentials would be lost on every
restart.
Sessions are validated on every request against this root, with a 2-second cache. A revocation on the control plane therefore takes effect on a runner within that window.
No SQLite database
FACE keeps no state in SQLite. The control plane’s application-metadata snapshot lives
in an appfs table, like a runner’s state, sealed under FACE’s key in the object store.
Nothing restores or replicates a database file, and the control plane’s /data
volume is scratch only.
An instance upgraded from an earlier build may still have face.db at SQLITE_PATH. On
boot, FACE opens it only if it exists (it never creates one), imports its rows once (the
snapshot only when the appfs table has none, plus the legacy session revocations and
connections), and never writes to it again. The file is left untouched as the rollback.
No deployment path replicates or restores a SQLite file. The operator starts the
server directly (/app/server serve); the image’s /app/entrypoint, for a local
podman run, starts the same server and nothing else.
Credential store
A connection’s credentials never sit in an environment variable or in the connection
record. The connection record has no field a secret would fit in. When a connection is
saved, secret-named fields are lifted into a credential bundle. Bundles are stored as rows
of the tenant table tenants/<t>/var/lib/connections (credentials/<ref>), sealed under
FACE’s derived key and bound to their path. Consensus replicates only the connection
metadata snapshot, never the credentials.
Where FACE writes to local disk (the local appfs root, the knowledge corpus without an
object store), directories are created 0700 and files 0600. The older layout
(FACE_CONNECTIONS_DIR, with connections.json and credentials/*.json) is imported
once on a persistent root, and each imported file is renamed <file>.migrated.
Migrations leave a rollback in place
Each move to a new storage layout copies the data once and leaves the old copy behind, so you can go back to an older image:
Legacy metadata files are imported once into their table. Each one is then replaced by an envelope-sealed
<file>.migrated. To read one back, use thefaceopstool that ships in the backend image, with the same KEK in its environment:/app/faceops metalog-unseal --table <table> <file>.migratedOlder object-store tables (
tables/face/<name>/in the object store’s own bucket) are copied into the appfs root on first open, and only into a table that has never been written. The originals are left in place.The retired vector corpus (
comet/in the object store) is not read or migrated. It is left in place, untouched. Re-run document ingestion to make documents searchable in the knowledge corpus.
Key custody
TIDE_ENVELOPE_KEK is a base64-encoded 32-byte key held in the tide-envelope-kek
Secret in face-system. face-operator creates it if it is absent, and never rotates
or overwrites it.
To change keys, keep the old key readable. TIDE_ENVELOPE_KEK_RETIRED takes a list of
earlier KEKs (base64, separated by commas or whitespace). FACE and the storage libraries
use them for decryption only, and seal new writes under TIDE_ENVELOPE_KEK. If you
replace the KEK without listing the old one as retired, everything sealed under the old
key becomes unreadable.
What to back up
- The
tide-envelope-kekSecret, escrowed offline. - The
face-install-configConfigMap, which names the tenant the data is filed under. - The object-store buckets FACE writes to on
face-objectd: the appfs bucket (FACE_APPFS_BUCKET, defaultface), the documents bucket (FACE_DOCUMENTS_BUCKET, defaultface-documents) and the bucket inOBJECTSTORE_BUCKET(the deployment setsrunink-face-db-nane1). - When there is no object store: the node-local connections volume
(
/var/lib/tide/face-connectionson the node), which contains the appfs root, and the knowledge directory. - The node-local raft volume (
/var/lib/tide/rafton the node, mounted at/raft), which holds the control plane’s consensus state (RAFT_DATA_DIR).
To restore, put the KEK Secret and the tenant ConfigMap back first, then the object-store data, and only then start the control plane.