Runners
A fetch runs on a runner, never on the control plane. The control plane receives
RunFetch, forwards it to a runner together with the caller’s session and the connection
credentials the fetch needs, and streams the runner’s progress back to the FACE app. At
startup each process logs which role it has:
🧵 Fetch role: CONTROL-PLANE — forwards fetches to the dedicated runner at <endpoint> (never runs them here).
🧵 Fetch role: RUNNER — executes the fetch pipeline in-process (dedicated compute).A runner is the FACE backend image started with SERVICE_ROLE=runner. Every boot check
on Prerequisites and ordering applies to a runner as
well, including the mesh CA, the KEK and the inference endpoint. A runner also mounts the
same appfs root as its control plane, because it validates the forwarded session there
and writes its fetch results there.
How to connect data sources to runners, and how to choose a runner for a source, is covered in Data sources. This page covers operating the runners themselves.
The in-cluster runner
A deployment has one in-cluster runner: the static face-runner Deployment in
face-system, from TIDE’s on-host bundle. The control plane dials it at
MANAGED_RUNNER_ENDPOINT (face-runner.face-system.svc:7102) over mutual TLS from the
shared mesh CA. It never falls back to plaintext.
| Task | How |
|---|---|
| Size the runner | Change the requests and limits in face-runner’s manifest in TIDE’s on-host bundle. No operator reconciles it. |
| Point the control plane at a runner | spec.managedRunnerEndpoint on the ControlPlane resource face-control-plane. |
| Refuse to run fetch work on the control plane | Set REQUIRE_RUNNER_ISOLATION=true in the ControlPlane env. An unreachable runner is then a hard failure. |
| Keep runner state across restarts | Keep RUNNER_ID stable. The manifest sets face-runner. |
| Keep runner and control plane on one tenant | Both read FACE_TENANT from the face-install-config ConfigMap. Do not set it anywhere else. |
Runners are not consensus voters. They leave RAFT_NODE_ID and RAFT_MTLS_ADDR
unset, and a runner’s /readyz reports the raft check as skipped. That is expected:
the control plane is the only voting member.
When a runner cannot be reached, the fetch fails with:
Analysis failed: dedicated runner unreachable — fetch does not run on the control plane: <reason><reason> explains the dial error. Look for a runner pod that is not Ready, an endpoint
that points at the wrong Service, or a mesh certificate that no longer chains to the CA
the control plane holds.
Self-hosted runners (outbound enrolment)
A self-hosted runner runs on your own machine and dials out to the control plane.
Nothing dials in to it. The runner authenticates with a service-account token issued
for your tenant by IdentityService.GenerateServiceAccountKey, holds one enrolment
stream open, and runs the fetches the control plane assigns over that stream.
| Variable | Purpose |
|---|---|
SERVICE_ROLE=runner | Starts the process as a runner. |
RUNNER_ENROLL_ADDR | The control plane’s gRPC address, as seen from the runner. Setting it turns enrolment on. |
RUNNER_ENROLL_TOKEN | The service-account token. It is required. |
RUNNER_ID | A stable id, so that a reconnect takes the same registry slot. |
RUNNER_NAME, RUNNER_VERSION | Shown in the control plane’s runner registry. |
RUNNER_ENROLL_TLS | Enrolment always uses TLS with the system roots. false is refused (Runner enrolment refused: RUNNER_ENROLL_TLS=false is not supported): terminate TLS in front of the control plane instead, so the token is never sent in cleartext. |
When enrolment starts, the runner logs:
🛰️ Outbound runner enrolment enabled: dialling <addr> (runner id "<id>")If RUNNER_ENROLL_ADDR is set without a token, the runner refuses to start:
❌ RUNNER_ENROLL_ADDR is set but RUNNER_ENROLL_TOKEN is empty — a runner enrols with a service-account token issued by IdentityService.GenerateServiceAccountKey; there is no anonymous enrolmentWhen the stream ends, the runner reconnects after 5 seconds.
You can start self-hosted runners yourself, as above, or deploy them from Settings › Integrations › Runners (below).
Deploying runners from FACE
An organisation admin can deploy, redeploy and decommission a runner from Settings › Integrations › Runners. Any other role, and any service account, is refused, and nothing is deployed. Every attempt is audited, and deployment records survive a restart.
| Target | What FACE does | Needs |
|---|---|---|
| On premises | Copies the runner image archive or binary this server holds to your machine over SSH, checked against its SHA-256. | FACE_RUNNER_IMAGE_ARCHIVE with FACE_RUNNER_IMAGE_ARCHIVE_SHA256 (and FACE_RUNNER_IMAGE_REF), or FACE_RUNNER_BINARY with FACE_RUNNER_BINARY_SHA256. |
| Google Cloud | Starts a Compute Engine VM in your project from an image in your registry. No service-account JSON key is used, and Cloud Run is refused. | FACE_RUNNER_GCP_IMAGE, a digest-pinned image. |
| Snowflake, Databricks | Starts a runner that connects out to FACE’s runner channel over mutual TLS. | FACE_RUNNER_CHANNEL_ADDR and FACE_RUNNER_CHANNEL_LISTEN, and the runner CA in FACE_RUNNER_CA_DIR. Without them the targets are listed as unavailable. |
Deploy credentials come from the connection’s sealed credentials and non-secret settings, and
from files under FACE_DEPLOY_SECRETS_DIR, read by path. See
Plans and billing for how runners in your own cloud are
charged on the Cloud plan.
Snowflake runners
runink runner create provisions a Snowflake Container Services runner. It reads
--compute-pool (or RUNNER_COMPUTE_POOL, which is required for that type),
RUNNER_CPU_LIMIT (default 1) and RUNNER_MEMORY_LIMIT (default 1G). It stops if the
compute pool is not named, because it cannot invent one.