Skip to content

Runners

A fetch runs on a runner, never on the control plane. The control plane receives RunFetch, forwards it to a runner together with the caller’s session and the connection credentials the fetch needs, and streams the runner’s progress back to the FACE app. At startup each process logs which role it has:

🧵 Fetch role: CONTROL-PLANE — forwards fetches to the dedicated runner at <endpoint> (never runs them here).
🧵 Fetch role: RUNNER — executes the fetch pipeline in-process (dedicated compute).

A runner is the FACE backend image started with SERVICE_ROLE=runner. Every boot check on Prerequisites and ordering applies to a runner as well, including the mesh CA, the KEK and the inference endpoint. A runner also mounts the same appfs root as its control plane, because it validates the forwarded session there and writes its fetch results there.

How to connect data sources to runners, and how to choose a runner for a source, is covered in Data sources. This page covers operating the runners themselves.

The in-cluster runner

A deployment has one in-cluster runner: the static face-runner Deployment in face-system, from TIDE’s on-host bundle. The control plane dials it at MANAGED_RUNNER_ENDPOINT (face-runner.face-system.svc:7102) over mutual TLS from the shared mesh CA. It never falls back to plaintext.

TaskHow
Size the runnerChange the requests and limits in face-runner’s manifest in TIDE’s on-host bundle. No operator reconciles it.
Point the control plane at a runnerspec.managedRunnerEndpoint on the ControlPlane resource face-control-plane.
Refuse to run fetch work on the control planeSet REQUIRE_RUNNER_ISOLATION=true in the ControlPlane env. An unreachable runner is then a hard failure.
Keep runner state across restartsKeep RUNNER_ID stable. The manifest sets face-runner.
Keep runner and control plane on one tenantBoth read FACE_TENANT from the face-install-config ConfigMap. Do not set it anywhere else.

Runners are not consensus voters. They leave RAFT_NODE_ID and RAFT_MTLS_ADDR unset, and a runner’s /readyz reports the raft check as skipped. That is expected: the control plane is the only voting member.

When a runner cannot be reached, the fetch fails with:

Analysis failed: dedicated runner unreachable — fetch does not run on the control plane: <reason>

<reason> explains the dial error. Look for a runner pod that is not Ready, an endpoint that points at the wrong Service, or a mesh certificate that no longer chains to the CA the control plane holds.

Self-hosted runners (outbound enrolment)

A self-hosted runner runs on your own machine and dials out to the control plane. Nothing dials in to it. The runner authenticates with a service-account token issued for your tenant by IdentityService.GenerateServiceAccountKey, holds one enrolment stream open, and runs the fetches the control plane assigns over that stream.

VariablePurpose
SERVICE_ROLE=runnerStarts the process as a runner.
RUNNER_ENROLL_ADDRThe control plane’s gRPC address, as seen from the runner. Setting it turns enrolment on.
RUNNER_ENROLL_TOKENThe service-account token. It is required.
RUNNER_IDA stable id, so that a reconnect takes the same registry slot.
RUNNER_NAME, RUNNER_VERSIONShown in the control plane’s runner registry.
RUNNER_ENROLL_TLSEnrolment always uses TLS with the system roots. false is refused (Runner enrolment refused: RUNNER_ENROLL_TLS=false is not supported): terminate TLS in front of the control plane instead, so the token is never sent in cleartext.

When enrolment starts, the runner logs:

🛰️  Outbound runner enrolment enabled: dialling <addr> (runner id "<id>")

If RUNNER_ENROLL_ADDR is set without a token, the runner refuses to start:

❌ RUNNER_ENROLL_ADDR is set but RUNNER_ENROLL_TOKEN is empty — a runner enrols with a service-account token issued by IdentityService.GenerateServiceAccountKey; there is no anonymous enrolment

When the stream ends, the runner reconnects after 5 seconds.

An enrolled runner handles one kind of work, fetch. It does not persist, queue or retry work. A fetch assigned while the runner is shutting down is lost, and the control plane sees a deadline. Rotate the service-account token the way you rotate any credential, then restart the runner with the new value.

You can start self-hosted runners yourself, as above, or deploy them from Settings › Integrations › Runners (below).

Deploying runners from FACE

An organisation admin can deploy, redeploy and decommission a runner from Settings › Integrations › Runners. Any other role, and any service account, is refused, and nothing is deployed. Every attempt is audited, and deployment records survive a restart.

TargetWhat FACE doesNeeds
On premisesCopies the runner image archive or binary this server holds to your machine over SSH, checked against its SHA-256.FACE_RUNNER_IMAGE_ARCHIVE with FACE_RUNNER_IMAGE_ARCHIVE_SHA256 (and FACE_RUNNER_IMAGE_REF), or FACE_RUNNER_BINARY with FACE_RUNNER_BINARY_SHA256.
Google CloudStarts a Compute Engine VM in your project from an image in your registry. No service-account JSON key is used, and Cloud Run is refused.FACE_RUNNER_GCP_IMAGE, a digest-pinned image.
Snowflake, DatabricksStarts a runner that connects out to FACE’s runner channel over mutual TLS.FACE_RUNNER_CHANNEL_ADDR and FACE_RUNNER_CHANNEL_LISTEN, and the runner CA in FACE_RUNNER_CA_DIR. Without them the targets are listed as unavailable.

Deploy credentials come from the connection’s sealed credentials and non-secret settings, and from files under FACE_DEPLOY_SECRETS_DIR, read by path. See Plans and billing for how runners in your own cloud are charged on the Cloud plan.

Snowflake runners

runink runner create provisions a Snowflake Container Services runner. It reads --compute-pool (or RUNNER_COMPUTE_POOL, which is required for that type), RUNNER_CPU_LIMIT (default 1) and RUNNER_MEMORY_LIMIT (default 1G). It stops if the compute pool is not named, because it cannot invent one.