Skip to content

Defence in depth

FACE layers independent checks rather than relying on any one of them. Every user request carries an authenticated session and passes a role check (see Identity & access). Services also authenticate each other with short-lived, rotating ECDSA certificates issued from a shared cluster certificate authority.

Service-to-service: mesh mTLS

Mesh mTLS is unconditional

When the backend starts, it builds a rotating TLS identity from the shared mesh CA, which it reads from the secrets MESH_CA_CERT and MESH_CA_KEY. In the cluster these come from the tide-mesh-ca Secret in FACE’s namespace, which TIDE mints and distributes (tide mesh ca). If either is missing, unreadable or expired, the process stops with mesh identity unavailable, refusing to start and does not serve.

No environment variable produces a plaintext mesh connection. There is no “mTLS off” setting, and no code path falls back to an insecure transport.

That identity secures:

  • FACE server to runner. The FACE server (the control plane) uses it to dial a runner, and that call carries connector credentials and tenant data.
  • Raft consensus peers. The raft service replicates configuration, runner and connection metadata between nodes.
  • Mesh gRPC. It is the transport for every peer that dials FACE’s gRPC services as a mesh member.

Rotating, short-lived ECDSA leaves

Identities come from Runink’s shared PKI library (the pki.Source type):

PropertyValueWhere it is set
Key typeECDSA P-256the shared PKI library
Leaf lifetime1 hourLeafTTL: time.Hour
Re-issue intervalevery 20 minutesRotateEvery: 20 * time.Minute
Minimum TLS version on mesh hopsTLS 1.3pki ServerConfig / ClientConfig
Server-side client authenticationtls.RequireAndVerifyClientCert against the mesh CApki.Source.ServerConfig

The re-issue interval must be shorter than the leaf lifetime, and pki refuses any configuration where it is not. A leaf is replaced well before it expires. Leaves can never outlive the CA that signed them, and the process readiness check tracks both leaf validity and CA validity. A pod whose mesh identity has lapsed reports itself not ready instead of continuing to answer health checks.

TLS 1.3 limits every mesh cipher suite to an AEAD construction (AES-GCM or ChaCha20-Poly1305). It also rules out renegotiation and static-RSA key exchange, and encrypts the handshake.

Chain verification, and what identity means

A mesh server requires a client certificate and verifies it against the mesh CA during the TLS handshake. A peer without a CA-signed certificate is rejected before any RPC is dispatched.

A mesh client verifies that the server’s certificate chains to the mesh CA, with the server-authentication key usage. On the mesh, identity means “holds a leaf signed by this cluster’s mesh CA”. It does not mean “is this particular hostname”. That lets peers be dialled by service name, pod address or IP without re-issuing certificates. It also means the CA private key is the root of trust for the whole mesh, and its custody is a platform responsibility.

No plaintext downgrade

A failed handshake means a failed call: the caller receives Unavailable with the x509 reason. FACE does not retry in plaintext and does not come up with a degraded transport.

Tests pin this behaviour:

  • TestRunnerDialDoesNotDowngradeToAPlaintextRunner and TestRunnerDialFailsClosedOnCASkew dial a real plaintext-only runner and a runner holding a certificate from a different CA. It requires both to be refused. A positive control proves each refusal came from the TLS layer and not from a dead port. TestRunnerRefusesAnAnonymousCaller makes a real anonymous TLS dial and requires the mutual half of mTLS to reject it.
  • TestNoInsecureTransportInAnyPackage parses every Go source file in the repository and fails on insecure.NewCredentials, grpc.WithInsecure, ALTS credentials or InsecureSkipVerify: true. It also fails on InsecureSkipVerify set from a variable. The only places it excuses are the MQTT and Kafka connectors, which read the connection’s insecure_skip_verify field. No stored connection can turn that field on: the connector security policy refuses any connection with it set, on write and on every read (see Network egress controls). Because the parse skips comments, prose that mentions these names can neither trip the check nor satisfy it.

Consensus is never served without mTLS

The raft service carries no user token, so it is exempt from bearer authentication (see the unauthenticated surface). To keep that exemption safe, the raft service is served only on the mTLS peer listener, never on a plain registrar. TestRaftServiceIsNeverOnTheApplicationListener walks every non-test file in the command package and fails if any code registers the raft service anywhere else.

People: the browser path

Browser traffic reaches FACE through TIDE’s edge in tide-system, which terminates the browser’s TLS. The edge dials the FACE control plane’s TLS sidecar with a cluster-CA client certificate, and the sidecar forwards to the application port. When TIDE’s operator injects that sidecar it also sets FACE_BIND_HOST=127.0.0.1, so the application port listens on loopback only; a bind host that is not an IP literal stops the backend at startup. A separate health listener (FACE_HEALTH_LISTEN) serves only /livez, /readyz and /auth/config so the kubelet can probe the pod. FACE does not ask a browser for a client certificate, because a browser cannot present a mesh identity. Because the sidecar delivers edge traffic over loopback, FACE never treats a loopback address alone as proof that a caller is local.

Every user request instead has to carry a session bearer token, which the auth interceptor validates on every call. A verified mTLS identity is recorded on the request as an attribute, but it never replaces a session token. TestAVerifiedMTLSPeerStillNeedsABearerToken pins that rule.

Transport identity on the request

FACE never trusts a header that claims a certificate was checked upstream. The caller’s transport identity is read only from TLS state that this process negotiated. A certificate counts only if the TLS stack populated its verified chains. A certificate that was merely presented does not count.

That identity is used in three places:

  • Server reflection (the gRPC service catalogue) is available only to a verified mTLS peer. Anyone else receives server reflection requires a verified mTLS peer.
  • The internal model service. ModelService answers without a session only when the caller is this process: a loopback peer that also holds a verified mesh certificate (see Identity & access).
  • The policy subject. When a verified certificate is present, its common name is attached to the policy subject as the attribute mtls_client_cn.

Typed request identity

After a token is verified, the tenant, role, subject and subscription tier go onto the request context under an unexported, typed key. No other package in the process, dependencies included, can collide with those keys or forge them. TestNoBareStringContextKeysInPackageCmd fails on any context.WithValue in the command package whose key is a string. The security policy engine reads a typed policy.Subject built from the same verified values. That subject’s internal-system flag is always false for a remote caller. Only an in-process audit hook sets it to true.

Scope of these controls

  • Mesh mTLS covers service-to-service connections only. It puts no client-certificate requirement on browser traffic. User requests are authenticated by session tokens, and they are only as strong as the session controls described in Identity & access.
  • The mesh CA is managed by the platform. Protecting the CA key, and distributing and rotating the CA, is done by the platform, not by FACE.
  • Envelope encryption applies at rest; TLS applies in transit. Data is sealed before it is written, as Encryption at rest describes. Confidentiality in transit comes from TLS.
  • The tests cover FACE’s own dial and listen code only. They say nothing about the certificates your cluster actually holds. Check CA validity on your own installation.