Defence in depth
FACE layers independent checks rather than relying on any one of them. Every user request carries an authenticated session and passes a role check (see Identity & access). Services also authenticate each other with short-lived, rotating ECDSA certificates issued from a shared cluster certificate authority.
Service-to-service: mesh mTLS
Mesh mTLS is unconditional
When the backend starts, it builds a rotating TLS identity from the shared mesh
CA, which it reads from the secrets MESH_CA_CERT and MESH_CA_KEY. In the
cluster these come from the tide-mesh-ca Secret in FACE’s namespace, which
TIDE mints and distributes (tide mesh ca). If either is missing, unreadable
or expired, the process stops with
mesh identity unavailable, refusing to start and does not serve.
No environment variable produces a plaintext mesh connection. There is no “mTLS off” setting, and no code path falls back to an insecure transport.
That identity secures:
- FACE server to runner. The FACE server (the control plane) uses it to dial a runner, and that call carries connector credentials and tenant data.
- Raft consensus peers. The raft service replicates configuration, runner and connection metadata between nodes.
- Mesh gRPC. It is the transport for every peer that dials FACE’s gRPC services as a mesh member.
Rotating, short-lived ECDSA leaves
Identities come from Runink’s shared PKI library (the pki.Source type):
| Property | Value | Where it is set |
|---|---|---|
| Key type | ECDSA P-256 | the shared PKI library |
| Leaf lifetime | 1 hour | LeafTTL: time.Hour |
| Re-issue interval | every 20 minutes | RotateEvery: 20 * time.Minute |
| Minimum TLS version on mesh hops | TLS 1.3 | pki ServerConfig / ClientConfig |
| Server-side client authentication | tls.RequireAndVerifyClientCert against the mesh CA | pki.Source.ServerConfig |
The re-issue interval must be shorter than the leaf lifetime, and pki refuses
any configuration where it is not. A leaf is replaced well before it expires.
Leaves can never outlive the CA that signed them, and the process readiness
check tracks both leaf validity and CA validity. A pod whose mesh identity has
lapsed reports itself not ready instead of continuing to answer health checks.
TLS 1.3 limits every mesh cipher suite to an AEAD construction (AES-GCM or ChaCha20-Poly1305). It also rules out renegotiation and static-RSA key exchange, and encrypts the handshake.
Chain verification, and what identity means
A mesh server requires a client certificate and verifies it against the mesh CA during the TLS handshake. A peer without a CA-signed certificate is rejected before any RPC is dispatched.
A mesh client verifies that the server’s certificate chains to the mesh CA, with the server-authentication key usage. On the mesh, identity means “holds a leaf signed by this cluster’s mesh CA”. It does not mean “is this particular hostname”. That lets peers be dialled by service name, pod address or IP without re-issuing certificates. It also means the CA private key is the root of trust for the whole mesh, and its custody is a platform responsibility.
No plaintext downgrade
A failed handshake means a failed call: the caller receives Unavailable with
the x509 reason. FACE does not retry in plaintext and does not come up with a
degraded transport.
Tests pin this behaviour:
TestRunnerDialDoesNotDowngradeToAPlaintextRunnerandTestRunnerDialFailsClosedOnCASkewdial a real plaintext-only runner and a runner holding a certificate from a different CA. It requires both to be refused. A positive control proves each refusal came from the TLS layer and not from a dead port.TestRunnerRefusesAnAnonymousCallermakes a real anonymous TLS dial and requires the mutual half of mTLS to reject it.TestNoInsecureTransportInAnyPackageparses every Go source file in the repository and fails oninsecure.NewCredentials,grpc.WithInsecure, ALTS credentials orInsecureSkipVerify: true. It also fails onInsecureSkipVerifyset from a variable. The only places it excuses are the MQTT and Kafka connectors, which read the connection’sinsecure_skip_verifyfield. No stored connection can turn that field on: the connector security policy refuses any connection with it set, on write and on every read (see Network egress controls). Because the parse skips comments, prose that mentions these names can neither trip the check nor satisfy it.
Consensus is never served without mTLS
The raft service carries no user token, so it is exempt from bearer
authentication (see the unauthenticated surface).
To keep that exemption safe, the raft service is served only on the mTLS
peer listener, never on a plain registrar.
TestRaftServiceIsNeverOnTheApplicationListener walks every non-test file in the
command package and fails if any code registers the raft service anywhere else.
People: the browser path
Browser traffic reaches FACE through TIDE’s edge in tide-system, which
terminates the browser’s TLS. The edge dials the FACE control plane’s TLS
sidecar with a cluster-CA client certificate, and the sidecar forwards to the
application port. When TIDE’s operator injects that sidecar it also sets
FACE_BIND_HOST=127.0.0.1, so the application port listens on loopback only;
a bind host that is not an IP literal stops the backend at startup. A separate
health listener (FACE_HEALTH_LISTEN) serves only /livez, /readyz and
/auth/config so the kubelet can probe the pod. FACE does not ask a browser
for a client certificate, because a browser cannot present a mesh identity.
Because the sidecar delivers edge traffic over loopback, FACE never treats a
loopback address alone as proof that a caller is local.
Every user request instead has to carry a session bearer token, which the auth
interceptor validates on every call. A verified mTLS identity is recorded on the
request as an attribute, but it never replaces a session token.
TestAVerifiedMTLSPeerStillNeedsABearerToken pins that rule.
Transport identity on the request
FACE never trusts a header that claims a certificate was checked upstream. The caller’s transport identity is read only from TLS state that this process negotiated. A certificate counts only if the TLS stack populated its verified chains. A certificate that was merely presented does not count.
That identity is used in three places:
- Server reflection (the gRPC service catalogue) is available only to a
verified mTLS peer. Anyone else receives
server reflection requires a verified mTLS peer. - The internal model service.
ModelServiceanswers without a session only when the caller is this process: a loopback peer that also holds a verified mesh certificate (see Identity & access). - The policy subject. When a verified certificate is present, its common name
is attached to the policy subject as the attribute
mtls_client_cn.
Typed request identity
After a token is verified, the tenant, role, subject and subscription tier go
onto the request context under an unexported, typed key. No other package in
the process, dependencies included, can collide with those keys or forge them.
TestNoBareStringContextKeysInPackageCmd fails on any context.WithValue in the command
package whose key is a string. The security policy engine reads a typed
policy.Subject built from the same verified values. That subject’s
internal-system flag is always false for a remote caller. Only an in-process
audit hook sets it to true.
Scope of these controls
- Mesh mTLS covers service-to-service connections only. It puts no client-certificate requirement on browser traffic. User requests are authenticated by session tokens, and they are only as strong as the session controls described in Identity & access.
- The mesh CA is managed by the platform. Protecting the CA key, and distributing and rotating the CA, is done by the platform, not by FACE.
- Envelope encryption applies at rest; TLS applies in transit. Data is sealed before it is written, as Encryption at rest describes. Confidentiality in transit comes from TLS.
- The tests cover FACE’s own dial and listen code only. They say nothing about the certificates your cluster actually holds. Check CA validity on your own installation.