Skip to content
Data audit & lineage

Data audit & lineage

Three views answer “what happened” questions:

  • Data audit assesses the data estate. It is the third tile of the Overview’s audit verdict strip, and ?tab=dataAudit opens its full panel.
  • Lineage shows how data moves. It is the second tab of DataEx › Estate (?tab=lineage).
  • Deploy lineage shows whether a commit shipped. Despite the name, it is about the CD pipeline, not data. It is the second card of DevEx › Audit Chain (?tab=deployLineage).

Data audit (?tab=dataAudit)

This is the data assessment. The Overview’s verdict strip shows three: Platform audit, Model & agent audit and Data audit. These pages are assessments, not the audit log. The log is DevEx › Audit chain.

The verdict

The page opens on one StatusIndicator verdict over checks it reads from existing endpoints:

CheckRead from
Source connection testsResolve · sources
Estate mapResolve · estate map
Declared vs observedResolve · reconciliation
Data quality (CapEx)CapEx · summary
Data governance findingsGET /api/data-governance-findings

A check that could not be read goes, with its reason, into a single not-measured list. It is never counted as a pass.

Actions and the runner picker

Resolve’s Test, Explore, Access patterns and Map estate, and CapEx’s pull, run on the pages that own them: the Data audit panel says where each one lives and links there. The runner is picked where the action is confirmed:

  • DataEx › Estate. Each Test, Explore, Access patterns and Map estate confirm dialog lists each source’s own runner (the default: no runner on the wire), Runink managed (console) and every registered self-hosted runner from /api/runners (a revoked one is shown and cannot be picked). The list comes from consoleRunnerOptions through the shared ui estate package’s EstateController.runnerOptions hook.
  • CapEx › Pull from a connection. The pull form draws the same picker, starting on the connection’s own runner.

A connection’s default binding is set in the Integrations › Connections wizard, with the same picker.

The choice goes on the request’s optional runner field. If a build cannot carry the field, a self-hosted choice is refused client-side with RunnerChoiceNotSent before any call is made. It is never silently dropped. The field has three states:

FieldMeans
absentUse the connection’s binding
consoleRunink managed
a nameThat runner

That is why managed is always sent explicitly as console.

A self-hosted runner that cannot take the audit answers with a runner outcome (runner_unknown, runner_revoked or runner_offline), and nothing runs anywhere. Audits run on the Runink managed runner (console). See Estate.

Agent reviews

An audit page starts an agent in exactly one way: by running an existing playbook with PlaybookService.RunNow. It uses the same call, the same confirmation text and the same refusal handling as Playbooks. There is no new dispatch path. Creating or editing a playbook stays on Playbooks. See Playbooks.

Lineage (?tab=lineage)

The Lineage page reads GET /api/data-lineage (session-gated, GET only). It combines:

  • the declared topology from the connection registry;
  • the observed flows that apps have published.

Observations come from two origins, and the page never mixes them. Every edge and publisher carries origin: "app" or origin: "resolve":

  • App edges carry movements and rows. The page draws graph links from app edges only.
  • Resolve edges carry structure. They are listed under “Observed by Runink TIDE Resolve”. The DECLARED_VIEW_DEPENDENCY edges are drawn as Declared derivations.

Resolve makes the structural half observed. The movement half needs an app to publish. With no app publishing, the gap is named in unmeasured[] as app flow lineage. Per-node DQ and consumers are absent.

How apps publish: POST /api/data-lineage/edges

Apps push their lineage. The console never reaches into an app pod. The body is a bounded rollup of the distinct source→destination pairs the app observed, with movement and row counts, plus the publisher’s own status: whether capture is durable, and how many records it observed and dropped.

CaseResponse
TIDE_HEALTH_INGEST_TOKEN unset503: observed lineage ingest is not configured (TIDE_HEALTH_INGEST_TOKEN unset; …)
Wrong bearer token403: bad ingest token
app missing400: app is required
app is tide-resolve403. That name is reserved for the console’s own Resolve observations
No observations and no edges400. A publisher with nothing to say must stay silent

The token is the same agent-health ingest credential the apps already hold. On the app side it is TIDE_HEALTH_URL + TIDE_HEALTH_TOKEN, mounted from a Secret that tide-operator’s provisioner generates. Edges without a connection id are dropped, and repeated pairs are folded into one. Reports expire after 30 days.

Deploy lineage (?tab=deployLineage)

GET /api/lineage (session-gated) answers “did my change ship?” from measurements. Per image, it follows six stages: merged → agents saw it → built → landed in the registry → rolled → running.

  • The delivery workflow records what it rolled, and the console reads it back.
  • The build leg comes from the image build’s registry probe of /v2/<repo>/tags/list.
  • Each stage is ok, warn, failed or unmeasured. Unmeasured is used freely: a build leg with no digest record, a commit older than the run store’s retention, or a workload whose pods could not be read.

Add ?sha=<commit> to ask about one commit. The answer starts from found: false and state unmeasured, and fills in only what it could measure.

Deploy lineage is not data lineage. It tracks container images through CI/CD. The Lineage page above tracks data between sources.