Data audit & lineage
Three views answer “what happened” questions:
- Data audit assesses the data estate. It is the third tile of the Overview’s audit verdict strip, and
?tab=dataAuditopens its full panel. - Lineage shows how data moves. It is the second tab of DataEx › Estate (
?tab=lineage). - Deploy lineage shows whether a commit shipped. Despite the name, it is about the CD pipeline, not data. It is the second card of DevEx › Audit Chain (
?tab=deployLineage).
Data audit (?tab=dataAudit)
This is the data assessment. The Overview’s verdict strip shows three: Platform audit, Model & agent audit and Data audit. These pages are assessments, not the audit log. The log is DevEx › Audit chain.
The verdict
The page opens on one StatusIndicator verdict over checks it reads from existing endpoints:
| Check | Read from |
|---|---|
| Source connection tests | Resolve · sources |
| Estate map | Resolve · estate map |
| Declared vs observed | Resolve · reconciliation |
| Data quality (CapEx) | CapEx · summary |
| Data governance findings | GET /api/data-governance-findings |
A check that could not be read goes, with its reason, into a single not-measured list. It is never counted as a pass.
Actions and the runner picker
Resolve’s Test, Explore, Access patterns and Map estate, and CapEx’s pull, run on the pages that own them: the Data audit panel says where each one lives and links there. The runner is picked where the action is confirmed:
- DataEx › Estate. Each Test, Explore, Access patterns and Map estate confirm dialog lists each source’s own runner (the default: no runner on the wire), Runink managed (
console) and every registered self-hosted runner from/api/runners(a revoked one is shown and cannot be picked). The list comes fromconsoleRunnerOptionsthrough the shared ui estate package’sEstateController.runnerOptionshook. - CapEx › Pull from a connection. The pull form draws the same picker, starting on the connection’s own runner.
A connection’s default binding is set in the Integrations › Connections wizard, with the same picker.
The choice goes on the request’s optional runner field. If a build cannot carry the field, a self-hosted choice is refused client-side with RunnerChoiceNotSent before any call is made. It is never silently dropped. The field has three states:
| Field | Means |
|---|---|
| absent | Use the connection’s binding |
console | Runink managed |
| a name | That runner |
That is why managed is always sent explicitly as console.
A self-hosted runner that cannot take the audit answers with a runner outcome (runner_unknown, runner_revoked or runner_offline), and nothing runs anywhere. Audits run on the Runink managed runner (console). See Estate.
Agent reviews
An audit page starts an agent in exactly one way: by running an existing playbook with PlaybookService.RunNow. It uses the same call, the same confirmation text and the same refusal handling as Playbooks. There is no new dispatch path. Creating or editing a playbook stays on Playbooks. See Playbooks.
Lineage (?tab=lineage)
The Lineage page reads GET /api/data-lineage (session-gated, GET only). It combines:
- the declared topology from the connection registry;
- the observed flows that apps have published.
Observations come from two origins, and the page never mixes them. Every edge and publisher carries origin: "app" or origin: "resolve":
- App edges carry movements and rows. The page draws graph links from app edges only.
- Resolve edges carry structure. They are listed under “Observed by Runink TIDE Resolve”. The
DECLARED_VIEW_DEPENDENCYedges are drawn as Declared derivations.
Resolve makes the structural half observed. The movement half needs an app to publish. With no app publishing, the gap is named in unmeasured[] as app flow lineage. Per-node DQ and consumers are absent.
How apps publish: POST /api/data-lineage/edges
Apps push their lineage. The console never reaches into an app pod. The body is a bounded rollup of the distinct source→destination pairs the app observed, with movement and row counts, plus the publisher’s own status: whether capture is durable, and how many records it observed and dropped.
| Case | Response |
|---|---|
TIDE_HEALTH_INGEST_TOKEN unset | 503: observed lineage ingest is not configured (TIDE_HEALTH_INGEST_TOKEN unset; …) |
| Wrong bearer token | 403: bad ingest token |
app missing | 400: app is required |
app is tide-resolve | 403. That name is reserved for the console’s own Resolve observations |
| No observations and no edges | 400. A publisher with nothing to say must stay silent |
The token is the same agent-health ingest credential the apps already hold. On the app side it is TIDE_HEALTH_URL + TIDE_HEALTH_TOKEN, mounted from a Secret that tide-operator’s provisioner generates. Edges without a connection id are dropped, and repeated pairs are folded into one. Reports expire after 30 days.
Deploy lineage (?tab=deployLineage)
GET /api/lineage (session-gated) answers “did my change ship?” from measurements. Per image, it follows six stages: merged → agents saw it → built → landed in the registry → rolled → running.
- The delivery workflow records what it rolled, and the console reads it back.
- The build leg comes from the image build’s registry probe of
/v2/<repo>/tags/list. - Each stage is
ok,warn,failedor unmeasured. Unmeasured is used freely: a build leg with no digest record, a commit older than the run store’s retention, or a workload whose pods could not be read.
Add ?sha=<commit> to ask about one commit. The answer starts from found: false and state unmeasured, and fills in only what it could measure.