Data audit & lineage

Three Intelligence pages answer “what happened” questions:

• Data audit assesses the data estate.
• Lineage shows how data moves.
• Deploy lineage shows whether a commit shipped. Despite the name, it is about the CD pipeline, not data.

Data audit (?tab=dataAudit)

This is the data assessment page. Each console category has its own audit page: Platform audit in DevEx, Model & agent audit in DataEx, and Data audit here. These pages are assessments, not the audit log. The log is DevEx › Audit chain.

Code: flutter/lib/features/audits/data_audit_screen.dart.

The verdict

The page opens on one StatusIndicator verdict over checks it reads from existing endpoints:

A check that could not be read goes, with its reason, into a single not-measured list. It is never counted as a pass.

Actions and the runner picker

Every action on the page is one the console already has: Resolve’s Test, Explore, Access patterns and Map estate, and CapEx’s pull. Each is admin-only, audited server-side, and behind the same confirmation Resolve uses.

Data audit is where an audit’s runner is picked (features/audits/audit_runner_picker.dart):

• Runink managed, sent as console. This is the default.
• A registered self-hosted runner from /api/runners, sent by name.

A connection’s default binding is set in the DataEx › Connections wizard, with the same picker.

The choice goes on the request’s optional runner field through features/intelligence/atlas/atlas_runner_field.dart. That file sets the field by name, so it starts sending the choice as soon as the regenerated Dart has the field. If a build cannot carry the field, a self-hosted choice is refused client-side with RunnerChoiceNotSent before any call is made. It is never silently dropped. The field has three states:

That is why managed is always sent explicitly as console.

A self-hosted runner that cannot take the audit answers with a runner outcome (runner_unknown, runner_revoked or runner_offline), and nothing runs anywhere. Until dispatch to runners exists (rollout step 6), every self-hosted choice ends this way. See Resolve.

Agent reviews

An audit page starts an agent in exactly one way: by running an existing playbook with PlaybookService.RunNow (features/audits/audit_playbooks.dart). It uses the same call, the same confirmation text and the same refusal handling as the Orchestrator. There is no new dispatch path. Creating or editing a playbook stays on the Orchestrator. See Playbooks.

Lineage (?tab=lineage)

The Lineage page reads GET /api/data-lineage (session-gated, GET only, internal/console/datalineage.go). It combines:

• the declared topology from the connection registry;
• the observed flows that apps have published.

Observations come from two origins, and the page never mixes them. Every edge and publisher carries origin: "app" or origin: "resolve":

• App edges carry movements and rows. The page draws graph links from app edges only.
• Resolve edges carry structure. They are listed under “Observed by CORE Resolve”. The DECLARED_VIEW_DEPENDENCY edges are drawn as Declared derivations.

Resolve makes the structural half observed. The movement half needs an app to publish. With no app publishing, the gap is named in unmeasured[] as app flow lineage. Per-node DQ and consumers are absent.

How apps publish: POST /api/data-lineage/edges

Apps push their lineage. The console never reaches into an app pod. The body is a bounded rollup of the distinct source→destination pairs the app observed, with movement and row counts, plus the publisher’s own status: whether capture is durable, and how many records it observed and dropped (datalineageingest.go).

The token is the same agent-health ingest credential the apps already hold. On the app side it is CORE_HEALTH_URL + CORE_HEALTH_TOKEN, mounted from a Secret that core-operator’s provisioner generates. Edges without a connection id are dropped, and repeated pairs are folded into one. Reports expire after 30 days.

Deploy lineage (?tab=deployLineage)

GET /api/lineage (session-gated, internal/console/lineage.go) answers “did my change ship?” from measurements. Per image, it follows six stages: merged → agents saw it → built → landed in the registry → rolled → running.

• core-cd.yml writes what it already knew into the core-console-lineage ConfigMap, and the console reads it back.
• The build leg comes from core-images-build.yml’s registry probe of /v2/<repo>/tags/list.
• Each stage is ok, warn, failed or unmeasured. Unmeasured is used freely: a build leg with no digest record, a commit older than the run store’s retention, or a workload whose pods could not be read.

Add ?sha=<commit> to ask about one commit. The answer starts from found: false and state unmeasured, and fills in only what it could measure.