Skip to content

Estate

DataEx › Estate (?tab=resolve) shows what the tenant’s registered sources actually contain:

  • how their datasets group into data domains;
  • how the sources are used;
  • whether what a source declares about itself agrees with what its data shows.

It is ported from FACE’s Reconcile and Data Connectors pages, not from Atlas. It reuses FACE’s estate engine (connect, graph, access).

Estate (the Resolve service) is the one place Runink TIDE dials a tenant source. The only other dialer is the admin-only connectorsync probe, POST /api/connectors/status/refresh, which covers the connectors TIDE publishes and never touches registry records. The agents (datagov included), tide connection register and granting a source to the workspace all dial nothing.

Two services, one set of documents

The page is the shared estate component (runink.ui.estate.v1.EstateService).

The older runink.core.atlas.v1.ResolveService stays registered as a deprecated alias for one release. The CapEx pull dialog, the connection wizard’s “Test after saving” and the audit pages still call it. Both services read and write the same three stored documents, so each sees the other’s writes.

EstateServiceResolveService (deprecated)Class
ListSources, GetSourceDescription, GetAccessPatterns, GetEstateMap, GetReconciliationListSources, GetSourceDescription, GetAccessPatterns, GetEstate, GetReconciliationread
TestSource, ExploreSource, ReadAccessPatterns, MapEstatethe same fouradmin-write

Reads never dial

The read RPCs answer from what the last dialing action committed. Nothing dials on a read, a timer or a page load. A test fails the engine if a read calls it.

The page re-reads GetEstate every 5 seconds only while a map run is RUNNING and the tab is on screen.

The four dialing actions

Each is admin-write (TIDE_ATLAS_ADMINS). Each is audited under resource atlas-estate with the outcome, including refusals, and each sits behind a confirm dialog on the page that states what it dials and what it keeps.

ActionAudit actionBudgetWhat it does
TestSourceatlas.resolve.test20 sChecks that the source answers with its credentials
ExploreSourceatlas.resolve.explore90 sReads the catalogue: datasets, columns, types, declared keys, view dependencies, and profile statistics where the engine has them
ReadAccessPatternsatlas.resolve.access90 sReads the source’s own access views over a window of window_days 1..90
MapEstateatlas.resolve.map15 min per run, 30 s per sourceBuilds the estate domain map across sources. It returns at once and runs in the background, one run in flight per console

A second map while one is running is refused: map run … started by … at … is still in flight; one run at a time.

Credentials are opened with the registry’s audited Credentials read for one engine call and then dropped. They are never stored, returned on the wire, logged or handed to an agent. Engine errors are scrubbed (resScrub) before they are recorded.

Choosing the runner

Every action, and CapexService.PullFeed, first decides the effective runner (resEffectiveRunner) from the request’s optional runner field:

runner on the requestEffective runner
absent (empty)The connection’s binding. An empty binding means console
consoleRunink managed, explicitly, even for a connection bound to a self-hosted runner
a runner nameThat registered runner

A name must be a DNS label or console. Anything else gets runner … is not a runner name (a DNS label, or "console" for the Runink managed runner).

console dials in-process. The runner registry is not even read. Actions run on console; a choice of any other runner is refused. No bundle is opened, the engine is not called, and there is no fallback to another runner. The refusal names the state it found:

State foundOutcomeWhat the detail says
Not registeredrunner_unknownEnrol that runner, or choose console
Revokedrunner_revokedWho revoked it and when
Pending (not enrolled yet)runner_offlineEnrolment is not complete
Active, not connectedrunner_offlineTIDE has no live connection to its address
Active and connectedrunner_offline“connected; …”. This never claims the runner is down
Registry unreadablefailed“runner registry unavailable”. Never runner_unknown

In a map run, the runner routes every source, and a refused source becomes a gap in the map. The effective runner, and whether it came from the action or the binding, is recorded in the audit detail and in ActionRecord.runner. Runners themselves are managed on DataEx › Runners.

The values boundary

The estate adapter is the boundary where values stop. It copies structure and counts field by field into record types that have no field for a cell, MIN/MAX, a sample row, a principal name or a credential. Tests guard this boundary.

KeptNever kept
Datasets, columns, types, nullability, declared keysSampled values, sample rows
Row counts, NULL counts, DISTINCT counts (with their basis)MIN / MAX / mean (a customer cell)
Access counts: reads, writes, distinct principalsPrincipal names
Outcome, time and actor of each actionCredentials

Explore profiling limits. Where the engine supports statistics, Explore computes per-column aggregates over at most 25 datasets (resMaxStatDatasets). Only DISTINCT and NULL counts cross the boundary. Explore also keeps the statistics pass’s counted row count.

No confidence number. Domains carry an evidence class (declared, measured or weak) and every edge carries an evidence kind. A GUESS edge is drawn dotted in the warning tone and is never styled like a declared key.

Storage

The documents are metadoc kinds in the console’s appfs root:

DocumentContentsBudget
var/lib/atlas-estatePer-source last action, access counts, map runs512 KiB
var/lib/atlas-estate-mapThe last committed map768 KiB
var/lib/atlas-estate-desc-<h>One source’s last Explore, with h = sha256(id)[:16]768 KiB

What Resolve records as data lineage

When an Explore, ReadAccessPatterns or MapEstate commits, the console records what it observed as data lineage. This happens in process: there is no HTTP hop and no ingest token. The edges go into the same observed-lineage store the apps publish into, under the publisher tide-resolve. The app ingest refuses that publisher name from anyone else with a 403.

Edge kindBetweenFrom
DECLARED_FOREIGN_KEYdataset → datasetExplore’s declared keys, and the map’s declared keys
INFERRED_* placementdataset → domainThe map, only for domains whose members come from two or more sources
OBSERVED_ACCESSdataset → principalsAccess counts, only where a stated read or write count is above zero
DECLARED_VIEW_DEPENDENCYbase → viewThe source’s own catalogue: Postgres pg_depend, MySQL VIEW_TABLE_USAGE. The Lineage page draws these as “Declared derivations”

Not recorded: GUESSED_COLUMN_OVERLAP (a guess is not an observation) and catalogue containment.

Each edge carries only qualified names, the kind, counts and provenance. It carries no constraint or column name, no detail text and no admin identity.

Edges are deduplicated source→destination. Each commit replaces its own scope’s claims (explore:<id>, access:<id>, map), and claims expire after 30 days. The Resolve lineage key is capped at 256 edges / 512 KiB, and the weakest edges are dropped first and counted.

A committed map also raises the playbook harness event estate / estate.mapped, with attributes domains, unplaced, gaps, drift, shadow and missing. See Playbooks.

Refusals you will see

  • On a console with no sign-in: this console has no sign-in configured, so dialing a tenant source would be unattributable — set GOOGLE_CLIENT_ID or CONSOLE_ADMIN_PASSWORD before testing, exploring or mapping sources.
  • On the page, the first PermissionDenied disables every dial and shows the console’s reason. A FailedPrecondition or InvalidArgument is drawn verbatim and gates nothing else.

More in Troubleshooting.