Skip to content

Apache Kafka

The test-estate safety rule: stand the test estate up only beside a test instance, never a production deployment. Its systems are throwaway test services with published credentials: change-me is every test credential. They are still encrypted and still ask for a login, because FACE refuses a connection that is not (see Connector security). Use a scratch machine or a scratch cluster.

Grade A · dataset: Stream (a timestamped recording) · type kafka

What it is

An Apache Kafka cluster, with a Confluent-compatible Schema Registry and ksqlDB. In the test estate it is the event bus: order, shipment, reefer-alert and freight-invoice events, and the topics ksqlDB computes from them. The connector reference is Streaming: Kafka.

This page is the recording: a bounded set of records you load once. For a broker on your machine that keeps receiving the estate’s sensor, drone, yard and mainframe events without end, see Apache Kafka, live estate.

Grade

A. Open source. It stands up locally from the download, with the model and its rows, end to end. Kafka is the fullest quick start: the recording holds the schemas, the Schema Registry’s subjects and the records.

See the grade table for all 32 types.

Settings and credentials

From the connection catalogue. Settings are ordinary fields on the connection record. Credentials are kept in a sealed bundle and never returned.

Names
Typekafka
Other spellings that resolveredpanda, apachekafka
Settingsbootstrap_servers, topics, client_id, security_protocol, sasl_mechanism, username, schema_registry_url, schema_registry_username, insecure_skip_verify, ssl_ca_pem
Credentialspassword, schema_registry_password

The test connection

Quillmere event bus

SettingTest value
bootstrap_serverskafka.quillmere.invalid:9092
topicsquillmere.
client_idface-discovery
security_protocolsasl_ssl
sasl_mechanismscram-sha-512
usernameface-reader
schema_registry_urlhttps://schema-registry.quillmere.invalid
schema_registry_usernameface-reader

topics is a prefix filter: quillmere. matches all nine topics. The connection record also carries one extra setting, ksqldb_url = https://ksqldb.quillmere.invalid:8088, in config.extra. When FACE maps the estate, it reads ksqlDB through it, as well as the Schema Registry.

Credentials: password, schema_registry_password. In the test estate, every credential is change-me. Credentials are write-only: they go into the sealed credential bundle and are never returned.

Stand up the test system

The safety rule at the top of this page comes first: a scratch machine or a scratch cluster, never a production or platform one.

  1. Stand up a broker with a SASL_SSL listener (SCRAM logins) and a Confluent-compatible Schema Registry that answers HTTPS only, with its own logins. FACE connects to nothing less. The live estate page stands up exactly that with Docker Compose, and writes the CA certificate (secrets/ca.crt) that the client configs below point at. Use a login that may create topics and produce.

  2. Download the recording files listed under Download below. Create the nine topics, with 3 partitions each. client.properties is a Kafka client config for that listener (security.protocol=SASL_SSL, sasl.mechanism=SCRAM-SHA-512, the login and the CA):

    for f in schemas/*-value.avsc; do
      kafka-topics.sh --bootstrap-server localhost:9092 --command-config client.properties --create --if-not-exists --topic "$(basename "$f" -value.avsc)" --partitions 3 --replication-factor 1
    done
  3. Register the subjects as the registry recording has them: every version in order, then each subject’s own compatibility level. Most value schemas use TopicNameStrategy (<topic>-value). The freight invoice topic uses TopicRecordNameStrategy, and two topics also have a key subject. Set SR to your Schema Registry’s https:// URL, and SR_AUTH to a registry login as user:password:

    cd schema-registry
    for d in subjects/*/; do
      s="$(basename "$d")"
      for v in $(jq -r '.[]' "$d/versions.json"); do
        jq '{schema, schemaType} | with_entries(select(.value != null))' "$d/versions/$v.json" \
          | curl -fsS -u "$SR_AUTH" -X POST -H 'Content-Type: application/vnd.schemaregistry.v1+json' --data @- "$SR/subjects/$s/versions"
      done
    done
    for f in config/*.json; do
      jq '{compatibility: .compatibilityLevel}' "$f" \
        | curl -fsS -u "$SR_AUTH" -X PUT -H 'Content-Type: application/vnd.schemaregistry.v1+json' --data @- "$SR/config/$(basename "$f" .json)"
    done
  4. Put the records on the bus. Each line of records.jsonl is one record: its topic, key and JSON value. Write one kcat file per topic, one key<TAB>value line per record, then produce them. kcat.conf holds the same listener settings in librdkafka form (security.protocol=SASL_SSL, sasl.mechanisms=SCRAM-SHA-512, sasl.username, sasl.password, ssl.ca.location):

    mkdir -p kcat
    for t in $(jq -r .topic records.jsonl | sort -u); do
      jq -r --arg t "$t" 'select(.topic == $t) | "\(.key | if type == "string" then . else tojson end)\t\(.value | tojson)"' records.jsonl > "kcat/$t.kcat"
    done
    for f in kcat/*.kcat; do
      kcat -F kcat.conf -P -b localhost:9092 -t "$(basename "$f" .kcat)" -K $'\t' -l "$f"
    done
  5. Add the connection in Settings › Integrations › Systems. FACE saves only a secured Kafka connection: security_protocol sasl_ssl, sasl_mechanism scram-sha-512 or scram-sha-256, a username and its password, and, when a Schema Registry is named, an https:// URL with schema_registry_username and its schema_registry_password. A broker or registry without encryption and a login is refused. The live estate page stands up a broker and registry that way and lists every setting.

  6. Where the deployment runs ksqlDB, set the ksqldb_url extra setting. The recording’s ksqldb/ folder holds the test ksqlDB’s answers to LIST STREAMS EXTENDED;, LIST TABLES EXTENDED; and LIST QUERIES;, which are what FACE reads from it.

What FACE finds

FACE reads the topic list through Metadata and the value schemas through the Schema Registry. Discovery reads no record.

DatasetKindDomainPrimary keyRowsFields
quillmere.order.created.v1topicordersnone246
quillmere.order.po-duplicates.v1topicordersnone14
quillmere.shipment.status.v1topicshipmentsnone696
quillmere.shipment.latest.v1topicshipmentsnone696
quillmere.shipment.export.v0topicshipmentsnone46
quillmere.reefer.alerts.v1topiccold_chainnone47
quillmere.coldchain.excursions.v1topiccold_chainnone45
quillmere.freight.invoice.posted.v1topicfreight_auditnone249
quillmere.freight.rate-variance.v1topicfreight_auditnone17

expected.json in the download has the full model: every field with its logical type, the source’s native type and nullability.

What the Schema Registry and ksqlDB add

The registry read adds metadata, not data. It sends no Fetch and no ksqlDB SELECT or PRINT. It keeps no query text and samples no value.

  • Schema Registry: 11 subjects. There are nine value subjects and two key subjects. Some subjects have more than one version: the shipment status and reefer alert schemas each dropped a field in version 2, and the freight invoice schema added an optional one. Every subject’s compatibility level is stated. Most use the global BACKWARD, the freight invoice subject uses FULL_TRANSITIVE, and the legacy export uses NONE. Each value schema’s Avro namespace names its business domain: invalid.quillmere.orders, .shipments, .coldchain or .freight.
  • ksqlDB: 9 sources and 4 persistent queries. ORDER_PO_DUPLICATES counts order-created events per customer PO and holds only the duplicate PO. SHIPMENT_LATEST keeps each shipment’s latest status. REEFER_EXCURSIONS aggregates the reefer alerts per shipment and holds only SHP-50004. INVOICE_RATE_VARIANCE flags invoices billed more than 5% over contract and holds only INV-90003. Each query records which topic flows into which.
  • Traffic. ksqlDB states each source’s message figures. For the legacy export, LEGACY_SHIPMENT_EXPORT, it states zero messages, and no query reads or writes it.

How the registries reduce noise in domain inference

The inference block of expected.json states what FACE’s estate map infers from this connection twice: from the topic names and columns alone, and with the registry read added.

Names and columns aloneWith the Schema Registry and ksqlDB
ordersorder.created, order.po-duplicatesorder.created, order.po-duplicates
shipmentsshipment.status, shipment.latest, shipment.export.v0shipment.status, shipment.latest
cold_chainnot placedreefer.alerts, coldchain.excursions
freight_auditfreight.invoice.posted, freight.rate-variancefreight.invoice.posted, freight.rate-variance
Withheldnoneshipment.export.v0: ksqlDB states it carries no traffic

The legacy export uses the same words as the live shipment feed, so names alone place a dead topic in the shipments domain. With the registry read, ksqlDB states that the export carries no traffic, and no recorded flow ties it to a live topic, so it is withheld. The excursion table’s columns (breach_count, peak_celsius, limit_celsius) share no vocabulary with the reefer alerts they are computed from, so names alone place neither topic. Their shared Avro namespace, and field docs that name the reefer alerts, put both topics in cold_chain.

How it joins the Quillmere universe

This source carries 9 of the universe’s shared keys. The same ID means the same thing in every source: a shipment here is the same shipment in the database, the event bus, the GPS track and the claim PDF.

Universe entityFields hereAlso in
carrierquillmere.freight.invoice.posted.v1.carrier_id, quillmere.freight.rate-variance.v1.carrier_idd365, document, files, guidewire, mysql, objectstore, postgres, redshift, sharepoint, tms, yms
customerquillmere.order.created.v1.customer_id, quillmere.order.po-duplicates.v1.customer_idfiles, hubspot, oms, postgres, redshift, salesforce, sap, snowflake
customer_poquillmere.order.created.v1.customer_po, quillmere.order.po-duplicates.v1.customer_pofiles, hubspot, oms, postgres, snowflake
devicequillmere.reefer.alerts.v1.device_idbigquery, cctv, databricks, files, gps, iot, mqtt, postgres, rfid, servicenow
dockquillmere.shipment.status.v1.dock_id, quillmere.shipment.latest.v1.dock_id, quillmere.shipment.export.v0.dock_idcctv, excel, files, logisticsapi, postgres, sensorgateway, whs, yms
invoicequillmere.freight.invoice.posted.v1.invoice_id, quillmere.freight.rate-variance.v1.invoice_idd365, files, mysql, objectstore, redshift
orderquillmere.order.created.v1.order_id, quillmere.shipment.status.v1.order_id, quillmere.shipment.latest.v1.order_id, quillmere.shipment.export.v0.order_iddocument, files, hubspot, oms, postgres, sap, snowflake
shipmentquillmere.shipment.status.v1.shipment_id, quillmere.shipment.latest.v1.shipment_id, quillmere.shipment.export.v0.shipment_id, quillmere.reefer.alerts.v1.shipment_id, quillmere.coldchain.excursions.v1.shipment_id, quillmere.freight.invoice.posted.v1.shipment_id, quillmere.freight.rate-variance.v1.shipment_idbigquery, d365, databricks, document, excel, files, github, guidewire, iot, logisticsapi, mqtt, mysql, objectstore, postgres, redshift, salesforce, sap, servicenow, sharepoint, tms, whs, yms
trailerquillmere.shipment.status.v1.trailer_id, quillmere.shipment.latest.v1.trailer_id, quillmere.shipment.export.v0.trailer_id, quillmere.reefer.alerts.v1.trailer_id, quillmere.coldchain.excursions.v1.trailer_idcctv, databricks, document, excel, iot, mqtt, postgres, servicenow, tms, yms

Evidence it holds for the planted anomalies:

  • ANOM-1, An overcharged freight invoice. streams/kafka/records.jsonl, look for "rate_per_mile":3.84,"contracted_rate_per_mile":3.25,"variance_pct":18.20 (the ksqlDB stream INVOICE_RATE_VARIANCE).
  • ANOM-3, A reefer temperature excursion. streams/kafka/records.jsonl, look for "shipment_id":"SHP-50004", and for "shipment_id":"SHP-50004","trailer_id":"TRL-102","breach_count":4,"peak_celsius":11.40 (the ksqlDB table REEFER_EXCURSIONS).
  • ANOM-4, A duplicate customer PO. streams/kafka/records.jsonl, look for "customer_po":"PO-0002-4194", and for "customer_po":"PO-0002-4194","order_count":2 (the ksqlDB table ORDER_PO_DUPLICATES).

See all connectors at once for the walkthrough across every source.

Download

Streams are dialled and held open, so their datasets are replay recordings under streams/kafka/, not a zip. Download the files, and use them as described above.

Every subject’s versions are under streams/kafka/schema-registry/subjects/<subject>/, and its own compatibility level, where it has one, under schema-registry/config/. The download manifest lists each file.

The whole universe is universe.json. manifest.json lists every download with its size and SHA-256, and SHA256SUMS is the same list for sha256sum -c.