Apache Kafka
change-me is every test credential. They are still encrypted and still ask for a login, because FACE refuses a connection that is not (see Connector security). Use a scratch machine or a scratch cluster.Grade A · dataset: Stream (a timestamped recording) · type kafka
What it is
An Apache Kafka cluster, with a Confluent-compatible Schema Registry and ksqlDB. In the test estate it is the event bus: order, shipment, reefer-alert and freight-invoice events, and the topics ksqlDB computes from them. The connector reference is Streaming: Kafka.
This page is the recording: a bounded set of records you load once. For a broker on your machine that keeps receiving the estate’s sensor, drone, yard and mainframe events without end, see Apache Kafka, live estate.
Grade
A. Open source. It stands up locally from the download, with the model and its rows, end to end. Kafka is the fullest quick start: the recording holds the schemas, the Schema Registry’s subjects and the records.
See the grade table for all 32 types.
Settings and credentials
From the connection catalogue. Settings are ordinary fields on the connection record. Credentials are kept in a sealed bundle and never returned.
| Names | |
|---|---|
| Type | kafka |
| Other spellings that resolve | redpanda, apachekafka |
| Settings | bootstrap_servers, topics, client_id, security_protocol, sasl_mechanism, username, schema_registry_url, schema_registry_username, insecure_skip_verify, ssl_ca_pem |
| Credentials | password, schema_registry_password |
The test connection
Quillmere event bus
| Setting | Test value |
|---|---|
bootstrap_servers | kafka.quillmere.invalid:9092 |
topics | quillmere. |
client_id | face-discovery |
security_protocol | sasl_ssl |
sasl_mechanism | scram-sha-512 |
username | face-reader |
schema_registry_url | https://schema-registry.quillmere.invalid |
schema_registry_username | face-reader |
topics is a prefix filter: quillmere. matches all nine topics. The connection record also carries one extra setting, ksqldb_url = https://ksqldb.quillmere.invalid:8088, in config.extra. When FACE maps the estate, it reads ksqlDB through it, as well as the Schema Registry.
Credentials: password, schema_registry_password. In the test estate, every credential is change-me. Credentials are write-only: they go into the sealed credential bundle and are never returned.
Stand up the test system
The safety rule at the top of this page comes first: a scratch machine or a scratch cluster, never a production or platform one.
Stand up a broker with a SASL_SSL listener (SCRAM logins) and a Confluent-compatible Schema Registry that answers HTTPS only, with its own logins. FACE connects to nothing less. The live estate page stands up exactly that with Docker Compose, and writes the CA certificate (
secrets/ca.crt) that the client configs below point at. Use a login that may create topics and produce.Download the recording files listed under Download below. Create the nine topics, with 3 partitions each.
client.propertiesis a Kafka client config for that listener (security.protocol=SASL_SSL,sasl.mechanism=SCRAM-SHA-512, the login and the CA):for f in schemas/*-value.avsc; do kafka-topics.sh --bootstrap-server localhost:9092 --command-config client.properties --create --if-not-exists --topic "$(basename "$f" -value.avsc)" --partitions 3 --replication-factor 1 doneRegister the subjects as the registry recording has them: every version in order, then each subject’s own compatibility level. Most value schemas use TopicNameStrategy (
<topic>-value). The freight invoice topic uses TopicRecordNameStrategy, and two topics also have a key subject. SetSRto your Schema Registry’shttps://URL, andSR_AUTHto a registry login asuser:password:cd schema-registry for d in subjects/*/; do s="$(basename "$d")" for v in $(jq -r '.[]' "$d/versions.json"); do jq '{schema, schemaType} | with_entries(select(.value != null))' "$d/versions/$v.json" \ | curl -fsS -u "$SR_AUTH" -X POST -H 'Content-Type: application/vnd.schemaregistry.v1+json' --data @- "$SR/subjects/$s/versions" done done for f in config/*.json; do jq '{compatibility: .compatibilityLevel}' "$f" \ | curl -fsS -u "$SR_AUTH" -X PUT -H 'Content-Type: application/vnd.schemaregistry.v1+json' --data @- "$SR/config/$(basename "$f" .json)" donePut the records on the bus. Each line of
records.jsonlis one record: itstopic,keyand JSONvalue. Write one kcat file per topic, onekey<TAB>valueline per record, then produce them.kcat.confholds the same listener settings in librdkafka form (security.protocol=SASL_SSL,sasl.mechanisms=SCRAM-SHA-512,sasl.username,sasl.password,ssl.ca.location):mkdir -p kcat for t in $(jq -r .topic records.jsonl | sort -u); do jq -r --arg t "$t" 'select(.topic == $t) | "\(.key | if type == "string" then . else tojson end)\t\(.value | tojson)"' records.jsonl > "kcat/$t.kcat" done for f in kcat/*.kcat; do kcat -F kcat.conf -P -b localhost:9092 -t "$(basename "$f" .kcat)" -K $'\t' -l "$f" doneAdd the connection in Settings › Integrations › Systems. FACE saves only a secured Kafka connection:
security_protocolsasl_ssl,sasl_mechanismscram-sha-512orscram-sha-256, ausernameand itspassword, and, when a Schema Registry is named, anhttps://URL withschema_registry_usernameand itsschema_registry_password. A broker or registry without encryption and a login is refused. The live estate page stands up a broker and registry that way and lists every setting.Where the deployment runs ksqlDB, set the
ksqldb_urlextra setting. The recording’sksqldb/folder holds the test ksqlDB’s answers toLIST STREAMS EXTENDED;,LIST TABLES EXTENDED;andLIST QUERIES;, which are what FACE reads from it.
What FACE finds
FACE reads the topic list through Metadata and the value schemas through the Schema Registry. Discovery reads no record.
| Dataset | Kind | Domain | Primary key | Rows | Fields |
|---|---|---|---|---|---|
quillmere.order.created.v1 | topic | orders | none | 24 | 6 |
quillmere.order.po-duplicates.v1 | topic | orders | none | 1 | 4 |
quillmere.shipment.status.v1 | topic | shipments | none | 69 | 6 |
quillmere.shipment.latest.v1 | topic | shipments | none | 69 | 6 |
quillmere.shipment.export.v0 | topic | shipments | none | 4 | 6 |
quillmere.reefer.alerts.v1 | topic | cold_chain | none | 4 | 7 |
quillmere.coldchain.excursions.v1 | topic | cold_chain | none | 4 | 5 |
quillmere.freight.invoice.posted.v1 | topic | freight_audit | none | 24 | 9 |
quillmere.freight.rate-variance.v1 | topic | freight_audit | none | 1 | 7 |
expected.json in the download has the full model: every field with its logical type, the source’s native type and nullability.
What the Schema Registry and ksqlDB add
The registry read adds metadata, not data. It sends no Fetch and no ksqlDB SELECT or PRINT. It keeps no query text and samples no value.
- Schema Registry: 11 subjects. There are nine value subjects and two key subjects. Some subjects have more than one version: the shipment status and reefer alert schemas each dropped a field in version 2, and the freight invoice schema added an optional one. Every subject’s compatibility level is stated. Most use the global
BACKWARD, the freight invoice subject usesFULL_TRANSITIVE, and the legacy export usesNONE. Each value schema’s Avro namespace names its business domain:invalid.quillmere.orders,.shipments,.coldchainor.freight. - ksqlDB: 9 sources and 4 persistent queries.
ORDER_PO_DUPLICATEScounts order-created events per customer PO and holds only the duplicate PO.SHIPMENT_LATESTkeeps each shipment’s latest status.REEFER_EXCURSIONSaggregates the reefer alerts per shipment and holds only SHP-50004.INVOICE_RATE_VARIANCEflags invoices billed more than 5% over contract and holds only INV-90003. Each query records which topic flows into which. - Traffic. ksqlDB states each source’s message figures. For the legacy export,
LEGACY_SHIPMENT_EXPORT, it states zero messages, and no query reads or writes it.
How the registries reduce noise in domain inference
The inference block of expected.json states what FACE’s estate map infers from this connection twice: from the topic names and columns alone, and with the registry read added.
| Names and columns alone | With the Schema Registry and ksqlDB | |
|---|---|---|
| orders | order.created, order.po-duplicates | order.created, order.po-duplicates |
| shipments | shipment.status, shipment.latest, shipment.export.v0 | shipment.status, shipment.latest |
| cold_chain | not placed | reefer.alerts, coldchain.excursions |
| freight_audit | freight.invoice.posted, freight.rate-variance | freight.invoice.posted, freight.rate-variance |
| Withheld | none | shipment.export.v0: ksqlDB states it carries no traffic |
The legacy export uses the same words as the live shipment feed, so names alone place a dead topic in the shipments domain. With the registry read, ksqlDB states that the export carries no traffic, and no recorded flow ties it to a live topic, so it is withheld. The excursion table’s columns (breach_count, peak_celsius, limit_celsius) share no vocabulary with the reefer alerts they are computed from, so names alone place neither topic. Their shared Avro namespace, and field docs that name the reefer alerts, put both topics in cold_chain.
How it joins the Quillmere universe
This source carries 9 of the universe’s shared keys. The same ID means the same thing in every source: a shipment here is the same shipment in the database, the event bus, the GPS track and the claim PDF.
| Universe entity | Fields here | Also in |
|---|---|---|
| carrier | quillmere.freight.invoice.posted.v1.carrier_id, quillmere.freight.rate-variance.v1.carrier_id | d365, document, files, guidewire, mysql, objectstore, postgres, redshift, sharepoint, tms, yms |
| customer | quillmere.order.created.v1.customer_id, quillmere.order.po-duplicates.v1.customer_id | files, hubspot, oms, postgres, redshift, salesforce, sap, snowflake |
| customer_po | quillmere.order.created.v1.customer_po, quillmere.order.po-duplicates.v1.customer_po | files, hubspot, oms, postgres, snowflake |
| device | quillmere.reefer.alerts.v1.device_id | bigquery, cctv, databricks, files, gps, iot, mqtt, postgres, rfid, servicenow |
| dock | quillmere.shipment.status.v1.dock_id, quillmere.shipment.latest.v1.dock_id, quillmere.shipment.export.v0.dock_id | cctv, excel, files, logisticsapi, postgres, sensorgateway, whs, yms |
| invoice | quillmere.freight.invoice.posted.v1.invoice_id, quillmere.freight.rate-variance.v1.invoice_id | d365, files, mysql, objectstore, redshift |
| order | quillmere.order.created.v1.order_id, quillmere.shipment.status.v1.order_id, quillmere.shipment.latest.v1.order_id, quillmere.shipment.export.v0.order_id | document, files, hubspot, oms, postgres, sap, snowflake |
| shipment | quillmere.shipment.status.v1.shipment_id, quillmere.shipment.latest.v1.shipment_id, quillmere.shipment.export.v0.shipment_id, quillmere.reefer.alerts.v1.shipment_id, quillmere.coldchain.excursions.v1.shipment_id, quillmere.freight.invoice.posted.v1.shipment_id, quillmere.freight.rate-variance.v1.shipment_id | bigquery, d365, databricks, document, excel, files, github, guidewire, iot, logisticsapi, mqtt, mysql, objectstore, postgres, redshift, salesforce, sap, servicenow, sharepoint, tms, whs, yms |
| trailer | quillmere.shipment.status.v1.trailer_id, quillmere.shipment.latest.v1.trailer_id, quillmere.shipment.export.v0.trailer_id, quillmere.reefer.alerts.v1.trailer_id, quillmere.coldchain.excursions.v1.trailer_id | cctv, databricks, document, excel, iot, mqtt, postgres, servicenow, tms, yms |
Evidence it holds for the planted anomalies:
- ANOM-1, An overcharged freight invoice.
streams/kafka/records.jsonl, look for"rate_per_mile":3.84,"contracted_rate_per_mile":3.25,"variance_pct":18.20(the ksqlDB stream INVOICE_RATE_VARIANCE). - ANOM-3, A reefer temperature excursion.
streams/kafka/records.jsonl, look for"shipment_id":"SHP-50004", and for"shipment_id":"SHP-50004","trailer_id":"TRL-102","breach_count":4,"peak_celsius":11.40(the ksqlDB table REEFER_EXCURSIONS). - ANOM-4, A duplicate customer PO.
streams/kafka/records.jsonl, look for"customer_po":"PO-0002-4194", and for"customer_po":"PO-0002-4194","order_count":2(the ksqlDB table ORDER_PO_DUPLICATES).
See all connectors at once for the walkthrough across every source.
Download
Streams are dialled and held open, so their datasets are replay recordings under streams/kafka/, not a zip. Download the files, and use them as described above.
streams/kafka/expected.jsonstreams/kafka/manifest.jsonstreams/kafka/records.jsonlstreams/kafka/ksqldb/list-queries.jsonstreams/kafka/ksqldb/list-streams-extended.jsonstreams/kafka/ksqldb/list-tables-extended.jsonstreams/kafka/schemas/quillmere.coldchain.excursions.v1-value.avscstreams/kafka/schemas/quillmere.freight.invoice.posted.v1-value.avscstreams/kafka/schemas/quillmere.freight.rate-variance.v1-value.avscstreams/kafka/schemas/quillmere.order.created.v1-value.avscstreams/kafka/schemas/quillmere.order.po-duplicates.v1-value.avscstreams/kafka/schemas/quillmere.reefer.alerts.v1-value.avscstreams/kafka/schemas/quillmere.shipment.export.v0-value.avscstreams/kafka/schemas/quillmere.shipment.latest.v1-value.avscstreams/kafka/schemas/quillmere.shipment.status.v1-value.avscstreams/kafka/schema-registry/subjects.jsonstreams/kafka/schema-registry/config.json
Every subject’s versions are under streams/kafka/schema-registry/subjects/<subject>/, and its own compatibility level, where it has one, under schema-registry/config/. The download manifest lists each file.
The whole universe is universe.json. manifest.json lists every download with its size and SHA-256, and SHA256SUMS is the same list for sha256sum -c.