Skip to content

Streaming: Kafka

FACE connects to Kafka (and to Redpanda, which speaks the same protocol) to enumerate its topic catalogue and read declared structure from a Confluent-compatible Schema Registry. FACE implements the handshake, ApiVersions, Metadata and the SASL exchange natively. No Kafka client library is used.

Kafka connections serve the topic catalogue and Schema Registry structure through Explore.

Type spellings

kafka, redpanda, apachekafka (the “Apache Kafka” card).

Azure Event Hubs and Amazon MSK are not aliased, because they require SASL mechanisms this connector does not speak.

Settings (KafkaConnectionConfig)

SettingMeaning
bootstrap_serversOne or more host:port values (default port 9092). Required, and there is no default broker. List several: Test succeeds as soon as any one of them answers.
topicsOptional prefix filters over the catalogue. An empty list is normal and enumerates every topic the principal may see. Reading the catalogue moves no data.
client_idSent in every request header. The default is runink-face-<connection id>.
security_protocolsasl_ssl. Required. Unset (which means plaintext), plaintext, sasl_plaintext and ssl are refused: ssl encrypts but carries no login.
sasl_mechanismscram-sha-512 or scram-sha-256. Required. SCRAM verifies the broker’s server signature and refuses the connection if it does not match. Any other mechanism is refused.
usernameThe SASL username (a setting). Required.
schema_registry_urlAn https:// base URL. Optional: without it, topics are listed with no columns. The URL must not contain credentials. http:// is refused.
schema_registry_usernameThe registry’s HTTP-basic user. Required when a registry URL is set.
ssl_ca_pemThe PEM CA certificate the broker’s TLS certificate must chain to, for a broker with a private CA. Empty means the public roots. Only these roots are trusted for the broker, its host name is still checked, and a value holding a private key is refused. It is a public certificate, not a credential. The Schema Registry keeps the system roots.

Credentials: password for SASL (required), and schema_registry_password for the registry (a separate credential, because the registry authenticates separately; required when a registry URL is set).

insecure_skip_verify is refused: the broker’s and the registry’s certificates are always checked. A connection that does not meet these rules cannot be saved, and a stored one is refused on every read until it is upgraded. See Security requirements.

Set allow_internal_network to control whether the connection may reach internal addresses.

Test

Test performs a real handshake against the bootstrap servers in order: dial, TLS, ApiVersions, and the SASL exchange. It returns success on the first broker that completes it. If none does, it returns no bootstrap server answered: … with each broker’s error.

A successful Test does not establish:

  • that the principal may list topics, because topic authorisation is checked per topic
  • that the Schema Registry exists or answers, because Test does not contact it
  • that any topic holds a record

Explore

Explore reads the topic catalogue (up to 5,000 topics) and, where a registry is configured, each topic’s value subject (<topic>-value). No record body is read.

TopicColumns
With a registered value schemaThe schema’s fields, with the Avro type verbatim, nullability from the union, enum symbols as allowed values, and each doc as a description. This is declared structure.
Without oneNone. FACE never infers fields from sampled messages.
Registry not asked, or unreadableNone. The schemas aspect shows asked=false or readable=false, so you can tell this case apart from “no schema”.

Topic names ending in .by-<key>.v<n> (for example acme.orders.events.by-order_id.v1) yield a partition key, marked with a name-match basis. It is never reported as declared. Key subjects and offsets are not requested, and the aspects say so.

For Kafka, use Explore. Sample and Profile apply to the SQL warehouses.