INTERNAL_KAFKA Connections

These connections map to an underlying Kafka Topic; if it does not already exist, then a new topic will be created. Transformers connected via an input of this type will be provided configuration on how to connect and authenticate with Kafka. Users are able to set topic configuration by adding a properties block to the connection JSON when submitting a pipeline:

"INPUT_TOPIC": {
  "conn_type": "INTERNAL_KAFKA",
  "ref": "some-input-topic",
  "properties": {
    "partitions": "2", // reserved key
    "min.insync.replicas": "1", // passed directly as a Kafka config
    "replicas": "1" // reserved key
  }
}

There are currently two reserved keys in the properties block:

  • partitions: If set, override the default number of partitions.

  • replicas: If set, override the default replication factor.

All other properties will be passed through directly as Kafka configuration. See Apache Kafka Configuration for a list of topic configuration options.

By default, created topics have:

  • 12 partitions

  • 3 replicas