Kafka Connector

Overview

The Kafka connector reads from and writes to Apache Kafka topics, with optional SASL/PLAIN authentication.

Connector Types:

• KafkaReader - Consume messages from a topic as part of a consumer group
• KafkaWriter - Produce messages to a topic

Kafka Reader

Consume from a topic using a consumer group:

{
  "type": "KafkaReader",
  "config": {
    "brokers": ["localhost:9092"],
    "topic": "sensor-events",
    "groupId": "meddle-consumers"
  }
}

With SASL/PLAIN Authentication

{
  "type": "KafkaReader",
  "config": {
    "brokers": [
      "kafka-1.example.com:9092",
      "kafka-2.example.com:9092",
      "kafka-3.example.com:9092"
    ],
    "topic": "factory.events",
    "groupId": "meddle-factory-consumers",
    "username": "kafka_user",
    "password": "kafka_password"
  }
}

Kafka Writer

Produce messages to a topic:

{
  "type": "KafkaWriter",
  "config": {
    "brokers": ["localhost:9092"],
    "topic": "processed-events"
  }
}

With SASL/PLAIN Authentication

{
  "type": "KafkaWriter",
  "config": {
    "brokers": [
      "kafka-1.example.com:9092",
      "kafka-2.example.com:9092"
    ],
    "topic": "factory.processed",
    "username": "kafka_user",
    "password": "kafka_password"
  }
}

Configuration Parameters

• brokers: List of bootstrap broker addresses (host:port)
• topic: Kafka topic name
• groupId: (Reader) Consumer group identifier
• username: (Optional) SASL/PLAIN username
• password: (Optional) SASL/PLAIN password

Consumer Groups

The groupId controls how Kafka partitions are distributed across readers:

• Multiple readers with the same groupId share partitions for horizontal scaling.
• Multiple readers with different groupId values each receive a full copy of the topic stream.
• Offsets are committed per consumer group; restarts resume from the last committed offset.

Partitioning

The writer uses the LeastBytes balancer, which spreads messages across partitions to minimise per-broker load. Messages within a single Meddle pipeline are not guaranteed to land on the same partition.

Authentication

• Authentication is enabled only when both username and password are supplied.
• The connector uses SASL/PLAIN. Pair with TLS at the network layer (broker config) when running in production.

Related Connectors

• MQTT - Lightweight pub/sub for devices
• AMQP - RabbitMQ alternative
• NATS - Cloud-native pub/sub alternative