You can use Confluent Hub to add your desired JARs, either by installing them at runtime or by creating a new Docker image. Note that containerized Connect via Docker will be used for many of the examples in this series.Ĭonfluent maintains its own image for Kafka Connect, cp-kafka-connect-base, which provides a basic Connect worker to which you can add your desired JAR files for sink and source connectors, single message transforms, and converters. 3.2.You can run a Kafka Connect worker directly as a JVM process on a virtual machine or bare metal, but you might prefer the convenience of running it in a container, using a technology like Kubernetes or Docker. The same logic applies for the kafka-1 and kafka-2 services, where they'll be listening on ports 2902, respectively. Although zookeeper-1 and zookeeper-2 are listening on port 2181, they're exposing it to the host via ports 2211, respectively. Moreover, each service must expose a unique port to the host machine. We must ensure that the service names and KAFKA_BROKER_ID are unique across the services. KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:9092,PLAINTEXT_HOST://localhost:39092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:9092,PLAINTEXT_HOST://localhost:29092

KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181 So, let's add configuration for one more node each for Zookeeper and Kafka services:. Connection Using Kafka ToolĪ cluster setup for Apache Kafka needs to have redundancy for both Zookeeper servers and the Kafka servers. With that, our Kafka setup is ready for use. Kafka_1 | INFO Started socket server acceptors and processors () Kafka_1 | INFO Started data-plane acceptor and processor(s) for endpoint : ListenerName(PLAINTEXT) ()

Kafka_1 | DEBUG Started partition state machine with initial state -> HashMap() () Kafka_1 | DEBUG Started replica state machine with initial state -> HashMap() () Start Kafka ServerĬonnection to localhost port 22181 succeeded!Ĭonnection to localhost port 29092 succeeded!Īdditionally, we can also check the verbose logs while the containers are starting up and verify that the Kafka server is up: $ docker-compose logs kafka | grep -i started Similarly, the kafka service is exposed to the host applications through port 29092, but it is actually advertised on port 9092 within the container environment configured by the KAFKA_ADVERTISED_LISTENERS property. However, for any client running on the host, it'll be exposed on port 22181. In this setup, our Zookeeper server is listening on port= 2181 for the kafka service, which is defined within the same container setup. KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 Let's create a simple docker-compose.yml file with two services, namely zookeeper and kafka: version: '2'

Docker for mac kafka