Hello Kafka 01

In the world of modern data architecture, the ability to process vast streams of real-time data is no longer a luxury—it’s a necessity. This is where Apache Kafka, a distributed event streaming platform, shines. For Java developers, integrating with Kafka opens up a world of possibilities for building robust, scalable, and real-time applications. This five-part series will be your guide to mastering the fundamentals of using Java with Kafka, starting with the very first step: producing your first message.

What is Kafka and Why Should You Care?

Think of Kafka as a highly durable and distributed log file. Applications can write messages (events) to it, and other applications can read those messages in real-time. This decoupling of data producers and consumers is incredibly powerful for building microservices, real-time analytics, and data integration pipelines. Kafka is renowned for its high throughput, reliability, and scalability, making it a cornerstone technology for thousands of companies, including giants like Netflix, Twitter, and Uber.

Setting the Stage: Your Local Kafka Environment

Before we dive into the Java code, you’ll need a running Kafka instance. For local development, the easiest way to get started is by using Docker. You’ll need both ZooKeeper (a coordination service for Kafka) and a Kafka broker (the server that stores the data).

You can use a simple docker-compose.yml file like the one below to spin up a basic environment:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Save this as docker-compose.yml and run docker-compose up -d in your terminal. You now have a Kafka broker listening on localhost:9092.

Next, let’s create a topic to send our messages to. A topic is like a category or a channel for your messages. You can do this from the command line:

docker-compose exec kafka kafka-topics --create --topic my-first-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Your First Java Producer: “Hello, Kafka!”

Now for the fun part: writing a Java application that sends a message to our newly created topic. You’ll need to add the Kafka client library to your project. If you’re using Maven, add this to your pom.xml:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>3.6.0</version>
</dependency>

Here is a simple Java class that acts as a Kafka producer:

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class SimpleProducer {

    public static void main(String[] args) {
        // 1. Create Producer Properties
        Properties properties = new Properties();
        properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        // 2. Create the Producer
        KafkaProducer<String, String> producer = new KafkaProducer<>(properties);

        // 3. Create a producer record
        ProducerRecord<String, String> record =
                new ProducerRecord<>("my-first-topic", "hello, kafka!");

        // 4. Send data - asynchronous
        producer.send(record);

        // 5. Flush and close producer
        producer.flush();
        producer.close();
    }
}

Dissecting the Code:

1. Create Producer Properties: We start by creating a Properties object to configure our producer. The most crucial property is BOOTSTRAP_SERVERS_CONFIG, which tells the producer where to find the Kafka broker(s). We also specify serializers for the message key and value, which convert our Java strings into bytes that Kafka can understand.

2. Create the Producer: With the properties set, we instantiate a KafkaProducer. This client will manage the connection to the Kafka cluster and handle sending messages.

3. Create a Producer Record: Each message we send is encapsulated in a ProducerRecord. This object contains the topic we’re sending the message to, and optionally a key and the message value itself. In this case, our value is the simple string “hello, kafka!”.

4. Send Data: The send() method is asynchronous. It adds the record to a buffer of pending records to be sent and returns immediately. This is what makes Kafka producers so fast.

5. Flush and Close: Since send() is asynchronous, we call flush() to ensure that any buffered records are sent before the application exits. Finally, close() releases any resources the producer is using.

After running this Java code, you have successfully sent your first message to a Kafka topic!

In the next installment of this series, we’ll explore the other side of the coin: writing a Java consumer to read the “hello, kafka!” message from our topic. Stay tuned