kafkacat

kafkacat is a command-line tool for interacting with Apache Kafka. It provides a simple, fast, and easy-to-use way to consume and produce messages.

Features

• Flexible configuration: kafkacat can be easily configured via command-line options or environment variables to fit different use cases.
• Versatile message format support: kafkacat can consume and produce messages in a wide range of formats, such as JSON, Avro, and Protobuf.
• Efficient message processing: kafkacat can consume and produce messages at high speed thanks to its efficient I/O and batching mechanisms.
• Convenient Kafka administration: kafkacat provides various subcommands for inspecting, altering, and managing Kafka topics and partitions.

Installation

kafkacat can be installed on most Unix-like systems (including macOS and Linux) via package managers such as Homebrew and APT, or manually from source code.

Homebrew (macOS)

To install via Homebrew, run:

brew install kafkacat

APT (Debian, Ubuntu)

To install via APT, run:

sudo apt-get update && sudo apt-get install kafkacat

Manual install from source code

To install manually from source code, follow the instructions in the official repository.

Usage

kafkacat has a wide range of usage scenarios, from basic message consumption and production to advanced Kafka administration. Here are some examples:

Consuming messages from a topic

To consume messages from a topic, run:

kafkacat -b kafka.example.com -t mytopic

This will consume messages from the mytopic topic on the kafka.example.com Kafka broker.

Producing messages to a topic

To produce messages to a topic, run:

echo "hello, world" | kafkacat -b kafka.example.com -t mytopic

This will produce a message with the text "hello, world" to the mytopic topic on the kafka.example.com Kafka broker.

Inspecting a topic

To inspect the metadata and configuration of a topic, run:

kafkacat -b kafka.example.com -L -t mytopic

This will print information about the mytopic topic, such as its number of partitions, replication factor, and retention policy.

Altering a topic

To alter the configuration of a topic, run:

kafkacat -b kafka.example.com -t mytopic -X cleanup.policy=compact

This will set the cleanup.policy configuration of the mytopic topic to compact, which changes how older messages are deleted to optimize disk space usage.

Conclusion

kafkacat is a powerful and versatile tool that can greatly simplify working with Apache Kafka. Its flexible configuration, versatile message format support, efficient message processing, and convenient Kafka administration make it a valuable addition to any Kafka developer's toolkit.