How to List All Kafka Topics in Java

Learn how to use the Kafka AdminClient API to list all the topics in a Kafka cluster using Java. This article also explains how to filter the topics by name or pattern.


Before we dive into the process of listing all Kafka topics, let's make sure you have everything set up and ready to go. Here's what you need to get started:

A Running Kafka Instance

Obviously you'll need a running instance of Apache Kafka to execute the commands and code examples in this article. If you don't already have Kafka installed and running on your system, don't worry, we've got you covered. You can download the latest version of Kafka from the official Apache Kafka website. After downloading, extract the archive and follow the quick start guide to get your Kafka instance up and running.

Once you've set up Kafka, start the Zookeeper and Kafka server with the following commands:

# Start Zookeeper
$ ./bin/ ./config/

# Start Kafka server
$ ./bin/ ./config/

Make sure your Zookeeper and Kafka server are running without any errors.

Note: It's important to note that this setup is suitable for local development and testing purposes. For a production environment, you should consider a more robust and secure configuration.

Access to Kafka's Command-Line Tools

Kafka comes with a set of handy command-line tools that make it easy to interact with the system. These tools are bundled with the Kafka installation and are typically located in the bin directory. Make sure that you have access to these tools and that they are in your system's PATH.

Now that you have a running Kafka instance and access to the command-line tools, you're all set to start listing Kafka topics.

Using Command-Line Tools to List Topics

To list all Kafka topics, open your terminal and navigate to the bin directory of your Kafka installation. Execute the following command, making sure to replace <broker-address> with the address of one of your Kafka brokers:

$ ./ --list --bootstrap-server <broker-address>

For example, if your Kafka broker is running on localhost:9092, the command would be:

$ ./ --list --bootstrap-server localhost:9092

This command will display a list of all the topics in your Kafka instance. If you don't see any topics, it's likely that you haven't created any yet.

Note: Kafka brokers are responsible for managing and storing messages, and you need to provide the address of at least one broker to execute most Kafka commands. The --bootstrap-server parameter specifies the address of a broker that the command should connect to.

Troubleshooting Common Issues

If you encounter any issues while listing Kafka topics, here are a few common problems and their solutions:

Error connecting to the broker: Make sure your Kafka broker is running and that you've entered the correct broker address in the --bootstrap-server parameter.

No topics listed: If no topics are displayed, it's possible that there are no topics created in your Kafka instance. To create a topic, you can use the script with the --create flag.

Permission issues: If you get a permission error while running the script, make sure the script has the correct permissions. You may need to use chmod +x to make it executable.

That's it! Now you can list all Kafka topics using the command-line tools. In the next section, we'll explore how to do this programmatically with Java and Python libraries.

Listing Topics Programmatically

In this section, we'll go beyond the command-line tools and explore how to list Kafka topics programmatically using the Kafka AdminClient API and some popular Python libraries.

Using Kafka AdminClient API

The Kafka AdminClient API allows you to interact with Kafka programmatically using Java. To list all topics, you can create an instance of the AdminClient class and call the listTopics method. Here's a code example to demonstrate this:


import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.ListTopicsResult;

import java.util.Properties;
import java.util.Set;
import java.util.concurrent.ExecutionException;

public class ListKafkaTopics {
    public static void main(String[] args) throws ExecutionException, InterruptedException {
        Properties properties = new Properties();
        properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

        try (AdminClient adminClient = AdminClient.create(properties)) {
            ListTopicsResult topics = adminClient.listTopics();
            Set<String> topicNames = topics.names().get();

            System.out.println("Topics in the Kafka cluster:");

Replace "localhost:9092" with your Kafka broker's address, and run the code. You'll see the list of topics printed in the console.

Utilizing Kafka Python Libraries

There are a couple of popular Python libraries you can use to interact with Kafka: confluent-kafka-python and kafka-python. Let's see how to list topics using both of these libraries.

  1. Using confluent-kafka-python:

First, install the confluent-kafka-python library if you haven't already:

$ pip install confluent-kafka

Now, you can use the following code snippet to list all Kafka topics:

from confluent_kafka.admin import AdminClient, NewTopic

conf = {'bootstrap.servers': 'localhost:9092'}
admin_client = AdminClient(conf)

topic_list = admin_client.list_topics().topics
print("Topics in the Kafka cluster:")
for topic in topic_list:
  1. Using kafka-python:

Install the kafka-python library with the following command:

$ pip install kafka-python

And here's a code snippet that lists all Kafka topics using the kafka-python library:

from kafka import KafkaAdminClient

admin_client = KafkaAdminClient(bootstrap_servers='localhost:9092')
topic_list = admin_client.list_topics()

print("Topics in the Kafka cluster:")
for topic in topic_list:

And that's it! Now you can not only list the topics from the command line, but programmatically as well.

#java #kafka 

How to List All Kafka Topics in Java
1.30 GEEK