This tutorial guides you on how to list all kafka topics using kafka testclient. Let’s say you are running kafka docker container in a kubernetes pod. And you would like to check or list all kafka topics then you are at the right place.
Deploy a test client in kubernetes cluster
You can use the following deployment YAML file (kafka-testclient.yaml) to deploy a kafka testclient pod in your kubernetes cluster.
apiVersion: v1 kind: Pod metadata: name: kafka-testclient namespace: kafka spec: containers: - name: kafka image: solsson/kafka:0.11.0.0 command: - sh - -c - "exec tail -f /dev/null"
First, connect to the kubernetes cluster. In case of GKE cluster you need to run the following command.
$ gcloud container clusters get-credentials cluster1 --zone us-central1-c --project sne5g21
Then, run the following kubectl command to deploy the kafka-testclient Pod.
$ kubectl apply -f kafka-testclient
Then using this kafka-testclient, you can create test topics as shown below.
$ kubectl -n mv exec -ti kafka-testclient -- ./bin/kafka-topics.sh --zookeeper zookeeper:2181 --topic topic1 --create --partitions 1 --replication-factor 1 $ kubectl -n mv exec -ti kafka-testclient -- ./bin/kafka-topics.sh --zookeeper zookeeper:2181 --topic topic2 --create --partitions 1 --replication-factor 1 $ kubectl -n mv exec -ti kafka-testclient -- ./bin/kafka-topics.sh --zookeeper zookeeper:2181 --topic topic3 --create --partitions 1 --replication-factor 1
The above commands would have created topics topic1, topic2 and topic3. Note, here we need to use the correct hostname for zookeeper cluster and the topic configuration.
List all kafka topics using kafka testclient
Now, let’s see how to list all kafka topics using the kafka-testclient Pod.
Run the following command to list all the kafka topics created.
$ kubectl -n mv exec -ti kafka-testclient -- ./bin/kafka-topics.sh --zookeeper zookeeper.mv.svc.clusterset.local:2181 --list Defaulting container name to kafka. Use 'kubectl describe pod/testclient -n mv' to see all of the containers in this pod. topic1 topic2 topic3
Similarly, you can test kafka producer by running the following command using kafka-testclient.
$ kubectl -n mv exec -ti kafka-testclient -- ./bin/kafka-console-producer.sh --broker-list kafka:9092 --topic topic1 > Hello !! > How are you doing ? >
And you can also test kafka consumer, so that we can consume the messages that were produced by running the following command in another terminal.
$ kubectl -n mv exec -ti testclient -- ./bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic topic1 --from-beginning Hello !! How are you doing ?
That’s it you had learnt how to deploy a kafka client as pod to communicate to zookeeper and kafka services deployed in the GKE Kubernetes Cluster. And did a quick test to list all the kafka topics using the testclient.
Hope it helped 🙂
- How to Explore Docker Container’s File System ?
- Docker EXPOSE Port only to the Host on Google Cloud
- Get Docker Container’s IP Address from the Host
- Copy Files between Host and Docker Container
- Install Anthos Service Mesh (Istio) on GKE Kubernetes Cluster ?
- kubectl unable to connect to server: x509: certificate signed by unknown authority
- How to list all Containers running in Kubernetes Pod ?
- How to find which users belongs to a specific group in linux
- Google cloud shell command to get the default PROJECT_ID – GCP
- How to remove an image tag from docker hub ?
- Build a Docker Image with a Dockerfile and Cloud Build in GCP?
- How to create GCP project on Google Cloud Platform
- MariaDB – How to set max_connections permanently ?
- Create GCP project on Google Cloud Platform
- What is ImagePullBackOff status on a Kubernetes pod ?
- Create non-root SSH user account and provide access to specific folders
- Delete docker repository from the docker hub
- Unable to connect to the server: dial tcp i/o timeout
- Replication factor: 1 larger than available brokers: 0 – Create Kafka Topic
- View Kafka messages using kafka testclient