This tutorial guides you on how to view kafka messages using kafka testclient. Let’s say you are running kafka docker container in the kubernetes cluster. And you would like to view kafka messages sent to the specific topic then you are at the right place.
Install testclient to view Kafka messages
Use the following deployment YAML file to deploy a kafka testclient pod in your kubernetes cluster.
apiVersion: v1 kind: Pod metadata: name: kafka-testclient namespace: kafka spec: containers: - name: kafka image: solsson/kafka:0.11.0.0 command: - sh - -c - "exec tail -f /dev/null"
First, you need to connect to the kubernetes cluster. In case of GKE cluster run the following command to make an entry in the kubeconfig.
$ gcloud container clusters get-credentials cluster1 --zone us-central1-c --project sne5g21
Then, run the following kubectl command to deploy the kafka-testclient Pod in the kubernetes cluster.
$ kubectl apply -f kafka-testclient
Also, you can check whether the specific topic exists or not using the following command.
$ kubectl -n mv exec -ti kafka-testclient -- ./bin/kafka-topics.sh --zookeeper zookeeper:2181 --list
The above command will list all the kafka topics.
View Kafka messages using kafka testclient
Now, let’s see how to view kafka messages in a specific topic using kafka testclient.
You can use kafka-console-consumer.sh script to view kafka messages produced on some topic as shown below.
$ kubectl -n mv exec -ti testclient -- ./bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic topic1 --from-beginning Hello !! How are you doing ?
Note, the kafka console consumer is a tool that reads data from Kafka topic and outputs it to standard output. The following are the options that you can use based on your needs. For example, if you would like kafka consumer to consume or read maximum 10 messages from a kafka topic before exiting, then you can use option –max-messages <Integer: num_messages>.
Below are some of the important list of options that you might use with kafka-console-consumer when needed.
–bootstrap-server <server> – The <server> to connect to.
–zookeeper <urls> -The connection string for the zookeeper connection in the form host:port.
–topic <topic> – The topic id to consume on.
–partition <Integer: partition> – The partition to consume from.
–from-beginning – To consume messages starting from the beginning instead of the latest message.
–max-messages <Integer: num_messages> – The maximum number of messages to consume before exiting. If not set, consumption is continual.
–offset <consume offset> – The offset id to consume from (a non-negative number).
For other options details please check kafka-console-consumer options.
That’s it you had learnt how to view kafka messages using kafka testclient utility.
Hope it helped 🙂
- How to Explore Docker Container’s File System ?
- Docker EXPOSE Port only to the Host on Google Cloud
- Get Docker Container’s IP Address from the Host
- Copy Files between Host and Docker Container
- Install Anthos Service Mesh (Istio) on GKE Kubernetes Cluster ?
- kubectl unable to connect to server: x509: certificate signed by unknown authority
- How to list all Containers running in Kubernetes Pod ?
- How to find which users belongs to a specific group in linux
- Google cloud shell command to get the default PROJECT_ID – GCP
- How to remove an image tag from docker hub ?
- Build a Docker Image with a Dockerfile and Cloud Build in GCP?
- How to create GCP project on Google Cloud Platform
- MariaDB – How to set max_connections permanently ?
- Create GCP project on Google Cloud Platform
- What is ImagePullBackOff status on a Kubernetes pod ?
- List all kafka topics using kafka testclient
- Delete docker repository from the docker hub
- Unable to connect to the server: dial tcp i/o timeout
- Replication factor: 1 larger than available brokers: 0 – Create Kafka Topic
- Connect to kafka docker from outside docker network not working ?