Kafka commonly used commands

1 minute read

Kafka CLI

To list topics:

  • kafka-topics --list --zookeeper localhost:2181

To consume data from a topic:

  • kafka-console-consumer --bootstrap-server kafka0:9092 --topic sk_rest_proxy --from-beginning

To create a new topic:

  • kafka-topics --bootstrap-server kafka:9092 --create --topic my-topic --partitions 1 --replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1

To check configuration of topic:

  • kafka-configs --zookeeper zookeeper:2181 --entity-type topics --entity-name my-topic --describe

To modify configuration of a topic:

  • kafka-configs --zookeeper zookeeper:2181 --entity-type topics --entity-name my-topic --alter --add-config max.message.bytes=128000

To delete all messages from a topic: One way to delete all messages inside a Kafka topic is by setting the retention.ms to 1 second (1000ms).

kafka-configs --zookeeper zookeeper:2181 --entity-type topics --entity-name com.kafka.producers.purchases --alter --add-config rentention.ms=1000
kafka-configs --zookeeper zookeeper:2181 --entity-type topics  --entity-name com.kafka.producers.purchases --describe
kafka-configs --zookeeper zookeeper:2181 --entity-type topics  --entity-name com.kafka.producers.purchases --alter --delete-config retention.ms
kafka-configs --zookeeper zookeeper:2181 --entity-type topics  --entity-name com.kafka.producers.purchases --describe

To delete a topic:

  • kafka-topics --zookeeper localhost:2181 --delete --topic <topic-name>
root@06d9f93d9c75:/home/workspace/producers# kafka-topics --zookeeper localhost:2181 --delete --topic TURNSTILE_SUMMARY
Topic TURNSTILE_SUMMARY is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

Kafka Connect

Kafka REST Proxy

Faust

KSQL

Schema registry

  • curl -X GET http://localhost:8081/subjects/ | python -m json.tool : This command is used to check what is currently registered in Schema Registry.
# curl -X GET http://localhost:8081/subjects/ | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   521  100   521    0     0  32562      0 --:--:-- --:--:-- --:--:-- 32562
[
    "org.chicago.cta.station.arrivals-key",
    "org.chicago.cta.station.arrivals-value",
    "weather-value",
    "weather-key",
    "org.chicago.cta.turnstiles-key",
    "org.chicago.cta.turnstiles-value"
]