Programmatically Get Number Of Items In Topic/partition

Hi,
How can I programmatically get the number of items in a topic, pending for
consumption?
If no programmatic way is avail, what other method is available?
Shlomi

Take a look at ConsumerOffsetChecker. It does just that: print the
offset and lag for each consumer and partition.
You can either use that class directly, or use it as a guideline for
your implementation

The other method is via command line
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group *groupName*
Refer :
https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-ConsumerOffsetChecker
https://apache.googlesource.com/kafka/+/0.8.0-beta1-candidate1/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala

Bingo. 10x!!

actually this tool is not a 100% match to what I need, since it can only
provide information on topics that have comsumers: Is there also another
equivalent tool/method of querying topics that have no consumers ? in this
case this tool will not help as it requires a group id as a mandatory
parameter...

Take a look into how Offset Requests are made and handled.  You could do
that to get the low and high watermarks for the stream.  With the high and
low watermark just subtract them to get total. This will be in the consumer
api part of whatever language client your using. The specs for this are
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
********************************************/
actually this tool is not a 100% match to what I need, since it can only
provide information on topics that have comsumers: Is there also another
equivalent tool/method of querying topics that have no consumers ? in this
case this tool will not help as it requires a group id as a mandatory
parameter...
https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-ConsumerOffsetChecker
https://apache.googlesource.com/kafka/+/0.8.0-beta1-candidate1/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
pending

Hi experts! I am currently working on spark streaming with kafka. I have couple of questions related to this task. 1) Is there a way to find number of partitions given a topic name? 2)Is there a way to detect whether kafka server is running or not ? Thanks View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/getting-number-of-partition-per-topic-in-kafka-tp20952
Is there any way to change the number of partitions topic wise on the fly without restarting kafka server?
hi.. I am new to Voldemort and I am not able to understand the concept of data partitioning and how to choose number of partitions can anyone explain or point to any resource. I tried reading design document but its not clear.
Hi All, I want to get the replication and partition count of a topic. I tried the following piece of code: java.util.Set topics = new HashSet(); topics.add("topicName"); Set topicMetadatas = AdminUtils.fetchTopicMetadataFromZk(JavaConversions.asScalaSet(topics), zkClient); Iterator topicMetadataIterator = topicMetadatas.iterator(); while (topicMetadataIterator
Hi, All I've run bin/kafka-producer-perf-test.sh on our kafka-production cluster, I found the number of partitions really have huge impacts on the producer performance, see: start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec 2014-12-22 19:53:27:392, 2014-12-22 19:54:25:581, 1, 3000, 200, 2861.02, 49.1678, 1000000
I notice that when using the SimpleConsumer javaapi and trying to consume from an existing topic and partition that has never been written to before, I get a SocketTimeoutException. Is there a way around this? I'm using kafka-0.8.0 beta1. Thanks, -Robert
Hey Guys, If at the beginning, I configure the producer to not use compression and produce a number of message to a topic. Then later, I reconfigure the producer to use compression and produce another batch of messages to the same topic. Can a consumer correctly read all the messages in the topic? Will the consumer stop reading correctly when it reach the compressed part? Thanks so much
Is there a way to compute the total number of records in each RDD partition? So say I had 4 partitions.. I’d want to have partition 0: 100 records partition 1: 104 records partition 2: 90 records partition 3: 140 records Kevin Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile
Greets, All: Sorry about the off-topic post, but, I lost my partition table on my Slack 7.0 Athlon workstation last night, and, need to find a way to get 500 Mbytes of data from my home directories. I used some stupid IBM utility while trying to set up a 3rd boot system for Win98 and it toasted the partition tables. I'm working with gpart which is finding the partitions, but, I don't
Hi , I have query regarding partition offset . While running kafka cluster for some time ,I noticed that the partition offset keeps on increasing and at some point the offset decreased by some number . In what scenarios does the offset of a topic partition reduces ? The problem I am facing is that my consumers are pulling the msgs from the topic but at somepoint in time it throws exception