Number Of Messages In A Kafka Topic

Hi Kafka Users!
how to check number of messages in a kafka topic using java api?
Thanks

This could be a little bit work... You can do the following,
use simple consumer,
1. Get all the partition information of the topic by issuing a topic
metadata request.
2. Send OffsetRequest to each of the partitions to get the earliest and
latest offset of a partition so you get number of messages there.
3. Sum up the numbers you got in step 2 from each partition.
You can also use KafkaConsumer,
1. Call seekToBeginning() and seeKToEnd() for each TopicPartition to get
the message number in each partition.
2. Sum up the numbers in step 1.
I�m not sure if there is a better way to do this...
Jiangjie (Becket) Qin

thanks Jiangjie (Becket) Qin
Regards: HAFIZ MUJADID

I currently have a topic with 1024 partitions. I know it's kind of going past the recommended limits, but I kept it like that because I am moving a legacy system to kafka and it has a 1024 parallel partitions. I wanted to understand the costs of having so many partitions a little bit more though. I understand that the brokers will be doing a lot more random disk IO with that many partitions unless
Hi, Please let me know how to find the total number of messages in a particular topic. Regards, Bhuvana
I see a large number of advisory topics (ActiveMQ.Advisory.*) in my list of topics, such as: ActiveMQ.Advisory.TempQueue What are these used for? Is it a problem if these Topics have consumers and none of the messages are dequeued? Thanks for any help.
Hi Kafka Users, Is it possible to monitor the number of messages in and out for per topic Their is a MBean for "AllTopicsMessagesInPerSec" but I couldn't find anything on per topic basis. Also what does the "Byte out rate" MBean indicates , because as per my understanding only the consumer maintains the offset based on message it has consumed so nothing actually comes out of broker,
Hi experts! I am currently working on spark streaming with kafka. I have couple of questions related to this task. 1) Is there a way to find number of partitions given a topic name? 2)Is there a way to detect whether kafka server is running or not ? Thanks View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/getting-number-of-partition-per-topic-in-kafka-tp20952
Hello, i'm using ActiveMQ 5.4.2 on Windows 7. I'm running a Ruby client which creates a topic named "/queue/ActiveMQFeed" using Stomp Protocol and publish on it an indexed message every 3 seconds. Now i want to get these messages through several clients's browsers. Now the problem is that if i run just a single browser i get a message every 3 seconds, if i run two browsers i get a message on both
I got a Kafka topic with messages in Avro format. I would like to display the live stream of these events in a web app. Are there any streaming consumer clients to convert the Avro messages into some readable format, if not any insight into how I can achieve this would be very helpful? Thanks!
Any help will be appreciated. Thanks, Xiang
Hello, I wrote a simple junit test to test a Kafka producer. public class KafkaProducerTest { private int brokerId = 0; private String topic = "test"; @Test public void producerTest() throws InterruptedException { // setup Zookeeper String zkConnect = TestZKUtils.zookeeperConnect(); EmbeddedZookeeper zkServer = new EmbeddedZookeeper(zkConnect
Hi I'm hoping to use AMQ for a stock price publishing system, so high throughput and low latency are important factors. To verify that the kind of throughput we are after is achievable I've created a very simple producer/consumer pair based on the sample code in http://www.activemq.org/site/nms.html. I've come across a potential problem where the consumer's receive rate dramatically falls after
Hi, I was wondering if there was some way to store messages on the ActiveMQ server. Let me clarify. Suppose you were keeping track of a sum of numbers and each number arrived in a message. Here is how it would work: 1. Some producer would constantly publish the numbers throughout the day 2. A number of consumers that are listening to the producer in #1 keep summing up the numbers. If some consumer
In 0.8, is there an undocumented way to increase the number of replicas for an existing topic? We created a number of topics with the wrong number of replicas. We could just delete and recreate but delete is flaky in 0.8. Was hoping someone figured out a way to do this w/out deleting the topics. Thanks, Paul
Hi, How can I programmatically get the number of items in a topic, pending for consumption? If no programmatic way is avail, what other method is available? Shlomi
I am wondering where does kafka cluster keep the topic metadata (name, partition, replication, etc)? How does a server recover the topic's metadata and messages after restart and what data will be lost? Thanks for anyone to answer my questions. best, Yuheng
I am using network of brokers to forward messages using AMQ 5.11.0 when I queue a message on server A it gets forwarded to server's B respective queue using the configuration above. However, it does not work for topics. When I send a message to my topic on server A it does not get forwarded to server's B Topic. I have a consumer
Hello, here are two scenarios that illustrate the point : - a logging system : some applications send exceptions messages to a topic/queue; a logging client will listen to the topic/queue. But each time the client picks a message, it is deleted from the queue and if the client stops and reconnects later it won't be able to see the previous error messages. - a synchronization system : each time a
Hi, I have the following problem: My Kafka consumer is consuming messages, but the processing of the message might fail. I do not want to retry until success, but instead want to quickly consume the next message. However at a later time I might still want to reprocess the failed messages. So I though about storing a list of offsets of the messages that have failed in the first try for later
Hi,I'm facing the problem in subject.The system is done by a broker (in future a cluster of), a couple of consumers and some producers.Initially I've setup one producer and messages sent arrive to one consumer only.I've the broker configured with two TransportConnectors:First consumer connects to "openwire" connector (follows the code) from the company intranet, this is the one that doesn't receive
Dear all, I am using Storm 0.9.1-incubating, along with kafka 0.8.1.1 with storm-kafka-0.8-plus. Initially, the library worked fine and was able to fetch messages from a kafka topic. However, after running the topology few times, now I am unable to fetch any message from the kafka topic. I believe it is a problem to set the offset from the beginning of the topic. I have tried setting kafkaConfig
Hi I have a few topics created with 1 partition. After running the cluster for a few days, I want to increase the partition to 10 to improve the consumer throughput. I learnt that it is not supported in 0.8. What is the recommendation of improving consumer throughput after a topic is created and the data volume increased? Regards, Cal