I currently have a topic with 1024 partitions. I know it's kind of going
past the recommended limits, but I kept it like that because I am moving a
legacy system to kafka and it has a 1024 parallel partitions. I wanted to
understand the costs of having so many partitions a little bit more though.
I understand that the brokers will be doing a lot more random disk IO with
that many partitions unless
Hi,
Please let me know how to find the total number of messages in a particular
topic.
Regards,
Bhuvana
I see a large number of advisory topics (ActiveMQ.Advisory.*) in my list of topics, such as:
ActiveMQ.Advisory.TempQueue
What are these used for? Is it a problem if these Topics have consumers and none of the messages are dequeued?
Thanks for any help.
Hi Kafka Users,
Is it possible to monitor the number of messages in and out for per topic
Their is a MBean for "AllTopicsMessagesInPerSec" but I couldn't find
anything on per topic basis.
Also what does the "Byte out rate" MBean indicates , because as per my
understanding only the consumer maintains the offset based on message it
has consumed so nothing actually comes out of broker,
Hi experts!
I am currently working on spark streaming with kafka. I have couple of
questions related to this task.
1) Is there a way to find number of partitions given a topic name?
2)Is there a way to detect whether kafka server is running or not ?
Thanks
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/getting-number-of-partition-per-topic-in-kafka-tp20952
Hello, i'm using ActiveMQ 5.4.2 on Windows 7. I'm running a Ruby client which
creates a topic named "/queue/ActiveMQFeed" using Stomp Protocol and publish
on it an indexed message every 3 seconds.
Now i want to get these messages through several clients's browsers. Now the
problem is that if i run just a single browser i get a message every 3
seconds, if i run two browsers i get a message on both
I got a Kafka topic with messages in Avro format. I would like to display
the live stream of these events in a web app. Are there any streaming
consumer clients to convert the Avro messages into some readable format, if
not any insight into how I can achieve this would be very helpful?
Thanks!
Any help will be appreciated.
Thanks,
Xiang
Hello,
I wrote a simple junit test to test a Kafka producer.
public class KafkaProducerTest {
private int brokerId = 0;
private String topic = "test";
@Test
public void producerTest() throws InterruptedException {
// setup Zookeeper
String zkConnect = TestZKUtils.zookeeperConnect();
EmbeddedZookeeper zkServer = new EmbeddedZookeeper(zkConnect
Hi
I'm hoping to use AMQ for a stock price publishing system, so high
throughput and low latency are important factors. To verify that the kind
of throughput we are after is achievable I've created a very simple
producer/consumer pair based on the sample code in
http://www.activemq.org/site/nms.html.
I've come across a potential problem where the consumer's receive rate
dramatically falls after
Hi,
I was wondering if there was some way to store messages on the ActiveMQ
server. Let me clarify. Suppose you were keeping track of a sum of numbers
and each number arrived in a message. Here is how it would work:
1. Some producer would constantly publish the numbers throughout the day
2. A number of consumers that are listening to the producer in #1 keep
summing up the numbers. If some consumer
In 0.8, is there an undocumented way to increase the number of replicas for an existing topic? We created a number of topics with the wrong number of replicas. We could just delete and recreate but delete is flaky in 0.8. Was hoping someone figured out a way to do this w/out deleting the topics.
Thanks,
Paul
Hi,
How can I programmatically get the number of items in a topic, pending for
consumption?
If no programmatic way is avail, what other method is available?
Shlomi
I am wondering where does kafka cluster keep the topic metadata (name,
partition, replication, etc)? How does a server recover the topic's
metadata and messages after restart and what data will be lost?
Thanks for anyone to answer my questions.
best,
Yuheng
I am using network of brokers to forward messages using AMQ 5.11.0
when I queue a message on server A it gets forwarded to server's B
respective queue using the configuration above. However, it does not work
for topics. When I send a message to my topic on server A it does not get
forwarded to server's B Topic. I have a consumer
Hello,
here are two scenarios that illustrate the point :
- a logging system : some applications send exceptions messages to a
topic/queue;
a logging client will listen to the topic/queue. But each time the client
picks a message, it is deleted from the queue and if the client stops and
reconnects later it won't be able to see the previous error messages.
- a synchronization system : each time a
Hi,
I have the following problem:
My Kafka consumer is consuming messages, but the processing of the message
might fail. I do not want to
retry until success, but instead want to quickly consume the next message.
However at a later time I might still want to reprocess the failed
messages.
So I though about storing a list of offsets of the messages that have
failed in the first try
for later
Hi,I'm facing the problem in subject.The system is done by a broker (in
future a cluster of), a couple of consumers and some producers.Initially
I've setup one producer and messages sent arrive to one consumer only.I've
the broker configured with two TransportConnectors:First consumer connects
to "openwire" connector (follows the code) from the company intranet, this
is the one that doesn't receive
Dear all,
I am using Storm 0.9.1-incubating, along with kafka 0.8.1.1 with
storm-kafka-0.8-plus. Initially, the library worked fine and was able to
fetch messages from a kafka topic. However, after running the topology few
times, now I am unable to fetch any message from the kafka topic. I believe
it is a problem to set the offset from the beginning of the topic. I have
tried setting
kafkaConfig
Hi I have a few topics created with 1 partition. After running the cluster
for a few days, I want to increase the partition to 10 to improve the
consumer throughput. I learnt that it is not supported in 0.8.
What is the recommendation of improving consumer throughput after a topic
is created and the data volume increased?
Regards,
Cal