Is it possible that using the same group name for two topics could cause a
conflict?
I have a situation where Im seeing vast numbers of records (more than 2x)
get duplicated in a topic. I was looking at consumer lag using
'kafka-consumer-groups ... --new-consumer' and noticed that I had another
app using the same group name for a different (much smaller) topic.
Would increasing partitions have an effect like this?
Added 33% more partitions to a topic yesterday and all sorts of havoc
occurred.
using 0.10.1.0
Jon Yeargers 's gravatar image asked Feb 17 2017 at 08:31 in Incubator-Kafka-Users by Jon Yeargers

0 Answers

Related Discussions

  • Kafka Java Consumer Processes Duplicate Messages in Incubator-kafka-users

  • Hi, I am kind of new to Kafka. I have set up a 3 node kafka (1 broker per machine) cluster with 3 node zookeer cluster. I am using Kafka 0.9.0.0 version. The set up works fine wherein from my single producer I am pushing a JSON string to Kafka to a topic with 3 partitions and replication factor of 2. At consumer end I have application with 3 consumer threads (I suppose each consumer thread...

  • Duplicate Records In Kafka 0.7 in Incubator-kafka-users

  • Hi, I would like to check to see if other people are seeing duplicate records with Kafka 0.7. I read the Jira's and I believe that duplicates are still possible when using message compression on Kafka 0.7. I'm seeing duplicate records from the range of 6-13%. Is this normal? If you're using Kafka 0.7 with message compression enabled, can you please let me know any duplicate records and if so...

  • Processing Older Records Kafka Consumer in Incubator-kafka-users

  • Hi, Is there any way to re-consume the older records from Kafka broker with kafka consumer? I am using kafka 0.9.0.0 In one of the scenario, I saw records for 2 days from today were not consumed as consumer was stuck. When the consumer restarted, it started processing records from today but older records for last 2 days are not processed. Is there any way to achieve the same? Any help...

  • New Consumer & Committed Offsets in Incubator-kafka-users

  • Hi, I've been trying out the new consumer and have noticed that i get duplicate messages when i stop the consumer and then restart (different processes, same consumer group). I consume all of the messages on the topic and commit the offsets for each partition and stop the consumer. On the next run i expect to get 0 messages, however i get a batch of records from each partition - in this ...

  • Kafka Consumer Poll in Incubator-kafka-users

  • Hi, I am new to Kafka. We are planning to use Kafka messaging for our application. I was playing with Kafka 0.9.0.1 version and i have following queries. Sorry for asking basic questions. 1) I have instantiated Kafka Consumer and invoked consumer.poll(Long.MAX_VALUE). Although i have specified timeout as Long.MAX_VALUE, i observe my consumer to fetch records whenever the publisher publishes...

  • Consumer Not Associating? Perpetual Rebalance in Incubator-kafka-users

  • Single app with single consumer. Pulling ~30 records / min. When I enter 'kafka-topics ... --new-consumer --group If I enter "kafka-consumer-offset-checker ...--topic --group "it responds with appropriate consumer position(s) but tells me "owner" is "none". I know my app is consuming records and if I stop/start it it picks up where it left off. Why is it marked as 'rebalancing'? ...

  • Kafka Streams Change Log Behavior in Incubator-kafka-users

  • Hi All, I'm new to Kafka Streams and have a question on change log. If a StreamTask fails and is restarted, will the change log be written to the old change log's topic partition ? Is it possible for some change log topic partition to have duplicate records so that log compaction is required ? Thanks, Manu Zhang...

  • New Consumer - ConsumerRecords Partitions in Incubator-kafka-users

  • Hello Apache Kafka community, New HLC poll returns ConsumerRecords. Do ConsumerRecords contain records for every partition that HLC is actively subscribed on for every poll request, or does it contain only records for partitions which had messages and which were retrieved in poll request? If latter, then please consider adding a method to ConsumerRecords class, "public Iterable getPartitions...

  • KafkaConsumer Poll Always Returns Null in Incubator-kafka-users

  • I came across this google group conversation that suggests KafkaConsumer will not be complete until the next release. (https://groups.google.com/forum/#!msg/kafka-clients/4VLb-_wI22c/imYRlxogo-kJ) ``` org.apache.kafka.clients.consumer.KafkaConsumer consumer = new org.apache.kafka.clients.consumer.KafkaConsumer(consumerProps); consumer.subscribe("project-created"); Map records = consumer....

  • Consumer Poll In 0.9.0.1 Hanging in Incubator-kafka-users

  • Hi, When using Apache Camel Kafka to consume messages, I notice that when the topic is not created the fetching here: org.apache.camel.component.kafka.KafkaConsumer.run.. ConsumerRecords records = consumer.poll(Long.MAX_VALUE); ... just hangs forever, even if I create the topic and publish messages. It seems that I need to create the topic *before* the "poll" is invoked, otherwise, the...

  • Consumer Poll Returns No Records Unless Called More Than Once, Why? in Incubator-kafka-users

  • //this code Properties consumerProps = new Properties(); consumerProps.put("bootstrap.servers", "localhost:9092"); //without deserializer it fails, which makes sense. the documentation however doesn't show this consumerProps.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); consumerProps.put("value.deserializer", "org...

  • Kafka Consumer Picking Up The Same Message Multiple Times in Incubator-kafka-users

  • Hi, I've recently started using kafka to read documents coming through a web crawler. What I'm noticing is when I'm dealing with few million documents, the consumer is processing the same message over and over again. Looks like the data is not getting committed for some reason. This is not the case when I'm testing the consumer with few hundred message. I'm using kafka high level consumer...

  • Java Client For Kafka Consumer(0.9) Not Polling The Records From Brokers Instantly in Incubator-kafka-users

  • Hi We are facing a issue where our Consumer component is not instantly logging the records polled from the Brokers. We have following the below architecture as attached. Following are the properties configured: Producer.properties bootstrap.servers=xx.xxx.xxx.140:9092,xx.xxx.xxx.140:9093,xx.xxx.xxx.140:9094,xx.xxx.xxx.141:9092,xx.xxx.xxx.141:9093,xx.xxx.xxx.141:9094,xx.xxx.xxx.142:9092,xx...

  • Consumer/Publisher Code Is Not Throwing Any Exception If My Kafka Broker Is Not Running in Incubator-kafka-users

  • Hi, I am new to Kafka, could you please let me know why my application is not getting any exception like *refuse to connect so and so host, *if the broker is down/unable to connect *Here is my producer code* private static Map producerConfigs() { Map props = new HashMap(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(ProducerConfig.CLIENT_ID_CONFIG, "...

  • Monitor The Lag For The Consumers That Are Assigned To Partitions Topic in Incubator-kafka-users

  • Hello! I'm working with Kafka 0.9.1 new consumer API. The consumer is manually assigned to a partition. For this consumer I would like to see its progress (meaning the lag). Since I added the group id consumer-tutorial as property, I assumed that I can use the command bin/kafka-consumer-groups.sh --new-consumer --describe --group consumer-tutorial-group --bootstrap-server localhost:9092...

  • Kafka Producer Dropping Records in Incubator-kafka-users

  • Hello, We have the following piece of code where we read lines from a file and push them to a Kafka topic : Properties properties = new Properties(); properties.put("bootstrap.servers", ); properties.put("key.serializer", StringSerializer.class.getCanonicalName()); properties.put("value.serializer", StringSerializer.class.getCanonicalName()); properties...

  • OffsetOutOfRange Errors in Incubator-kafka-users

  • Hello, I understand what this error means, just not sure why I keep running into it after 24-48 hrs of running fine consuming > 300 messages / second. What happens when a kafka log rolls over and some old records are aged out? I mean what happens to the offsets? We are using a python client which stores the offsets in ZK. But in the middle of the run, say after 2 days or so, suddenly...

  • Specifying Exact Number Of Records To Fetch in Incubator-kafka-users

  • Hello, Do Kafka protocol & KafkaConsumer (java) client address the following need? The caller specifies that it needs N number of records with a max wait time of Tn milliseconds. If N records are available within Tn, the records are returned to the caller. If Tn expires, then the caller gets whatever number of records (< N) that were available. If this functionality not readily supported...

  • Generate Specific Throughput Load in Incubator-kafka-users

  • Hi, Is there a way to generate a specified amount of throughput? I'm using the Stats class here to measure the throughput. Though I need to be able to precisely control the amount of load. For example, 1000 records per second. Thanks, Josh...

  • Properducertest On Multiple Nodes in Incubator-kafka-users

  • Hi, I am trying to run 20 performance test on 10 nodes using pbsdsh. The messages will send to a 6 brokers cluster. It seems to work for a while. When I delete the test queue and rerun the test, the broker does not seem to process incoming messages: [[email protected] kafka_2.10-0.8.2.1]$ bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance speedx1 50000000 100 -1 ...