We upgraded our Kafka clusters from 0.8.2.1 to 0.10.0.1, but most of our
consumers use older libraries that do not support the new message format.
So we set the brokers' log.message.format.version to 0.8.2 while we work on
upgrading our consumers.
In the meantime, I'm worried about a performance problem with consumers
that have upgraded and are requesting messages using the new Kafka 10
versions of those API calls.
I may be misunderstanding, but it seems logical that the performance
problem isn't just about old consumers with a new broker. I would think the
performance problem would also exist if we take new brokers, set the log
format to an old version, then have our consumers make API calls using the
Kafka 10 API calls. The broker would need to do on-the-fly conversion from
the 0.8.2 log format up to the 0.10.0 format to send to the new consumers.
This is the inverse problem of what's mentioned here:
https://kafka.apache.org/documentation/#upgrade_10_performance_impact
Is this a valid problem?
Jeff Widman 's gravatar image asked Jan 11 2017 at 10:51 in Incubator-Kafka-Users by Jeff Widman

1 Answers

Hi Jeff,
The new consumer also supports the old message format without requiring
conversion.
Ismael
Ismael Juma 's gravatar image answered Jan 11 2017 at 12:16 by Ismael Juma

Related Discussions

  • New Consumer Client Compatible With Old Broker in Incubator-kafka-users

  • Hi team, Does new consumer client (the one in trunk) work with 0.8.2.x broker? I am planning to use the new consumer in our development but don't want to upgrade the broker to the latest. is it possible to do that?...

  • Migrating Old Consumer Offsets To New Consumer in Incubator-kafka-users

  • I’m running the old 0.8 consumer storing offsets in Zookeeper, want to migrate to the new consumer introduced in 0.9 . I don’t see anything in the docs about how to do that while preserving offsets. Do I need to follow the steps from the FAQ to migrate to committing offsets to Kafka, then I can swap in the new consumer?...

  • Questions About New Consumer API in Incubator-kafka-users

  • Hi guys, I'm interested in the new Consumer API. http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/ I have couple of question. 1. In this doc it says kafka consumer will automatically do load balance. Is it based on throughtput or same as what we have now balance the cardinality among all consumers in same ConsumerGroup? In a real case different partitions could have...

  • Is There A Way To Make Sure Two Consumers Receive The Same Message From The Broker? in Incubator-kafka-users

  • is there a way to make sure two consumers receive the same message from the kafka broker in a atomic way? such that if consumer 1 gets a message consumer 2 should also get that message and in case one of the consumer fails for whatever reason consumer 2 should also rollback to previous offset or invalidate or something like that. is that possible?...

  • Migrating Old Consumer Offsets To New Consumer in Incubator-kafka-users

  • I’m running the old 0.8 consumer storing offsets in Zookeeper, want to migrate to the new consumer introduced in 0.9 . I don’t see anything in the docs about how to do that while preserving offsets. Do I need to follow the steps from the FAQ to migrate to committing offsets to Kafka, then I can swap in the new consumer?...

  • There Is An Excepiton When Consumer Get Data From One Broker. in Incubator-kafka-users

  • Hi,all There is an exception when consumer get data from one broker. And can not get new data from this broker. But there are some data on this broker that are not consumed. The exception is like:[kafka.consumer.FetcherRunnable$$anonfun$run$3.apply(FetcherRunnable.scala:91)]: FetchRunnable-0 kafka.consumer.FetcherRunnable - error in FetcherRunnable for Memcache2Sql:1-0: fetched offset = 2983409308...

  • Kafka-consumer-groups.sh Delete Group With New-consumer in Incubator-kafka-users

  • Hi all When I use kafka-consumer-groups.sh to delete new-consumer's group, the shell note that "Option [delete] is not valid with [new-consumer]. Note that there's no need to delete group metadata for the new consumer as it is automatically deleted when the last member leaves." How to understand it is automatically deleted when the last member leaves? Does it means the consumer's...

  • Questions On The New Kafka Consumer in Incubator-kafka-users

  • I was reading the documentation for the new Kafka consumer API at https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java and came across this: "Each Kafka consumer must specify a consumer group that it belongs to." Currently we use Kafka but WITHOUT the consumer group functionality since I do the partition distribution based on...

  • New Consumer Api? in Incubator-kafka-users

  • Hello guys, A while ago i read that the new consumer api was going to be released sometime in July as part of the 0.8.3/0.9 release. https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan Do we have an update when we think that can happen? Thanks, Jalpesh...

  • New Consumer APIs in Incubator-kafka-users

  • Hi, Would like to if New Consumer API is GAed. Regards, Rajeswari...

  • New Code Not Receiving Messages After Release Where Old Code Left Off in Incubator-kafka-users

  • Thanks Jun, yup, that is how I understand it. I just figured out the error is a red herring to my problem since that is caught properly when logged in ZookeeperConsumerConnector.commitOffsets. it seems that the problem I am running into is this (any help is appreciated) one code base consumer = apophis_starscream.site1.medialets.com-1322840963912-9c58256d (from the logs) so when i kill that consumer...

  • Is Kafka0.9.0.1's New Consumer Ready For Production in Incubator-kafka-users

  • hi : i am a kafka developer, our company want to upgrade kafka cluster from 0.8.2.1 to 0.9.0.1. At the 0.9.0.0 doc, the new Consumer is beta state, also the release notes of 0.9.0.1 do not point out the state change. can any body tell me if it is ready for production Thanks & Regards, Bruce Yan...

  • Is The New Consumer API Ready? in Incubator-kafka-users

  • I'm updating the kafka APIs we use to the new standalone ones, but it look like the new consumer isn't ready yet (the code has got lots of placeholders etc), and there's only the producer in the Javadoc at http://kafka.apache.org/082/javadoc/index.html. Is there an ETA on when the new consumer will be ready? SimonC This message, and any files/attachments transmitted together with it, is intended...

  • Kafka New Consumer Is Ready For Production Setup? in Incubator-kafka-users

  • Hi, I want to use kafka new consumer. But in the documentation of 0.10.0 version, it is mentioned that "The code is considered beta quality." I am using the latest version which is 0.10.1. I am not able to find out any documentation for this version. Can you please tell me, if the new consumer is ready for production use or it is still in the beta version? Thanks and Regards, Gaurav...

  • Is There A Way To Get The Offset Of A Consumer Of A Topic? in Incubator-kafka-users

  • Hi, I was wondering if there's an easy way to get the current offset of a specified consumer? What we need is something to check the consumer offset every 5 min. If it's changing at a certain rate every time we check, good. If not, then we have a problem. I was going to write a monitoring program to check the broker or consumer for this offset to see if it changed and how much it changed...

  • Is Kafka 8 User Compatible With Kafka 10 Broker? in Incubator-kafka-users

  • Hi guys, I created a new cluster with kafka 10 broker. I have the following settings in server properties file: inter.broker.protocol.version=0.10.0.0 message.format.version=0.10.0 log.message.format.version=0.10.0 When I try to read from old console consumer it still succeeds. Online documents say old consumer is not compatible as long as message format version is not set to 0.8.0. ...

  • Consumer Not Getting Data When There Is A Big Lag In The Topic in Incubator-kafka-users

  • Hi, I am been facing some JAVA high level consumer related issues lately and would like to understand more on this. We have 9 bare-metals (48 core, 250 GB, Terabytes of Hard disks) running *Kafka 0.8.2* and 5 independent VM (8 core, 60 GB) running zookeeper. I have a topic that has key as metadata and value as a file. The file can be as large as *180 MB.* We have a topic with 90 partitions...

  • Consumer Doesn't Start Picking Existing Messages From Topic Until New Data Is Added in Incubator-kafka-users

  • Hi All I am facing issues with the kafka 0.8 consumer with the consumer not picking existing messages in the topic for consumption. Following is a snippet from my code : - public void run() { ConsumerIterator it = stream.iterator(); System.out.println("************************In Run : before while loop"); //*1st SYSO* while (it.hasNext()) { ...

  • New Consumer - Discover Consumer Groups in Incubator-kafka-users

  • Hi, Assuming i am using the latest kafka (trunk), exclusively with the new consumer, and i want to monitor consumer lag across all groups - how would i go about discovering the consumer groups? Is there an API call for this? Thanks, Damian...

  • Not Able To Monitor Consumer Group Lag With New Consumer And Kerberos in Incubator-kafka-users

  • Hi, I'm not able to use kafka-consumer-groups.sh to monitor the lag of my consumers when my cluster is kerberized. I'm using kafka version 0.9.0 installed on an hortonworks hdp 2.4.0 cluster. I've replicated my setup on two sandboxes, one without kerberos and one with kerberos. ./kafka-consumer-groups.sh --list --new-consumer --bootstrap-server sandbox.hortonworks.com:6667 When...