Kafka Metrics In 0.8

Hi
I am getting various kafka parameters (like NumFetchRequests,
FetchRequestsPerSecond, ProduceRequestsPerSecond etc.) through JMX in case
of kafka-0.7.
In kafka-0.8 i am doing the same but not able to get parameters.
There are some changes in reporting metrics in kafka-0.8 as compared to
kafka-0.7. I have already noticed that. According to changes in kafka-0.8 i
am trying to get kafka metrics using JMX through jmxtrans api.
Here is sample java code of getting jmx parameters using jmxtrans api:
       Server kafkaJMXServer = new Server("127.0.0.1", "9999");
        Query query1 = new Query();
query1.setObj("kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesInPerSec");
        query1.addAttr("MeanRate");
        kafkaJMXServer.addQuery(query1);
        JmxProcess process = new JmxProcess(kafkaJMXServer);
        JmxUtils.execute(process);
        System.out.println(query1.getResults());
Please let me know what i am missing?
*Thanks & Regards*
*Hanish Bansal*

It seems that the mbean name that you used is wrong. The mbean names
registered by metrics 2.2.0 have quotes in them.
Thanks,
Jun

Okay i'll try for same.
Also want to know that in kafka-0.7 there is single MBean -
SocketServerStats which provides all kafka Metrics. But in Kafka-0-8 there
are individual MBeans (like AllTopicsBytesInPerSec, AllTopicsBytesOutPerSec
etc.) for getting each parameter like ByteInRate, ByteOutRate etc.
Does kafka-0.8 provide any single MBean from which we can get all
parameters?
*Thanks & Regards*
*Hanish Bansal*

In 0.8, there are more mbeans than 0.7.
Thanks,
Jun

Thanks Jun
*Thanks & Regards*
*Hanish Bansal*

Hi All,
As kafka-0.8's main feature in data replication.
We are running two kafka nodes(lets say: kafka-1 and kafka-2) in cluster
mode. If we produce data into kafka using replication factor=2 then its
replica is created on second machine.
I am using jconsole to monitor kafka. if messages are going on kafka-1 then
i am able to see kafka metrics on jconsole of kafka-1. But by connecting to
jconsole for kafka-2 i am not seeing any metrics, i want to know why it is
happening ??
For more clarity i am asking same question in other way:
If one kafka node is storing replicas of data, then will it be visible on
jconsole or not ?
*Thanks & Regards*
*Hanish Bansal*

Does the Kafka data show up in the log dir in kafka-2? Since only the
leader can serve any traffic, if there is only have one partition or all
leaders are on one broker, not all metrics (e.g., produce/fetch rate) will
show up on both brokers.
Thanks,
Jun

Yes, Kafka data shows up in the log dir in kafka-2. We are having 2
partitions on all brokers.
*Thanks & Regards*
*Hanish Bansal*

Could you run kafka-list-topics to see whether the leader of those
partitions are distributed on both brokers?
Thanks,
Jun

I am trying to send kafka metrics for display to ganglia server using latest download from https://github.com/adambarthelson/kafka-ganglia. Here's my kafka_metrics.json { "servers" : [ { "port" : "9999", "host" : "ecokaf1", "queries" : [ { "outputWriters" : [ { "@class" : "com.googlecode.jmxtrans.model.output.GangliaWriter", "settings" : { "
Good evening. I would like to preserve logs in kafka for 2 days. Due to the size of data it must be compressed otherwise it will take enormous amount of disk space. Its my understanding that the only way to store messages in kafka compressed is to enable compression on the producer side, is that correct? Will they be sent to consumer also in compressed format? How the consumer can check if stream
Hello- Is CallbackHandler supported in Kafka 0.8 for async producers? If yes, can I use it to alter the batched messages before they are pushed to broker? For example, I may want to delete some of the messages in the batch based on some business logic in my application? If no, is there any alternate way? I want to do some kind of single instancing on messages pushed in kafka in last
Hi there, I am using kafka 0.8 (currently 0.8.0-beta1-candidate1) and I want my consumer to fail if it can not reliably find out the last consumed offset from zookeeper. According to https://kafka.apache.org/08/configuration.html for "https://kafka.apache.org/08/configuration.html" .. But it looks like the exception is always thrown, not only when there is no initial offset found in Zookeper
I noticed that ProducerData has been removed in 0.8 branch If I'd wish to send a complex message type (encoded message) , how would I do it in 0.8? In 0.7.X (a snippet from Neha's example) val producer = new Producer[Message, MemberRecord](config); // send a single message val message = new MemberRecord(1, "John", "US") val producerData = new ProducerData[Message, MemberRecord
We are looking at using kafka 0.8-beta1 and high level consumer. kafka 0.7 consumer supported backoff.increment.ms to avoid repeatedly polling a broker node which has no new data. It appears that this property is no longer supported in 0.8. What is the reason? Instead there is fetch.wait.max.ms which is the maximum amount of time the server will block before answering the fetch request if
I'm trying to get the Ontology plugin to work in an 0.8 environment. I set extension.ontology.extension-name to org.apache.nutch.ontology.jena.OntologyImpl This seemed to be the correct thing to do according to nutch-default.xml which says: Use the specified online ontology extension. If empty, the first available extension will be used. The "name" here refers to an 'id' attribute of the
Hi folks, In my project, we want to perform to update our active Kafka 0.8 cluster to Kafka 0.8.1.1 without downtime and losing any data. The process (after reading http://kafka.apache.org/documentation.html#upgrade) looks to me like this. For each broker in turn: 1. Bring the broker down. 2. Update Kafka to 0.8.1.1 on the broker node. 3. Start the broker. 4. Run preferred-replica-election
Hello, Just wanted to know where we are with the beta release for 0.8? More importantly, is 0.8 going to be publicly available from a maven repository? How about different versions of 0.8 built for different versions of scala? (for example scala 2.8 vs 2.9 etc.) Much appreciated. Soby Chacko
RGVhciBhbGwsDQoNCkkgYW0gd3JpdGluZyB0byB0aGlzIGxpc3QgdG8gY2hlY2sgaWYgdGhlcmUg aXMgYW55IG5ldyBwcm9ncmVzcyBjb25jZXJuaW5nIDAuOCBXaW5kb3dzIEMrKyBjbGllbnRzIGFu ZCBpZiB0aGVyZSBhcmUgZXhwZXJpZW5jZXMgd2UgY2FuIHNoYXJlLg0KDQpNeSBjb21wYW55ICh3 ZSBhcmUgaW4gdGhlIGNvbnRleHQgb2YgcGVyZm9ybWFuY2UgYWR2ZXJ0aXNpbmcgbmV0d29ya3Mp IGlzIHVzaW5nIEthZmthIDAuOCBhcyB0aGUgYmFja2JvbmUgZm9yIHRoZSBldmVudCBpbmZyYXN0 cnVjdHVyZS4gQXMgdGhlIGNvcmUgY29tcG9uZW50cyBhcmUgcnVubmluZyB3aXRoIEMrKy9XaW5k