Metrics To Monitor In Kafka

Hi,
What are the important metrics to monitor in Kafka (as explained here
) and raise alarms
when the value reaches certain value? I am looking for some guidelines
similar to this excellent monitoring doc
about ElasticSearch
I am using kafka in a logstash deployment.
Logstash Forwarder --- Logstash -----Kafka ---- Logstash --- ES Cluster
Its a single broker and single zookeeper deployment running in a single
machine.
Thanks,
Debraj

This document explains some important metrics
http://docs.confluent.io/1.0/kafka/monitoring.html
Also, you can check the kafka demo at https://apps.sematext.com/demo.
As a side thing, as I have done, you can report every stat to Ganglia and
then see, according to your domain, what are the most important ones to use
and monitor. Ganglia does it really well in terms of aggregating form
various kafka nodes
Regards,
Prabhjot
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"

Consumer offset/lag is what people are always after. :)
See also: http://search-hadoop.com/?q=kafka+otis+important+metrics
Otis
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/

Hi, I'm working on a Trident/Kafka Topology which uses a OpaqueTridentKafkaSpout. I'm currently trying to obtain some metrics about this topology ( number of tuples emitted by the OpaqueTridentKafkaSpout, process time for each spout/bolt, ...) I first tried to use Storm UI and the REST API but the number of tuples emitted by the spout was increasing really fast even though I had no messages
Hi We have any monitor tool which monitor kafka offset? I need some opensource admin tool for kafka. Please guide me. Regards, -Anandh Kumar
I am integrating the Kafka JMX metrics into our own system. I am able to view the Kafka metrics via JConsole. However, pulling the metrics into my own system, using my JSON files is failing. It is unable to find the mbeans ("javax.management.InstanceNotFoundException: kafka.server:type=BrokerTopicMetrics,name=AllTopicsMessagesInPerSec"). In JConsole, "mxbean" is set to false for this, and all other
We have make a simple web console to monitor some kafka informations like consumer offset, logsize. https://github.com/shunfei/DCMonitor Hope you like it and offer your help to make it better :) Regards Flow
How to configure, to emit kafka broker metrics to graphite. t SunilKalva
Hi I am getting various kafka parameters (like NumFetchRequests, FetchRequestsPerSecond, ProduceRequestsPerSecond etc.) through JMX in case of kafka-0.7. In kafka-0.8 i am doing the same but not able to get parameters. There are some changes in reporting metrics in kafka-0.8 as compared to kafka-0.7. I have already noticed that. According to changes in kafka-0.8 i am trying to get kafka
Hi, I have enabled JMX_PORT for KAfka server and am trying to understand some of the metrics that are being exposed. I have two questions: 1. what are the best metrics to monitor to quickly spot unhealthy Kafka cluster? 2. what do these metrics mean: ReplicaManager -> LeaderCount ? and ReplicaManager -> PartitionCount ?I have three topics created, with one partition each, and replication = 1,
Please I need help for sending metrics to Graphite. Can anyone help me in resolving this. Thank You. Regards, Sanjay Mengani Extn : 4060 Mobile : +91-9985267763
Hi, we started a new github project for using Kafka as the central point for all application and infrastructure metrics https://github.com/stealthly/metrics-kafka/. We started off with implementing a Metrics Reporter (for Coda Hale's metrics) which produces (reports) the statistics to a Kafka topic. We also created an example consumer for monitoring those metrics using Riemann. More details
I am trying to send kafka metrics for display to ganglia server using latest download from https://github.com/adambarthelson/kafka-ganglia. Here's my kafka_metrics.json { "servers" : [ { "port" : "9999", "host" : "ecokaf1", "queries" : [ { "outputWriters" : [ { "@class" : "com.googlecode.jmxtrans.model.output.GangliaWriter", "settings" : { "