Hi,
I ran my application on AWS machine having 32GB ram. This application is
written in Java using Kafka-Streaming low level api. When I ran it for 72
hours or more and load (messages) are keep on pumping to kafka-topic then
entire memory i.e. 32 GB of aws machine gets exhausted. While I ran my java
application with -xms4gb and -xmx12gb. As per Kafka-Streaming memory
management, RocksDB uses off-heap space which might be causing this issue.
Can you please help me to resolve this problem.
Regards,
Deepak
Deepak Pengoria 's gravatar image asked Feb 17 2017 at 05:30 in Incubator-Kafka-Users by Deepak Pengoria

1 Answers

Hi Deepak,
Could you tell us a bit more on what your app is doing (e.g., aggregates, or joins)? Also which version of Kafka are you using?
Have you changed any default configs?
Thanks
Eno
Eno Thereska 's gravatar image answered Feb 17 2017 at 07:16 by Eno Thereska

Related Discussions

  • Kafka Streaming in Incubator-kafka-users

  • I am trying to run the examples from git. While running the wordcount example I see this error: Caused by: *java.lang.RuntimeException*: librocksdbjni-win64.dll was not found inside JAR. Am I expected to include this jar locally?...

  • Kafka Connect RequestTaskReconfiguration in Incubator-kafka-users

  • Hey kafka-users, Is it normal for a Kafka Connect source connector that calls requestTaskReconfiguration to cause all the connectors on the kafka-connect distributed system to be stopped and started? that will occasionally invoke `context.requestTaskReconfiguration()` when it detects new work that it wants to distribute to its tasks. When this occurs, I observe all three of the connectors...

  • Is Running Kafka Streaming Application Advisable On High Latency WAN Setup in Incubator-kafka-users

  • Hi folks, I needed bit of feedback from you based on your experiences using kafka streaming application. We have a replicated kafka cluster running in a data center in one city. We are running a kafka streaming application which reads from a source topic from that cluster and commits the output into local database in its own data center. The distance between these two data center is about...

  • One Kafka Broker Went Rogue in Incubator-kafka-users

  • Hi, I encountered a strange issue in our kafka cluster, where randomly a single broker entered a state where it seemed to think it was the only broker in the cluster (it shrank all of its ISRs to just existing on itself). Some details about the kafka cluster: - running in an EC2 VPC on AWS - 3 nodes (d2.xlarge) - Kafka version : 0.10.1.0 More information about the incident: Around...

  • Kafka Streaming Message Loss in Incubator-kafka-users

  • Hi I'm trialling Kafka Streaming for a large stream processing job, however I'm seeing message loss even in the simplest scenarios. I've tried to boil it down to the simplest scenario where I see loss which is the following: 1. Ingest messages from an input stream (String, String) 2. Decode message into a type from JSON 3. If succesful, send to a second stream and update an atomic counter...

  • Exception When Running Kafka in Incubator-kafka-users

  • Hi, I'm new to kafka (and scala). I downloaded 0.7.1 (I also tried the latest from svn) and followed the quick start guide. When I try to start the server, I get the following exception. Does anybody know what's going on and how I can fix it? Thanks! [[email protected] kafka-0.7.1-incubating]$ bin/kafka-server-start.sh config/server.properties Exception in thread "main" java.lang.NoClassDefFoundError...

  • Kafka Colapsed Due To OutOfMemory in Incubator-kafka-users

  • Hi, I have a 3 node kafka in production with 1gb in each JVM. Suddenly one day, two of them when down and the other tried to make all the work. The log errors were that they could not sincronize. We applied rebalance to them and it never ends. Unfortunately, we have not GC log, so we had to add it and then realized than each JVM was out of memory. We changed specifically the line 29 of kafka...

  • Kafka GIT 7c334752 Issue With Metrics in Incubator-kafka-users

  • Hi, I updated our test system to use Kafka from latest revision 7c33475274cb6e65a8e8d907e7fef6e56bc8c8e6 and now I'm seeing: [2015-11-03 14:07:01,554] ERROR [KafkaApi-2] error when handling request Name:LeaderAndIsrRequest;Version:0;Controller:3;ControllerEpoch:1;CorrelationId:5;ClientId:3;Leaders:BrokerEndPoint(3,192.168.60.168,21769);PartitionState:(5c700e33-9230-4219-a3e1-42574c175d62...

  • Only One ProducerSendThread Thread When Running With Multiple Brokers (kafka 0.8) in Incubator-kafka-users

  • Hi, I am using kafka 0.8. I have 3 machines each running kafka broker. I am using async mode of my Producer. I expected to see 3 different threads with names starting with ProducerSendThread- (according to this article: http://engineering.gnip.com/kafka-async-producer/) However I can see only one thread with the name *ProducerSendThread-* This is my producer configuration: server=...

  • Kafka Consumer Does Not Work From Eclipse When Kafka Running Inside VM in Incubator-kafka-users

  • Kafka 0.8.2.2.3 and zookeper both are running inside VM. I was able to run both producer and consumer within VM successfully using kafka-console-producer.sh and kafka-console-consumer.sh respectively. Even I was able to consume Kafka messages from host machine using kafka-console-consumer.sh. But when I tried to run the consumer using java from eclipse then zookeeper logs following error 2015...

  • Kafka Consumer Does Not Work From Eclipse When Kafka Running Inside VM in Incubator-kafka-users

  • Kafka 0.8.2.2.3 and zookeper both are running inside VM. I was able to run both producer and consumer within VM successfully using kafka-console-producer.sh and kafka-console-consumer.sh respectively. Even I was able to consume Kafka messages from host machine using kafka-console-consumer.sh. But when I tried to run the consumer using java from eclipse then zookeeper logs following error 2015...

  • Sys CPU And Load Average Spikes With Kafka Cluster On Virtual Box in Incubator-kafka-users

  • Hi All, We are running into a situation where System CPU (as seen in top) is increasing to 67%. Also the load average of system (as seen in top) increases to 25 Following is the setup. 1. One Zookeeper, 3 kafka brokers 2. This cluster is running on Centos 6.5 OS * Centos OS is the guest OS * Virtual box is the hypervisor * Windows OS is the host OS 3. The...

  • Getting Kafka_metrics Errors When Running Kafka 0.8 in Incubator-kafka-users

  • Hi,I cloned Kafka 0.8 and built it with Scala 2.9.2 I launched it successfully but when a message is sent, I got a huge stack trace regarding kafka_metrics For example: java.io.IOException: Unable to create kafka_metrics/kafka.server.BrokerTopicMetrics.AllTopicsBytesOutPerSec.csv at com.yammer.metrics.reporting.CsvReporter.createStreamForMetric(CsvReporter.java:145) at com.yammer.metrics.reporting...

  • Getting None.get Exception When Running Kafka-reassign-partitions.sh in Incubator-kafka-users

  • Checked the json file and everything seems normal. When I run the script and I got the error: Partitions reassignment failed due to None.get java.util.NoSuchElementException: None.get at scala.None$.get (Option.scala:185) my json file: {partitions": [{"topic": "topicA", "partition": 0}] } thanks in advance, Cal...

  • RocksDB Error When Running Kafka With A Larger Data Set in Incubator-kafka-users

  • Hi Kafka crew, I'm rolling out a Kafka Streams application on a pretty large dataset, and I see some exceptions that worry me, which I haven't seen with smaller data sets (Using Kafka streams trunk of a few days ago) 2017-02-15 15:25:14,431 | INFO | StreamThread-42 | StreamTask | 86 - com.dexels.kafka.streams - 0.0.53.201702141103 | task [9_4] Initializing state stores 2017...

  • Migration Tool Consumer Rebalance Error When Attempting To Migrate From Kafka 0.7 To Kafka 0.8 in Incubator-kafka-users

  • Hello. I am trying to migrate from Kafka 0.7 to 0.8, and the first stage of this is to use the Migration Tool. I have setup a local Kafka 0.8 cluster consisting of 3 nodes on my local Vagrant (along with a zookeeper running locally). The Migration Tool Producer will connect to these three nodes in order to replicate the data. The Migration Tool Consumer(s) will connect to my Staging boxes...

  • How To Get And Reset Kafka Stream Application's Offsets in Incubator-kafka-users

  • Hi I am running a streaming application with streamsProps.put(StreamsConfig.APPLICATION_ID_CONFIG, "test"); How do I find out the offsets for each of the source, intermediate and internal topics associated with this application. And how can I reset them to some specific value via shell of otherwise. Thanks Sachin...

  • Exception Connecting To Kafka When IPv6 Textual Representation Of The Address Is Returned in Incubator-kafka-users

  • Hi, I keep running into the issue already discussed here:https://mail-archives.apache.org/mod_mbox/kafka-dev/201503.mbox/%3CJIRA.[email protected]%3E where I am getting the "javax.management.MalformedObjectNameException: Invalid character ':' in value part of property" error... due to IPv6 address representation returned by my Mac. Is there a known or...

  • Kafka-list-topic Showing Leader: -1 When Brokers 300 Or 301 Would Be The Leader in Incubator-kafka-users

  • Our main concern is: how to nuke or repair things so that brokers 300 and 301 can rejoin the leader and isr lists properly. We have a 5-host cluster, each host has both zookeeper-3.3.6 and kafka-0.8 running on it. I sniffed around in zkCli.sh and didn't notice anyting particularly wrong or corrupt. kafka-list-topic is showing leader: -1 when brokers 300 or 301 would be the leader: # /usr...

  • Running Kafka/zookeeper Running Locally in Incubator-kafka-users

  • Hi Any one tried running zookeeper and kafka locally , which can be useful for automating the test cases for API built on kafka ? SunilKalva...