QnaList > Groups > Spark-User > Mar 2016
faq

How To Release Data Frame To Avoid Memory Leak

Hi
I have data frames created every 5 minutes. I use a dict to keep the recent
1 hour data frames. So only 12 data frame can be kept in the dict. New data
frame come in, old data frame pop out.
My question is when I pop out the old data frame, do I have to call
dataframe.unpersist to release the memory?
For example 
    If currentTime == fiveMinutes:
        myDict[currentTime] = dataframe
        oldestDataFrame = myDict.pop(oldest)
Now do I have to call oldestDataFrame.unpresist?  Because I think python
will automatically release unused variable
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-release-data-frame-to-avoid-memory-leak-tp26656.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

asked Mar 31 2016 at 21:19

kramer2009126.com 's gravatar image



Related discussions

Tagged

Group Spark-user

asked Mar 31 2016 at 21:19

active Mar 31 2016 at 21:19

posts:1

users:1

Spark-dev

Spark-user

©2013 QnaList.com