Hello,
I'm trying to do the following: have a set of short strings (as a queue) with expiration times (have items get dropped if they aren't processed fast enough).
It's probably because I'm well tired now, but I can't seem to find a way to do that: SADD queue v4lu3 and then EXPIRE that new item.
Is there a way, a workaround or just plain nothing?
Thank you for your time and good night.
Adrin Navarro's gravatar image asked May 1 2012 at 16:54 in Redis-Db by Adrin Navarro

4 Answers

You can only expire entire keys (a whole set, hash, zset, list, or string) at a time.
Generally, sets are not terribly good queues, as fast read/remove access patterns for individual items involve randomly popping items.
Most people choose to use lists as queues, as they have obvious first-in-first-out (or last-in-first-out) behavior.
To get a queue with expiration behavior, I can think of a two different ways of building it with Redis.
First way: 1. Store your queue items in a list or set or whatever 2. When you add an item X to the queue, also set a key 'item:X' to an empty string with an expiration. 3. When you fetch an item from the queue, delete the representative key. If the deletion occurred, then execute the queue item, otherwise discard it (because it expired)
Second way: 1. Store your queue items in a sorted set, with members being your queue items, and the score being your expiration time 2. You manually expire old items by removing items with expiration times in the past 3. You can fetch items based on their expiration times with "ZRANGE QUEUE 0 0", or randomly with "count = ZCARD QUEUE; index = floor(random() * count); ZRANGE QUEUE index index", but you need to ZREM items once you've pulled them from the queue.
Regards, - Josiah
Josiah Carlson's gravatar image answered May 1 2012 at 17:44 by Josiah Carlson
Got it thanks!
I'm also thinking about adding a timestamp in the value SADD queue 1335971908:itemslug and just do processing on the client. Not worried about performance, just a few queries a minute.
SADD queue time:value, SPOP queue, and process or discard depending on situation. Sounds good to me.
Thank you for your ideas and making clear that there is really no built-in way to do it.
Have a nice day!
Adrin Navarro's gravatar image answered May 2 2012 at 08:30 by Adrin Navarro
If you are adding timestamps, then uniqueness doesn't matter.
If uniqueness doesn't matter, why are you using a set? Are you looking to get random queue ordering? Why not use a list?
Regards, - Josiah
Josiah Carlson's gravatar image answered May 2 2012 at 09:57 by Josiah Carlson
Uniqueness did matter, sorting did not And I didn't think about it.
I've 'forgotten' the expiring times for now, I'll go back into that later :(
Adrin Navarro's gravatar image answered May 3 2012 at 14:36 by Adrin Navarro

Related Discussions

  • Lua Generated Keys Internal To Script (or A Better Way To Handle A Set Of Keys To Expiring Items) in Redis-db

  • I was looking up a way to handle expiring items in a set who reference expiring keys, and I found that a method using pubsub will exist in 2.8, unfortunately, that doesn't help me at the moment, so the least crappy way I could figure out how to do this is to remove the items next time I do a get of the set and the best way is to do it via a lua script, but I had some concerns. I was told via chat...

  • Redis SET Command - Order In Which Items Are Entered Into The Key Store in Redis-db

  • Hello, I am testing and implementing a number of redis instances for use as a high speed RAM only event store, which will read and write events from different software components, as well as another instance to act as a persistent event store for configuration events (less frequent but requires safety by writing all changes to the append only file). I am currently performing tests on inserting...

  • Possible To Expire Set Items? in Redis-db

  • I'm new to Redis, and as far as I can tell, it seems like I'm looking for something like what SADDEX or SETNXEX would be. A way to add an item, only if it does not exist, and have it use the expiry feature without having to have logic on the client and another call to the database. I've seen the posts about SETNXEX and how adding a feature that is a simple combination of other features has ...

  • Idle Functionality in Redis-db

  • I have a patch I'd like to contribute, if you guys would like. This adds a new feature; I'm not sure if this conflicts with other development or sort of steps on other functionality but let me give you the background and what I did. For an e-commerce site I'm working on we needed a solution for fast server-side carts. After looking at a few options Redis made the most sense. We brought up...

  • The Larger Lists' Items The Less Effectively Redis Uses Memory? in Redis-db

  • Hi all! I've made a simple test by inserting 10 mln items into 100k keys as lists. In the first run each item has 150 chars, in the second - 24. It turned out that in the first test redis took ~2.5GB and in the second - 0.28GB that gives us 150 useful bytes from 250 taken (+100 bytes overhead per item) in the first case and 24 out of 28 (+4 bytes overhead) in the second. Then I ran a few more...

  • Redis List Items, TTL And Ranking By Score in Redis-db

  • Hi Everyone, I posted this question on stackoverflow but I only got 8 views and no responses over the past 24 hours. I would like to sort blog articles by page hits over the past 5000 views. I was experimenting with the following (200, 205, 202 are id's of the blog articles): ZINCRBY blog_hits 1 200 ZINCRBY blog_hits 1 200 ZINCRBY blog_hits 1 200 ZINCRBY blog_hits 1 205 ZINCRBY blog_hits...

  • Issue 649 In Redis: LRANGE Suboptimal For Tail Items in Redis-db

  • Status: New Owner: ---- Labels: Type-Defect Priority-Medium New issue 649 by didier...@gmail.com: LRANGE suboptimal for tail items http://code.google.com/p/redis/issues/detail?id=649 What version of Redis you are using, in what kind of Operating System? Redis 2.2 and 2.4 branches What is the problem you are experiencing? LRANGE can be used to retrieve the last n items of a list. Unfortunately, ...

  • [redis-db] Redis Cache Serialization/Deserialization Takes Too Much Time And Hence Application Performace Degreded in Redis-db

  • Hi All, We have implemented Redis Cache for caching items in Asp.net MVC application . we have to serialized while adding items in to redis cache and have to deserialized cached items while getting from the redis cache. due to that application becomes slow .pages takes too much to load. we are using class to add , get ,remove cached items(please find the attached file). has anybody...

  • [redis-db] How Do I Remove A Range Of Items From List? in Redis-db

  • I thought that lrange in combination with ltrim would do the trick but it won't You received this message because you are subscribed to the Google Groups "Redis DB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected] To post to this group, send email to [email protected] Visit this group at http://groups...

  • Issue 656 In Redis: BRPOPLPUSH And A Single LPUSH With Several Items Doesn't Work Well Together in Redis-db

  • Status: New Owner: ---- Labels: Type-Defect Priority-Medium New issue 656 by hampus.w...@gmail.com: BRPOPLPUSH and a single LPUSH with several items doesn't work well together http://code.google.com/p/redis/issues/detail?id=656 This is in the unstable branch (but possibly in the other ones too). Try doing the following on a clean Redis instance (in this order): 1. Client A blocks on 'BRPOPLPUSH...

  • Efficiently Storing 100 Million Items Into Redis. What Is The Best Way To Do This? in Redis-db

  • I have a file which has around 100 million strings each no more than 20 chars in length. I need to find whether a string that the user enters exists within these strings. I am using Amazon ElastiCache to run Redis. The features are: STANDARD Small Cache Node (cache.m1.small): 1.3 GB memory, 1 ECU (1 virtual core with 1 ECU), 64-bit platform, Moderate I/O Capacity 64-bit platform, Moderate I/O Capacity...

  • [redis-db] BRPOPLPUSH No Longer Pop's When List Renamed To Match Source Key in Redis-db

  • Hi, ive been using the LPUSH / BRPOPLPUSH pattern for queues successfully for several years. Previously if a list was RENAME'ed blocked BRPOPLPUSH requests with no timeout would see the new items as if they had been LPUSH'ed. After upgrading from 2.6.x to 2.8.4 BRPOPLPUSH only pops items that have been LPUSH'ed. Is there a way to get the old behaviour without having a timeout ...

  • [redis-db] Provide GEOPERPDIST For Calculating Distance From A Geospatial Item To A Line Spanning Two Other Geospatial Items in Redis-db

  • Hey, A good use case for the new GEOADD is to track movement (vehicles for instance) in real time by inserting many, many [lat, lon] pairs in a sorted set. An immediate concern when dealing with this big set is the noise, so Douglas Peucker) require calculations related to how far a [lat, lon] pair is from a line formed by two others (the orthogonal distance). Since Redis already provides...

  • How To Push Many Items To The List Fast in Redis-db

  • I need to store quite large list (thousands of items) and perform LRANGE operations on it later. Looks like the only solution is to RPUSH items one by one, but it takes significant amount of time for a list with many items. Is it possible to speedup the process of pushing many items to the list?...

  • Question About Keeping An Age-sorted Group Of Items in Redis-db

  • Suppose I'm implementing something like a forums system / BBS. Each forum has a number of threads; each thread has a number of comments. I want getting the first page (or first few pages) of the forum thread list (sorted by last comment) fast, and I want it to still be possible to browse back history, even for a fairly big forum topic that lives for a long time. My first approach to mapping...

  • Is Single Hash Capable Of Handling 10000 Items? in Redis-db

  • I have more than 10000 topics which needs to be stored as part of hash structure. Is there any performance penalty if the number of items in the hash increases to more number?...

  • Most Efficient Way To Pop Atomically Lots Of List Items in Redis-db

  • Hi! What is the most efficient way to pop atomically a large amount of items from a list? I'm thinking of usage of the following set of commands: MULTI LRANGE foo X Y LTRIM foo X Y EXEC What do you think? P.S. Wouldn't it be nice to have an optional argument for LTRIM which would make it return the trimmed stuff? -- Best regards, Pavel...

  • How To Get All Items With The Same Prefix in Redis-db

  • Folks, I was originally using list, and call lrange to get all items. But since I also need to remove an item from the list, and it takes much longer time. I thus decide to switch to use keys with prefix instead: just put all the messages with a prefix: key value prefix:1 msg1 prefx:2 msg2 ... My question is, what is the most efficient way to get all the messages with the same...

  • Processing Queued Items In Bulk, Flag Is Processing, Remove W/complete in Redis-db

  • I need to have a queue where I insert a single item in the queue at the rate of upto 2-3K per second, where is message body (xml) maybe be 20-50K in size. Then a worker process will fetch items in the queue in batches (few hundred? not sure what is reasonable yet), this is because I need to analyze all of them before hitting the database (to rule out duplicates etc.). How would this work in redis...

  • Sorted Sets: Count Elements In Range in Redis-db

  • First i have to thank you Salvatore, redis is a really great datastore, implementing data structures was the thing i missed in other nosql-datastores. But i am actually missing a feature to count the number of elements in a given range. The application i am working for needs to track the number of items added to a scored set (by unixtimestamp in milliseconds) in exactly the last 5 minutes. ...