I'm trying to do the following: have a set of short strings (as a queue) with expiration times (have items get dropped if they aren't processed fast enough).
It's probably because I'm well tired now, but I can't seem to find a way to do that: SADD queue v4lu3 and then EXPIRE that new item.
Is there a way, a workaround or just plain nothing?
Thank you for your time and good night.
Adrián Navarro's gravatar image asked May 1 2012 at 16:54 in Redis-Db by Adrián Navarro

4 Answers

You can only expire entire keys (a whole set, hash, zset, list, or string) at a time.
Generally, sets are not terribly good queues, as fast read/remove access patterns for individual items involve randomly popping items.
Most people choose to use lists as queues, as they have obvious first-in-first-out (or last-in-first-out) behavior.
To get a queue with expiration behavior, I can think of a two different ways of building it with Redis.
First way: 1. Store your queue items in a list or set or whatever 2. When you add an item X to the queue, also set a key 'item:X' to an empty string with an expiration. 3. When you fetch an item from the queue, delete the representative key. If the deletion occurred, then execute the queue item, otherwise discard it (because it expired)
Second way: 1. Store your queue items in a sorted set, with members being your queue items, and the score being your expiration time 2. You manually expire old items by removing items with expiration times in the past 3. You can fetch items based on their expiration times with "ZRANGE QUEUE 0 0", or randomly with "count = ZCARD QUEUE; index = floor(random() * count); ZRANGE QUEUE index index", but you need to ZREM items once you've pulled them from the queue.
Regards, - Josiah
Josiah Carlson's gravatar image answered May 1 2012 at 17:44 by Josiah Carlson
Got it thanks!
I'm also thinking about adding a timestamp in the value SADD queue 1335971908:itemslug and just do processing on the client. Not worried about performance, just a few queries a minute.
SADD queue time:value, SPOP queue, and process or discard depending on situation. Sounds good to me.
Thank you for your ideas and making clear that there is really no built-in way to do it.
Have a nice day!
Adrián Navarro's gravatar image answered May 2 2012 at 08:30 by Adrián Navarro
If you are adding timestamps, then uniqueness doesn't matter.
If uniqueness doesn't matter, why are you using a set? Are you looking to get random queue ordering? Why not use a list?
Regards, - Josiah
Josiah Carlson's gravatar image answered May 2 2012 at 09:57 by Josiah Carlson
Uniqueness did matter, sorting did not And I didn't think about it.
I've 'forgotten' the expiring times for now, I'll go back into that later :(
Adrián Navarro's gravatar image answered May 3 2012 at 14:36 by Adrián Navarro

Related Discussions