groups of keys and dependent keys
sgrimm at facebook.com
Mon Oct 16 18:40:01 UTC 2006
Sounds good if you can make it work efficiently. Here are a few
questions I'd want to see answers for before anyone starts implementing
A key and its dependencies may be spread across multiple servers,
memcached being a distributed system. How will a server know when it's
safe to delete something? Bear in mind that at some installations (ours
included, but others as well) "multiple servers" means well in excess of
100, so "broadcast to all the other servers when something happens" is
not an option unless you're VERY frugal in the use of broadcasts.
What happens when a server with one of the dependent keys crashes or its
memcached instance is restarted?
Can you set different expiration times on different keys in a dependency
graph? What happens if the parent key's expiration time passes and you
try to fetch a child key? (Expiration is lazy in memcached.)
What will the performance impact be on operations not involving key
dependencies? Busy memcached installations handle a couple tens of
thousands of requests a second, so whatever the implementation for this
looks like, it needs to be able to keep up with that kind of load.
Don't get me wrong -- if what you describe can be made to work well, it
will definitely be useful! But I suspect it'll be trickier than it
appears to make it bulletproof.
Martin Kihlgren wrote:
> Hello list.
> My name is Martin Kihlgren, and I am currently employed by a company
> that heavily relies on memcached.
> We are doing lots of caching, simply.
> And, unfortunately, a whole lot of invalidations.
> Today we got to know of
> and I thought about the implications.
> It would be so very much nicer to have this functionality built into
> memcached :O
> The scheme I would imagine is:
> * The addition of key dependency, in at least one level.
> * A key can be declared depending on another key.
> * A key can not be dropped unless all keys depending on it are
> * If a key having dependent keys MUST be dropped, the whole
> set of dependent keys are dropped as well.
> This would make it very easy for us to create group keys having
> dependent keys, and then just delete the group key thus forcing all
> dependent keys to be dropped.
> Much simpler and more efficient than storing the set of keys in the
> group in memcache (thus having to worry about memcache dropping the
> group key before the individual keys in the group are dropped) and
> also much better than storing the group keys on disk or in db.
> Is this something you have discussed? Or is it already in some edge
> version of memcached? Or is it something that is possible to do with
> the current codebase?
> //Martin Kihlgren
More information about the memcached