groups of keys and dependent keys

[Steven Grimm]

Sounds good if you can make it work efficiently. Here are a few 
questions I'd want to see answers for before anyone starts implementing 
this:

A key and its dependencies may be spread across multiple servers, 
memcached being a distributed system. How will a server know when it's 
safe to delete something? Bear in mind that at some installations (ours 
included, but others as well) "multiple servers" means well in excess of 
100, so "broadcast to all the other servers when something happens" is 
not an option unless you're VERY frugal in the use of broadcasts.

What happens when a server with one of the dependent keys crashes or its 
memcached instance is restarted?

Can you set different expiration times on different keys in a dependency 
graph? What happens if the parent key's expiration time passes and you 
try to fetch a child key? (Expiration is lazy in memcached.)

What will the performance impact be on operations not involving key 
dependencies? Busy memcached installations handle a couple tens of 
thousands of requests a second, so whatever the implementation for this 
looks like, it needs to be able to keep up with that kind of load.

Don't get me wrong -- if what you describe can be made to work well, it 
will definitely be useful! But I suspect it'll be trickier than it 
appears to make it bulletproof.

-Steve


Martin Kihlgren wrote:
> Hello list.
>
> My name is Martin Kihlgren, and I am currently employed by a company
> that heavily relies on memcached.
>
> We are doing lots of caching, simply.
>
> And, unfortunately, a whole lot of invalidations.
>
> Today we got to know of
> http://search.cpan.org/dist/Cache-Memcached-Managed/lib/Cache/Memcached/Managed.pm#SYNOPSIS
> and I thought about the implications.
>
> It would be so very much nicer to have this functionality built into
> memcached :O
>
> The scheme I would imagine is:
>
>  * The addition of key dependency, in at least one level.
>    * A key can be declared depending on another key.
>  * A key can not be dropped unless all keys depending on it are
>    dropped.
>    * If a key having dependent keys MUST be dropped, the whole
>      set of dependent keys are dropped as well.
>
> This would make it very easy for us to create group keys having
> dependent keys, and then just delete the group key thus forcing all
> dependent keys to be dropped.
>
> Much simpler and more efficient than storing the set of keys in the
> group in memcache (thus having to worry about memcache dropping the
> group key before the individual keys in the group are dropped) and
> also much better than storing the group keys on disk or in db.
>
> Is this something you have discussed? Or is it already in some edge
> version of memcached? Or is it something that is possible to do with
> the current codebase?
>
> regards,
> //Martin Kihlgren
>
>