namespace bids

[Brian Moon]

Let me preface this by saying that I have not opened up the code to look 
at how this might be done.  Likely, Brad and co. have already run 
through these ideas in their head.

We (dealnews) have pondered adding wild card deletes to memcached.  The 
servers would have to walk the cache for any matching keys though.  That 
could be bad.

So, I was pondering namespaces with my spare cycles the other day.  This 
would mean the servers would have to keep a list of keys per namespace. 
  It could then simply delete those namespaces.  I guess it could also 
keep some sort of flush time for the namespace.  It could simply set 
this when the delete came in and then any requests for that namespace 
would return nothing the cache was older than the namespace flush time. 
  Adds would need to be aware of this too so they would not fail.

Either solution would require that the clients send the deletes to every 
server.  That means more than memcached will have to be changed to 
accommodate this.  It also means that memcached would need to 
accommodate clients that are not namespace aware.

We have worked around this in our application.  Any cache we create that 
is not quantifiable (like unnamed sql query cache) is logged to a mysql 
table using insert delayed.  We store various information about it that 
could be considered a namespace.  We then query this table and delete 
any keys found in there.  We then delete any rows from the table older 
than when we queried that match our criteria.  It leaves a small window 
of error, but so far, its not a problem.

For other data, we simply delete all keys we can create such as article 
cache.  We just loop through all known article ids.  Its quite fast. 
Have you tried looping through all your "fred" data and removing it? 
You might be surprised.

Brian Moon
dealnews.com
--------------
How to go broke saving money.
http://dealnews.com/

Cahill, Earl wrote:
> For some time we have had the same memcached servers handling several 
> different nodes of information.  As we have seen memcached handle our 
> loads, we have cached more, and memcached has yet to miss a beat.  We 
> currently have millions of items cached, though, yes we could probably 
> expire some of those.
> 
>  
> 
> Recently we added a new ‘node’ of stuff, call it fred.  Well, it turns 
> out fred changes a bit more than the rest, and sometimes we would like 
> to wipe everything that has anything to do with fred.  Problem is the 
> key names in fred are rather dynamic, combining several types of data, 
> one of which is an ordered list.  That makes it kind of hard to know all 
> of fred’s keys.  There are potentially a few hundred thousand such 
> keys.  It also doesn’t appear to be possible to walk everything 
> currently in a memcached instance for sufficiently large slabs.  We 
> don’t want to restart the whole instance, as we would lose several 
> million confs and have to slam our netapp to rewarm the cache.  Ttls are 
> fine, but it would be nice to just wipe the whole node.
> 
>  
> 
> Seems like namespaces are what we want here, and I think we’d be willing 
> to pay for it.  As a pre-empt, the perl API does not really have name 
> space support.  In my reading, all it does is combine your ‘namespace’ 
> to keys, which as far as I can tell, aids little to our problem.
> 
>  
> 
> So my questions are, who can we pay to develop namespace support, and 
> how much would it cost?  We would want all the code contributed back to 
> trunk, and we would like to work off of trunk and not some branch or the 
> like.  We would like to be able to handle lots of namespaces, like 
> adding a namespace wouldn’t be much more taxing than adding a normal key.
> 
>  
> 
> Maybe just a question for brad, but if namespace support just happened, 
> in a good way, and all you had to was perhaps merge a few things, would 
> you be ok with it?
> 
>  
> 
> I think namespaces could help improve what I think is a pretty great 
> product.  Don’t want to start a flame war, with other potential uses, 
> but I think there are many.
> 
>  
> 
> Thanks,
> 
> Earl
>