Listing Keys in Cache ??

[Todd Fisher]

Hi all,

  I've been working on implementing a ESI (Edge Side Include) Proxy caching
server.   The idea is very similar to SSI (Server Side Include), except the
includes are remote instead of local.

There's a very nice write up of using SSI in this context with nginx and
memcached here =>
http://blog.kovyrin.net/2007/08/05/using-nginx-ssi-and-memcache-to-make-your-web-applications-faster/

My project is mongrel-esi (http://code.google.com/p/mongrel-esi) and I'd
like to use it as an example of how one can use memcache and still
invalidate URLs using regular expressions.

The basic idea is to store the regular expressions that have been picked up
by processing an invalidation rule in memcache.   Before retrieving a key
from memcache checking if the key matches any of the regular expressions in
the invalidation-key and that the object being cached is older then the
invalidation rule that matched.   We can probably be clever about how we
name the invalidation-key so that it too can expire.   So long as the
invalidation-key doesn't expire before a predetermined max-expiration time
that all keys get.  Then we can be certain that either before we request a
key, or the keys expiration time the objects marked to expire by the regex
will expire as requested by surrogate.

For me this realization is great because it means I can finish implementing
the proxy server using memcached as the cache storage.   Hopefully, it also
will be a helpful solution for others looking to access all the keys.   And
of course, I may  be  missing something important, so please correct me if I
have.

With this solution I think it's important to get the clever part about
making sure the expire rules have an expire time on them that is greater
then or equal to the expire time of all keys.  There may also be some ways
to name the keys based on the specific domain to have more then one
invalidation key to avoid letting it grow into a large dataset that needs to
be retrieved everytime.   It also means you have two hits on memcache per
key lookup instead of the one before...   There are probably other
techniques for solving this problem...  Just hoping it makes sense and is
useful to others...

-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20070909/a1bf8025/attachment.html