Listing all keys on a server: why?

[Serhat Sakarya]

How would you deal with entries that depend indirectly on user jsmith?

A situation I'm trying to solve involves a gallery of up to 50k items that
can be sorted by a limited number of criteria. Memcached could store each
page (of 12 items) as function of order type and page number. The problem is
that whenever any of these items change, all relevant pages must be
invalidated (e.g. pushing an item to rank #1 will invalidate all following
pages).

One approach I'm considering involves 1. adding timestamps for each cached
entry and 2. keeping track of invalidation times for each group:

# cached entries as function of order and page
"xxx-rank-5" --> { 1000s, array(...) }         // invalidated
"xxx-rank-20" --> { 1080s, array(...) }       // valid
"xxx-rank-123" --> { 900s, array(...) }       // invalidated
"xxx-name-500" --> { 1040s, array(...) }   // valid
....etc

# meta-entries
"group-xxx-rank" --> { 1050s }
"group-xxx-name" --> { 1010s }

In this scenario, each page load would involve two memcache requests
followed by a timestamp delta check. If invalid, the entry can be replaced
with fresh values.

Such an approach might emulate the namespaces concept a bit...

The general problem with this case is that when there are many changes (e.g.
lots of people voting), the effective TTL for each entry decreases and so
does performance.

Maybe this is a bit of an extreme case though... however, I'm somewhat
curious about the merits of writing a custom storage engine (ab)using the
memcache protocol to return dynamically generated data in response to "get"
requests. This may give many more possibilities for optimization, especially
in those cases where a full-featured RDBMS is not necessary (e.g. keeping
track of visitor activity).

Regards,

Serhat

On 7/27/06, Jacob Coby < jcoby at listingbook.com> wrote:
>
> I think a lot of that comes from the lack of namespaces in memcached. You
> can't just say 'invalidate all cache entries for user jsmith.'  ...
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20060727/862df7e4/attachment.htm