Holding Collections in memcache

[Chris Hondl]

One thing that we do is store collections in multiple keys.

For example we keep a collection of all users currently running the latest
release.  This collection has about 30,000 items.  We have a set of numbered
keys "release_user_00" through "release_user_99" that each contain a comma
separated list of ids.  We take the id of the user module 100 and store the
id into the corresponding key.  Makes it relatively lightweight to update
the date, and easy to pull down the entire set as necessary.

We use this approach for handling sets of objects in several parts of our
system.

Chris


On 7/5/07, Steve Grimm <sgrimm at facebook.com> wrote:
>
> On 7/5/07 5:12 AM, "Paul Stacey" <paul.stacey at fatsoma.com> wrote:
> > Events contains an array of event
>
> A better way to deal with this kind of thing is with a two-phase fetch. So
> instead of directly caching an array of event data, instead cache an array
> of event IDs. Query that list, then use it construct a list of the keys of
> individual event objects you want to fetch, then multi-get that list of
> keys.
>
> In addition to greatly increasing the maximum size of the list (memcached
> has a 1-megabyte hard limit on item size, and obviously you can fit a lot
> more IDs than full data in that limit) you will also, if you're using
> multiple memcached servers, spread the load around by virtue of your
> multi-get hitting basically every server once your key list is a certain
> size. If you stick all the data in one frequently-requested item, then you
> will pound the particular server that happened to get that item while all
> the others sit idle.
>
> Another advantage of a scheme like this is that you can update an item's
> data without having to read then write every list that contains that item.
> Just update it by ID (like you'd do in your database queries) and all the
> lists that contain it will magically get the correct information.
>
> All the above said, it is true that dealing with lists of things is not
> currently memcached's strong suit. For example, there is no "append"
> operation. It is much better at caching individual objects.
>
> -Steve
>
>


-- 
http://avatars.imvu.com/chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20070705/bebeb2c3/attachment.htm