Holding Collections in memcache
paul.stacey at fatsoma.com
Thu Jul 5 16:44:28 UTC 2007
Steve, thanks for the advice.
Chris, I did think of a similar approach. I was thinking of putting the
items in chunks. For example I would use event_0_500 for the first 500
then 501_1000 for the next and so on. However, as ours is a list that
represents a timeline i.e. from todays date to some future date, the
problem is that a new object could land inside an existing chunk thus
pushing the edge of the list into the other list meaning all the chunks
would need updating.
Chris Hondl wrote:
> One thing that we do is store collections in multiple keys.
> For example we keep a collection of all users currently running the
> latest release. This collection has about 30,000 items. We have a
> set of numbered keys "release_user_00" through "release_user_99" that
> each contain a comma separated list of ids. We take the id of the
> user module 100 and store the id into the corresponding key. Makes it
> relatively lightweight to update the date, and easy to pull down the
> entire set as necessary.
> We use this approach for handling sets of objects in several parts of
> our system.
> On 7/5/07, *Steve Grimm* < sgrimm at facebook.com
> <mailto:sgrimm at facebook.com>> wrote:
> On 7/5/07 5:12 AM, "Paul Stacey" < paul.stacey at fatsoma.com
> <mailto:paul.stacey at fatsoma.com>> wrote:
> > Events contains an array of event
> A better way to deal with this kind of thing is with a two-phase
> fetch. So
> instead of directly caching an array of event data, instead cache
> an array
> of event IDs. Query that list, then use it construct a list of the
> keys of
> individual event objects you want to fetch, then multi-get that
> list of
> In addition to greatly increasing the maximum size of the list
> has a 1-megabyte hard limit on item size, and obviously you can
> fit a lot
> more IDs than full data in that limit) you will also, if you're using
> multiple memcached servers, spread the load around by virtue of your
> multi-get hitting basically every server once your key list is a
> size. If you stick all the data in one frequently-requested item,
> then you
> will pound the particular server that happened to get that item
> while all
> the others sit idle.
> Another advantage of a scheme like this is that you can update an
> data without having to read then write every list that contains
> that item.
> Just update it by ID (like you'd do in your database queries) and
> all the
> lists that contain it will magically get the correct information.
> All the above said, it is true that dealing with lists of things
> is not
> currently memcached's strong suit. For example, there is no "append"
> operation. It is much better at caching individual objects.
More information about the memcached