experiences setting max object size > 1MB
dormando at rydia.net
Mon Mar 17 07:18:49 UTC 2008
The higher the max page size is set, the more slab classes it'll
generate. As the class size increases the memory overhead per class
Each class needs a minimum of one POWER_BLOCK-sized page allocated to
it. So as you increase POWER_BLOCK in size, it needs more POWER_BLOCK's
allocated up front. Just as an example; normally you'd have 34 or 39
slab classes of 1MB each. Then you might need 70 of 16MB each just to
get off the ground.
So as a side effect of the way the slabber is designed, POWER_BLOCK
turns into suck crank. Also, if you're loading 5 megabyte objects and
demarshalling them for a web request, you have larger issues. Might as
well spool those monsters to file or persist them in memory... I think
it appears to just be that for web requests, 1MB being the upper bound
of what you can fetch in a single go is a handy sanity check.
Nick Grandy wrote:
> Hi all,
> I've been searching around for recent experiences storing larger than
> normal objects in memcached. Since I didn't find much, I'm hoping
> people will share their experiences.
> I've installed 1.2.5 and edited the slabs.c file as follows:
> #define POWER_BLOCK 16777216
> This has the effect (I believe!) of setting the max object size to
> 16MB, and it seems to work. Running with the -vv option shows that
> there is a nice distribution of slabs created up to 16MB, and
> memcached does work. So I'm optimistic.
> Now here are the questions. Have other people used this technique
> successfully? What sort of 'gotchas' might be waiting around the
> corner? Perhaps related, I am curious why the memcached protocol
> limits the max size to 1MB. Would it make sense to make the max slab
> size a command line option?
> I guess not that many people need to store large objects, or this
> would come up more often. In my case, I am running a web app in Rails
> that makes use of large hashtables. I realize there are other
> workarounds; eg, I could refactor so the data is stored in smaller
> chunks <1MB ( but that seems fairly arbitrary); or share the lookup
> tables between processes by dumping to a file. But, sharing via
> memcached seems more flexible and robust - assuming it doesn't blow
> up! I'm running on EC2 with ample memory, so the potential
> inefficiency of allocating large slabs is not currently a concern.
> So, in short, is setting a large slab size a reasonable thing to do,
> or am I making a big mistake?
> Thoughts appreciated.
> And huge thanks to the memcached contributors for such a valuable tool!
More information about the memcached