first time user with out of memory question
sgrimm at facebook.com
Mon Jun 12 18:08:12 UTC 2006
Paul T wrote:
> I don't follow your logic.
> Either :
> "it is so simple - it can not have bugs" or
> "I just fixed the last bug in that part of code"
> - it can not be *both* at the same time, right?
The benefit of using a very simple memory allocator in memcached is not
that there is no possibility of bugs (of course any code can have bugs,
even simple code), but that it's faster. A well-written fixed-chunk-size
slab allocator takes less CPU time to manage its memory than a
well-written heap manager like the system malloc() because there is less
metadata to manipulate. It also never suffers from memory fragmentation
because fragmentation is a side effect of variable-sized allocation.
On the other hand, memcached's slab allocator replaces fragmentation and
greater CPU consumption with other behaviors that may or may not be
worthwhile tradeoffs for a particular application: potentially large
amounts of per-item overhead depending on how well your item sizes match
the chunk sizes, and per-chunk-size LRU behavior rather than true
cache-wide LRU. I attempted to reduce the first of those problems by
allowing finer-grained chunk sizes, but it's impossible to avoid completely.
It bears mentioning that the slab allocator is a compile-time option in
memcached. If those tradeoffs are not worthwhile for your application,
you can recompile it to use the system malloc().
> Unfortunately, memcached is an asynchronous
> application (based on libevent). For that kind of
> applications (timing/pattern/load/OS sesnitive) -- "it
> runs in my environment" does not mean that it would
> work for a different setup.
I completely agree with you: it's always advisable to make sure a
particular tool is the one you need for a particular job. "Your mileage
may vary" is generally true of all software, asynchronous or otherwise.
If someone tries out the Facebook version of memcached and finds that it
doesn't work as well as the stock 1.1.12, they should absolutely use
1.1.12. (And send a message to the mailing list detailing their
experience in case it's a sign of a bug that needs fixing.)
More information about the memcached