GNU malloc vs. slab alloc
jeff.horner at vanderbilt.edu
Thu Apr 27 19:52:45 UTC 2006
Not having looked at your patch, I have a simple question: Does you
patch support storing objects of size > 1Mb? I ask b/c I submited a
patch that uses the slab allocator for objects < 1Mb, but for larger
sizes it uses malloc().
Torsten Foertsch wrote:
> what is the best way to submit patches to memcached and get them accepted?
> I have a patch for revision 260 ready that implements:
> - item expiry relative to the least recent access
> If a negative expiry timeout is set the actual expiry time is
> - a "touch" operation
> adjusts it->time. This is actually a "get" without transferring data.
> - active expiry
> expired items are removed by delete_handler() a function that is called
> every 5 seconds.
> - using system malloc instead of the slab allocator is turned from a compile
> time define into a command line option, see below.
> - a new command line option that emulates a full cache, i.e. all set
> operations return error (for testing clients)
> Situations when GNU malloc isn't bad at all
> After reflecting on the fragmentation problem it occurred to me that with
> active expiry and most of the items having an expiry time assigned using GNU
> malloc wouldn't be that bad. I wrote then a test program that stresses the
> server with a set operation with a random key followed by 2 get operations
> with keys randomly selected from the 200 recently generated keys. Between
> these operations the program sleeps for 0.15 sec. Thus, the server is
> stressed with approx. 2 set and 4 get operations per second. Item sizes vary
> randomly between 1 byte and 1 megabyte. Expiry times vary from 300 to 900
> seconds after last usage.
> The server was started with the -M option and the test program counts failed
> and succeeded set operations.
> The program ran for over a week against 3 servers, GNU malloc with 256 and 320
> MB limit and slab alloc with 384 MB limit. Each server processed over a
> million set operations. The out of memory rate for the slab allocator with
> 384 MB was more than 6 times higher than that for GNU malloc with 256 MB. GNU
> malloc with 320 produced only 0.05% of the out of mem errors of the slab
> allocator, in fact only 29 errors out of 1,286,612 set operations.
> To estimate the effect of fragmentation on the error rate I drew a chart
> showing the number of errors depending on the number of set operations. For
> the slab allocator I expected a very straight line for there is no
> fragmentation in principle. If fragmentation was an issue for GNU malloc I
> would have expected a nonlinear curve with a lower slope a t start and a
> higher slope at the end. But that did not happen. The curve is very straight,
> as well.
> To draw a conclusion I would say that for this usage pattern GNU malloc is to
> be preferred because it utilizes the memory much better than the slab
> allocator and does not suffer from fragmentation problems.
> What is wrong with these considerations?
> memcached.png: the chart
> bigpatch: the patch
> x.pl: the test program
Jeffrey Horner Computer Systems Analyst School of Medicine
615-322-8606 Department of Biostatistics Vanderbilt University
More information about the memcached