<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
You are actually noticing a couple different things. First, the
difference between bytes_used and the maximum size of the cache:
memcached's memory manager is optimized for speed rather than space
efficiency. It allocates memory in 1MB slabs, which are divided into
fixed-size chunks. The possible chunk sizes are determined at startup.
(Run the 1.2 version with "-vv" to see a dump of the sizes.) That means
that, especially for large objects, a certain amount of memory can be
wasted on each item in the cache.<br>
<br>
220MB of data out of 300MB maximum sounds low to me; we typically see
about 85% memory efficiency. But if your object sizes are just right,
you might get efficiency that low. Try playing with the -f and -n
options in 1.2 to tune the chunk sizes; this can be a huge win
especially if you have a lot of fixed-size objects in your cache.<br>
<br>
As for memcached exceeding its memory limit, the -m option only
controls the maximum amount of memory used by the cache itself. As you
guessed, each incoming connection also eats some memory. If you are
sending large "get" requests for lots of keys, memcached needs to
allocate enough buffer space for the entire request. There is code in
1.2.0 to release those buffers if they exceed a certain size, but even
so, with enough connections you'll definitely see an increase in memory
used. An extra 70MB doesn't sound out of line at all; you don't say how
big a "huge" number of connections actually is, but for example, on the
memcached instances where we aren't using UDP, with about 34000
connections we see an extra 950MB of memory used above the configured
limit. (Which is a major reason we added UDP support in the first
place.)<br>
<br>
There's one other place where memcached can exceed its memory
limit: if a "set" request comes in whose object size requires a fixed
chunk size that hasn't been seen yet, memcached will always allocate a
1MB slab for objects of that size, even if the cache has already
reached its
size limit. That ensures that you don't get bogus "out of memory"
errors after filling up your cache with, say, 1000-byte objects, then
sending in a 1500-byte object.<br>
<br>
It's arguable that rather than allocating memory above and beyond the
cache, memcached should instead free stuff from the cache to make room
for connection data, so as to make the best use of the available
memory. Truth be told we would probably prefer it that way, but for now
we just keep track of roughly how big the processes get and lower our
limits accordingly.<br>
<br>
Assuming you have enough physical memory, you might try creating, say,
a 100MB instance and seeing how big it gets. As far as I know -- and we
use memcached VERY heavily -- it does not actually leak memory, so its
size should reach a stable equilibrium if you let it run for a while.<br>
<br>
Memcached's memory allocation is definitely an area that we (Facebook)
will be looking at improving at some point. It is already much better
in 1.2 than in 1.1.x, but that was just a matter of fiddling with some
settings rather than a fundamental algorithmic change. 85% memory
efficiency is good but 99% would be better! We have a few ideas about
how to get there without losing memcached's hard-won CPU efficiency,
but some experimentation will be required.<br>
<br>
-Steve<br>
<br>
<br>
<br>
Olga Khenkin wrote:
<blockquote cite="mid0J84009KZ2HUD291@mtaout3.barak.net.il" type="cite">
<meta http-equiv="Content-Type" content="text/html; ">
<meta name="Generator" content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
text-align:justify;
line-height:18.0pt;
font-size:10.0pt;
font-family:Arial;}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
<div class="Section1">
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;">Hi
all, <o:p></o:p></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;">I
have the following problem with memcached: I run it with certain memory
limit,
let’s say <i><span style="font-style: italic;">–m 300</span></i> (300
Mb). In a few hours it reaches 370 Mb and continues to grow. <o:p></o:p></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;">When
I get stats from this instance, it shows max_available between 300 and
320 Mb,
and bytes_used about 220 Mb. What happens with the rest of memory?<o:p></o:p></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;">Our
site works with memcached through PHP Memcache extension, in an array
of 20
instances working together. It works with huge number of connections,
so I
would suspect connection structures of eating the memory, but in times
of
relatively low traffic, when connections get released, the memory is
not.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;">We
basically use version 1.1.12, but today I tried 1.2 and met the same
problem. So
if it’s a bug in memcached, it wasn’t fixed. But maybe I just use
it the wrong way?<o:p></o:p></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;">Maybe
somebody got the same problem? Or maybe you have any ideas about the
cause of
this memory issue? We’re happy to use memcached, but with those memory
issues we have to restart it every few days and, naturally, lose all
cache at
restart.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;">I
will be grateful for any ideas, before I enter deep debugging…<o:p></o:p></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size: 10pt;">Olga.<o:p></o:p></span></font></p>
</div>
</blockquote>
<br>
</body>
</html>