Key Size

Philip Neustrom philipn at gmail.com
Mon Jan 8 18:27:34 UTC 2007


Most people don't need long cache keys, but I think that our
discussion has sort of assumed that we're using ascii keys, for which
250 bytes is really just 250 characters, and that's more than enough
for most of our english-speaking identifiers.  For something like
utf8, you're looking at a situation where even 70-some-odd characters
could push you over 250 bytes.

Using a hash seems like a good idea, but it's additional logic, and
there's the risk of collisions (and with it, serving up data that
doesn't belong to you or losing/corrupting data).

Steve mentions that with tens of millions of items in a cache, that
the cost of having an additional byte is too great.  If I'm not
mistaken, this is only 10MB of extra memory used, and in this
situation (10 million items), you're looking at a cache which is
likely in the 100's of GBs.  This doesn't seem like a measurable
difference to me (less than .01 percent?)  It would be a different
case if we were forcing every key to be of the maximum length, but it
doesn't s
eem like memcached does that (we use the length on a key-by-key basis).

Perhaps my thinking is wrong, but I'm not sure why we don't just bump
up from char to short, which would allow all sane key sizes (65530
bytes).

--Philip Neustrom

On 1/5/07, Steven Grimm <sgrimm at facebook.com> wrote:
> I believe the key length is stored in a 1-byte field in the item data
> structure at the moment. See memcached.h. You'll need to change it from
> "unsigned char" to "short" or "int" if you want keys longer than 256
> characters. I *think* it should work if you do that and increase
> KEY_MAX_LENGTH as you've done, but I've never tried it. (Don't have the
> source in front of me at the moment or I'd tell you the field name.)
>
> The length is 1 byte because (a) most people don't need long cache keys
> -- I think our longest one is maybe 40 bytes -- and (b) if your cache
> has tens of millions of items, saving a couple bytes here and there
> actually makes a measurable difference in cache capacity.
>
> -Steve
>
>
> Renato Silveira wrote:
> > Hi,
> >
> > I´m using memcached as a URL keyword cache. The url is the key and the
> > cached object is a string with relevant keywords of the site. Well,
> > it´s working fine for my purpose. The only problem is the limit of key
> > size to 250 bytes(because there can be url bigger than 250 bytes).
> > Well, I tried to increase the value of the constant KEY_MAX_LENGTH to
> > 2048 bytes. I recompile memcached and it´s not working. Should I have
> > to make any modification on the client API? Does anyone can helpme
> > with this issue?
> >
> > tks!
> >
> > Renato
>
>


More information about the memcached mailing list