How best to deal with lots of small objects - currently failing?

[Ben [skiddle.com]]

Hi

I must admit I'm not that clued up on the memcached workings. 

I've got a test to put strings into the cache so I'll run it after
restarting memcached then after it's warmed up a bit.

These were the memcached stats this morning, but this is only just after a
restart yesterday. I'll repost once it's been running for a bit longer as
that's when we start to get the problems:

Array ( [localhost:11211] => Array ( [pid] => 85890 [uptime] => 43464 [time]
=> 1203594632 [version] => 1.2.4 [pointer_size] => 32 [rusage_user] =>
14.426167 [rusage_system] => 55.931405 [curr_items] => 34513 [total_items]
=> 70581 [bytes] => 283048099 [curr_connections] => 1 [total_connections] =>
268597 [connection_structures] => 17 [cmd_get] => 268594 [cmd_set] => 70581
[get_hits] => 165008 [get_misses] => 103586 [evictions] => 0 [bytes_read] =>
533057498 [bytes_written] => 4530839326 [limit_maxbytes] => 1073741824
[threads] => 1 ) )

I've also noticed that we were using 'localhost' instead of '127.0.0.1' as
mentioned in another thread which was giving us connection errors on about
1% of the connection attempts.

I'll report back with stats later

Cheers

-----Original Message-----
From: dormando [mailto:dormando at rydia.net] 
Sent: 21 February 2008 07:53
To: ben at benseb.co.uk
Cc: memcached at lists.danga.com
Subject: Re: How best to deal with lots of small objects - currently
failing?

What's your cache eviction rate?

Do you get errors when set'ing new variables into the instance? Out of
memory, etc?

The big deal with memcached so far as exacerbating memory loss is when
you change the item profile over time. Extra slab pages are assigned out
of main ram as necessary. So if you initially assign 50% into some small
slabs, and 50% into some big slabs by usage, then become 90% small
slabs, your small slabs will be more likely to get evictions.

Also, I swear there's a bug somewhere. Which is why I asked the above
questions :)

Also, I second the stuff brian moon said.

-Dormando

Ben [skiddle.com] wrote:
> Hi
> 
>  
> 
> I need to cache a string (only about 120 bytes but requires lots of
> lookups to generate) - the string will always be small, and there will
> be up to 40,000 of them
> 
>  
> 
> I'm presuming that each memcached daemon only has a limited number of
> 'slots' for each size of data?? Is that correct?
> 
>  
> 
> At the moment I only have 1 large memcached daemon running (2.5Gb) - the
> problem I have is that after a while, even though the daemon is only
> using about 500mb out of its 2.5Gb allocation, these strings stop
> getting cached (ie its always a miss, I then cache it and next time i
> get a miss again - even with a long expiration time) - this is annoying
> as there is obviously enough memory available so I presume the problem
> must be the number stored rather than memory allocated?
> 
>  
> 
> I also need to store a range of other data, so would it be best to setup
> a memcached daemon that's somehow (?) tuned for this size of string and
> push all request through to that, leaving another daemon for general use?
> 
>  
> 
> Thanks
> 
>  
> 
> *Ben Sebborn***
> 
>  
> 






++++ Message Scanning REF:961-avs4-1203580368 ++++