Dropping Keys...?

[xing at fantasyball.net]

You need to provide the list with more info so the developers can get a 
better grasp:

1) Which version of memcached?

2) What type memcache client? Version?

3) Libevent version?

4) Are you saying you have 10 memcached servers with 64mb each so you 
only have 640MB total? If you are saturating 100Mb traffic on queries, I 
  imagine that's a lot of data to be cached. What happens when you give 
128MB to each memcached?

5) Check "stats" output from memcached server and look at the cache 
misss/hit raitio to see if something is wrong or memory exhausting is 
forcing premature cache flush. Pasting some "stats" info here would help 
a lot.

Are you memcache client app servers synced in terms of clock? Just to 
make sure stuff gets expired on expected time.

Just give the list more info. I'm not a developer but even I have hard 
time sorting out what you have and what you are using.

Skylos wrote:
> Greetings.
> 
> This week I have attempted to impliment using memcached as an an
> antifraud information cache, backed by persistant storage in mysql
> database.  I need a fast cache because the system is very large and
> busy.  So busy, in fact, that the systems routinely are running at the
> max their 100bTX network cards can handle - full input from the nfs
> server, and full output to the web.  They are not consuming all of
> their processor though.
> 
> I configured each of the 10 farm servers, each with a 64 meg cache on
> the internal network interface.  The systems share configuration
> files, so each of them gets the same module memcached connect that
> lists all 10 servers.  I concluded initial testing on a staging
> server.  things were working as expected, though I once in awhile
> noticed dropped keys - or at least, values I expected to be there that
> were returning undefined.  thinking it was probably just randomness, I
> unfortunately ignored it.
> 
> When I brought it online, disaster struck.  I expected an initial
> flurry of db hits to populate the cache - but they didn't stop!  There
> were so many cache misses that my code that connects to the db to
> repopulate missing elements went nuts, connecting way too much and
> resulting in overloading the database's connection pool.
> 
> I'm still deleting error messages from my email.
> 
> So, what do you think went wrong?  Why was I seeing data apparently
> drop from the cache?  with ~1000 siimultaneous users storing ~25 small
> (5-10 byte) keys per user, was I saturating my cache memory?  Was the
> excessive traffic on the interfaces causing delays or dropped packets,
> that the memcached client would just return undef instead of getting
> the data?
> 
> Thoughts?  Will adding network cards (or upgrading to gigabit on the
> back end) fix the problem?  I hope that your collective expertise can
> shed some light on it.  I look bad because I failed disastrously in
> this implimentation after pushing hard for using memcached, I need to
> know what went wrong.  I sure hope it was hardware overload.
> 
> David