Dropping Keys...?

[Skylos]

Greetings.

This week I have attempted to impliment using memcached as an an
antifraud information cache, backed by persistant storage in mysql
database.  I need a fast cache because the system is very large and
busy.  So busy, in fact, that the systems routinely are running at the
max their 100bTX network cards can handle - full input from the nfs
server, and full output to the web.  They are not consuming all of
their processor though.

I configured each of the 10 farm servers, each with a 64 meg cache on
the internal network interface.  The systems share configuration
files, so each of them gets the same module memcached connect that
lists all 10 servers.  I concluded initial testing on a staging
server.  things were working as expected, though I once in awhile
noticed dropped keys - or at least, values I expected to be there that
were returning undefined.  thinking it was probably just randomness, I
unfortunately ignored it.

When I brought it online, disaster struck.  I expected an initial
flurry of db hits to populate the cache - but they didn't stop!  There
were so many cache misses that my code that connects to the db to
repopulate missing elements went nuts, connecting way too much and
resulting in overloading the database's connection pool.

I'm still deleting error messages from my email.

So, what do you think went wrong?  Why was I seeing data apparently
drop from the cache?  with ~1000 siimultaneous users storing ~25 small
(5-10 byte) keys per user, was I saturating my cache memory?  Was the
excessive traffic on the interfaces causing delays or dropped packets,
that the memcached client would just return undef instead of getting
the data?

Thoughts?  Will adding network cards (or upgrading to gigabit on the
back end) fix the problem?  I hope that your collective expertise can
shed some light on it.  I look bad because I failed disastrously in
this implimentation after pushing hard for using memcached, I need to
know what went wrong.  I sure hope it was hardware overload.

David