Intermittent failed cache GET since upgrade

[James M]

Hey guys, I have been running 1.2.1 for quite some time with no issues on my
site.  I run a Linux 2.6 system (compiled 64-bit) on a dual Intel Xeon box.
However, I recently did a bunch of system updates including:

memcached to 1.2.4 (from 1.2.1)
gcc to 4.2.2 (from 4.1.2)
glibc to 2.7 (from 2.6.1)
libevent to 1.4.1_beta1 (from 1.3e)
Linux kernel to 2.6.24 (from 2.6.22)

Now, I have an occasional problem where a client hits a page that does a
cache GET and it returns nothing (even though the data is populated).  But
if they refresh the page, usually in one or two attempts, the cache GET
returns the data successfully.  I also have backend standalone processes
that do cache updates throughout the day.  However, this was always the case
and never caused a problem before.  I thought that maybe one of the upgrades
caused some type of concurrency or timing issue or maybe it had to do with
the new threading support in newer memcached versions.

I tried downgrading libevent to 1.3e and memcached back to 1.2.1 (and
compiled both with the old compiler gcc 4.1.2).  However, the issue was not
resolved.  None of my code that accesses/updates the cache was changed, so
it has to be something with the system updates I did (maybe glibc or updated
kernel).

I am almost at the last resort of editing my code to implement retry logic
on cache GETs but I am reluctant to do this since it will be a performance
hit and since this was never an issue before.

Does anyone have any idea what could be the cause here?  I am completely
lost :(  By the way, most of my code base is in Perl (using Cache::Memcached
and Cache::Memcached::Managed - both are most current versions and were not
upgraded).  Thanks in advance.

James