Libmemcache/php-mcache issues

[Garrett Rooney]

Russ Garrett wrote:
> OK, today I tried moving our live site over to the mcache PHP extension. 
> Previously we were using the PECL extension with a custom multi-server 
> wrapper, however this caused problems with failover (i.e., it doesn't), 
> and we'd much rather be running on libmemcache which seems like a much 
> more reliable solution.
> 
> So the setup is:
> 
> mcache version 1.2.0-beta8
> libmemcache version 1.4.0.b5
> memcached version 1.1.11
> 
> We are (or rather, were, since we've rolled back) getting loads of 
> errors like this (probably 5 per second):
> 
> Sep  4 15:54:20 local at www5 httpd[3256]: [error] PHP Fatal error:  
> memcache(4) protocol error - \r\nWEEKLY UK AUDIOSCROBBLER ARTIST 
> CHART\r\nW...
> Sep  4 15:54:16 local at www3 httpd[27668]: [error] PHP Fatal error:  
> memcache(4) protocol error - VALUE c012ba4abdd9f4f8b4ab1f1386957dd5 1 
> 4043\r\nO:5:"group":..
> Sep  4 15:54:17 local at www2 httpd[3275]: [error] PHP Fatal error:  
> read(2) failed - server unexpectedly closed connection in 
> /nfsroot/web/site/www.last.fm/core/memcached.php on line 43
> Sep  4 15:53:45 local at www3 httpd[27673]: [error] PHP Fatal error:  
> memcache(4) protocol error - no \\r before \\n in 
> /nfsroot/web/site/www.last.fm/core/memca
> ched.php on line 43
> 
> Firstly, why are these fatal errors? Surely libmemcache should simply 
> log a notice, and then close/reopen the connection to reset it, hence 
> not completely breaking the site if something goes wrong with memcached.
> 
> Is this some sort of version incompatibility, or an issue in libmemcache 
> itself? It only seems to happen on a certain (small) subset of values. I 
> hope to isolate these keys and see if I can reproduce these errors on 
> our development site.

I'm not sure if you're running into the same sort of problems we did, 
but we recently tried to use libmemcache and eventually threw up our 
hands and switched to apr_memcache instead, despite the extra 
dependencies it introduced into the system.

libmemcache seems to have this lovely habit of logging an awful lot of 
data to stderr in some situations (specifically it got into loops when 
it couldn't talk to a server for some reason where all it would do is 
keep dumping messages to stderr) and even more fun, deciding that an 
entire class of errors are fatal and exiting in response.  The beautiful 
thing is that this class of errors often seems to be a result of bugs in 
the actual libmemcache library (the big one we found was checking errno 
after a strtol call without zeroing errno out first and thus picking up 
an old errno value, thus incorrectly assuming an error had occurred when 
in fact nothing had gone wrong).

We fixed the strtol bugs locally, but the looping on failure to connect 
to the server was just too much (it didn't help that the code in 
question was rather #ifdef happy and hard to follow), so we gave up and 
ported our code.  No problems with apr_memcache so far.

-garrett