Further reliability patches for libmemcache, mcm_get_line()

John McCaskey johnm at klir.com
Mon Dec 13 15:23:51 PST 2004


Sean,

I sent you a patch late last week which improved the reliability of
mcm_get_line by checking for the \r\n at the end of a response, and if
not present trying to read more data from the socket.

Unfortunately as I mentioned then it wasn't a 100% solution as the data
in a stored object may have \r\n's that just happen to come up at the
end of a read, but do not represent the actual end of the data.

Here is a further patch which passes the type of command to
mcm_get_line, and then inside the function checks for a proper end of
line response.  Ie, for a get command, the data is only all read if a
END\r\n is found, or an error response is sent.  For a store command a
STORED\r\n, or NOT_STORED\r\n is valid.  

This results in a few extra comparisons each time data is read from the
server but in my use it improves reliability considerably (actually
wasn't usuable before as not all data would get read, then on the next
read command a protocol error would return, and the entire app would
exit(1)).

I'm only aware of one situation where this new patch could still result
in a protocol error with valid feedback from the memcached server.  If a
value contains the sequence "END\r\n" in the actual data and a read
command just happens to finish there, but we are still awaiting the
final END\r\n.  I believe the only way around this is to parse through
the VALUE ...\r\n responses as each comes in and make sure that the
amoutn of data read is equal to the amount of bytes specified in the
response, plus the other overhead.  The performance hit and parsing for
this doesn't seem worthwhile to me as I don't expect to ever be storing
anything that could trigger this scenario.  So I'll leave it to you to
decide whether this needs to be addressed or not.  

I think obviously the best way to address all of this is the binary
protocol, but for now this patch has given me the reliability I need.
Hope it helps.

The patch file is against the memcache.c and .h 1.1.0 presently up on
your site.

-- 
John A. McCaskey
Software Development Engineer
Klir Technologies, Inc.
johnm at klir.com
206.902.2027
-------------- next part --------------
A non-text attachment was scrubbed...
Name: reliable_get_line.patch
Type: text/x-patch
Size: 15043 bytes
Desc: not available
Url : http://lists.danga.com/pipermail/memcached/attachments/20041213/f8e79981/reliable_get_line.bin


More information about the memcached mailing list