Read vs. write performance (sloppiness in memcache)

[Brad Fitzpatrick]

I've been investigating the findings that writes are faster than reads.

I did a bunch of network sniffing and found the problem is how memcached
returns results.  It sends way too many packets, resulting in tons of
round-trips.

In the first case, where the cache is empty, and the client sets 500 keys,
it does a get (one packet), and the server sends back a reply saying it
has nothing (one packet), the client sets it (one packet), and the server
says okay (one packet).  Every packet in the transaction except the setup
and teardown is a PSH+ACK.  That is, it's sending new data and
acknowledging the data it just got.

However, memcached is a pig when it comes to sending results.

Client:   get foo, bar
Server:   I have foo!
Client:   Okay, I heard you.  I'm bored.  (ACK)
Server:   Here is foo.
Client:   Okay.  I'm still waiting (ACK)
Server:   I have bar!
Client:   *yawn* (ACK)
Server:   Here is bar.  And we're done.
Client:   Okay. (ACK)

It should be:

Client:   get foo, bar (PSH)
Server:   I have foo and bar.  here they are.  we're done. (ACK,PSH)
Client:   Okay. (ACK)

We'll fix this.

In the meantime, don't let this deter you from using memcache.
LiveJournal is running with this "slow" code and it's still ridiculously
fast (and way faster than a database).  This finding just means we can
go even faster.


- Brad