binary protocol notes from the facebook hackathon
sgrimm at facebook.com
Wed Jul 11 10:07:43 UTC 2007
Disclaimer: I was not in the binary protocol discussion at the hackathon
apart from popping in for a few minutes at one point, so I'm coming at
this as a relatively uninformed observer.
Dustin Sallings wrote:
>> I'm a little confused by your use of the term cork. As in the
>> TCP_CORK flag right? TCP_CORK will have no effect on UDP sockets for
>> obvious reasons ;)
> Yes, this is a specific optimization for TCP.
I thought the term "cork" wasn't a reference to TCP_CORK specifically,
but more to the concept of not sending output until an uncork-ish
command. That would apply to UDP just as well as TCP, and is how we
retain the system CPU time savings that multigets provide in the current
protocol: we can do one giant sendmsg() call for TCP, or one sendmsg()
per outgoing packet for UDP, even if there are a bunch of results packed
together. As in the current memcached, if you do that there is never any
need to do an actual TCP_CORK; the corking happens at the application
level, which is much more desirable since it minimizes system calls.
Now, having just written that, I see a problem with it, maybe one that
you guys discussed when I wasn't in the room: it implies that we have to
accumulate incoming commands without processing them until we have the
final uncork-ish command. On the other hand, maybe that's not so bad,
given that it's exactly what happens in the protocol today when a large
"get" comes in; it gets buffered up until we reach the end of line, and
only then parsed.
> UDP responses must match UDP requests. Each request has an opaque
> that must line up in the response. Multiple UDP packets may be sent
> in response to a single UDP request. I believe that covers all of it.
And I'll add one note to that: we should get our terminology *very*
straight so we don't confuse anyone (including ourselves), since we're
talking about "requests" and "responses" at two different levels in UDP
land. Can we standardize on something like this?
UDP *request* (always 1 *packet*) contains some number of binary
protocol *commands*, always ending in a non-quiet *command*
UDP *response* (may span multiple *packets*) contains at least the
*results* of all the non-quiet *commands*, and possibly other *results*
from quiet *commands* as well.
TCP has no *request* or *response*, just *command* and *result*.
I am not attached to those particular terms, I just think we need to put
a stake in the ground and pick different words for those different
things to make things easier to follow down the road.
More information about the memcached