binary protocol notes from the facebook hackathon

[Steven Grimm]

Disclaimer: I was not in the binary protocol discussion at the hackathon 
apart from popping in for a few minutes at one point, so I'm coming at 
this as a relatively uninformed observer.


Dustin Sallings wrote:
>> I'm a little confused by your use of the term cork. As in the 
>> TCP_CORK flag right? TCP_CORK will have no effect on UDP sockets for 
>> obvious reasons ;)
>     Yes, this is a specific optimization for TCP.

I thought the term "cork" wasn't a reference to TCP_CORK specifically, 
but more to the concept of not sending output until an uncork-ish 
command. That would apply to UDP just as well as TCP, and is how we 
retain the system CPU time savings that multigets provide in the current 
protocol: we can do one giant sendmsg() call for TCP, or one sendmsg() 
per outgoing packet for UDP, even if there are a bunch of results packed 
together. As in the current memcached, if you do that there is never any 
need to do an actual TCP_CORK; the corking happens at the application 
level, which is much more desirable since it minimizes system calls.

Now, having just written that, I see a problem with it, maybe one that 
you guys discussed when I wasn't in the room: it implies that we have to 
accumulate incoming commands without processing them until we have the 
final uncork-ish command. On the other hand, maybe that's not so bad, 
given that it's exactly what happens in the protocol today when a large 
"get" comes in; it gets buffered up until we reach the end of line, and 
only then parsed.

>     UDP responses must match UDP requests.  Each request has an opaque 
> that must line up in the response.  Multiple UDP packets may be sent 
> in response to a single UDP request.  I believe that covers all of it.

And I'll add one note to that: we should get our terminology *very* 
straight so we don't confuse anyone (including ourselves), since we're 
talking about "requests" and "responses" at two different levels in UDP 
land. Can we standardize on something like this?

UDP *request* (always 1 *packet*) contains some number of binary 
protocol *commands*, always ending in a non-quiet *command*

UDP *response* (may span multiple *packets*) contains at least the 
*results* of all the non-quiet *commands*, and possibly other *results* 
from quiet *commands* as well.

TCP has no *request* or *response*, just *command* and *result*.

I am not attached to those particular terms, I just think we need to put 
a stake in the ground and pick different words for those different 
things to make things easier to follow down the road.

-Steve