Binary Protocol...

[James Mastros]

Sean Chittenden wrote:
> In the interests of feature growth and moving away from the  
> convenient, but rather expensive text protocol, I'd like to propose the  
> binary memcache protocol.  
I'm not clear that the protocol is that expensive, or that it matters 
terribly much.  Are your servers or users CPU-bound?  Is all that much 
CPU used in the parsing of the protocol?  Are they network bound, and if 
so, is the protocol overhead really that much more then the data you're 
slinging about?  Remember that all the techniques for forcing things 
into one packet -- disabling Nangle's Algo, all that jazz -- are 
available with textual protocols too.

Text-based protocols are easier to debug, and they're easier to extend 
by multiple people without them stepping on each-other's toes.

 > If you haven't ever worked with binary
> protocols, please refrain from the technical aspects of this discussion  
> as the potential for a bikeshed[1] is huge.  
I've worked with binary protocols some (though never a purticularly 
well-written one), and have worked with a fair number of binary file 
types, which are largely the same thing.  Some of my issues may seem 
like bikeshedding to some, but they do have a fair bit of vaildity.

> The HELLO Packet:
I'd rather refer to these as "message", and make explicit that you can 
have more then one of them in a TCP/IP packet.

>  0                   1                   2                   3
>  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> |    Version    |    Options    |  User Length  | Passwd Length |
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> |                         Key Space ID                          |
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> /                           Username                            /
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> /                           Password                            /
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
I'd prefer to see each length field go immediately before the thing that 
it's counting, and all length fileds be the same size.  (We probably 
don't need an explicit statement of endianness, but it couldn't hurt.) 
This may seem bikeshedish, but it allows for reuse of routines to pack 
and unpack them into the languge's native strings.  In perl, it even 
allows using a single pack/unpack function call.

> Options (required):
>     These bits refer to the bits in the Options Byte.
> 
>     Bit 0:    Connection provides authentication information
>     Bit 1:    This client connection requires TLS
>     Bit 2:    Disconnect if TLS can not be negotiated
>     Bit 3-7:    Not designated
What's the difference between 1 and 2?  Why have 0 different from just a 
0-length username and passwd?  What are you doing running memcached 
across a sniffable network, anyway?  Doesn't using TLS add in overhead 
more then enough to nullify any help that a binary protocal would help?

>     If Bit 2    of the Options 1 Byte is set, this value specifies the
>     expiration of a key in seconds from the Epoch.  
Either "from the UNIX Epoch" or "in seconds since Dec 31, 1969 at 
23:59:59 GMT", please.

> Options 1 (required):
Auxilury actions:
>     These bits refer to the bits in the Options 1 Byte.
> 
>     Bit 0:    If this key exists and has a relative expiration, reset
>             the expiration to be be relative to the current time.
>     Bit 1:    Request that the server delete the key after sending the
>             value to the client.
>     Bit 2:    After the server has processed this request, close the
>             connection.
>     Bit 3:    If the key exists, include the expiration of the key in
>             the response from the server.
>     Bit 4:    If the key exists, include the number of fetch requests
>             left for this key.
>     Bit 5-7:    Not designated
Bit 5: do not return the data, only do the other actions in the auxilury 
actions byte.

> Key (required):
>     The key for the given request.  Keys are not padded by a null  
> character.
There is a certian danger in allowing the user to specify keys that 
cannot be retreived by the normal (textual) protocol.  I'm really not 
sure if we should say "you get what you deserve, then", or dissallow it. 
  (For that matter, I can't quite recall if there really is such a beast.)

> The ERROR Packet:
> 
> The ERROR Packet is one of the ways a server responds to client  
> requests.  Not all ERROR Packets are fatal errors and indeed, the  
> server responds with an ERROR Packet after a STORE Packet has been  
> processed by the server.
I'm not sure this is a good idea.  Shouldn't we imply good by the lack 
of an error packet, if we wish to be efficent?

>     Major Status "S"'s available Minor Status Codes:
>         ' ':    No problems
I'd prefer this be \0 or '-'; either is more readable.

> Message Length (required):
>     If an additional error message is sent by the server, this field
>     will contain a non-zero response.
s/response/value/

> Response (required):
>     All packets from the server contain a Response Byte.  Each
>     value for the request byte is unique for the protocol version.
>     The RESPONSE packet has a request value of 'r'.
Um, this is very poorly worded.

> Additional Notes:
> 
> If a client connects and sends an invalid request that is out of bounds  
> for the protocol, the server with a plain text error message and closes  
> the connection.  The format for the plain text error response is:
> 
> ERROR [code]: [message]\n
> [custom message]\n
> <server closes connection>
I hope this just got in this spec by accident -- haven't we already 
covered this with the error packet?

	-=- James Mastros