binary protocol notes from the facebook hackathon
alexs at advfn.com
Wed Jul 11 10:20:34 UTC 2007
On 11 Jul 2007, at 10:38, Dustin Sallings wrote:
> On Jul 11, 2007, at 2:11, Alex Stapleton wrote:
>> Packing multiple responses into a single UDP packet seems pretty
>> doable but the server is going to need some information about how
>> many keys its being asked for to do this with any efficiency.
>> Maybe we need a message to specify the start of a multikey request
>> which contains the number of requests that will be made, followed
>> by the requests? (In fact, even for TCP it makes a lot of sense,
>> doesn't need this NOOP command stuck on the end that way.)
> UDP responses must match UDP requests. Each request has an opaque
> that must line up in the response. Multiple UDP packets may be
> sent in response to a single UDP request. I believe that covers
> all of it.
> The NOOP is a command that will have a response so you can tell
> when your quiet commands have completed.
>> Also regarding UDP, is the server going to be capable of parsing
>> multiple messages in a single packet?
> Yes, this is part of the compromise of killing of multikey gets.
>> Also, what does the server do if the NOOP command gets sent in its
>> own packet, and that packet goes missing? Sit in "corked" mode
>> forever? Treat it as an uncork? What if one of the packets with a
>> get request in goes missing? Does the server report an error? Does
>> the server just pretend like the client never requested that key(s)?
> What do you mean missing? Either the packets on a TCP connection
> are delivered in order to the application, or the connection is
> invalid and should be shut down, right?
Sorry, I was referring to UDP requests. I am concerned about exactly
how the server is intending to deal with UDP requests with this new
protocol. Although it sounds like you guys have thought it through
already :) I know UDP functionality is at the very least important to
I am wondering how the server will be able to handle a scenario like
client sends request for n keys + NOOP.
NOOP goes missing somehow.
server sits in corked mode waiting for an uncork.
client times out the request.
client sends request for m keys + NOOP.
server gets NOOP and responds to n+m keys.
The client is going to have to compare every incoming key to every
key in its requested list to make sure it really wants them? The
client should send a NOOP before every multikey request to make sure
the server is uncorked? (Remember that this UDP port could previously
have been used by a client that had a NOOP send failure and then just
exited. Leaving a bunch of requests pending.)
I know I am coming up with some rather elaborate failure scenarios
here but I feel that previously memcached handling of UDP was not
that great (atomic operations become rather tricky) and now it seems
to only be getting worse, or at least less well specified.
alexs at advfn.com
T: 0207 0700 956
More information about the memcached