mecached - text protocol?

[Brad Fitzpatrick]

Avva,

While I mostly agree with you, I think a binary protocol would still be
nice to have in addition.  I started working on it this morning and have
it kinda working.

In particular, I want to change:

   -- no get_multi support on the server side in the binary protocol.
      the client should just pipeline multiple gets at once and match up
      the responses.

   -- binary responses in response to binary requests.  less CPU used
      on the client.  (the top CPU eater on LiveJournal)

What I did was:

int try_read_command(conn *c) {
    char *el, *cont;

    if (!c->rbytes)
        return 0;

    /* binary protocol */
    if (c->rbuf[0] == (char) 0) {           <<<<<<<<<<< new logic
        unsigned int packet_len;
        int i;

        /* has the length header? */
        if (c->rbytes <= 4)
            return 0;

        packet_len = ntohl(* (uint32_t*) c->rbuf) + 4;
        if (c->rbytes < packet_len)
            return 0;
       ....


So if a request starts with a zero, it's a binary command, and the
first 4 bytes are the length.

I wrote up this so far:

Alternatively, a binary protocol is available.

You can intermix binary protocol requests and text protocol requests
on the same connection.  Some protocol requests (the infrequently used
ones) have only text representations.

The binary protocol header is:

  4 byte big endian length of command request, not including
    this length.

  2 byte big endian command number

    1: get
    2: add
    3: set
    4: replace
    5: incr
    6: decr
    7: delete

What follows is based on the command:

     get <key>
     add <key> <flags> <exptime> <value>
     set <key> <flags> <exptime> <value>
     replace <key> <flags> <exptime> <value>
     incr <key> <delta>
     decr <key> <delta>
     delete <key> <exptime>

Where the parameters are:

      <key>:  1 byte: length of following key, without NULL byte (1-250, inclusive)
              the key
              \0 byte

      <value>: 4 byte big endian lenth of value
               data bytes, no null termination

      <delta>: 4 byte big endian delta value

      <exptime>: 4 byte time (or zero)  see protocol.txt for description

----

Just tinkering.  I was going to benchmark and get approval on the
protocol before I committed.  Not incredibly serious with any of this
yet.  My sole interest is in making the responses faster to parse (and
fixing the whitespace in keys issue).

- Brad


On Thu, 15 Jul 2004, Anatoly Vorobey wrote:

> On Thu, Jul 15, 2004 at 09:17:28AM -0700, Brad Fitzpatrick wrote:
> > Michal,
> >
> > A binary protocol would be a nice addition.  In particular, the whitespace
> > problem in keys bugs me as well.  I want that fixed.
> >
> > And while we're happy with the current performance, I'm sure it could
> > improve.
> >
> > Let's discuss both the format of the binary protocol and its
> > implementation on this list.  It should be very easy to add.
>
> I disagree. There's a reason why all successful network protocols happen
> to be text protocols. FTP, SMTP, NNTP, HTTP, you name it. And it wasn't
> done this way to make testing with telnet easier.
>
> This in particular:
>
> > > process_command() function is horrorible, all these "if(strncmp...)" takes
> > > some CPU and these are unnecessary.
>
> is just ridiculous. These CPU cycles are are a negligible fraction of
> what is spent reading/writing to the network, looking things up, or, for
> that matter, calculating hash values! I mean, rewriting memcached's hash
> function in optimised x86 assembly would "speed things up" 3-4 times more
> than what we'd win by moving to a binary protocol and eliminating all the
> horrible strcmp() calls. And the speedup, although 3-4 times larger,
> would also be almost as negligible, because it's also far from being the
> bottleneck.
>
> Text protocols are easy to implement both on client and on server sides
> (and on the client side, we have several different programming
> languages, where working with text is always more straightforward).
> They are, more importantly, easy to maintain, to debug and to extend.
> Adding a new minor command is just a matter of sticking another if
> clause on the server, and putting some text to the socket on the client.
> In a binary format, you have to carefully allocate the command id, check
> that the existing command struct is adequate for your needs, swear and
> extend it for everyone if it isn't, match the numeric id->symbolic
> name mapping on the client side, encode parameters, decode parameters
> (matter of simple sprintf/sscanf or similar functions in a text
> protocol), etc. etc. It just becomes too bothersome to try out and
> experiment with stuff.
>
> It's not that I'm happy with everything about the current protocol. It
> has, for instance, a very painful drawback of not specifying the command
> line's length in advance, so when the server reads data in, it can't
> read in the exact number of bytes that's the command, it has to mingle
> the command buffer and the data buffer at some point. A binary protocol
> would help with that, but this reason wasn't even named so far. Things
> like strcmp() cpu cycles are goofy reasons to switch to binary, not real
> reasons. And the stuff I just mentioned doesn't really necessitate
> a switch to a binary protocol either, it could be solved by prepending
> each command line with its length (in decimal, padded to three bytes
> with spaces on the left if necessary). Though it'd make telnetting
> harder, so I'm not all that eager to propose it ;-)
>
> Whitespace characters - I don't know about that. Do we really suffer
> from not being able to use them in keys? Do other people on this list?
> We could always have spaces but not \n's in keys and delimit them with
> \n's.
>
> --
> avva
> "There's nothing simply good, nor ill alone" -- John Donne
>
>