<HTML>
<HEAD>
<TITLE>Re: binary wire protocol in memcached</TITLE>
</HEAD>
<BODY>
<FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'>Hi,<BR>
<BR>
We have a working binary wire protocol implementation (client and server) though it does not reflect the changes that occurred at the last hackathon (at Sun). Some key differences:<BR>
<BR>
1) The magic byte is different, and also unique to request and response. We also maintain a notion of binary protocol versions for possible future (albeit limited) expansion.<BR>
<BR>
</SPAN></FONT><FONT SIZE="2"><FONT FACE="Courier New"><SPAN STYLE='font-size:10.0px'>// version number and magic byte.<BR>
#define BP_VERSION 0x0<BR>
#define BP_REQ_MAGIC_BYTE (0x50 | BP_VERSION)<BR>
#define BP_REP_MAGIC_BYTE (0xA0 | BP_VERSION)<BR>
</SPAN></FONT></FONT><FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'><BR>
2) We specified the command IDs a bit differently, grouping them by request/response type.<BR>
<BR>
</SPAN></FONT><FONT SIZE="2"><FONT FACE="Courier New"><SPAN STYLE='font-size:10.0px'>#define BIT(x) (1ULL << (x))<BR>
#define FIELD(val, shift) ((val) << (shift))<BR>
<BR>
#define BP_E_E FIELD(0x0, 4)<BR>
#define BP_E_S FIELD(0x1, 4)<BR>
#define BP_K_V FIELD(0x2, 4)<BR>
#define BP_KV_E FIELD(0x3, 4)<BR>
#define BP_KN_E FIELD(0x4, 4)<BR>
#define BP_KN_N FIELD(0x5, 4)<BR>
#define BP_N_E FIELD(0x6, 4)<BR>
#define BP_S_E FIELD(0x7, 4)<BR>
#define BP_S_S FIELD(0x8, 4)<BR>
<BR>
#define BP_QUIET BIT(3)<BR>
<BR>
typedef enum bp_cmd {<BR>
// these commands go as an empty_req and return as an empty_rep.<BR>
BP_ECHO_CMD = (BP_E_E | FIELD(0x0, 0)),<BR>
BP_QUIT_CMD = (BP_E_E | FIELD(0x1, 0)),<BR>
<BR>
// these commands go as an empty_req and return as a string_rep.<BR>
BP_VER_CMD = (BP_E_S | FIELD(0x0, 0)),<BR>
BP_SERVERERR_CMD = (BP_E_S | FIELD(0x1, 0)), // this is actually not a<BR>
// command. this is solely<BR>
// used as a response when<BR>
// the server wants to<BR>
// indicate an error status.<BR>
<BR>
// these commands go as a key_req and return as a value_rep.<BR>
BP_GET_CMD = (BP_K_V | FIELD(0x0, 0)),<BR>
BP_GETQ_CMD = (BP_K_V | BP_QUIET | FIELD(0x0, 0)),<BR>
<BR>
// these commands go as a key_value_req and return as an empty_rep.<BR>
BP_SET_CMD = (BP_KV_E | FIELD(0x0, 0)),<BR>
BP_ADD_CMD = (BP_KV_E | FIELD(0x1, 0)),<BR>
BP_REPLACE_CMD = (BP_KV_E | FIELD(0x2, 0)),<BR>
BP_APPEND_CMD = (BP_KV_E | FIELD(0x3, 0)),<BR>
<BR>
BP_SETQ_CMD = (BP_KV_E | BP_QUIET | FIELD(0x0, 0)),<BR>
BP_ADDQ_CMD = (BP_KV_E | BP_QUIET | FIELD(0x1, 0)),<BR>
BP_REPLACEQ_CMD = (BP_KV_E | BP_QUIET | FIELD(0x2, 0)),<BR>
BP_APPENDQ_CMD = (BP_KV_E | BP_QUIET | FIELD(0x3, 0)),<BR>
<BR>
// these commands go as a key_number_req and return as an empty_rep.<BR>
BP_DELETE_CMD = (BP_KN_E | FIELD(0x0, 0)),<BR>
BP_DELETEQ_CMD = (BP_KN_E | BP_QUIET | FIELD(0x0, 0)),<BR>
<BR>
// these commands go as a key_number_req and return as a number_rep.<BR>
BP_INCR_CMD = (BP_KN_N | FIELD(0x0, 0)),<BR>
BP_DECR_CMD = (BP_KN_N | FIELD(0x1, 0)),<BR>
<BR>
// these commands go as a number_req and return as an empty_rep.<BR>
BP_FLUSH_ALL_CMD = (BP_N_E | FIELD(0x0, 0)),<BR>
<BR>
// these commands go as a string_req and return as an empty_rep.<BR>
BP_FLUSH_REGEX_CMD = (BP_S_E | FIELD(0x0, 0)),<BR>
<BR>
// these commands go as a string_req and return as a string_rep.<BR>
BP_STATS_CMD = (BP_S_S | FIELD(0x0, 0)),<BR>
} bp_cmd_t;<BR>
</SPAN></FONT></FONT><FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'><BR>
3) We have no support for 64-bit arithmetic commands. We have retained decr (so far).<BR>
<BR>
4) The return value for arithmetic commands is binary and not a string. I wasn’t at the hackathon during which this was decided so I’d be interested in finding out why a string is preferable.<BR>
<BR>
Thanks,<BR>
Tony<BR>
<BR>
On 8/27/07 8:39 PM, "Dustin Sallings" <dustin@spy.net> wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'><BR>
<BR>
I just got memcached passing all of my tests with the binary wire <BR>
protocol (tcp). Sorry it took me so long to get going on it.<BR>
<BR>
I've implemented this as a patch stack (using mercurial queues) over <BR>
trunk r608 with Evan Miller's 64-bit counter patch applied (since <BR>
everyone is excited by big numbers). I've got no idea where (if <BR>
anywhere) active development is going on. I can pick it up and move <BR>
it needed.<BR>
<BR>
I don't know what clients are ready. I haven't done a release of my <BR>
java client with binary protocol support yet, but I can roll out a <BR>
pre-release if someone wants to try it (all of my interop tests pass <BR>
between my java client and memcached in binary mode, so I just need <BR>
to clean it up a bit). My test python client is functional for at <BR>
least interactive use as well.<BR>
<BR>
<BR>
Known issues:<BR>
<BR>
I didn't implement a timeout on delete. I suppose I should, but do <BR>
people actually use that?<BR>
<BR>
I didn't implement a timeout on flush. I'd rather avoid that one if <BR>
possible because the semantics are really confusing.<BR>
<BR>
I basically ignored managed buckets because I have no idea how they <BR>
work.<BR>
<BR>
My 64-bit ntohl kind of thing might not work on big endian machines <BR>
(I haven't tried it yet). Moreover, I would hope something like that <BR>
would already exist somewhere, so I didn't bother making it fast.<BR>
<BR>
I've only implemented version, flush, noop, set, add, replace, get, <BR>
getq, delete, and incr. Besides stats (for which we have no clearly <BR>
defined packet format), there seem to be other commands in there that <BR>
don't make much sense to me.<BR>
<BR>
There may be more, but I don't know about them yet.<BR>
<BR>
--<BR>
Dustin Sallings<BR>
<BR>
<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'><BR>
</SPAN></FONT>
</BODY>
</HTML>