memcached and streaming

[Tomash Brechko]

Hello,

In memcached.c there's a line,

  setsockopt(sfd, IPPROTO_TCP, TCP_NODELAY, (void *)&flags, sizeof(flags));

We should realise that this is a serious performance killer for
streaming.  For instance, C::M::F has multi-update commands that are
capable to bundle several requests into one network packet.  However
replies are always sent _one at a time_.  I.e. tons of packets with
single 'STORED' etc. (>40 bytes header and 6 bytes of data---not the
best ratio).  Not to say the the greater number of packets means a
huge increase of latency in a congested network.

memcached should have in its TODO "Disable TCP_NODELAY and enable
TCP_CORK/TCP_NOPUSH for streaming".  The server may tell if the
streaming is used either by extending the protocol to support new
keyword, 'more' (specified for all commands but last; my experiments
with norepy keyword show that "modifying the parser would kill the
performance" are just evil lies ;)).  Or it may use automatic
heuristic: if it parsed a command, but there's more data in the input
buffer, then it doesn't have to push the reply until it process that
data too.  Or both.

get with 100 keys has about 3x smaller wallclock time than 100 single
key gets bundled together, on otherwise free network (and even about
2x on loopback).


-- 
   Tomash Brechko