Extensible command syntax

Dustin Sallings dustin at spy.net
Wed Nov 7 22:10:35 UTC 2007

On Nov 7, 2007, at 13:18 , Tomash Brechko wrote:

> But could you please first explain how current commands can solve
> Brian's problem?  As you recall, he said that with mandatory
> positional parameters he can't omit expiration time nor flags while
> doing update.  You correctly answered that CAS case is a no issue, but
> what about other updates?

	I can't speak for append or prepend.  These are commands I've not  
used.  I can say that I implemented the binary incr in such a way that  
flags and expiration are both required, but depending on the  
expiration value, it may be considered an error if the value doesn't  
exist.  If it does, it should ignore the flags and expiration and just  
update an existing value.

	That is, they only exist for the case where you're creating a new  

> The fundamental flaw with positional parameters is that you can
> neither add nor even omit anything.  I'm sure you realize this
> yourself, and the addition of CAS clearly shows it: the scheme is
> fixed, so let's add another two or more commands with another fixed
> set of parameters.  What if I want to use CAS will all update
> commands, not "set-with-cas" only?  For Brian's case, it's possible to
> say, "let's introduce some special exptime value, which would mean 'do
> not change'", and the problem will be fixed, really.

	I won't argue against a CAS token on update being a cleaner approach  
in general.

> To my mind, the whole protocol is broken, I think it was never
> thoroughly designed, rather fixed here and there over time.  This is
> called "patchy design", as I was told.

	There are definitely cases where this is true.  I've not been  
successful arguing against some of these, but a fresh approach might  
be good.

> For instance, 'get key key
> key...'  returns nothing for found keys that weren't found, while it
> could return NOT_FOUND.  This alone breaks all pipelining, because
> there's no direct correspondence between requests and replies, and
> instead of simply counting the results one has to _compare strings_ to
> know which result goes where.

	No, this get mechanism works fine, and I pipeline everything heavily  
with great success.  You can get values back in any order and you are  
notified when all of the results are available.  A get across a couple  
of thousand keys is a one line response in a case where none exist (or  
one message response in the binary protocol).

	That end message closes the request in both cases.

> We may as well put aside all talks
> about performance bit-tweaking until all such flaws are fixed.
> Sending a potentially large chunk of data just to learn that the key
> is locked now, sorry, is another example.  I also have my doubt about
> "flags" field.  Since they are opaque, couldn't you use some byte
> range of the data itself?

	This is certainly done in some clients in spite of the flags.  I get  
good use out of them, though.

> I do understand your worry, every significant change is painful,
> especially when you have lots of derived work.  But this particular
> change actually doesn't break much.  Now, instead of adding another
>  } else if (strcmp("newcmd", ...
> you add it elsewhere, and that's all.  It won't close any door, but it
> will open many more.  And it will be the same O(1), just for smaller
> values of 1 :).


	No, that's not my worry about the large part of the gperf work.  It's  
that my protocol implementation doesn't use string compares at all, so  
I don't know how much work I need to do to get both mechanisms in  
place.  Patch itself might do all of the work, but I haven't looked yet.

> When you add data to the cache, you can't do anything with the
> response anyway.  Success?--Good.  Failure?---The data wasn't cached,
> maybe will have luck next time.

	I assume users of my client don't generally care and the entire  
process can occur asynchronously as they throw away the results.  If  
they do care, I have the answer for them.  I can't infer who doesn't  
or doesn't care, but I do need to know when the command is processed  
so the next one in the pipeline can begin.

	Of course, this is slightly less of an issue with the binary protocol  
where responses are tagged to the request that issued them.

> But you know, I'm not sure myself.  Let me try it, and maybe I'll be
> ably to show you some nice charts :).

	Well, my argument is that you're modifying the server for something  
that you may be able to make more efficient in your client.

	Try the difference between your client and something like netcat  
(just draining the responses).  I consider this the ``terminal  
velocity'' to approach.  I don't think you'll see a huge difference  
between the two models when doing such a test.

> I was putting it the other way around.  Client may speak only one
> protocol, text or binary.  And server upon reading the request can
> easily distinguish the two: is first byte a zero?  If yes---binary,
> not---text.

	Understood.  It's *possible* to do that now.  I just don't know that  
it's worth having the server auto-detect.  Especially if people want  
to add more protocols.  At some point, the detection would get too  

>> 	This makes sense in abstract, but I'm failing to see the deficiency
>> you think warrants it.
> Please be more open-minded.  I don't yet know who's'who on the list, I
> learned about memcached only a week ago, but you sound like you may
> set no-go to my patches.

	I'm not terribly special, but I do support some of your work and I  
think it's always good to have more people going over stuff.

	My concern is that you're solving problems we don't have.  For  
example, you feel that pipelining is impossible in the current text  
protocol.  I absolutely rely on it in both protocols.

> And I'm not going to push on you anyways ;).
> I'll continue to post patches unless someone would object, but it's up
> to you to decide if this is a no issue, or there are some problems
> with current protocol indeed.

	It's a given that the current protocol isn't perfect.  That's why we  
made a new one.  You should complain about that one more.  :)

Dustin Sallings

More information about the memcached mailing list