CAS operation

[Chris Goffinet]

Anyone interested in the patches I did for the CAS implementation for  
memcache 1.2.3, and libmemcache? It's prototype and could be improved  
but its working :-)


Chris Goffinet
goffinet at yahoo-inc.com



On Sep 8, 2007, at 6:44 PM, dormando wrote:

> Thanks for the example, that cleared it right up! Personally I'll  
> agree that this might be the first actually useful server extension  
> I've heard of for a long while. It removes (at least) one round  
> trip, reduces polling, and helps deal with the LRU for the usual  
> workaround case.
>
>> Are there enough potential server-side operations to really justify  
>> a scripting interface?  When it's been brought up in meetings, the  
>> response has varied from interest, to a warning about crossing  
>> streams or something.
>> Both a CAS and a scripting interface provide a foundation for  
>> building atomic data structure manipulation without having to  
>> define all of the data structures and all of the operations at the  
>> protocol level.  Each has downside.  With CAS, you may have to  
>> retry a few times under contention and this would be amplified by  
>> the latency.  With scripting, you may easily slow down the whole  
>> server, and you have to deal with how parameters are passed in and  
>> results are returned -- and this would mean very different things  
>> by protocol.
>
> I remember floating the idea of embedding lua at the last meet and  
> not especially getting torn up by it. Paul Querna had some good  
> ideas that I've been wanting to get into my other C/lua project  
> sometime soon. It's in my big fancy TODO to attempt a proof of  
> concept for lua + memcached after I get a few more releases of my  
> other lua project out the door (so a month or two out).
>
> Embedding lua should be trivial to try out, and has some nifty ideas  
> to it:
>
> - Dynamically add/remove commands to the server (Paul's idea?).
> - Any wild n' crazy "needs to be atomic" operation you can think of.
> - Easily implement a slab-realloc thread, using pluggable strategies.
> - Coordination algorithms, subscription connections, blah blah.
>
> Tons of downsides:
>
> - Gives new users a swiss army chainsaw with which to decapitate  
> themselves.
> - High end users are unlikely to use it at all, given their needs  
> for the binary protocol already. Needing a sun T2000 or quad socket  
> barcelona to run memcached isn't what was advertised.
> - Will add a lot more latency to any command which uses scripting,  
> even something as fast as lua, with pre-compiled commands. You could  
> try a scripting approach like Varnish and dynamically compile/link  
> in plugins?...
> - If using lua, the garbage collector will need to be carefully  
> managed. I'm going to be tuning this a lot for my own project, which  
> needs to manage potentially hundreds of millions of objects over  
> time. Otherwise memcached will randomly block operations.
> - Again, if using lua at least, the default allocator will need to  
> be replaced with memcached's own slabber, or a new slabber.  
> Otherwise the same malloc() issues which we've been avoiding until  
> now will come back in spades.
> - Turns threading into a nightmare. There're a few options with lua,  
> but you end up having to do tons of internal locking. If you decide  
> to run completely independent lua instances per thread instead,  
> you're back to square one without anything being atomic.
> - Again with the swiss army chainsaw.
>
> Any better/different alternatives? Maybe don't even try scripting? :)
>
> I'd be up for attempting a proof-of-concept lua embedding once the  
> binary protocol stuff settles a little more. Could be accelerated if  
> someone who sucks less at C helps out or we'd meet up somewhere and  
> just  write it.
>
> *shrug*,
> -Dormando

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20070909/537443bc/attachment.html