goffinet at yahoo-inc.com
Sun Sep 9 10:49:16 UTC 2007
Anyone interested in the patches I did for the CAS implementation for
memcache 1.2.3, and libmemcache? It's prototype and could be improved
but its working :-)
goffinet at yahoo-inc.com
On Sep 8, 2007, at 6:44 PM, dormando wrote:
> Thanks for the example, that cleared it right up! Personally I'll
> agree that this might be the first actually useful server extension
> I've heard of for a long while. It removes (at least) one round
> trip, reduces polling, and helps deal with the LRU for the usual
> workaround case.
>> Are there enough potential server-side operations to really justify
>> a scripting interface? When it's been brought up in meetings, the
>> response has varied from interest, to a warning about crossing
>> streams or something.
>> Both a CAS and a scripting interface provide a foundation for
>> building atomic data structure manipulation without having to
>> define all of the data structures and all of the operations at the
>> protocol level. Each has downside. With CAS, you may have to
>> retry a few times under contention and this would be amplified by
>> the latency. With scripting, you may easily slow down the whole
>> server, and you have to deal with how parameters are passed in and
>> results are returned -- and this would mean very different things
>> by protocol.
> I remember floating the idea of embedding lua at the last meet and
> not especially getting torn up by it. Paul Querna had some good
> ideas that I've been wanting to get into my other C/lua project
> sometime soon. It's in my big fancy TODO to attempt a proof of
> concept for lua + memcached after I get a few more releases of my
> other lua project out the door (so a month or two out).
> Embedding lua should be trivial to try out, and has some nifty ideas
> to it:
> - Dynamically add/remove commands to the server (Paul's idea?).
> - Any wild n' crazy "needs to be atomic" operation you can think of.
> - Easily implement a slab-realloc thread, using pluggable strategies.
> - Coordination algorithms, subscription connections, blah blah.
> Tons of downsides:
> - Gives new users a swiss army chainsaw with which to decapitate
> - High end users are unlikely to use it at all, given their needs
> for the binary protocol already. Needing a sun T2000 or quad socket
> barcelona to run memcached isn't what was advertised.
> - Will add a lot more latency to any command which uses scripting,
> even something as fast as lua, with pre-compiled commands. You could
> try a scripting approach like Varnish and dynamically compile/link
> in plugins?...
> - If using lua, the garbage collector will need to be carefully
> managed. I'm going to be tuning this a lot for my own project, which
> needs to manage potentially hundreds of millions of objects over
> time. Otherwise memcached will randomly block operations.
> - Again, if using lua at least, the default allocator will need to
> be replaced with memcached's own slabber, or a new slabber.
> Otherwise the same malloc() issues which we've been avoiding until
> now will come back in spades.
> - Turns threading into a nightmare. There're a few options with lua,
> but you end up having to do tons of internal locking. If you decide
> to run completely independent lua instances per thread instead,
> you're back to square one without anything being atomic.
> - Again with the swiss army chainsaw.
> Any better/different alternatives? Maybe don't even try scripting? :)
> I'd be up for attempting a proof-of-concept lua embedding once the
> binary protocol stuff settles a little more. Could be accelerated if
> someone who sucks less at C helps out or we'd meet up somewhere and
> just write it.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the memcached