Hackathon notes (non-binary protocol thread)

[Tobias Lütke]

Agreed,

I think the tagging solves the same problem as the wildcard deletes in
a more elegant way. The concept is so simple that you can explain it
in a single sentence which is a indicator of a good feature.

On 7/16/07, BUSTARRET, Jean-francois <jfbustarret at wat.tv> wrote:
>
> What about tagging (as discussed last week on the mailing-list) ? IMHO, it would be more useful than wildcard deletes/namespaces/...
>
> I'm looking forward to be able to test the new features...
>
> Jean-François
>
> -----Message d'origine-----
> De : memcached-bounces at lists.danga.com [mailto:memcached-bounces at lists.danga.com] De la part de Paul Lindner
> Envoyé : samedi 14 juillet 2007 16:12
> À : memcached at lists.danga.com
> Objet : Hackathon notes (non-binary protocol thread)
>
> Other stuff was done besides the binary protocol last monday at FBHQ.
>
> We had presentations from Facebook about how they use Memcached and status of libmemcache and the PHP client.  Also there was some interesting discussion of how to use MySQL replication to propagate cache invalidation to multiple data centers.
>
> I transcribed the FB slides for their server presentation.  [FB folks-- let me know if it's okay to splash that info across the mailing list..]
>
> Here's some more notes from our initial brainstorming discussions:
>
> * Pluggable Storage Backends
>
> We didn't discuss or even consider replacing the in-memory model with any other form of storage (durable or not...)
>
> * Dump/Restore functionality
>
> Some discussion.  Basically defining the correct conditions where this might be useful (for data that never expires and is long-term durable)
>
> Basic idea was to generate protocol stream and feed that into another memcached instance.
>
> * Abstract Data Types
>
> Some pushback on doing things like queues and such in the server.
> Many people mentioned that these could be constructed using the existing memcached get/set/add/incr/decr operations.
>
> I suggested we might want to come up with a standard way for clients to store abstract data types so we have some kind of cross platform way of sharing data.  For example a queue can be constructed with 10 data buckets and a counter, all in memcache...
>
> No action taken AFAIK.
>
> * Replace slab allocator
>
> There was some discussion on abstracting out slabs.c/slabs.h.  As a proof of concept it was suggested that someone write a simplistic free/malloc implementation to show how it could be done.
>
> Another interesting possibility was using a 'bulldozer thread' to optimize memory storage.
>
> No action taken AFAIK.
>
> * Make I/O buffers count as mem usage.
>
> We put that on the list of things to try.  I don't think anyone ran with it.
>
> * Multidimensional keys
>
> There was some discussion on how multidimensional keys could be used in practice.  Most people seemed to latch on using it as a way to expire large quantities of data when the underlying format changes due to new releases etc.
>
> It was mentioned that there was an FAQ entry on how to do this (fetch a 'generation' prefix key and use that as part of your individual data keys.  There was also talk of adding a generation identifier to the protocol(?).  The main benefit was that this would allow you to expire known stale data instead of having valid data expired out using the LRU.
>
> I don't know if anything was done on this other than discussion.
>
> * Documentation improvements
>
> Some people said they were going to work on docs.  Not sure what happened with that.
>
>
> * Client Library improvements
> * Wildcard (regex deletes)
> * Cache replication
>
> The Pizza came before we could discuss this, and then we dived into coding.
>
> * Append functionality
>
> Steven and a few others worked through how to modify the memcached support code.  His work is currently in the append branch.  I promised to write some unit tests for this, but haven't gotten around to it yet.
>
>
> I spent the rest of the night cleaning up a few more things in trunk and slogging through the Fedora RPM submission/build process.  I also put in some hooks for eventual publication of internal docs with Doxygen.
>
>
> So... that's my notes, maybe others can contribute their experiences?
>
>
> --
> Paul Lindner        ||||| | | | |  |  |  |   |   |
> lindner at inuus.com
>


-- 
Tobi
http://shopify.com       - modern e-commerce software
http://typo.leetsoft.com - Open source weblog engine
http://blog.leetsoft.com - Technical weblog