Hackathon notes (non-binary protocol thread)

[BUSTARRET, Jean-francois]

What about tagging (as discussed last week on the mailing-list) ? IMHO, it would be more useful than wildcard deletes/namespaces/...

I'm looking forward to be able to test the new features...

Jean-François

-----Message d'origine-----
De : memcached-bounces at lists.danga.com [mailto:memcached-bounces at lists.danga.com] De la part de Paul Lindner
Envoyé : samedi 14 juillet 2007 16:12
À : memcached at lists.danga.com
Objet : Hackathon notes (non-binary protocol thread)

Other stuff was done besides the binary protocol last monday at FBHQ.

We had presentations from Facebook about how they use Memcached and status of libmemcache and the PHP client.  Also there was some interesting discussion of how to use MySQL replication to propagate cache invalidation to multiple data centers.

I transcribed the FB slides for their server presentation.  [FB folks-- let me know if it's okay to splash that info across the mailing list..]

Here's some more notes from our initial brainstorming discussions:

* Pluggable Storage Backends 
 
We didn't discuss or even consider replacing the in-memory model with any other form of storage (durable or not...)

* Dump/Restore functionality

Some discussion.  Basically defining the correct conditions where this might be useful (for data that never expires and is long-term durable)

Basic idea was to generate protocol stream and feed that into another memcached instance.

* Abstract Data Types

Some pushback on doing things like queues and such in the server.
Many people mentioned that these could be constructed using the existing memcached get/set/add/incr/decr operations.

I suggested we might want to come up with a standard way for clients to store abstract data types so we have some kind of cross platform way of sharing data.  For example a queue can be constructed with 10 data buckets and a counter, all in memcache...

No action taken AFAIK.

* Replace slab allocator

There was some discussion on abstracting out slabs.c/slabs.h.  As a proof of concept it was suggested that someone write a simplistic free/malloc implementation to show how it could be done.

Another interesting possibility was using a 'bulldozer thread' to optimize memory storage.

No action taken AFAIK.

* Make I/O buffers count as mem usage.

We put that on the list of things to try.  I don't think anyone ran with it.

* Multidimensional keys

There was some discussion on how multidimensional keys could be used in practice.  Most people seemed to latch on using it as a way to expire large quantities of data when the underlying format changes due to new releases etc.

It was mentioned that there was an FAQ entry on how to do this (fetch a 'generation' prefix key and use that as part of your individual data keys.  There was also talk of adding a generation identifier to the protocol(?).  The main benefit was that this would allow you to expire known stale data instead of having valid data expired out using the LRU.

I don't know if anything was done on this other than discussion.

* Documentation improvements

Some people said they were going to work on docs.  Not sure what happened with that.


* Client Library improvements
* Wildcard (regex deletes)
* Cache replication

The Pizza came before we could discuss this, and then we dived into coding.

* Append functionality 

Steven and a few others worked through how to modify the memcached support code.  His work is currently in the append branch.  I promised to write some unit tests for this, but haven't gotten around to it yet.


I spent the rest of the night cleaning up a few more things in trunk and slogging through the Fedora RPM submission/build process.  I also put in some hooks for eventual publication of internal docs with Doxygen.


So... that's my notes, maybe others can contribute their experiences?


-- 
Paul Lindner        ||||| | | | |  |  |  |   |   |
lindner at inuus.com