Hackathon / Multidimensional keys / Wildcard deletes

[josh rotenberg]

So some of this hits exactly on what I've been working on lately. I
was thinking about discussing it at the hackathon.

We were looking at having some kind of simple, distributed key/value
storage system for storing chunks of stuff (html, xml, whatever) with
simple, namespaced keys. Stuff that doesn't belong in Mysql, and fits
more along the lines of Berkeley DB, but is accessible over the
network and can be accessed with a protocol instead of an API. We
already use memcached, and we use at least 4 different languages, so
coming up with a new client library for each of those didn't sound so
fun, so the memcached protocol fit almost perfectly.

There were a few other things we wanted: persistent storage, some
redundancy, deterministic key routing, and ease of configuration.  I
looked at two options to accomplish these things. The first was to
extend the memcached source to handle it all, and the other was as an
Apache 2 module, which is the route I ended up taking.

Some things that made the Apache route sound pretty good: first, the
ability to layer an HTTP interface over the storage access was already
there, although the initial version is actually a protocol handler
that doesn't do HTTP (yet). My idea was that eventually, some kind of
simple REST interface could map directly to the same functions that
also handled the memcached protocol calls. The second was the
existence of an easy way to configure all of these things. And the
third was modularity. Although my current version is a hardcoded
memcached-in/memcached-out proxy, my plan is to wire up something
similar to mod_cache whereby the protocol itself is handled in one
module, and one or more storage providers can be written to handle
what goes in/comes out of the backend, so that storage can be just
about anything, as long as you implement the callback interface to fit
within the "SET/GET/DELETE" etc. interface.

So yeah, I've been thinking about/poking at some of this stuff for a
little while now, and I'd love to talk more about the details.

Josh

(snipped out the quoted stuff here) ....