Memory Management...

[Kyle R. Burton]

Hello all.

I've just started looking at memcached to use it where I work for a
project that will be able to benefit from a large amount of in-ram
caching.  We're looking at setting up a cache system with somewhere
between 32Gb and 56Gb of memory across 4 to 8 servers with 8Gb of ram
each.

Since memcached is a stable pre-existing project we are evaluating it
before embarking on developing a from scratch solution.  Our
requirements for the caching system are somewhat different from what
looks like the most common use of memcached.  

The data we'll be caching will be largely static, with no automatic
expiration time (it looks like memcached already supports this).

We want to be able to organize our keys under logical containers and
will want to flush entire containers from the cache on demand (it
looks like we can create logical containers with memcached by
prefixing keys with a container identifier).  We have just begun to
look at the implementation of the memory management sub-system with
regards to it's allocation, de-allocation and compaction approaches.
Is there any documentation or discussion of how this subsystem
operates? (slabs.c?)

We are also need the cache system to appear as and acts as a single
entity to the systems that will utilize it.  Towards this end, we're
currently looking at creating a server that will sit between the
client API and the memcached servers that can implement the types of
services we need (flushing of containers, possibly re-distribute or
expire records according to our own LRU-style algorithm, etc.).  One
other feature that we need is to be able to bring machines in and out
of the cache cluster as necessary while having as little impact as
possible on the system as a whole.


Is there anyone else doing anything similar that they can talk about?
Is there anything about what we're looking to do that memcached may
already support that we're unaware of, or is not well suited to do?


Thanks,

Kyle R. Burton



-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mortis@voicenet.com                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------