Memcache on a block device...

[Jason Edgecombe]

Ed Roskos wrote:
>
> Howdy Folks,
>
>    I was curious whether anyone knows of any efforts, besides Tugela
> cache, to use a block device with memcache.  I work for a company,
> Violin Memory, which is completing work on a 2U 1/2 TB DRAM appliance
> which acts as an SSD over a PCIe cable to a host computer.  We are
> planning on prototyping a server which serves data via a 10 Gb
> Ethernet port.  Our goal is to serve data at a rate of about a GB/sec
> for memcache queries (with very low latency).  It would be nice to
> allow objects to serve from both the server DRAM (for very small
> objects which may change via the incr and decr commands) and from our
> appliance.
>
>    For the curious, our appliance has both block and char drivers
> available, each driving DMA transfers in multiples of 128B (up to 64
> KiB) to/from the appliance.  We could do a simple implementation of
> mmap() on top of the block device, but we can get better performance
> by doing DMA transfers in block-sized chunks via our character driver
> (assuming all data are hot).  We are therefore considering
> implementing a slab allocator on the server which manages the space on
> the appliance.  In such a scenario, the memcache associative array
> could contain a value which points to where on the appliance the value
> lives, and a new field in the item structure would indicate whether
> the modified memcache implementation would interpret the value as a
> pointer to the appliance or a value for a small object in server
> memory.  Opinions?
>
>    Thanks in advance!
>
>      --Ed.
>
> If there are any content cache/delivery software development folks in
> the job market, feel free to toss a resume to
> careers at violin-memory.com.
>
Are you using Linux?

If so, you can add the block device as the highest priority swap device. 
After that make memcache use more than the available ram. It will swap 
to the DRAM appliance at that point.

It might not be optimal, but it should work.

I read an article once where you could use the unused video ram in a 
video card as hig-performance swap that was almost as fast as main RAM.

You other option would be to make a filesystem on the device and then 
use tugela cache. tmpfs or ext2 might work. the only issue might be the 
undesirable fs caching. I don't know if tmpfs does caching since it 
lives in memory and swap. I suggest trying tmpfs before ext2 using tmpfs 
requires the using the DRAM appliance as swap like the first option..

Sincerely,
Jason