MemcacheFS

[Erik Osterman]

We implemented something quick yesterday using the fuse examples and 
libmemcache (replicating that python example in C). We configured it to 
just wrap an existing GlusterFS volume. Our initial test very crude test 
was just to accelerate stating. Stating 18500 files over GlusterFS took 
consistently ~22.5 seconds without the memcache layer. Stating the same 
files (after hydrating the cache) took ~4.5 seconds consistently with 
the memcache caching layer. We didn't really do anything for caching 
content, no more time to play with it. . . didn't even try to do any 
optimizations. I was just happy it helped validate idea.

If implemented truely as a caching solution, you shouldn't have to worry 
about losing files either. Cache misses would fall back to the 
filesystem and just rehydrate the cache. Or like you said, you could 
have it fall back to a database instead of the filesystem. What I like 
about using the filesystem as opposed to a database, is that it can be 
used more generally to accelerate any exported filesystem.


Erik Osterman


Peter van Dijk wrote:
>
> On 9-mei-2007, at 20:21, Erik Osterman wrote:
>
>> Python proof of concept code.
>>
>> http://blog.koshigoe.jp/archives/2007/04/fuse.html
>>
>> It doesn't appear to cache contents, only stat.
>>
>> Also, on a philisophical side note. I am floored by how often almost 
>> identical ideas are conceived of by different people at the same 
>> time. Look at the date on that blog entry!
>
> You have no idea how much this is scaring me right now ;)
>
> Over the last two weeks i spent a lot of time discussing a memcachefs 
> (fuse-based) with two fellow geeks - applications that came to mind 
> were (a) the smarty cache (b) php sessions; for both cases, losing 
> files (as a whole, not random parts inside) is ok and readdir is 
> irrelevant, which allows cutting a lot of corners.
>
> We're also pondering using mysql as the write-through bit, instead of 
> an actual filesystem, which wouldmake the storage stable and allow 
> readdir. No code so far, though.
>
> Cheers, Peter
>
>