MemcacheFS

[Erik Osterman]

Interestingly enough, we use mapserv too. We serve cached tiles from 
Amazon's S3, but fallback to our own servers when the file doesn't 
exist. First we check our local cache on GlusterFS, then generate the 
tiles if still not found. We do a lot of stuff like this, even rendering 
short animations.

I don't think it should be too difficult to implement for small files, 
e.g. maptiles and xml. In the next few weeks, I'll probably take a stab 
at it if no one else does first.

Best,

Erik Osterman


Cal Heldenbrand wrote:
> This is really cool!  One section of our application is mapping, which 
> uses MapServer <http://mapserver.gis.umn.edu/> along with ka-map 
> <http://ka-map.maptools.org/>.  For tiles & shape files, we have an 
> NFS backend (a bigass NetApp box) to serve all of these up.  Granted, 
> the netapp performs really well, and serving up static tiles seems to 
> be an easy task for it at the moment.  If this particular service 
> becomes really popular, I could see that a large cache could be 
> helpful to our architecture.
>
> I would definitely be excited if someone started working on this.  (I 
> might even help out if I can!)
>
> --Cal
>
> -- 
> Cal Heldenbrand
>    FBS Data Systems
>    E-mail:   cal at fbsdata.com <mailto:cal at fbsdata.com>
>
> On 5/9/07, *Erik Osterman* <e at osterman.com <mailto:e at osterman.com>> 
> wrote:
>
>     MogileFS is pretty sweet; I'll give it that. We considered it
>     pretty seriously before going with GlusterFS. Since MogileFS
>     relies on application modifications, which isn't a possibility
>     when the application is blackbox, we ultimately decided against
>     it. I don't think they finished the fuse module either, and the
>     web page still says, "We've prototyped a FUSE binding, so you
>     could use MogileFS without application support, but it's not
>     production-ready."
>
>     Also, I'm less thrilled about managing MogileFS with all of its
>     Perl depenencies. Memcache and GlusterFS are a cakewalk in
>     comparison to setup and configure. Granted, GlusterFS isn't yet
>     fully production worthy, it's been stable for us.
>
>     Note that CacheFS is not dependent on the underlying filesystem,
>     so a MemcacheFS developed in the same fashion could straddle
>     multiple exported filesystem types, such as NFS, GlusterFS, SMB,
>     OpenAFS, and even MogileFS. In much the same way Memcache doesn't
>     tie you to one database, MemcacheFS wouldn't tie you to one type
>     of networked filesystem.
>
>
>     Thanks for the suggestion...
>
>     Erik Osterman
>
>     Bruce Wang wrote:
>>
>>
>>     On 5/9/07, *Erik Osterman* <e at osterman.com
>>     <mailto:e at osterman.com>> wrote:
>>
>>         Right, this is along the same lines, but not generalized.
>>         They made Lighty Memcache aware, but that doesn't do much for
>>         other applications, e.g. our XSLT processor which can only
>>         access files. Further more, since it's not generalized, the
>>         cached data cannot easily be shared by many unrelated
>>         applications. So, if different applications employ their own
>>         Memcache caching strategy, a lot of memory is waisted on
>>         duplicate data. Though, embracing this idea, one could use
>>         Lighty + modmemcache + webdav + fuse but that sounds very slow :)
>>
>>
>>     http://danga.com/mogilefs/
>>
>>     Is this what you want? Also by Brad Fitzpatrick
>>     <http://bradfitz.com/>
>>
>>
>>
>>
>>
>>     -- 
>>     simple is good
>>     http://brucewang.net
>>     skype: number5 
>
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20070509/0e285d4e/attachment-0001.htm