Next Major Memcached release / roadmap?

[Steve Grimm]

I don't like making it a runtime option. A compile-time option, I could
maybe live with. But I don't want to have to install BDB headers and
libraries just to build memcached when I have no interest in BDB support,
which is presumably what runtime switchability would imply.

Also, dynamically selecting a backend at runtime seems to me to imply at
least a little performance hit, given that the backend storage isn't
particularly abstracted out at this point. Maybe not a huge enough hit to be
statistically significant, in which case fine. But I don't want to make my
memcached instances run slower for the sake of a feature I'm not using.

I suspect, however, that making a BDB backend work really well would require
a more radical restructuring of memcached, e.g., to allow the communication
with the persistent storage to take place in another thread such that it
wouldn't block requests for data that are in a local write-through cache. Or
would you just let the BDB libraries handle all that? I don't know if they
have options for "return this data if you have it in memory already,
otherwise return rather than blocking for disk I/O." Under no circumstances,
it seems to me, do you want to force clients whose requests can be serviced
from RAM to wait around for some other request to be serviced from disk.

-Steve


On 6/16/07 8:23 AM, "Iain Wade" <iwade at optusnet.com.au> wrote:

> I would like to see tugela-cache's BDB backend merged directly into memcached.
> 
> tugela-cache was a fork of an old memcache version for use in
> media-wiki/wikipedia. It ripped out the memory cache code and replaced
> it with BDB.
> 
> Subsequent enhancements to memcache have not been kept up to date in
> tugela-cache. UDP support is not there, for example.
> 
> If I coded/merged up a selectable backend patch which allowed a
> runtime choice, would it be acceptable to people?
> 
> --Iain
> 
> On 5/30/07, Paul Lindner <lindner at inuus.com> wrote:
>> I've been thinking of where memcached is now and where it might go in
>> the future.  I was hoping the community could come together and define
>> a roadmap of possible features to be added/incorporated in future
>> memcached releases.
>> 
>> Some patches that are floating around and are not yet committed
>> include:
>> 
>>  * non-expiring entries patch from Paul G
>>  * Ring buffers patch from Nathan.
>>  * Append data to existing entry from Filipe
>> 
>> Also I have tasked myself with integrating some of Hi5's custom server
>> code into memcached.  This code would include variants of the
>> memcached storage architecture and some possible background worker
>> threads.
>> 
>> Open issues that need consensus:
>> 
>>   * Protocol changes -- do we want to extend memcached protocol to
>>     support new features?  If so, what is the logical way to do this.
>> 
>>   * "Magic" values?  Do we want to allow some kind of mapping of
>>     client data to variations in server behavior?
>> 
>>     For example the non-expiration patch allows you to configure a
>>     value that results in a non-expiring item.
>> 
>>     Others have proposed that special features be enabled for a common
>>     prefix -- say all keys that have a  "ringbuffer:" prefix use that
>>     code path / storage engine..
>> 
>>   * How do we accomodate new features without affecting performance.
>> 
>>   * More? I'm sure there is..
>> 
>> 
>> Hopefully this gets things going.  I hope we can agree on the right
>> approach so we can incorporate as many community contributions as is
>> possible and feasible.
>> 
>> 
>> --
>> Paul Lindner        ||||| | | | |  |  |  |   |   |
>> lindner at inuus.com
>> 
>>