Upcoming memcached releases + rambling.

Thu Feb 7 17:09:48 UTC 2008

   Hey, sounds great overall.  One minor thing, though:   I submitted  
a binary patch a while back to finalize incr/decr transport.  I think  
it's important that that goes in before clients can be ready.

   While mercurial is obviously better than git at everything but  
managing svn repos, I'd be quite happy if I could simply have my  
changes published somewhere so I feel they weren't lost or becoming  
stale and others could try them and tell me what I did wrong (mostly  
so I could declare my client done sooner).

   Would it make sense for someone to provide a repo conversion to  
work from outside of danga and just declare svn officially unsupported?

-- 
Dustin Sallings (mobile)

On Feb 7, 2008, at 0:56, dormando <dormando at rydia.net> wrote:

> Yo all,
>
> The original plan for today was to pop on the list and yell "Hey!  
> Have some -rc's to test! Go nuts! Thanks for all the patches!" - but  
> my awesome  fever for the last few days has pushed that off by ... a  
> few more days.
>
> But it's totally cool, because it's happening. For the first time in  
> a long time memcached will act like it's releasing early, often.  
> Thanks to everyone for all of the recent code submissions and  
> bugfixes :)
>
> So we have two -rc's which are coming up as soon as I can wrap a few  
> bug reports and merge all this code:
>
> 1.3.0-rc (the binary tree, in its current state)
> 1.2.5-rc (non-binary branch with all of the recent fixes and patches)
>
> There might not be an immediate merge of 1.2.5 -> 1.3.0, but I'm not  
> convinced that matters immediately.
>
> I believe should probably give the binary tree a couple of real  
> development releases until we've had a few client implementations.  
> So in the case of the protocol having to change due to issues with  
> actual client integration, we won't be backing out on a stable API.
>
> SO! I'd like to lay out my own wishlist for memcached, some of which  
> is underway, some not:
>
> - Binary protocol. It's been too long, lets finish it. We need more  
> native tests, and we need to benchmark its speed relative to the  
> text protocol, make sure we're actually coming out ahead here.
>
> - Dynamic storage engines. Toru has a promising proof of concept  
> that I'd love to merge for an eventual 1.4 release.
>
> Most of the memcached forks have been due to folks not liking the  
> storage method. While I disagree with how a lot of these forks are  
> done, there's definitely merit in the flexibility. A lot of these  
> folks start from scratch, but I believe memcached should be a  
> simple, efficient, network platform for dead simple key=value  
> datastores, no matter how they're actually implemented.
>
> - Dedupe the error messages. We have 11 instances of "SERVER_ERROR  
> out of memory". This makes it impossible to even guess what  
> happened, even with full context. Did we fail to allocate a msghdr?  
> Did do_item_alloc fail? Who knows. We can do this for 1.2.5 if  
> there're people on board. I'll float the patches.
>
> - Add stats about failures. I'd love counters for all of the times  
> something failed without causing an eviction. Sometimes machines  
> will be hosed for a long time before we realize the hit rate is way  
> down due to the OS refusing 50% of the mallocs. Hard to tell if  
> you've hit maxconns, blah blah blah. More easy work, maybe for 1.2.5.
>
> - Method of dumping refcounts of "locked" items. I'm tracking down  
> what I suspect is a refcount bug, and I expect us to have more of  
> these in the future. It'll be great to be able to verify these  
> offhand. If someone isolates a bum server, they can dump slabs,  
> refcount, etc, and send to the list.
>
> - Full item dump. I guess we should do this with the appropriate  
> caveats. There's a potential patch floating around, we'll see about  
> it.
>
> - Fixing a lot of dynamic memory buffers. Per connection, per  
> freelist, whatever. A lot of these can be turned into linked lists  
> if, say, the 'item' struct had a 'next' pointer. This should remove  
> most of the various realloc logic.
>
> - Clean up the network iov's to work like above, and also manage  
> generic freelisted buffers. Which would remove the need for the  
> specialized cas suffix buffer. These should be wrapped so they can  
> be linklisted, instead of managed via dynamic arrays per connection.  
> Could also be used with the binary protocol to reduce syscalls.
>
> - TCP_CORK/TCP_NOPUSH where appropriate. I totally agree here, and  
> it used to explicitly do this.
>
> - New slabber engine. I view this as a bonus to making the slabber  
> code dynamic. We can start cooking an experimental new slabber to do  
> slab-reassignment, fragmentation avoidance, etc. Without really  
> hosing mainline.
>
> - More tests. We need better test coverage. Please submit tests!
>
> - "Standard benchmarks". Memcached's contributions have historically  
> been based on "we ran this in production, here're some tests and  
> proof it works fine". Recently it's been more about rapid cleanups  
> and the new storage engine independence. It'd be really great to  
> have some (say libmemcached based) tests to exercize (under  
> realistic AND unrealistic measures!) everything.
>
> - More documentation. I figure eventually someone will write docs  
> with better english, but there're tons of ideas floating around. We  
> should get them all down. Along with standard troubleshooting info,  
> of which there is practically none!
>
> - Cache::Memcached updates. Uh, whoops. This doesn't support any of  
> the recent features.
>
> ---
> On community related measures:
>
> - Buildbot. I totally started on this, but have not completed it. I  
> want to offer a buildbot network to the developer community. If you  
> want a platform supported, provide a slave. Otherwise we will not  
> target it (besides a couple of standby's).
>
> - GIT. SVN is dead. We'll need to move on to git or hg in order to  
> scale the developer pace. It's my understanding that just as many  
> folks on the list prefer GIT as they do Mercurial, and SVN is mainly  
> a convenience. Anything will be better. I could crack out weekly  
> releases if I didn't have to worry about SVN branches/merges.
>
> - Add more committers. I promise you guys; we're in an awkward  
> position with commit access, but people at sixapart absolutely  
> acknowledge and care about it. I've been having meetings,  
> discussions, etc, about dealing with this sooner than later. I  
> mostly help with memcached on my own time. However, sixapart does  
> love memcached and the rest of the danga projects. I can't say  
> anything definitive, of course, but I feel like I can share the  
> feeling of "we give a shit" with all of you folks.
>
> Part of having more committers means we need more dedicated  
> reviewers. Memcached is a stable project trusted by many, many  
> websites.
>
> - We should do more, less formal, hackathons. These are great for  
> hashing out minor issues, and I feel with more frequence they'd also  
> be really good for cranking out code, too ;)
>
> ---
> That's all for now.
>
> I'm missing a number of line items (please don't be offended if I  
> missed mentioning your submissions!) If it's in my inbox, it'll get  
> a fair review and probably go in.
>
> have fun,
> -Dormando
>
>