Upcoming memcached releases + rambling.
dustin at spy.net
Thu Feb 7 17:09:48 UTC 2008
Hey, sounds great overall. One minor thing, though: I submitted
a binary patch a while back to finalize incr/decr transport. I think
it's important that that goes in before clients can be ready.
While mercurial is obviously better than git at everything but
managing svn repos, I'd be quite happy if I could simply have my
changes published somewhere so I feel they weren't lost or becoming
stale and others could try them and tell me what I did wrong (mostly
so I could declare my client done sooner).
Would it make sense for someone to provide a repo conversion to
work from outside of danga and just declare svn officially unsupported?
Dustin Sallings (mobile)
On Feb 7, 2008, at 0:56, dormando <dormando at rydia.net> wrote:
> Yo all,
> The original plan for today was to pop on the list and yell "Hey!
> Have some -rc's to test! Go nuts! Thanks for all the patches!" - but
> my awesome fever for the last few days has pushed that off by ... a
> few more days.
> But it's totally cool, because it's happening. For the first time in
> a long time memcached will act like it's releasing early, often.
> Thanks to everyone for all of the recent code submissions and
> bugfixes :)
> So we have two -rc's which are coming up as soon as I can wrap a few
> bug reports and merge all this code:
> 1.3.0-rc (the binary tree, in its current state)
> 1.2.5-rc (non-binary branch with all of the recent fixes and patches)
> There might not be an immediate merge of 1.2.5 -> 1.3.0, but I'm not
> convinced that matters immediately.
> I believe should probably give the binary tree a couple of real
> development releases until we've had a few client implementations.
> So in the case of the protocol having to change due to issues with
> actual client integration, we won't be backing out on a stable API.
> SO! I'd like to lay out my own wishlist for memcached, some of which
> is underway, some not:
> - Binary protocol. It's been too long, lets finish it. We need more
> native tests, and we need to benchmark its speed relative to the
> text protocol, make sure we're actually coming out ahead here.
> - Dynamic storage engines. Toru has a promising proof of concept
> that I'd love to merge for an eventual 1.4 release.
> Most of the memcached forks have been due to folks not liking the
> storage method. While I disagree with how a lot of these forks are
> done, there's definitely merit in the flexibility. A lot of these
> folks start from scratch, but I believe memcached should be a
> simple, efficient, network platform for dead simple key=value
> datastores, no matter how they're actually implemented.
> - Dedupe the error messages. We have 11 instances of "SERVER_ERROR
> out of memory". This makes it impossible to even guess what
> happened, even with full context. Did we fail to allocate a msghdr?
> Did do_item_alloc fail? Who knows. We can do this for 1.2.5 if
> there're people on board. I'll float the patches.
> - Add stats about failures. I'd love counters for all of the times
> something failed without causing an eviction. Sometimes machines
> will be hosed for a long time before we realize the hit rate is way
> down due to the OS refusing 50% of the mallocs. Hard to tell if
> you've hit maxconns, blah blah blah. More easy work, maybe for 1.2.5.
> - Method of dumping refcounts of "locked" items. I'm tracking down
> what I suspect is a refcount bug, and I expect us to have more of
> these in the future. It'll be great to be able to verify these
> offhand. If someone isolates a bum server, they can dump slabs,
> refcount, etc, and send to the list.
> - Full item dump. I guess we should do this with the appropriate
> caveats. There's a potential patch floating around, we'll see about
> - Fixing a lot of dynamic memory buffers. Per connection, per
> freelist, whatever. A lot of these can be turned into linked lists
> if, say, the 'item' struct had a 'next' pointer. This should remove
> most of the various realloc logic.
> - Clean up the network iov's to work like above, and also manage
> generic freelisted buffers. Which would remove the need for the
> specialized cas suffix buffer. These should be wrapped so they can
> be linklisted, instead of managed via dynamic arrays per connection.
> Could also be used with the binary protocol to reduce syscalls.
> - TCP_CORK/TCP_NOPUSH where appropriate. I totally agree here, and
> it used to explicitly do this.
> - New slabber engine. I view this as a bonus to making the slabber
> code dynamic. We can start cooking an experimental new slabber to do
> slab-reassignment, fragmentation avoidance, etc. Without really
> hosing mainline.
> - More tests. We need better test coverage. Please submit tests!
> - "Standard benchmarks". Memcached's contributions have historically
> been based on "we ran this in production, here're some tests and
> proof it works fine". Recently it's been more about rapid cleanups
> and the new storage engine independence. It'd be really great to
> have some (say libmemcached based) tests to exercize (under
> realistic AND unrealistic measures!) everything.
> - More documentation. I figure eventually someone will write docs
> with better english, but there're tons of ideas floating around. We
> should get them all down. Along with standard troubleshooting info,
> of which there is practically none!
> - Cache::Memcached updates. Uh, whoops. This doesn't support any of
> the recent features.
> On community related measures:
> - Buildbot. I totally started on this, but have not completed it. I
> want to offer a buildbot network to the developer community. If you
> want a platform supported, provide a slave. Otherwise we will not
> target it (besides a couple of standby's).
> - GIT. SVN is dead. We'll need to move on to git or hg in order to
> scale the developer pace. It's my understanding that just as many
> folks on the list prefer GIT as they do Mercurial, and SVN is mainly
> a convenience. Anything will be better. I could crack out weekly
> releases if I didn't have to worry about SVN branches/merges.
> - Add more committers. I promise you guys; we're in an awkward
> position with commit access, but people at sixapart absolutely
> acknowledge and care about it. I've been having meetings,
> discussions, etc, about dealing with this sooner than later. I
> mostly help with memcached on my own time. However, sixapart does
> love memcached and the rest of the danga projects. I can't say
> anything definitive, of course, but I feel like I can share the
> feeling of "we give a shit" with all of you folks.
> Part of having more committers means we need more dedicated
> reviewers. Memcached is a stable project trusted by many, many
> - We should do more, less formal, hackathons. These are great for
> hashing out minor issues, and I feel with more frequence they'd also
> be really good for cranking out code, too ;)
> That's all for now.
> I'm missing a number of line items (please don't be offended if I
> missed mentioning your submissions!) If it's in my inbox, it'll get
> a fair review and probably go in.
> have fun,
More information about the memcached