Upcoming memcached releases + rambling.

[dormando]

Yo all,

The original plan for today was to pop on the list and yell "Hey! Have 
some -rc's to test! Go nuts! Thanks for all the patches!" - but my 
awesome  fever for the last few days has pushed that off by ... a few 
more days.

But it's totally cool, because it's happening. For the first time in a 
long time memcached will act like it's releasing early, often. Thanks to 
everyone for all of the recent code submissions and bugfixes :)

So we have two -rc's which are coming up as soon as I can wrap a few bug 
reports and merge all this code:

1.3.0-rc (the binary tree, in its current state)
1.2.5-rc (non-binary branch with all of the recent fixes and patches)

There might not be an immediate merge of 1.2.5 -> 1.3.0, but I'm not 
convinced that matters immediately.

I believe should probably give the binary tree a couple of real 
development releases until we've had a few client implementations. So in 
the case of the protocol having to change due to issues with actual 
client integration, we won't be backing out on a stable API.

SO! I'd like to lay out my own wishlist for memcached, some of which is 
underway, some not:

- Binary protocol. It's been too long, lets finish it. We need more 
native tests, and we need to benchmark its speed relative to the text 
protocol, make sure we're actually coming out ahead here.

- Dynamic storage engines. Toru has a promising proof of concept that 
I'd love to merge for an eventual 1.4 release.

Most of the memcached forks have been due to folks not liking the 
storage method. While I disagree with how a lot of these forks are done, 
there's definitely merit in the flexibility. A lot of these folks start 
from scratch, but I believe memcached should be a simple, efficient, 
network platform for dead simple key=value datastores, no matter how 
they're actually implemented.

- Dedupe the error messages. We have 11 instances of "SERVER_ERROR out 
of memory". This makes it impossible to even guess what happened, even 
with full context. Did we fail to allocate a msghdr? Did do_item_alloc 
fail? Who knows. We can do this for 1.2.5 if there're people on board. 
I'll float the patches.

- Add stats about failures. I'd love counters for all of the times 
something failed without causing an eviction. Sometimes machines will be 
hosed for a long time before we realize the hit rate is way down due to 
the OS refusing 50% of the mallocs. Hard to tell if you've hit maxconns, 
blah blah blah. More easy work, maybe for 1.2.5.

- Method of dumping refcounts of "locked" items. I'm tracking down what 
I suspect is a refcount bug, and I expect us to have more of these in 
the future. It'll be great to be able to verify these offhand. If 
someone isolates a bum server, they can dump slabs, refcount, etc, and 
send to the list.

- Full item dump. I guess we should do this with the appropriate 
caveats. There's a potential patch floating around, we'll see about it.

- Fixing a lot of dynamic memory buffers. Per connection, per freelist, 
whatever. A lot of these can be turned into linked lists if, say, the 
'item' struct had a 'next' pointer. This should remove most of the 
various realloc logic.

- Clean up the network iov's to work like above, and also manage generic 
freelisted buffers. Which would remove the need for the specialized cas 
suffix buffer. These should be wrapped so they can be linklisted, 
instead of managed via dynamic arrays per connection. Could also be used 
with the binary protocol to reduce syscalls.

- TCP_CORK/TCP_NOPUSH where appropriate. I totally agree here, and it 
used to explicitly do this.

- New slabber engine. I view this as a bonus to making the slabber code 
dynamic. We can start cooking an experimental new slabber to do 
slab-reassignment, fragmentation avoidance, etc. Without really hosing 
mainline.

- More tests. We need better test coverage. Please submit tests!

- "Standard benchmarks". Memcached's contributions have historically 
been based on "we ran this in production, here're some tests and proof 
it works fine". Recently it's been more about rapid cleanups and the new 
storage engine independence. It'd be really great to have some (say 
libmemcached based) tests to exercize (under realistic AND unrealistic 
measures!) everything.

- More documentation. I figure eventually someone will write docs with 
better english, but there're tons of ideas floating around. We should 
get them all down. Along with standard troubleshooting info, of which 
there is practically none!

- Cache::Memcached updates. Uh, whoops. This doesn't support any of the 
recent features.

---
On community related measures:

- Buildbot. I totally started on this, but have not completed it. I want 
to offer a buildbot network to the developer community. If you want a 
platform supported, provide a slave. Otherwise we will not target it 
(besides a couple of standby's).

- GIT. SVN is dead. We'll need to move on to git or hg in order to scale 
the developer pace. It's my understanding that just as many folks on the 
list prefer GIT as they do Mercurial, and SVN is mainly a convenience. 
Anything will be better. I could crack out weekly releases if I didn't 
have to worry about SVN branches/merges.

- Add more committers. I promise you guys; we're in an awkward position 
with commit access, but people at sixapart absolutely acknowledge and 
care about it. I've been having meetings, discussions, etc, about 
dealing with this sooner than later. I mostly help with memcached on my 
own time. However, sixapart does love memcached and the rest of the 
danga projects. I can't say anything definitive, of course, but I feel 
like I can share the feeling of "we give a shit" with all of you folks.

Part of having more committers means we need more dedicated reviewers. 
Memcached is a stable project trusted by many, many websites.

- We should do more, less formal, hackathons. These are great for 
hashing out minor issues, and I feel with more frequence they'd also be 
really good for cranking out code, too ;)

---
That's all for now.

I'm missing a number of line items (please don't be offended if I missed 
mentioning your submissions!) If it's in my inbox, it'll get a fair 
review and probably go in.

have fun,
-Dormando