spymemcached, ketama, and adding new servers

[Dave Peckham]

Hi,

I wrote a simple junit test to challenge my assumptions about spy
memcached's behavior in the "add new servers" use case. Here's what I
found.

1) fire up 10 server instances
2) create a spy memcached client using ketama hash:

    ConnectionFactory connFactory = new DefaultConnectionFactory(
        DefaultConnectionFactory.DEFAULT_OP_QUEUE_LEN,
        DefaultConnectionFactory.DEFAULT_READ_BUFFER_SIZE,
        HashAlgorithm.KETAMA_HASH);

    client = new MemcachedClient(connFactory, addresses);

3) write 100 never-expiring objects to the client. (quick check of
cur_items on each server showed very even distribution)
4) fire up an 11th server instance
5) destroy old client and create another one, with the new server
address added to the previous address list
6) try to fetch the 100 objects and count the hits

The result is that I only got 14 hits against the 100 entries. This
doesn't seem like a practical improvement over mod-N hashing, so I
hope I am doing something wrong or completely misunderstanding the
concept of consistent hashing.

Ideas?

Thanks,
Dave