spymemcached, ketama, and adding new servers

[Dave Peckham]

Hi Richard,

Thanks for the reply. This is encouraging in that you prove my
understanding and expectations are in fact correct. That means that
it's worth my time to continue to figure out what I am doing wrong!

At 100,000 entries, I still got only ~10% hit rate when adding a new
server to my client.

I'm going to create a new project that illustrates my problem using
simpler code and if my new tests don't provide me with results like
yours, I'll at least have some code that I can share (perhaps with
Dustin) to clarify my incorrect usage.

Cheers,
Dave

On Thu, Mar 6, 2008 at 10:37 AM, Richard Jones <rj at last.fm> wrote:
> I updated "make test" for libketama yesterday to add a downed server test that
>  was posted to this list a couple of months back:
>
>  $ wc -l ../ketama.servers
>  10 ../ketama.servers
>  $ ./downed_server_test.sh
>  Downed server test, setting up..
>   Running on original set of servers to get baseline
>   Running on original set of servers minus one
>   Analyzing results...WARNING 106615 items disturbed (10%)
>   Running on original set of servers plus one
>   Analyzing results...WARNING 40536 items disturbed (4%)
>  Finished downed server test.
>
>  Seems like a sane amount of changes.
>  ketama_test runs with 1,000,000 keys - do you see better results if you use
>  more keys?
>
>  RJ
>
>
>  PS, latest ketama stuff is here: svn://svn.audioscrobbler.net/misc/ketama
>  altho there haven't been many changes of late.
>
>
>
>
>
>
>  On Thursday 06 March 2008 17:25:56 Dave Peckham wrote:
>  > Hi,
>  >
>  > I wrote a simple junit test to challenge my assumptions about spy
>  > memcached's behavior in the "add new servers" use case. Here's what I
>  > found.
>  >
>  > 1) fire up 10 server instances
>  > 2) create a spy memcached client using ketama hash:
>  >
>  >     ConnectionFactory connFactory = new DefaultConnectionFactory(
>  >         DefaultConnectionFactory.DEFAULT_OP_QUEUE_LEN,
>  >         DefaultConnectionFactory.DEFAULT_READ_BUFFER_SIZE,
>  >         HashAlgorithm.KETAMA_HASH);
>  >
>  >     client = new MemcachedClient(connFactory, addresses);
>  >
>  > 3) write 100 never-expiring objects to the client. (quick check of
>  > cur_items on each server showed very even distribution)
>  > 4) fire up an 11th server instance
>  > 5) destroy old client and create another one, with the new server
>  > address added to the previous address list
>  > 6) try to fetch the 100 objects and count the hits
>  >
>  > The result is that I only got 14 hits against the 100 entries. This
>  > doesn't seem like a practical improvement over mod-N hashing, so I
>  > hope I am doing something wrong or completely misunderstanding the
>  > concept of consistent hashing.
>  >
>  > Ideas?
>  >
>  > Thanks,
>  > Dave
>
>
>
>  --
>  Richard Jones
>  Last.fm Ltd. | http://www.last.fm/
>  Office: +44 (0) 207 780 7080
>