System architecture best practices - memcached + webfarms

[Richard Jones]

I've been thinking about what changes we may have to make to our memcached 
installation in future as we continue to grow - our webfarm is approaching 
100 servers, each with 4GB ram, ~2GB of which is dedicated to memcached on 
each machine.

As we add more nodes, the usefulness of get_multi decreases - it's possible 
for a single page to hit almost all of the memcached instances. I read 
somewhere that facebook partition their memcached cluster to improve 
get_multi performance (eg, all user data on a subset of mc nodes). Can anyone 
comment on the effectiveness of this?

Are we fighting a losing battle here - perhaps we should try and cram as much 
ram as possible into a small number of machines (what do you do?). get_multi 
would be more useful, but it costs more and doesn't seem as elegant :(

Can anyone comment on how many memcache lookups they make per page? 
Traditionally our developers treated memcache lookups as "free", but when 
you're doing a few hundred memcache gets per page it soon adds up.. 

Thanks,
RJ



-- 
Richard Jones
Last.fm Ltd. | http://www.last.fm/
Office: +44 (0) 207 780 7080