System architecture best practices - memcached + webfarms

[Earl Cahill]

FWIW, I just bought one of these, which can hold 32 gigs of ram (16 x 2)

http://www.sun.com/servers/x64/x2200/

So, that might replace fourteen or fifteen of your two gig boxes.  They say the box can support 64 gigs, but I am not sure I believe them.

Earl

----- Original Message ----
From: Timo Ewalds <timo at tzc.com>
To: Brad Fitzpatrick <brad at danga.com>
Cc: memcached at lists.danga.com; Richard Jones <rj at last.fm>
Sent: Tuesday, July 3, 2007 11:54:57 AM
Subject: Re: System architecture best practices - memcached + webfarms

One technique that should work and that would make this easier (as you 
wouldn't have to pass the userid every time), would be to use a 
consistent hash on the key. (If you don't know what that is, look at 
http://lists.danga.com/pipermail/memcached/2006-October/002919.html for 
how it can be applied to adding/removing servers nicely). In the example 
below, "user17" is the same for the two keys, so if you can put them 
close on the number line, they're likely to go to the same server. I 
figured the way to do that would be have the top 24 bits be a hash of 
the prefix (ie "user17"), and the low 8 bits a hash of the suffix (ie 
"name" or "location"). Assuming all your requests in the get_multi have 
the same prefix (which I've found to be true in my application), you'll 
only hit a couple servers. The problem with this method of course is 
that it has the potential to be very unbalanced (ie some servers will be 
hit hard, while others will be empty), since the keys will be grouped 
much more strongly. I think the balance vs grouping issue can be solved 
with a bit of experimentation with the hash function and how many bits 
go to the prefix vs the suffix.

Timo

Brad Fitzpatrick wrote:
> In the Perl client, at least, your key can contain an explicit
> numeric hashing value, which has two benefits: 1) it's quicker to do $int
> % $num_servers than do a crc32, and 2) locality... you tend to hit one
> server if most the memcache objects you're requesting are, say, from the
> same user.
>
> So instead of:  (contrived example:)
>
>    $memc->get_multi("user17:name", "user17:location")
>
> which would do two crc32s, and likely hit two servers, we do:
>
>    $memc->get_multi([17, "user17:name"], [17, "user17:location"])
>
> No crc32s, and they'll both go to the same node.
>
> That's one trick.
>
> But even without that, your web nodes should already have all the TCP
> connections open to all the memcaches (or most of them), so then your
> get_multi implementation should just do a non-blocking write to all of
> them "at once" (actually in serial, but you're not waiting for a reply, so
> network latency, so it's pretty much "immediate"), then you select/epoll
> on them all at once, reading from them in order.  If your implementation
> does a serial get_multi to each one, then the network latency over 100
> requests will kill you, and you should fix the client API you're using.
>
> So basically you can avoid hitting a lot of servers usually, but even if
> you have to, it's shouldn't be _that_ bad.
>
> - Brad
>
>
> On Mon, 2 Jul 2007, Richard Jones wrote:
>
>   
>> I've been thinking about what changes we may have to make to our memcached
>> installation in future as we continue to grow - our webfarm is approaching
>> 100 servers, each with 4GB ram, ~2GB of which is dedicated to memcached on
>> each machine.
>>
>> As we add more nodes, the usefulness of get_multi decreases - it's possible
>> for a single page to hit almost all of the memcached instances. I read
>> somewhere that facebook partition their memcached cluster to improve
>> get_multi performance (eg, all user data on a subset of mc nodes). Can anyone
>> comment on the effectiveness of this?
>>
>> Are we fighting a losing battle here - perhaps we should try and cram as much
>> ram as possible into a small number of machines (what do you do?). get_multi
>> would be more useful, but it costs more and doesn't seem as elegant :(
>>
>> Can anyone comment on how many memcache lookups they make per page?
>> Traditionally our developers treated memcache lookups as "free", but when
>> you're doing a few hundred memcache gets per page it soon adds up..
>>
>> Thanks,
>> RJ
>>
>>
>>
>> --
>> Richard Jones
>> Last.fm Ltd. | http://www.last.fm/
>> Office: +44 (0) 207 780 7080
>>
>>
>>     
>
>   







 
____________________________________________________________________________________
Be a PS3 game guru.
Get your game face on with the latest PS3 news and previews at Yahoo! Games.
http://videogames.yahoo.com/platform?platform=120121
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20070708/c1e45d54/attachment.html