Optimising use and using multiple memcache servers in a pool

[Jason Edgecombe]

Hi Alan,

After reading your email, I still sense some confusion. I just wanted to 
throw out some examples to clarify things. I apologize if this is 
unnecessary.

Memcache is a distributed cache.  Given three memcache servers and 
objects number 1..9, the objects may be distributed as follows:
server1: 1, 4, 7
server2: 2, 5, 8
server3: 3, 6, 9

(I don't know how the hashing algorithm works, this is a simplification)

When client asks for object 1, it fetches it from server1, even if the 
client is server2 or server3.

The main reason to run multiple memcache servers is to increase the size 
of your cache to store more data or to ensure that only a portion of the 
cache is lost if a server goes down.

The main overhead with memcache vs APC would be the network latency.

If you have 3 memcache servers, then at worst, each client has a tcp 
connection to each server (assuming one thread on each client).

non-memcache workflow is as follows:
client read: select from db
client write: update db

the typical memcache workflow is as follows:

client read: fetch object from cache if available, else select from db. 
put object in cache if it didn't exist.
  other clients will fetch the cached copy after it's cached

client write: fetch object from db/cache, update db, update cache.

The client write procedure can be a little tricky. Fetching directly 
from the db avoids cache inconsistencies. memcache doesn't have 
transactions, so I'm not sure if some type of "I'm updating the db" 
token might be useful to store, or just have the updater overwrite 
what's in the cache.


Alan Jay wrote:
> Thanks Marcus,
>
>   
>>> In my simple implementation I have 7 web servers each running their
>>> own copy of "memcached" on each server the local php code selects
>>> that local memcached server and checks to see if the article it is
>>> loading has been cached.  And loads it if it has.  The Article
>>> Management System when someone edits an article resets the expire
>>> time from 6hrs to 5 seconds on each of the servers in turn when the
>>> article is updated.
>>>       
>
> To explain further - in this particular application only ONE machine (in fact
> different from front end server) can update the records.
>
>   
>>> This seems to be working OK this morning and we will get more stats
>>> over time but the pecl-memcahced interface has the ability to run
>>> multiple servers by adding and opening them all for selection.  But
>>> if (as I am) my script opens a single file it seems like quite a
>>> lot of potential overhead and I'm not sure what the advantages to
>>> doing it like that are.
>>>       
>
>   
>> Hm. I think you've also missed what memcached does to some extent
>> (see below). I'm not sure where the breakpoint will lie. I'd be
>> tempted to try running just a few instances (say 3-4) that are used
>> by all servers. That way you lower the overhead of having so many
>> connections to make, though of course it's also dependent on how much
>> stuff you are putting in the cache. Remember that a single cache/
>> retrieve operation will only ever result in a single request to one
>> memcached server, no matter how many servers you have.
>>     
>
> OK my worry was that each time a users requests an article then you have to
> open up ALL the memcached servers which sounds like quite a lot of overhead
> for a single transaction.   Though you knowledgeable people may tell me this
> is not actually an issue.
>
> I think understand the trade off you are explaining and it makes sense. 
>
>   
>>> At present the code already fails safe in that if it is unable to
>>> connect to the local memecached server it just gets the data from
>>> mySQL server direct.
>>>
>>> What advantages are there to using the multiple memcached servers
>>> added to the connection pool?
>>>       
>
>   
>> One server can benefit from getting something from the cache that was
>> put there by another server. 
>>     
>
> OK so when you add to the pool it adds to ALL the servers.
>
>   
>> By not sharing your servers, you're
>> missing out on this big feature. Also, if you're only doing local
>> caching, you're far better off doing it in APC as it will be much
>> faster. 
>>     
>
> We are already using APC but I thought that memcached was doing something
> different.  In my context it is the difference between caching the PHP and the
> data from the mySQL database (which is what we are trying to protect).
>
> This particular application is slightly unusual in that our goal is to reduce
> the activity on our mySQL database for example if an article is read by 10,000
> over a couple of hours it is a great improvement for the article to be read
> once for each cache 8 times rather than 10,000. (or have I missed something).
>
> However I do see what you are saying and I think I am beginning understanding
> what the implications are.  It all comes back to the amount of overhead
> imposed on opening up the pool instead of a single server.
>  
>   
>> Also, by not sharing your caches, you're just asking for
>> coherency problems - where different servers end up with different
>> versions of the same item, e.g.
>>
>> server A fetches object 1, stores in local cache
>> server B fetches object 1, stores in local cache
>> server A saves a change to object 1, updates its local cache
>>
>> Server B now has outdated version of object 1.
>>
>> By having all servers use the same list of memcached servers, you
>> avoid this problem.
>>     
>
> OK - in this context this is not an issue but I can see how it could be an
> issue and the advantages of doing it properly.
>
> In this context this application (I think) seems to work fine, but we do have
> another application (a forum) that can use memcached if we upgrade to the
> latest version.  So I suspect we will have to alter in line with the way this
> does things in the future.
>
> As you can tell I am just dipping into this complex issue primarily because
> our site was under very heavy load the last few days up from 2.5 million pages
> a day to 4 million pages per day and we have been trying to reduce the amount
> of access to the database by this application so that our other applications
> don't cause the database server to crash.
>
> Re-reading the description on the DANGA website it suggests placing a cache on
> every HTML server.  But again my question about loading the entire pool each
> time a query is required and the overhead this imposes is a curiosity.
>
> Marcus from your comments you seem to suggest that a good compromise might be
> to local 4 memcached servers and use them as a pool as opposed to 8 server in
> the pool or 8 servers run individually.
>
> I'd be interested in any other comments or thoughts from some of you who have
> been using this tool for some time.
>
> Alan
>
> PS After a very difficult day on Thursday, Friday went off with out any major
> issues after implementing this code early Friday morning.  It was our second
> busiest day with 4 million page views for the day and a peak hour at 9pm with
> 324,000 page views.
>  
>   
>> Marcus
>> --
>> Marcus Bointon
>>     
>
>
>