Best practice for a web farm?

[Brian Moon]

>> My point: if you want your web nodes to *take a beating* (and I've seen
>> this happen repeatedly from spambots and trackback botnets) don't put
>> memcache on your webnodes. Put your memcache on nodes that are well
>> protected from memory starvation ...like dedicated boxes or an NFS server.
> 
> Funny, my webnodes don't do that... Regardless, anything I say is
> assuming the rest of your system is tuned properly. If your webserver
> will run out of RAM before it runs out of CPU, you obviously don't have
> RAM free to put memcached there.

I agree with dormando here.  There is something Rasmus Lerdorf told me a 
long time ago that I laughed at the time.  He suggested starting all the 
apache processes that server will ever use at start up and not having a 
chance to have a siege on your apache daemon.  IMO, having Apache 
attempt to fork hundreds of children on demand is really bad for your 
server.  I have adopted this system for our servers at dealnews.com and 
my personal servers I run.

We run Apache with the worker MPM and mod_PHP.  We are cowboys in some 
people's eyes for doing that.  But, it works quite well when you control 
the systems.

<IfModule worker.c>
     StartServers         16
     MaxClients         1024
     MinSpareThreads    1024
     MaxSpareThreads    1024
     ThreadsPerChild      64
     MaxRequestsPerChild   0
</IfModule>

Yes, that is 1,024 MaxClients.  And yes, they run all the time with no 
problems.  Each processes uses about 100MB of ram each.  So, Apache 
uses 1.6GB of RAM and never grows.  If we need more power than that, we 
need to buy another server or refactor something.

Each of these server has a 512MB memcached instance running on them and 
have a total of 4GB of ram.  They never hit swap.  They maintain 1GB+ of 
free RAM at all times.

These servers are our front facing servers.  They don't do heavy CPU 
work.  They serve static content and light weight scripts like 
redirects.  Their main job is serving as a caching proxy.  We have a 
custom application proxy that uses PHP and memcached to serve nearly all 
of our pages.  It is much like perlbal for dormando.

On our appliation servers, we run with these settings:

<IfModule worker.c>
     StartServers            10
     MaxClients             250
     MinSpareThreads        250
     MaxSpareThreads        250
     ThreadsPerChild         25
     MaxRequestsPerChild   1000
</IfModule>

Each apache process on these servers uses about 30MB.  So, at the most, 
  Apache would use 300MB of RAM.  We have a PHP memory limit set to 
ensure that our scripts can't exceed our memory as well.  These servers 
have a 256MB memcached instance on them.  So, like the others, they have 
1GB+ free at all times.

In all cases, like dormando said, if you are running out of RAM before 
you run out of CPU, you are doing it wrong.

For the record, I am not telling you to use Apache Worker, but I would 
suggest you not dismiss it as bad until you have tested out your 
application with it.  It saves us loads of RAM since the PHP engine uses 
the shared memory of the process instead of each child having its own 
block of memory.

-- 

Brian Moon
Senior Developer/Engineer
------------------------------
When you care enough to spend the very least.
http://dealnews.com/