automatically spawn new workers
joe at digg.com
Wed Dec 26 00:38:12 UTC 2007
> "You'd need to have a process monitoring how many workers are being
> actively worked and then spawn more from that." -- How can we
> understand that a worker is actively working at the moment?
A bit tricky to be totally honest. The Net_Gearman package (and the
Python package it's modeled after) allow you to pass a monitoring
function to the workers that will tell you how much they're being
worked (roughly), but not really tell you if it's actually working.
You can have that gather statistics about how often the job is working
and use the other listeners for logging how long the average jobs are
taking. Based on those numbers you could probably say "Well, on
average we run 600 jobs a minute, which means we have 10 workers a
second busy and we're only running 15 workers. Maybe we should bump
that to 20 workers."
Hope that helps. I'm not entirely sure how to do this as we've only
started using it in production and in favor of saving time, we just
fired up a boatload of workers (also we know we get X submissions a
day which is the only place we use Gearman on Digg right now so it's
easy to estimate how many workers we need).
More information about the Gearman