automatically spawn new workers

Eugene Toropov jt at
Wed Dec 26 01:03:04 UTC 2007

I see. Thank you very much. I think we'll start with boatload of workers 
too. Then we'll analyze results and think if we need to patch gearmand for 
to be able to have a list of free workers or not.


>> "You'd need to have a process monitoring how many workers are being 
>> actively worked and then spawn more from that." -- How can we  understand 
>> that a worker is actively working at the moment?
> A bit tricky to be totally honest. The Net_Gearman package (and the 
> Python package it's modeled after) allow you to pass a monitoring 
> function to the workers that will tell you how much they're being  worked 
> (roughly), but not really tell you if it's actually working.  You can have 
> that gather statistics about how often the job is working  and use the 
> other listeners for logging how long the average jobs are  taking. Based 
> on those numbers you could probably say "Well, on  average we run 600 jobs 
> a minute, which means we have 10 workers a  second busy and we're only 
> running 15 workers. Maybe we should bump  that to 20 workers."
> Hope that helps. I'm not entirely sure how to do this as we've only 
> started using it in production and in favor of saving time, we just  fired 
> up a boatload of workers (also we know we get X submissions a  day which 
> is the only place we use Gearman on Digg right now so it's  easy to 
> estimate how many workers we need).
> --Joe

More information about the Gearman mailing list