automatically spawn new workers
jt at aaanet.ru
Wed Dec 26 01:03:04 UTC 2007
I see. Thank you very much. I think we'll start with boatload of workers
too. Then we'll analyze results and think if we need to patch gearmand for
to be able to have a list of free workers or not.
>> "You'd need to have a process monitoring how many workers are being
>> actively worked and then spawn more from that." -- How can we understand
>> that a worker is actively working at the moment?
> A bit tricky to be totally honest. The Net_Gearman package (and the
> Python package it's modeled after) allow you to pass a monitoring
> function to the workers that will tell you how much they're being worked
> (roughly), but not really tell you if it's actually working. You can have
> that gather statistics about how often the job is working and use the
> other listeners for logging how long the average jobs are taking. Based
> on those numbers you could probably say "Well, on average we run 600 jobs
> a minute, which means we have 10 workers a second busy and we're only
> running 15 workers. Maybe we should bump that to 20 workers."
> Hope that helps. I'm not entirely sure how to do this as we've only
> started using it in production and in favor of saving time, we just fired
> up a boatload of workers (also we know we get X submissions a day which
> is the only place we use Gearman on Digg right now so it's easy to
> estimate how many workers we need).
More information about the Gearman