automatically spawn new workers

Joe Stump joe at digg.com
Wed Dec 26 00:38:12 UTC 2007


> "You'd need to have a process monitoring how many workers are being  
> actively worked and then spawn more from that." -- How can we  
> understand that a worker is actively working at the moment?

A bit tricky to be totally honest. The Net_Gearman package (and the  
Python package it's modeled after) allow you to pass a monitoring  
function to the workers that will tell you how much they're being  
worked (roughly), but not really tell you if it's actually working.  
You can have that gather statistics about how often the job is working  
and use the other listeners for logging how long the average jobs are  
taking. Based on those numbers you could probably say "Well, on  
average we run 600 jobs a minute, which means we have 10 workers a  
second busy and we're only running 15 workers. Maybe we should bump  
that to 20 workers."

Hope that helps. I'm not entirely sure how to do this as we've only  
started using it in production and in favor of saving time, we just  
fired up a boatload of workers (also we know we get X submissions a  
day which is the only place we use Gearman on Digg right now so it's  
easy to estimate how many workers we need).

--Joe



More information about the Gearman mailing list