Gearman is slow. More workers == Slower

Joe Stump joe at digg.com
Wed Jan 9 18:39:52 UTC 2008


So after a few weeks of banging my head against why the Perl client  
code was so much faster I realized that I was benchmarking the Perl  
client incorrectly. After fixing that I'm happy to report the PHP code  
is quite a bit faster than the regular Perl client with lots of  
workers (a little slower with only 1 or 2 workers). That being said,  
I've done some benchmarking and I'm convinced of two things:

1.) Gearman is slow for doing "parallel" data requests.
2.) More workers makes Gearman slow down and not speed up as one might  
expect (e.g. if 1 worker takes 20ms to run two tasks adding another  
worker doesn't drop that time down to 10 or 12ms ran in parallel).

My tests were this:
1.) Write a simple noop test in Perl, PHP and Python
2.) Run a task set with 25 tasks in it against 1 worker, 2 workers and  
50 workers.
3.) Using only a single gearmand server.

Here are the results:
http://bugs.joestump.net/code/Net_Gearman/benchmarks/results.html

I'm especially concerned that adding more workers (and supposedly  
running tasks in parallel) doesn't actually speed things up in any  
way. Is anyone actually using Gearman in production for parallel data  
requests? If so, what is your setup? How are you able to get this  
running any faster than serial data requests? How many workers do you  
run per gearmand?

--Joe


More information about the Gearman mailing list