Gearman is slow. More workers == Slower
joe at digg.com
Wed Jan 9 18:39:52 UTC 2008
So after a few weeks of banging my head against why the Perl client
code was so much faster I realized that I was benchmarking the Perl
client incorrectly. After fixing that I'm happy to report the PHP code
is quite a bit faster than the regular Perl client with lots of
workers (a little slower with only 1 or 2 workers). That being said,
I've done some benchmarking and I'm convinced of two things:
1.) Gearman is slow for doing "parallel" data requests.
2.) More workers makes Gearman slow down and not speed up as one might
expect (e.g. if 1 worker takes 20ms to run two tasks adding another
worker doesn't drop that time down to 10 or 12ms ran in parallel).
My tests were this:
1.) Write a simple noop test in Perl, PHP and Python
2.) Run a task set with 25 tasks in it against 1 worker, 2 workers and
3.) Using only a single gearmand server.
Here are the results:
I'm especially concerned that adding more workers (and supposedly
running tasks in parallel) doesn't actually speed things up in any
way. Is anyone actually using Gearman in production for parallel data
requests? If so, what is your setup? How are you able to get this
running any faster than serial data requests? How many workers do you
run per gearmand?
More information about the Gearman