Perl's autoflush() and Gearman workers

Chris Goffinet goffinet at yahoo-inc.com
Wed Jan 9 07:21:46 UTC 2008


Yeah I found out that gearman is super slow :-D Will be working to see  
if we can improve. This is only when you do normal job types in  
parallel, not background jobs. We have been benchmarking more workers  
== greater latency for X tasks. So if i submitted 25 tasks with 4  
workers running idle (that just returned the number 1) and I started  
10 more workers, latency with gearman increases dramatically. Over  
100ms in some cases.

I am interested in the gearman c version and will be doing tests using  
libevent.


-- 
Chris Goffinet
MyBlogLog Engineer
Yahoo!
San Francisco, CA
United States

On Jan 8, 2008, at 4:22 PM, Joe Stump wrote:

> So I've been digging and digging and would love for someone who  
> actually works on Gearman to read this message and possibly reply  
> back to me. I've been working with Chris G. from Yahoo! on a PHP  
> library called Net_Gearman (which I've been spamming the list with  
> updates about).
>
> I've finally got my PHP code running as fast as the Perl one does  
> with a single worker running a few tasks. The problem, however, is  
> that when I add workers with the Perl code the time it takes the  
> tasks to run halves (which is obvious as it's running them in  
> parallel). The same is exactly the opposite with PHP. It goes from  
> 11ms to run 25 tasks with a single worker to taking 50ms. I've  
> verified that each worker is, in fact, getting an equal number of  
> jobs, etc.
>
> The problem is that a socket_select() waits around on the client  
> side for anywhere from 30 and 45ms waiting for data when I know the  
> worker has already written data back to gearmand.
>
> I have a nagging feeling this has something to do with Perl's  
> autoflush on it's IO::Socket, but I can't for the live of me figure  
> out what $| = 0 and $| = 1 actually does under the hood to "auto  
> flush" the socket. My thinking is that the socket isn't autoflushing  
> in my PHP code and, as a result, isn't sending data as soon as I  
> write it to the socket from the worker.
>
> Thoughts?
>
> --Joe
> _______________________________________________
> Gearman mailing list
> Gearman at lists.danga.com
> http://lists.danga.com/mailman/listinfo/gearman



More information about the Gearman mailing list