Size limit for tasks?
Paul Goracke
paul at goracke.org
Mon Apr 21 15:39:15 UTC 2008
Is there a known size limitation for submitted tasks? I'm running into
problems with tasks failing if they exceed a certain size. To make
matters more frustrating, the size doesn't seem to be a constant: At
first it seemed to be the magic 64k, but that number has varied as
I've been testing. One of the strange aspects of this is that I'm only
sending out data I have received from other workers--so Gearman is
handling apparently arbitrarily-sized task results, but not task params.
I realize some may consider this an abuse of Gearman, but if so I sure
would have liked to have known it when our project started. If anybody
knows any workarounds or pointers to the relevant source (I am
continuing to dig into it) I would appreciate it. Following are the
test scripts I'm currently using.
First noticed on Ubuntu hardy x86-64 kernel 2.6.24-11, packages:
gearman-server 1.09-1, libgearman-client-perl 1.09-1, libdanga-socket-
perl 1.57-1, libsys-syscall-perl 0.22-2. Also exhibited on Mac OS X
Intel (although at a higher size limit) 10.5.2, Gearman and
Gearman::Server 1.09, Danga::Socket 1.57, Sys::Syscall 0.22
Thanks for any insight.
pg
# 8<----------------------------------
#!/usr/bin/perl
# testworker.pl
# Assumes gearmand running on localhost
use strict; use warnings;
use Gearman::Worker;
my $worker = Gearman::Worker->new( job_servers => ['127.0.0.1:7003'] );
# return a string of a given length
$worker->register_function(
'string_of_length',
sub {
my $size = $_[0]->arg();
return 'x' x $size;
}
);
# return the length of a given string
$worker->register_function(
'length_of_string',
sub {
my $string = $_[0]->arg();
return length($string);
}
);
$worker->work() while 1;
# 8<----------------------------------
#!/usr/bin/perl
# tasklength.t
# Assumes gearmand running on localhost, and testworker.pl registered
use strict; use warnings;
use Test::More qw( no_plan );
BEGIN { use_ok 'Gearman::Client'; }
isa_ok( my $client = Gearman::Client->new( job_servers =>
['127.0.0.1:7003'] ), 'Gearman::Client' );
# test some easy sizes, then jump the size to guarantee failures
for my $size ( 50, 500, 100_000, 500_000 ) {
my $result = $client->do_task( 'string_of_length', $size );
is( $$result, 'x' x $size, "received string of length $size" );
# this fails at 100_000 on ubuntu, 500_000 on mac
my $result2 = $client->do_task( 'length_of_string', $result );
is( $$result2, $size, "received correct length ($size) of string" );
}
# 8<----------------------------------
More information about the Gearman
mailing list