Size limit for tasks?
paul at goracke.org
Mon Apr 21 16:50:27 UTC 2008
On Apr 21, 2008, at 9:33 AM, Joe Stump wrote:
> On Apr 21, 2008, at 8:39 AM, Paul Goracke wrote:
>> Is there a known size limitation for submitted tasks? I'm running
>> into problems with tasks failing if they exceed a certain size. To
>> make matters more frustrating, the size doesn't seem to be a
>> constant: At first it seemed to be the magic 64k, but that number
>> has varied as I've been testing. One of the strange aspects of this
>> is that I'm only sending out data I have received from other
>> workers--so Gearman is handling apparently arbitrarily-sized task
>> results, but not task params.
> When you say "failing" do you mean a hard fail or that they fail
> when things are unserialized?
It's a hard fail. do_task() returns undef; running in a taskset will
call a on_retry handler the appropriate number of times, then on_fail.
> With my PHP client/worker code I was running into UTF8 problems when
> using strlen() (which isn't unicode aware). If your results can be
> in UTF8 and the Perl code isn't accounting for that then things
> could be blowing up for that reason. Again, this was a PHP-only
> problem I ran into ...
That was one of my initial suspects; the "real product" is dealing in
UTF8 strings within Perl objects serialized via Storable's nfreeze()
for task submission. (The length() call in Perl is unicode-aware, but
is not used in my product, just these test scripts.) But the test I
attached still fails just by creating a string of 'x' characters yet
properly handles the same characters in a shorter string, so I have
ruled it out--at least for this issue.
Very preliminary debugging on the server side seems to indicate the
server _is_ getting all of the task but then for some reason is being
failed off the job queue--I have never seen it reach the worker.
Curiouser and curiouser.
More information about the Gearman