Not enough checking in MogileFS::Client, chunked uploads not respecting the chunk size asked for.

[dormando]

I believe it was set down that low (in a couple places! Perlbal, 
mogilefs...) because larger values were occasionally crashing perl for 
some people. Making it optional looks totally reasonable though.

I'll review/apply this if someone doesn't beat me to it. At a glance it 
looks fine but I like doing quick style/test checks ;)

-Dormando

Tomas Doran wrote:
> Hiya
> 
> I've found (what I think are) 2 problems in the Mogile client library.
> 
> 1) In the store_file method, there is a loop which reads segments of the
> file, and prints them to the file handle (which depending on which
> client backend you're using, does different things). The block size is
> fixed to 8192 bytes here, I've changed it to use the supplied
> readLineChunkSize param. 
> 
> We're doing chunked (range PUT, with ClientHTTPFile) uploads of some
> very big files, and this change makes a *massive* difference to the
> throughput that we see when we upload chunks of 4Mb in size, rather than
> 8k.
> 
> 2) In the same method - a the print on the filehandle can fail, and this
> is currently silently ignored. As we're doing multiple PUT requests per
> file then occasionally one of these requests times out, and I'd like to
> retry once per file.
> 
> Therefore I also added a 'retries' parameter to the options to
> store_file, and fixed the bug with not checking the return value of the
> print method on the file handle.
> 
> Patch attached, any feedback you have welcome :)
> 
> Cheers
> Tom
> 
>