Not enough checking in MogileFS::Client, chunked uploads not
respecting the chunk size asked for.
dormando at rydia.net
Thu Apr 10 21:26:06 UTC 2008
I believe it was set down that low (in a couple places! Perlbal,
mogilefs...) because larger values were occasionally crashing perl for
some people. Making it optional looks totally reasonable though.
I'll review/apply this if someone doesn't beat me to it. At a glance it
looks fine but I like doing quick style/test checks ;)
Tomas Doran wrote:
> I've found (what I think are) 2 problems in the Mogile client library.
> 1) In the store_file method, there is a loop which reads segments of the
> file, and prints them to the file handle (which depending on which
> client backend you're using, does different things). The block size is
> fixed to 8192 bytes here, I've changed it to use the supplied
> readLineChunkSize param.
> We're doing chunked (range PUT, with ClientHTTPFile) uploads of some
> very big files, and this change makes a *massive* difference to the
> throughput that we see when we upload chunks of 4Mb in size, rather than
> 2) In the same method - a the print on the filehandle can fail, and this
> is currently silently ignored. As we're doing multiple PUT requests per
> file then occasionally one of these requests times out, and I'd like to
> retry once per file.
> Therefore I also added a 'retries' parameter to the options to
> store_file, and fixed the bug with not checking the return value of the
> print method on the file handle.
> Patch attached, any feedback you have welcome :)
More information about the mogilefs