mogile2.0 still doesn't like big file ?

[komtanoo.pinpimai at livetext.com]

Hi All,

I think I've found the reason why mogilefs2 fails to replicate big file.
It occurs mostly when network congests or encounters very huge files, I'm
testing my cluster on a 10/100 hub, so it's easy to trigger the error.

The problem is mogstored kills any PUT connection taking longer than 30
seconds, I looked into Perlbal/ClientHTTPBase.pm, I found
----
sub max_idle_time { 30; }
----
, after I change it to something like 5000;, the problem has gone, it can
replicate 700M without a failure.

My question is,

1. When max_idle_time is 30, why I can download hugh file taking longer
than 30 secs, but I can't PUT things longer than 30 secs ?

2. should max_idle_time of PUT operation be separated from general
max_idle_time or just make it forever ?

thanks
-kem

On Mon, October 2, 2006 9:48 pm, komtanoo.pinpimai at livetext.com wrote:
>> Do you have a patch which fixes this?
>>
>
> I'll try to patch it tomorrow, I'd found a problem in replicator of CVS
> version, I think it's the same one, I'll will let you know.
>
>> What server are you using for your storage nodes?  >mogstored (which
>>
> version?), or something else?
>
> It's original mogstored in revision 421, I didn't do anything fancy.
>
>
> -kem
>
>
> On Mon, October 2, 2006 7:45 pm, Brad Fitzpatrick wrote:
>
>> Do you have a patch which fixes this?
>>
>>
>>
>> What server are you using for your storage nodes?  mogstored (which
>> version?), or something else?
>>
>>
>> On Mon, 2 Oct 2006 komtanoo.pinpimai at livetext.com wrote:
>>
>>
>>
>>> Hi,
>>>
>>>
>>>
>>> I worked on the mogile1 on CVS for a while and it doesn't like big
>>> file, especially more than 200M. As in the current svn, 421, the
>>> replicator seems to have the same problem.
>>>
>>> ---------------------------------------------------------------------
>>> --
>>> ------
>>> [monitor(4333)] Monitor running; scanning usage files
>>> [monitor(4333)] Monitor running; scanning usage files
>>> [replicate(4321)] Error: wrote 720896; expected to write 1048576;
>>> failed putting to /dev1/0/000/000/0000000015.fid [replicate(4321)]
>>> Failed
>>> copying fid 15 from devid 5 to devid 1 (error type: dest_error)
>>> ---------------------------------------------------------------------
>>> ---
>>> -----
>>>
>>>
>>>
>>> And it keeps trying without success. One side effect when this
>>> happens is client can't connect this tracker, even the "mogadm check"
>>> shows REQUEST
>>> FAILURE.
>>>
>>>
>>>
>>> I could apply a quick fix patch that I wrote for the CVS version to
>>> it, but it would be nice if mogile2 could get rid of this problem. I
>>> wonder if nobody has big file replicating problem on mogilefs2 ?
>>>
>>> thanks -kem
>>>
>>>
>>>
>>
>
>