Large number of mogstored processes

[mischa.z]

After looking at the code of mogstored and the method aio_threads() I 
understand that I *should* expect 100 aio processes (we've got 10 disks 
on every node). The big question now is why on some of the nodes 
mogstored does *not* create those processes ;)

e.g.
ps -ef | grep mog
root     29231     1  1 Feb21 ?        00:17:13 /usr/bin/perl 
/usr/bin/mogstored --daemon
root     29232 29231  0 Feb21 ?        00:00:42 mogstored [diskusage]
root     29234 29231  0 Feb21 ?        00:00:14 mogstored [iostat]

I checked $aio_threads before
Perlbal::run_manage_commands("SERVER aio_threads = $aio_threads", sub { 
print STDERR "$_[0]\n"; });
gets called:

num aio_threads: 100 at /usr/bin/mogstored line 137.

Seems to make sense.

Hm, any idea what could go wrong here?


mischa.z wrote:
> But why processes? Why not using threads? (Sorry if I ask dumb questions 
> but my knowledge of Perl is rather limited:/). Altogether those 
> processes use up a substantial amount of memory without actually doing 
> anything (at that time there weren't any PUT requests + GET is handled 
> by lighttpd). And what really strikes me is that I see a lot of child 
> processes on only 2 out of 5 nodes.
> e.g. another node that seems to be Ok:
> ps -ef | grep mog
> root      9673     1  0 Feb21 ?        00:00:00 /usr/bin/perl 
> /usr/bin/mogstored --daemon
> root      9674  9673  2 Feb21 ?        00:31:55 /usr/bin/perl 
> /usr/bin/mogstored --daemon
> root      9675  9674  0 Feb21 ?        00:00:48 mogstored [diskusage]
> root      9677  9674  0 Feb21 ?        00:00:23 mogstored [iostat]
> 
> Cheers.
> 
> Brad Fitzpatrick wrote:
>> Run ps with the 'f' option.  You'll see they're child processes used for
>> disk AIO.  (IO::AIO or Linux::AIO)
>>
>> On Thu, 22 Feb 2007, mischa.z wrote:
>>
>>> Hi,
>>>
>>> some of our storage nodes (we are using lighttpd for GET) show a large
>>> number of mogstored processes (right after start-up) just sitting there.
>>> This only happens on nodes with use > 90% - might be a coincidence?
>>> Is this normal behaviour or is something going wrong here?
>>>
>>> Thanks,
>>> Mischa
>>>
>>> svn740
>>>
>>> ps aux | grep mogstored | wc -l
>>> 105
>>>
>>> ps aux | grep mogstored
>>> root      1573  1.6  0.4 46864 16436 ?       S    11:32   1:02
>>> /usr/bin/perl /usr/bin/mogstored --daemon
>>> root      1574  0.0  0.4 46864 16436 ?       S    11:32   0:00
>>> /usr/bin/perl /usr/bin/mogstored --daemon
>>> root      1575  0.0  0.4 46864 16436 ?       S    11:32   0:00
>>> /usr/bin/perl /usr/bin/mogstored --daemon
>>> root      1576  0.0  0.4 46864 16436 ?       S    11:32   0:00
>>> /usr/bin/perl /usr/bin/mogstored --daemon
>>> ...
>>>
>>>
>>>
>>
> 
>