CPU Utilization by the mogilefsd process

Brad Fitzpatrick brad at danga.com
Wed Feb 7 22:41:30 UTC 2007


So one of the child processes died and the parent process has two sockets
open to it which are spinning writable...

Interesting.

We should be able to fix it without the info, but for my curiosity:  can
you tell which child process is dying?  If you run it in the foreground,
does it say "Unexpected death from ........" or similar.  Showing pid and
function of child.


On Wed, 7 Feb 2007, Carl Forsythe wrote:

> Brad Fitzpatrick wrote:
> > What version?
>
> Latest one from svn just installed on the nodes on Friday of last week
> (727?). Version 2 as a more general version number.
> >
> > strace it and tell me what it's doing?  And show output of lsof -p <pid>
> > as well?
> strace output:
> epoll_wait(7, {{EPOLLOUT, {u32=10, u64=10}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=11, u64=11}}}, 1000, 250) = 3
> time(NULL)                              = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=11, u64=11}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=10, u64=10}}}, 1000, 250) = 3
> time(NULL)                              = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=10, u64=10}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=11, u64=11}}}, 1000, 250) = 3
> time(NULL)                              = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=11, u64=11}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=10, u64=10}}}, 1000, 250) = 3
> time(NULL)                              = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=10, u64=10}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=11, u64=11}}}, 1000, 250) = 3
> time(NULL)                              = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=11, u64=11}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=10, u64=10}}}, 1000, 250) = 3
> time(NULL)                              = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=10, u64=10}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=11, u64=11}}}, 1000, 250) = 3
> time(NULL)                              = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=11, u64=11}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=10, u64=10}}}, 1000, 250) = 3
> time(NULL)                              = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=10, u64=10}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=11, u64=11}}}, 1000, 250) = 3
> time(NULL)                              = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=11, u64=11}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=10, u64=10}}}, 1000, 250) = 3
> time(NULL)                              = 1170886869
>
> and so on and so on and so on.
>
>
> lsof output:
> [root at mogdb00 ~]# lsof -p 6558
> COMMAND    PID      USER   FD   TYPE             DEVICE     SIZE    NODE
> NAME
> mogilefsd 6558 mogilefsd  cwd    DIR                8,2     4096       2 /
> mogilefsd 6558 mogilefsd  rtd    DIR                8,2     4096       2 /
> mogilefsd 6558 mogilefsd  txt    REG                8,2    21024 4978702
> /usr/bin/perl
> mogilefsd 6558 mogilefsd  mem    REG                8,2 48520720 4977414
> /usr/lib/locale/locale-archive
> mogilefsd 6558 mogilefsd  mem    REG                8,2    19520 5040752
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/IO/IO.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2    23632 5040927
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/Socket/Socket.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2   123664 5040765
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2    52044 5042813
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2     8040 5040954
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2    16128 5040957
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2    84080 5320464
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2    34352 5040758
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/List/Util/Util.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2    68390 5320352
> /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi/auto/BSD/Resource/Resource.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2   315483 5203911
> /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi/auto/DBI/DBI.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2   330140 5320581
> /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi/auto/DBD/mysql/mysql.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2    10384 6156230
> /lib64/libcom_err.so.2.1
> mogilefsd 6558 mogilefsd  mem    REG                8,2    56902 6156030
> /lib64/libnss_files-2.3.4.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2    23009 6156028
> /lib64/libnss_dns-2.3.4.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2   105080 6156227
> /lib64/ld-2.3.4.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2  1493409 6156228
> /lib64/tls/libc-2.3.4.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2    17943 6156232
> /lib64/libdl-2.3.4.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2   613297 6156236
> /lib64/tls/libm-2.3.4.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2    79336 4987377
> /usr/lib64/libz.so.1.2.1.2
> mogilefsd 6558 mogilefsd  mem    REG                8,2    30070 6156238
> /lib64/libcrypt-2.3.4.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2    91412 6156231
> /lib64/libresolv-2.3.4.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2   107187 6156054
> /lib64/libnsl-2.3.4.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2   106203 6156073
> /lib64/tls/libpthread-2.3.4.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2  1385848 5042812
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/CORE/libperl.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2   464040 4987375
> /usr/lib64/libkrb5.so.3.2
> mogilefsd 6558 mogilefsd  mem    REG                8,2   145456 4983599
> /usr/lib64/libk5crypto.so.3.0
> mogilefsd 6558 mogilefsd  mem    REG                8,2    17367 6156075
> /lib64/libutil-2.3.4.so
> mogilefsd 6558 mogilefsd  mem    REG                8,2    93832 4987376
> /usr/lib64/libgssapi_krb5.so.2.2
> mogilefsd 6558 mogilefsd  mem    REG                8,2   244288 6156234
> /lib64/libssl.so.0.9.7a
> mogilefsd 6558 mogilefsd  mem    REG                8,2  1230168 6156233
> /lib64/libcrypto.so.0.9.7a
> mogilefsd 6558 mogilefsd  mem    REG                8,2  1494376 5318825
> /usr/lib64/mysql/libmysqlclient.so.15.0.0
> mogilefsd 6558 mogilefsd    0u   CHR                1,3             2133
> /dev/null
> mogilefsd 6558 mogilefsd    1u   CHR                1,3             2133
> /dev/null
> mogilefsd 6558 mogilefsd    2u   CHR                1,3             2133
> /dev/null
> mogilefsd 6558 mogilefsd    3r   REG                8,2     2131 4987369
> /usr/bin/mogilefsd
> mogilefsd 6558 mogilefsd    4u  sock                0,4            16156
> can't identify protocol
> mogilefsd 6558 mogilefsd    5u  unix 0x000001007cfb46c0            16165
> socket
> mogilefsd 6558 mogilefsd    6u  IPv4              16166              TCP
> *:afs3-callback (LISTEN)
> mogilefsd 6558 mogilefsd    7r  0000                0,8        0   16167
> eventpoll
> mogilefsd 6558 mogilefsd    8u  unix 0x000001000a1f0c40            16168
> socket
> mogilefsd 6558 mogilefsd    9u  unix 0x000001000a1f0640            16170
> socket
> mogilefsd 6558 mogilefsd   10u  unix 0x000001007cfb49c0            16172
> socket
> mogilefsd 6558 mogilefsd   11u  unix 0x000001007c826980            16174
> socket
> mogilefsd 6558 mogilefsd   12u  unix 0x000001007c826c80            16176
> socket
> mogilefsd 6558 mogilefsd   13u  unix 0x000001007c860380            16178
> socket
> mogilefsd 6558 mogilefsd   14u  unix 0x000001007cffc640            16180
> socket
> mogilefsd 6558 mogilefsd   15u  unix 0x000001007cffc040            16182
> socket
> mogilefsd 6558 mogilefsd   16u  unix 0x0000010066740040            34322
> socket
> mogilefsd 6558 mogilefsd   17u  unix 0x000001006f820c80            16186
> socket
> mogilefsd 6558 mogilefsd   18u  unix 0x000001000938ac80            16188
> socket
> mogilefsd 6558 mogilefsd   19u  unix 0x000001000938a680            16190
> socket
> mogilefsd 6558 mogilefsd   20u  unix 0x000001000938a080            16192
> socket
> mogilefsd 6558 mogilefsd   21u  unix 0x00000100088039c0            16194
> socket
> mogilefsd 6558 mogilefsd   22u  unix 0x00000100088033c0            16196
> socket
> mogilefsd 6558 mogilefsd   23u  unix 0x00000100084dcc40            16198
> socket
> mogilefsd 6558 mogilefsd   24u  unix 0x00000100084dc640            16200
> socket
> mogilefsd 6558 mogilefsd   25u  unix 0x000001000a1f0040            16202
> socket
> mogilefsd 6558 mogilefsd   26u  unix 0x000001000805d980            16204
> socket
> >
> >
> >
> > On Wed, 7 Feb 2007, Carl Forsythe wrote:
> >
> >> Hi all,
> >>
> >> On our trackers I see one mogilefsd process in top (the parent process)
> >> sitting at 100% cpu utilization, is this typical behavior for the parent
> >> mogilefsd? The tracker platform is CentOS 4.4 64 bit running on AMD
> >> Opteron processors.
> >>
> >> Thanks,
> >>
> >> -Carl
> >>
> >>
> >>
> >>
>
>


More information about the mogilefs mailing list