CPU Utilization by the mogilefsd process
Brad Fitzpatrick
brad at danga.com
Thu Feb 8 23:32:12 UTC 2007
Carl,
I was just looking into fixing this, studying how it could've happened.
What version of Danga::Socket do you have, out of curiosity?
An old version might explain something.
Thanks,
Brad
On Wed, 7 Feb 2007, Carl Forsythe wrote:
> Brad Fitzpatrick wrote:
> > What version?
>
> Latest one from svn just installed on the nodes on Friday of last week
> (727?). Version 2 as a more general version number.
> >
> > strace it and tell me what it's doing? And show output of lsof -p <pid>
> > as well?
> strace output:
> epoll_wait(7, {{EPOLLOUT, {u32=10, u64=10}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=11, u64=11}}}, 1000, 250) = 3
> time(NULL) = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=11, u64=11}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=10, u64=10}}}, 1000, 250) = 3
> time(NULL) = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=10, u64=10}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=11, u64=11}}}, 1000, 250) = 3
> time(NULL) = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=11, u64=11}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=10, u64=10}}}, 1000, 250) = 3
> time(NULL) = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=10, u64=10}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=11, u64=11}}}, 1000, 250) = 3
> time(NULL) = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=11, u64=11}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=10, u64=10}}}, 1000, 250) = 3
> time(NULL) = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=10, u64=10}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=11, u64=11}}}, 1000, 250) = 3
> time(NULL) = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=11, u64=11}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=10, u64=10}}}, 1000, 250) = 3
> time(NULL) = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=10, u64=10}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=11, u64=11}}}, 1000, 250) = 3
> time(NULL) = 1170886869
> epoll_wait(7, {{EPOLLOUT, {u32=11, u64=11}}, {EPOLLOUT, {u32=9, u64=9}},
> {EPOLLOUT, {u32=10, u64=10}}}, 1000, 250) = 3
> time(NULL) = 1170886869
>
> and so on and so on and so on.
>
>
> lsof output:
> [root at mogdb00 ~]# lsof -p 6558
> COMMAND PID USER FD TYPE DEVICE SIZE NODE
> NAME
> mogilefsd 6558 mogilefsd cwd DIR 8,2 4096 2 /
> mogilefsd 6558 mogilefsd rtd DIR 8,2 4096 2 /
> mogilefsd 6558 mogilefsd txt REG 8,2 21024 4978702
> /usr/bin/perl
> mogilefsd 6558 mogilefsd mem REG 8,2 48520720 4977414
> /usr/lib/locale/locale-archive
> mogilefsd 6558 mogilefsd mem REG 8,2 19520 5040752
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/IO/IO.so
> mogilefsd 6558 mogilefsd mem REG 8,2 23632 5040927
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/Socket/Socket.so
> mogilefsd 6558 mogilefsd mem REG 8,2 123664 5040765
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
> mogilefsd 6558 mogilefsd mem REG 8,2 52044 5042813
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
> mogilefsd 6558 mogilefsd mem REG 8,2 8040 5040954
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/Sys/Hostname/Hostname.so
> mogilefsd 6558 mogilefsd mem REG 8,2 16128 5040957
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
> mogilefsd 6558 mogilefsd mem REG 8,2 84080 5320464
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
> mogilefsd 6558 mogilefsd mem REG 8,2 34352 5040758
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/auto/List/Util/Util.so
> mogilefsd 6558 mogilefsd mem REG 8,2 68390 5320352
> /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi/auto/BSD/Resource/Resource.so
> mogilefsd 6558 mogilefsd mem REG 8,2 315483 5203911
> /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi/auto/DBI/DBI.so
> mogilefsd 6558 mogilefsd mem REG 8,2 330140 5320581
> /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi/auto/DBD/mysql/mysql.so
> mogilefsd 6558 mogilefsd mem REG 8,2 10384 6156230
> /lib64/libcom_err.so.2.1
> mogilefsd 6558 mogilefsd mem REG 8,2 56902 6156030
> /lib64/libnss_files-2.3.4.so
> mogilefsd 6558 mogilefsd mem REG 8,2 23009 6156028
> /lib64/libnss_dns-2.3.4.so
> mogilefsd 6558 mogilefsd mem REG 8,2 105080 6156227
> /lib64/ld-2.3.4.so
> mogilefsd 6558 mogilefsd mem REG 8,2 1493409 6156228
> /lib64/tls/libc-2.3.4.so
> mogilefsd 6558 mogilefsd mem REG 8,2 17943 6156232
> /lib64/libdl-2.3.4.so
> mogilefsd 6558 mogilefsd mem REG 8,2 613297 6156236
> /lib64/tls/libm-2.3.4.so
> mogilefsd 6558 mogilefsd mem REG 8,2 79336 4987377
> /usr/lib64/libz.so.1.2.1.2
> mogilefsd 6558 mogilefsd mem REG 8,2 30070 6156238
> /lib64/libcrypt-2.3.4.so
> mogilefsd 6558 mogilefsd mem REG 8,2 91412 6156231
> /lib64/libresolv-2.3.4.so
> mogilefsd 6558 mogilefsd mem REG 8,2 107187 6156054
> /lib64/libnsl-2.3.4.so
> mogilefsd 6558 mogilefsd mem REG 8,2 106203 6156073
> /lib64/tls/libpthread-2.3.4.so
> mogilefsd 6558 mogilefsd mem REG 8,2 1385848 5042812
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/CORE/libperl.so
> mogilefsd 6558 mogilefsd mem REG 8,2 464040 4987375
> /usr/lib64/libkrb5.so.3.2
> mogilefsd 6558 mogilefsd mem REG 8,2 145456 4983599
> /usr/lib64/libk5crypto.so.3.0
> mogilefsd 6558 mogilefsd mem REG 8,2 17367 6156075
> /lib64/libutil-2.3.4.so
> mogilefsd 6558 mogilefsd mem REG 8,2 93832 4987376
> /usr/lib64/libgssapi_krb5.so.2.2
> mogilefsd 6558 mogilefsd mem REG 8,2 244288 6156234
> /lib64/libssl.so.0.9.7a
> mogilefsd 6558 mogilefsd mem REG 8,2 1230168 6156233
> /lib64/libcrypto.so.0.9.7a
> mogilefsd 6558 mogilefsd mem REG 8,2 1494376 5318825
> /usr/lib64/mysql/libmysqlclient.so.15.0.0
> mogilefsd 6558 mogilefsd 0u CHR 1,3 2133
> /dev/null
> mogilefsd 6558 mogilefsd 1u CHR 1,3 2133
> /dev/null
> mogilefsd 6558 mogilefsd 2u CHR 1,3 2133
> /dev/null
> mogilefsd 6558 mogilefsd 3r REG 8,2 2131 4987369
> /usr/bin/mogilefsd
> mogilefsd 6558 mogilefsd 4u sock 0,4 16156
> can't identify protocol
> mogilefsd 6558 mogilefsd 5u unix 0x000001007cfb46c0 16165
> socket
> mogilefsd 6558 mogilefsd 6u IPv4 16166 TCP
> *:afs3-callback (LISTEN)
> mogilefsd 6558 mogilefsd 7r 0000 0,8 0 16167
> eventpoll
> mogilefsd 6558 mogilefsd 8u unix 0x000001000a1f0c40 16168
> socket
> mogilefsd 6558 mogilefsd 9u unix 0x000001000a1f0640 16170
> socket
> mogilefsd 6558 mogilefsd 10u unix 0x000001007cfb49c0 16172
> socket
> mogilefsd 6558 mogilefsd 11u unix 0x000001007c826980 16174
> socket
> mogilefsd 6558 mogilefsd 12u unix 0x000001007c826c80 16176
> socket
> mogilefsd 6558 mogilefsd 13u unix 0x000001007c860380 16178
> socket
> mogilefsd 6558 mogilefsd 14u unix 0x000001007cffc640 16180
> socket
> mogilefsd 6558 mogilefsd 15u unix 0x000001007cffc040 16182
> socket
> mogilefsd 6558 mogilefsd 16u unix 0x0000010066740040 34322
> socket
> mogilefsd 6558 mogilefsd 17u unix 0x000001006f820c80 16186
> socket
> mogilefsd 6558 mogilefsd 18u unix 0x000001000938ac80 16188
> socket
> mogilefsd 6558 mogilefsd 19u unix 0x000001000938a680 16190
> socket
> mogilefsd 6558 mogilefsd 20u unix 0x000001000938a080 16192
> socket
> mogilefsd 6558 mogilefsd 21u unix 0x00000100088039c0 16194
> socket
> mogilefsd 6558 mogilefsd 22u unix 0x00000100088033c0 16196
> socket
> mogilefsd 6558 mogilefsd 23u unix 0x00000100084dcc40 16198
> socket
> mogilefsd 6558 mogilefsd 24u unix 0x00000100084dc640 16200
> socket
> mogilefsd 6558 mogilefsd 25u unix 0x000001000a1f0040 16202
> socket
> mogilefsd 6558 mogilefsd 26u unix 0x000001000805d980 16204
> socket
> >
> >
> >
> > On Wed, 7 Feb 2007, Carl Forsythe wrote:
> >
> >> Hi all,
> >>
> >> On our trackers I see one mogilefsd process in top (the parent process)
> >> sitting at 100% cpu utilization, is this typical behavior for the parent
> >> mogilefsd? The tracker platform is CentOS 4.4 64 bit running on AMD
> >> Opteron processors.
> >>
> >> Thanks,
> >>
> >> -Carl
> >>
> >>
> >>
> >>
>
>
More information about the mogilefs
mailing list