<p>
<strong>Memory & Overall stability testing</strong>
</p>
<p>The following test were conducted by generating newfile/getpath
traffic to the same 3 nodes continueously, including creating various
sizes of files, getting their paths, fetching them through mogstored in
48hours. What's interesting to see is how much memory each process
takes or can the system tolerate the marathon benchmark, etc.. (For
elaborate detail, 21 geting_path+fetching from mostored in every 1.5
seconds, 1k-1M new file in every 100 seconds, 1M-10M file in every 2000
seconds. About 2419686 getting_path+fetching requests and 3662 new
files were done to the mogile cluster within 48 hours. )
</p>
<p>
The total replicator = 2 in node1 and node2. Node 3 has no replicator.
</p>
<p>
The initial process information:
</p>
<p>*You might notice that there is only one mogstored in n2. I guess
it's because the incompatiblity of linux distributions, n2 is FC5 while
n1 and n3 are debian.
</p>
<pre class="wiki">n1:~# ps auxh | grep mogstored
root 9964 1.2 1.6 12932 8384 pts/2 S+ 18:03 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 9966 0.0 1.6 12932 8384 pts/2 S+ 18:03 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 9967 0.0 1.6 12932 8384 pts/2 S+ 18:03 0:00 /usr/bin/perl -w /usr/bin/mogstored
n1:~# ps auxh | grep mogilefsd
livetext 9970 1.1 1.8 12748 9612 pts/3 S+ 18:03 0:00 /usr/bin/perl /usr/bin/mogilefsd
livetext 9971 0.0 1.8 12748 9640 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [replicate]
livetext 9972 0.0 1.8 12748 9608 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [delete]
livetext 9973 0.0 1.8 12748 9588 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [queryworker]
livetext 9974 0.0 1.8 12748 9588 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [queryworker]
livetext 9975 0.0 1.8 12748 9588 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [queryworker]
livetext 9976 0.0 1.8 12748 9588 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [queryworker]
livetext 9977 0.0 1.8 12748 9588 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [queryworker]
livetext 9978 0.0 1.8 12748 9588 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [queryworker]
livetext 9979 0.0 1.8 12748 9588 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [queryworker]
livetext 9980 0.0 1.8 12748 9588 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [queryworker]
livetext 9981 0.0 1.8 12748 9588 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [queryworker]
livetext 9982 0.0 1.8 12748 9588 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [queryworker]
livetext 9983 0.1 2.0 13672 10516 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [monitor]
livetext 9984 0.0 1.8 12748 9636 pts/3 S+ 18:03 0:00 /usr/bin/mogilefsd [reaper]
n1:~#
[root@livetext ~]# ps aux | grep mogstored
root 20406 1.7 1.5 22040 8076 pts/2 Sl+ 18:03 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 20428 0.0 0.1 3916 680 pts/0 S+ 18:04 0:00 grep mogstored
[root@livetext ~]# ps aux | grep mogilefsd
livetext 20411 1.1 1.9 16604 10060 pts/1 S+ 18:04 0:00 /usr/bin/perl /usr/bin/mogilefsd
livetext 20412 0.0 1.7 16604 8760 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [replicate]
livetext 20413 0.0 1.6 16604 8460 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [delete]
livetext 20414 0.0 1.5 16604 7996 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [queryworker]
livetext 20415 0.0 1.5 16604 7996 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [queryworker]
livetext 20416 0.0 1.5 16604 7996 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [queryworker]
livetext 20417 0.0 1.5 16604 7996 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [queryworker]
livetext 20418 0.0 1.5 16604 7996 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [queryworker]
livetext 20419 0.0 1.5 16604 7996 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [queryworker]
livetext 20420 0.0 1.5 16604 7996 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [queryworker]
livetext 20421 0.0 1.5 16604 7996 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [queryworker]
livetext 20422 0.0 1.5 16604 7996 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [queryworker]
livetext 20423 0.0 1.5 16604 7996 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [queryworker]
livetext 20424 0.1 2.0 17664 10104 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [monitor]
livetext 20425 0.0 1.7 16604 8656 pts/1 S+ 18:04 0:00 /usr/bin/mogilefsd [reaper]
root 20431 0.0 0.1 3916 676 pts/0 S+ 18:04 0:00 grep mogilefsd
[root@livetext ~]#
n3:~# ps aux | grep mogstored
root 11537 1.0 1.6 10896 8376 pts/1 S+ 13:06 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 11538 0.0 1.6 10896 8376 pts/1 S+ 13:06 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 11539 0.0 1.6 10896 8376 pts/1 S+ 13:06 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 11540 0.0 1.6 10896 8376 pts/1 S+ 13:06 0:00 /usr/bin/perl -w /usr/bin/mogstored
n3:~# ps aux | grep mogilefsd
livetext 11543 0.9 1.8 12744 9604 pts/2 S+ 13:06 0:00 /usr/bin/perl /usr/bin/mogilefsd
livetext 11544 0.0 1.8 12744 9616 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [delete]
livetext 11545 0.0 1.8 12744 9584 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [queryworker]
livetext 11546 0.0 1.8 12744 9584 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [queryworker]
livetext 11547 0.0 1.8 12744 9584 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [queryworker]
livetext 11548 0.0 1.8 12744 9584 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [queryworker]
livetext 11549 0.0 1.8 12744 9584 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [queryworker]
livetext 11550 0.0 1.8 12744 9584 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [queryworker]
livetext 11551 0.0 1.8 12744 9584 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [queryworker]
livetext 11552 0.0 1.8 12744 9584 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [queryworker]
livetext 11553 0.0 1.8 12744 9584 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [queryworker]
livetext 11554 0.0 1.8 12744 9584 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [queryworker]
livetext 11555 0.2 2.0 13668 10516 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [monitor]
livetext 11556 0.0 1.8 12744 9636 pts/2 S+ 13:06 0:00 /usr/bin/mogilefsd [reaper]
n3:~#
</pre><p>
<i>After 20 hours</i>
</p>
<pre class="wiki">n1:~# ps aux | grep mogstored; ps aux | grep mogilefsd
root 9964 8.3 1.2 27392 6256 pts/2 R+ Jul31 99:49 /usr/bin/perl -w /usr/bin/mogstored
root 9966 0.0 1.2 27392 6256 pts/2 S+ Jul31 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 9967 0.0 1.2 27392 6256 pts/2 S+ Jul31 0:52 /usr/bin/perl -w /usr/bin/mogstored
root 9998 0.0 1.2 27392 6256 pts/2 S+ Jul31 0:52 /usr/bin/perl -w /usr/bin/mogstored
root 10001 0.0 1.2 27392 6256 pts/2 S+ Jul31 0:52 /usr/bin/perl -w /usr/bin/mogstored
root 10002 0.0 1.2 27392 6256 pts/2 S+ Jul31 0:50 /usr/bin/perl -w /usr/bin/mogstored
root 11525 0.0 0.1 1820 560 pts/0 R+ 14:00 0:00 grep mogstored
livetext 9970 0.7 1.0 12748 5248 pts/3 S+ Jul31 9:08 /usr/bin/perl /usr/bin/mogilefsd
livetext 9971 0.0 3.2 27480 16380 pts/3 S+ Jul31 1:11 /usr/bin/mogilefsd [replicate]
livetext 9972 0.0 0.3 12980 1872 pts/3 S+ Jul31 0:01 /usr/bin/mogilefsd [delete]
livetext 9973 0.0 0.8 14868 4108 pts/3 S+ Jul31 0:48 /usr/bin/mogilefsd [queryworker]
livetext 9974 0.0 0.8 14828 4108 pts/3 S+ Jul31 0:49 /usr/bin/mogilefsd [queryworker]
livetext 9975 0.0 0.8 14868 4136 pts/3 S+ Jul31 0:48 /usr/bin/mogilefsd [queryworker]
livetext 9976 0.0 0.8 14816 4128 pts/3 S+ Jul31 0:49 /usr/bin/mogilefsd [queryworker]
livetext 9977 0.0 0.8 14872 4076 pts/3 S+ Jul31 0:49 /usr/bin/mogilefsd [queryworker]
livetext 9978 0.0 0.8 14836 4120 pts/3 S+ Jul31 0:48 /usr/bin/mogilefsd [queryworker]
livetext 9979 0.0 0.8 14852 4108 pts/3 S+ Jul31 0:49 /usr/bin/mogilefsd [queryworker]
livetext 9980 0.0 0.8 14900 4096 pts/3 S+ Jul31 0:50 /usr/bin/mogilefsd [queryworker]
livetext 9981 0.0 0.8 14812 4148 pts/3 S+ Jul31 0:50 /usr/bin/mogilefsd [queryworker]
livetext 9982 0.0 0.8 14860 4104 pts/3 S+ Jul31 0:49 /usr/bin/mogilefsd [queryworker]
livetext 9983 0.0 1.4 14916 7488 pts/3 S+ Jul31 0:42 /usr/bin/mogilefsd [monitor]
livetext 9984 0.0 0.7 13008 3628 pts/3 S+ Jul31 0:02 /usr/bin/mogilefsd [reaper]
n1:~#
[root@livetext ~]# ps aux | grep mogstored; ps aux | grep mogilefsd
root 20406 5.0 2.9 59232 14660 pts/2 Sl+ Jul31 60:16 /usr/bin/perl -w /usr/bin/mogstored
root 27020 0.0 0.1 3912 676 pts/0 S+ 14:04 0:00 grep mogstored
livetext 20411 0.6 2.0 16736 10144 pts/1 S+ Jul31 7:25 /usr/bin/perl /usr/bin/mogilefsd
livetext 20412 0.1 4.8 31956 24316 pts/1 S+ Jul31 1:35 /usr/bin/mogilefsd [replicate]
livetext 20413 0.0 1.7 16836 8692 pts/1 S+ Jul31 0:02 /usr/bin/mogilefsd [delete]
livetext 20414 0.0 2.3 19296 11672 pts/1 S+ Jul31 0:57 /usr/bin/mogilefsd [queryworker]
livetext 20415 0.0 2.3 19264 11704 pts/1 S+ Jul31 0:58 /usr/bin/mogilefsd [queryworker]
livetext 20416 0.0 2.3 19200 11672 pts/1 S+ Jul31 0:57 /usr/bin/mogilefsd [queryworker]
livetext 20417 0.0 2.3 19280 11684 pts/1 S+ Jul31 0:57 /usr/bin/mogilefsd [queryworker]
livetext 20418 0.0 2.3 19240 11692 pts/1 S+ Jul31 0:58 /usr/bin/mogilefsd [queryworker]
livetext 20419 0.0 2.3 19284 11664 pts/1 S+ Jul31 0:57 /usr/bin/mogilefsd [queryworker]
livetext 20420 0.0 2.3 19208 11660 pts/1 S+ Jul31 0:57 /usr/bin/mogilefsd [queryworker]
livetext 20421 0.0 2.3 19204 11696 pts/1 S+ Jul31 0:58 /usr/bin/mogilefsd [queryworker]
livetext 20422 0.0 2.3 19308 11700 pts/1 S+ Jul31 0:57 /usr/bin/mogilefsd [queryworker]
livetext 20423 0.0 2.3 19180 11664 pts/1 S+ Jul31 0:58 /usr/bin/mogilefsd [queryworker]
livetext 20424 0.0 2.0 17900 10420 pts/1 S+ Jul31 0:53 /usr/bin/mogilefsd [monitor]
livetext 20425 0.0 1.7 16868 8916 pts/1 S+ Jul31 0:03 /usr/bin/mogilefsd [reaper]
root 27022 0.0 0.1 3916 680 pts/0 S+ 14:04 0:00 grep mogilefsd
[root@livetext ~]#
n3:~# ps aux | grep mogstored; ps aux | grep mogilefsd
root 11537 10.3 1.2 26952 6536 pts/1 R+ Jul31 123:42 /usr/bin/perl -w /usr/bin/mogstored
root 11538 0.0 1.2 26952 6536 pts/1 S+ Jul31 0:44 /usr/bin/perl -w /usr/bin/mogstored
root 11539 0.0 1.2 26952 6536 pts/1 S+ Jul31 0:45 /usr/bin/perl -w /usr/bin/mogstored
root 11540 0.0 1.2 26952 6536 pts/1 S+ Jul31 0:45 /usr/bin/perl -w /usr/bin/mogstored
livetext 11543 0.7 1.0 12744 5256 pts/2 S+ Jul31 8:29 /usr/bin/perl /usr/bin/mogilefsd
livetext 11544 0.0 0.3 12976 1880 pts/2 S+ Jul31 0:00 /usr/bin/mogilefsd [delete]
livetext 11545 0.0 0.8 14788 4236 pts/2 S+ Jul31 0:46 /usr/bin/mogilefsd [queryworker]
livetext 11546 0.0 0.8 14852 4184 pts/2 S+ Jul31 0:46 /usr/bin/mogilefsd [queryworker]
livetext 11547 0.0 0.8 14876 4236 pts/2 S+ Jul31 0:45 /usr/bin/mogilefsd [queryworker]
livetext 11548 0.0 0.8 14800 4224 pts/2 S+ Jul31 0:45 /usr/bin/mogilefsd [queryworker]
livetext 11549 0.0 0.8 14832 4264 pts/2 S+ Jul31 0:45 /usr/bin/mogilefsd [queryworker]
livetext 11550 0.0 0.8 14860 4224 pts/2 S+ Jul31 0:45 /usr/bin/mogilefsd [queryworker]
livetext 11551 0.0 0.8 14852 4196 pts/2 S+ Jul31 0:46 /usr/bin/mogilefsd [queryworker]
livetext 11552 0.0 0.8 14848 4268 pts/2 S+ Jul31 0:46 /usr/bin/mogilefsd [queryworker]
livetext 11553 0.0 0.8 14788 4268 pts/2 S+ Jul31 0:48 /usr/bin/mogilefsd [queryworker]
livetext 11554 0.0 0.8 14868 4276 pts/2 S+ Jul31 0:46 /usr/bin/mogilefsd [queryworker]
livetext 11555 0.0 1.5 14916 7716 pts/2 S+ Jul31 0:55 /usr/bin/mogilefsd [monitor]
livetext 11556 0.0 0.4 13004 2532 pts/2 S+ Jul31 0:00 /usr/bin/mogilefsd [reaper]
n3:~#
</pre><p>
<i>After 24 hours</i>
</p>
<pre class="wiki">n1:/usr/local/mysql/bin# ps aux | grep mogstored ; ps aux | grep mogilefsd
root 9964 8.3 1.3 27392 6688 pts/2 R+ Jul31 119:31 /usr/bin/perl -w /usr/bin/mogstored
root 9966 0.0 1.3 27392 6688 pts/2 S+ Jul31 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 9967 0.0 1.3 27392 6688 pts/2 S+ Jul31 1:02 /usr/bin/perl -w /usr/bin/mogstored
root 9998 0.0 1.3 27392 6688 pts/2 S+ Jul31 1:02 /usr/bin/perl -w /usr/bin/mogstored
root 10001 0.0 1.3 27392 6688 pts/2 S+ Jul31 1:03 /usr/bin/perl -w /usr/bin/mogstored
root 10002 0.0 1.3 27392 6688 pts/2 S+ Jul31 1:01 /usr/bin/perl -w /usr/bin/mogstored
root 12690 0.0 0.1 1820 564 pts/0 R+ 17:51 0:00 grep mogstored
livetext 9970 0.7 1.0 12748 5248 pts/3 S+ Jul31 11:01 /usr/bin/perl /usr/bin/mogilefsd
livetext 9971 0.0 3.2 28372 16424 pts/3 S+ Jul31 1:24 /usr/bin/mogilefsd [replicate]
livetext 9972 0.0 0.3 12980 1864 pts/3 S+ Jul31 0:01 /usr/bin/mogilefsd [delete]
livetext 9973 0.0 0.8 15320 4328 pts/3 S+ Jul31 0:58 /usr/bin/mogilefsd [queryworker]
livetext 9974 0.0 0.8 15292 4324 pts/3 S+ Jul31 0:59 /usr/bin/mogilefsd [queryworker]
livetext 9975 0.0 0.8 15208 4360 pts/3 S+ Jul31 0:58 /usr/bin/mogilefsd [queryworker]
livetext 9976 0.0 0.8 15304 4348 pts/3 S+ Jul31 1:00 /usr/bin/mogilefsd [queryworker]
livetext 9977 0.0 0.8 15208 4268 pts/3 S+ Jul31 0:59 /usr/bin/mogilefsd [queryworker]
livetext 9978 0.0 0.8 15208 4320 pts/3 S+ Jul31 0:59 /usr/bin/mogilefsd [queryworker]
livetext 9979 0.0 0.8 15228 4308 pts/3 S+ Jul31 0:59 /usr/bin/mogilefsd [queryworker]
livetext 9980 0.0 0.8 15244 4304 pts/3 S+ Jul31 0:59 /usr/bin/mogilefsd [queryworker]
livetext 9981 0.0 0.8 15284 4300 pts/3 S+ Jul31 1:00 /usr/bin/mogilefsd [queryworker]
livetext 9982 0.0 0.8 15228 4248 pts/3 S+ Jul31 1:00 /usr/bin/mogilefsd [queryworker]
livetext 9983 0.0 1.4 15128 7300 pts/3 S+ Jul31 0:51 /usr/bin/mogilefsd [monitor]
livetext 9984 0.0 0.4 13008 2308 pts/3 S+ Jul31 0:03 /usr/bin/mogilefsd [reaper]
root 12692 0.0 0.1 1820 564 pts/0 R+ 17:51 0:00 grep mogilefsd
[root@livetext bin]# ps aux | grep mogstored ; ps aux | grep mogilefsd
root 20406 5.0 2.9 59232 14660 pts/2 Sl+ Jul31 72:19 /usr/bin/perl -w /usr/bin/mogstored
root 28400 0.0 0.1 3912 676 pts/0 S+ 17:50 0:00 grep mogstored
livetext 20411 0.6 2.0 16736 10144 pts/1 S+ Jul31 8:56 /usr/bin/perl /usr/bin/mogilefsd
livetext 20412 0.1 5.0 32848 25268 pts/1 S+ Jul31 1:52 /usr/bin/mogilefsd [replicate]
livetext 20413 0.0 1.7 16952 8736 pts/1 S+ Jul31 0:02 /usr/bin/mogilefsd [delete]
livetext 20414 0.0 2.4 19676 12172 pts/1 S+ Jul31 1:09 /usr/bin/mogilefsd [queryworker]
livetext 20415 0.0 2.4 19736 12200 pts/1 S+ Jul31 1:10 /usr/bin/mogilefsd [queryworker]
livetext 20416 0.0 2.4 19708 12164 pts/1 S+ Jul31 1:08 /usr/bin/mogilefsd [queryworker]
livetext 20417 0.0 2.4 19768 12168 pts/1 S+ Jul31 1:08 /usr/bin/mogilefsd [queryworker]
livetext 20418 0.0 2.4 19740 12180 pts/1 S+ Jul31 1:09 /usr/bin/mogilefsd [queryworker]
livetext 20419 0.0 2.4 19764 12176 pts/1 S+ Jul31 1:09 /usr/bin/mogilefsd [queryworker]
livetext 20420 0.0 2.4 19708 12172 pts/1 S+ Jul31 1:09 /usr/bin/mogilefsd [queryworker]
livetext 20421 0.0 2.4 19708 12196 pts/1 S+ Jul31 1:10 /usr/bin/mogilefsd [queryworker]
livetext 20422 0.0 2.4 19816 12184 pts/1 S+ Jul31 1:09 /usr/bin/mogilefsd [queryworker]
livetext 20423 0.0 2.4 19680 12144 pts/1 S+ Jul31 1:10 /usr/bin/mogilefsd [queryworker]
livetext 20424 0.0 2.0 18020 10476 pts/1 S+ Jul31 1:03 /usr/bin/mogilefsd [monitor]
livetext 20425 0.0 1.7 17000 8968 pts/1 S+ Jul31 0:03 /usr/bin/mogilefsd [reaper]
root 28402 0.0 0.1 3916 676 pts/0 S+ 17:50 0:00 grep mogilefsd
n3:/usr/local/mysql/bin# ps aux | grep mogstored ; ps aux | grep mogilefsd
root 11537 10.3 1.1 26952 5980 pts/1 R+ Jul31 147:37 /usr/bin/perl -w /usr/bin/mogstored
root 11538 0.0 1.1 26952 5980 pts/1 S+ Jul31 0:53 /usr/bin/perl -w /usr/bin/mogstored
root 11539 0.0 1.1 26952 5980 pts/1 S+ Jul31 0:53 /usr/bin/perl -w /usr/bin/mogstored
root 11540 0.0 1.1 26952 5980 pts/1 S+ Jul31 0:54 /usr/bin/perl -w /usr/bin/mogstored
livetext 11543 0.7 1.0 12744 5256 pts/2 S+ Jul31 10:12 /usr/bin/perl /usr/bin/mogilefsd
livetext 11544 0.0 0.3 12976 1856 pts/2 S+ Jul31 0:00 /usr/bin/mogilefsd [delete]
livetext 11545 0.0 0.8 15252 4392 pts/2 S+ Jul31 0:56 /usr/bin/mogilefsd [queryworker]
livetext 11546 0.0 0.8 15220 4368 pts/2 S+ Jul31 0:55 /usr/bin/mogilefsd [queryworker]
livetext 11547 0.0 0.8 15208 4380 pts/2 S+ Jul31 0:55 /usr/bin/mogilefsd [queryworker]
livetext 11548 0.0 0.8 15256 4384 pts/2 S+ Jul31 0:54 /usr/bin/mogilefsd [queryworker]
livetext 11549 0.0 0.8 15216 4384 pts/2 S+ Jul31 0:55 /usr/bin/mogilefsd [queryworker]
livetext 11550 0.0 0.8 15236 4364 pts/2 S+ Jul31 0:55 /usr/bin/mogilefsd [queryworker]
livetext 11551 0.0 0.8 15212 4316 pts/2 S+ Jul31 0:55 /usr/bin/mogilefsd [queryworker]
livetext 11552 0.0 0.8 15308 4448 pts/2 S+ Jul31 0:55 /usr/bin/mogilefsd [queryworker]
livetext 11553 0.0 0.8 15272 4412 pts/2 S+ Jul31 0:57 /usr/bin/mogilefsd [queryworker]
livetext 11554 0.0 0.8 15248 4396 pts/2 S+ Jul31 0:55 /usr/bin/mogilefsd [queryworker]
livetext 11555 0.0 1.4 15124 7412 pts/2 S+ Jul31 1:05 /usr/bin/mogilefsd [monitor]
livetext 11556 0.0 0.4 13004 2520 pts/2 S+ Jul31 0:00 /usr/bin/mogilefsd [reaper]
</pre><p>
<i>After 39 hours</i>
</p>
<pre class="wiki">n1:~# ps aux | egrep "mogstored|mogilefsd"
root 9964 8.2 1.4 27392 7112 pts/2 R+ Jul31 193:36 /usr/bin/perl -w /usr/bin/mogstored
root 9966 0.0 1.4 27392 7112 pts/2 S+ Jul31 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 9967 0.0 1.4 27392 7112 pts/2 S+ Jul31 1:43 /usr/bin/perl -w /usr/bin/mogstored
livetext 9970 0.7 1.0 12748 5248 pts/3 S+ Jul31 18:32 /usr/bin/perl /usr/bin/mogilefsd
livetext 9971 0.0 3.5 31796 18032 pts/3 S+ Jul31 2:14 /usr/bin/mogilefsd [replicate]
livetext 9972 0.0 0.3 13212 1900 pts/3 S+ Jul31 0:02 /usr/bin/mogilefsd [delete]
livetext 9973 0.0 1.0 16920 5352 pts/3 S+ Jul31 1:39 /usr/bin/mogilefsd [queryworker]
livetext 9974 0.0 1.0 16932 5364 pts/3 S+ Jul31 1:40 /usr/bin/mogilefsd [queryworker]
livetext 9975 0.0 1.0 16868 5360 pts/3 S+ Jul31 1:39 /usr/bin/mogilefsd [queryworker]
livetext 9976 0.0 1.0 16860 5308 pts/3 S+ Jul31 1:40 /usr/bin/mogilefsd [queryworker]
livetext 9977 0.0 1.0 16872 5208 pts/3 S+ Jul31 1:39 /usr/bin/mogilefsd [queryworker]
livetext 9978 0.0 1.0 16892 5284 pts/3 S+ Jul31 1:39 /usr/bin/mogilefsd [queryworker]
livetext 9979 0.0 1.0 16880 5240 pts/3 S+ Jul31 1:40 /usr/bin/mogilefsd [queryworker]
livetext 9980 0.0 1.0 16876 5244 pts/3 S+ Jul31 1:40 /usr/bin/mogilefsd [queryworker]
livetext 9981 0.0 1.0 16944 5312 pts/3 S+ Jul31 1:40 /usr/bin/mogilefsd [queryworker]
livetext 9982 0.0 1.0 16964 5296 pts/3 S+ Jul31 1:40 /usr/bin/mogilefsd [queryworker]
livetext 9983 0.0 1.6 16072 8432 pts/3 S+ Jul31 1:23 /usr/bin/mogilefsd [monitor]
livetext 9984 0.0 0.4 13260 2392 pts/3 S+ Jul31 0:04 /usr/bin/mogilefsd [reaper]
root 9998 0.0 1.4 27392 7112 pts/2 S+ Jul31 1:41 /usr/bin/perl -w /usr/bin/mogstored
root 10001 0.0 1.4 27392 7112 pts/2 S+ Jul31 1:45 /usr/bin/perl -w /usr/bin/mogstored
root 10002 0.0 1.4 27392 7112 pts/2 S+ Jul31 1:41 /usr/bin/perl -w /usr/bin/mogstored
root 13837 0.0 0.1 1820 564 pts/0 R+ 09:16 0:00 grep -E mogstored|mogilefsd
[root@livetext ~]# ps aux | egrep "mogstored|mogilefsd"
root 20406 5.0 2.9 59232 14660 pts/2 Sl+ Jul31 119:31 /usr/bin/perl -w /usr/bin/mogstored
livetext 20411 0.6 2.0 16736 10144 pts/1 S+ Jul31 15:03 /usr/bin/perl /usr/bin/mogilefsd
livetext 20412 0.1 5.7 36764 29156 pts/1 S+ Jul31 3:04 /usr/bin/mogilefsd [replicate]
livetext 20413 0.0 1.7 17068 8912 pts/1 S+ Jul31 0:03 /usr/bin/mogilefsd [delete]
livetext 20414 0.0 2.8 21784 14196 pts/1 S+ Jul31 1:57 /usr/bin/mogilefsd [queryworker]
livetext 20415 0.0 2.8 21848 14232 pts/1 S+ Jul31 1:58 /usr/bin/mogilefsd [queryworker]
livetext 20416 0.0 2.8 21760 14192 pts/1 S+ Jul31 1:56 /usr/bin/mogilefsd [queryworker]
livetext 20417 0.0 2.8 21776 14208 pts/1 S+ Jul31 1:55 /usr/bin/mogilefsd [queryworker]
livetext 20418 0.0 2.8 21804 14220 pts/1 S+ Jul31 1:57 /usr/bin/mogilefsd [queryworker]
livetext 20419 0.0 2.8 21800 14212 pts/1 S+ Jul31 1:56 /usr/bin/mogilefsd [queryworker]
livetext 20420 0.0 2.8 21772 14184 pts/1 S+ Jul31 1:56 /usr/bin/mogilefsd [queryworker]
livetext 20421 0.0 2.8 21788 14240 pts/1 S+ Jul31 1:57 /usr/bin/mogilefsd [queryworker]
livetext 20422 0.0 2.8 21796 14236 pts/1 S+ Jul31 1:57 /usr/bin/mogilefsd [queryworker]
livetext 20423 0.0 2.8 21772 14224 pts/1 S+ Jul31 1:56 /usr/bin/mogilefsd [queryworker]
livetext 20424 0.0 2.1 18140 10696 pts/1 S+ Jul31 1:45 /usr/bin/mogilefsd [monitor]
livetext 20425 0.0 1.8 17132 9160 pts/1 S+ Jul31 0:06 /usr/bin/mogilefsd [reaper]
root 1000 0.0 0.1 3916 676 pts/0 S+ 09:16 0:00 egrep mogstored|mogilefsd
n3:~# ps aux | egrep "mogstored|mogilefsd"
root 11537 10.3 1.3 26952 6960 pts/1 S+ Jul31 243:57 /usr/bin/perl -w /usr/bin/mogstored
root 11538 0.0 1.3 26952 6960 pts/1 S+ Jul31 1:29 /usr/bin/perl -w /usr/bin/mogstored
root 11539 0.0 1.3 26952 6960 pts/1 S+ Jul31 1:29 /usr/bin/perl -w /usr/bin/mogstored
root 11540 0.0 1.3 26952 6960 pts/1 S+ Jul31 1:29 /usr/bin/perl -w /usr/bin/mogstored
livetext 11543 0.7 1.0 12744 5256 pts/2 S+ Jul31 17:17 /usr/bin/perl /usr/bin/mogilefsd
livetext 11544 0.0 0.3 13208 1932 pts/2 S+ Jul31 0:00 /usr/bin/mogilefsd [delete]
livetext 11545 0.0 1.1 16908 5612 pts/2 S+ Jul31 1:34 /usr/bin/mogilefsd [queryworker]
livetext 11546 0.0 1.0 16932 5556 pts/2 S+ Jul31 1:34 /usr/bin/mogilefsd [queryworker]
livetext 11547 0.0 1.0 16820 5400 pts/2 S+ Jul31 1:33 /usr/bin/mogilefsd [queryworker]
livetext 11548 0.0 1.0 16852 5500 pts/2 S+ Jul31 1:32 /usr/bin/mogilefsd [queryworker]
livetext 11549 0.0 1.0 16920 5516 pts/2 S+ Jul31 1:34 /usr/bin/mogilefsd [queryworker]
livetext 11550 0.0 1.0 16904 5468 pts/2 S+ Jul31 1:33 /usr/bin/mogilefsd [queryworker]
livetext 11551 0.0 1.0 16860 5404 pts/2 S+ Jul31 1:34 /usr/bin/mogilefsd [queryworker]
livetext 11552 0.0 1.0 16812 5508 pts/2 S+ Jul31 1:33 /usr/bin/mogilefsd [queryworker]
livetext 11553 0.0 1.0 16932 5520 pts/2 S+ Jul31 1:36 /usr/bin/mogilefsd [queryworker]
livetext 11554 0.0 1.0 16920 5464 pts/2 S+ Jul31 1:32 /usr/bin/mogilefsd [queryworker]
livetext 11555 0.0 1.7 16072 8752 pts/2 S+ Jul31 1:46 /usr/bin/mogilefsd [monitor]
livetext 11556 0.0 0.5 13260 2544 pts/2 S+ Jul31 0:00 /usr/bin/mogilefsd [reaper]
</pre><p>
<i>After 48 hours</i>
</p>
<pre class="wiki">n1:/etc# ps aux | egrep "mogstored|mogilefsd"
root 9964 8.1 1.3 27392 6888 pts/2 R+ Jul31 233:04 /usr/bin/perl -w /usr/bin/mogstored
root 9966 0.0 1.3 27392 6888 pts/2 S+ Jul31 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 9967 0.0 1.3 27392 6888 pts/2 S+ Jul31 2:05 /usr/bin/perl -w /usr/bin/mogstored
livetext 9970 0.7 1.0 12748 5248 pts/3 S+ Jul31 22:36 /usr/bin/perl /usr/bin/mogilefsd
livetext 9971 0.0 3.6 33576 18316 pts/3 S+ Jul31 2:40 /usr/bin/mogilefsd [replicate]
livetext 9972 0.0 0.3 13212 1884 pts/3 S+ Jul31 0:03 /usr/bin/mogilefsd [delete]
livetext 9973 0.0 1.1 17752 5788 pts/3 S+ Jul31 2:00 /usr/bin/mogilefsd [queryworker]
livetext 9974 0.0 1.1 17756 5768 pts/3 S+ Jul31 2:02 /usr/bin/mogilefsd [queryworker]
livetext 9975 0.0 1.1 17788 5816 pts/3 S+ Jul31 2:01 /usr/bin/mogilefsd [queryworker]
livetext 9976 0.0 1.1 17776 5828 pts/3 S+ Jul31 2:01 /usr/bin/mogilefsd [queryworker]
livetext 9977 0.0 1.1 17816 5740 pts/3 S+ Jul31 2:02 /usr/bin/mogilefsd [queryworker]
livetext 9978 0.0 1.1 17840 5888 pts/3 S+ Jul31 2:01 /usr/bin/mogilefsd [queryworker]
livetext 9979 0.0 1.1 17796 5852 pts/3 S+ Jul31 2:01 /usr/bin/mogilefsd [queryworker]
livetext 9980 0.0 1.1 17808 5828 pts/3 S+ Jul31 2:02 /usr/bin/mogilefsd [queryworker]
livetext 9981 0.0 1.1 17796 5860 pts/3 S+ Jul31 2:02 /usr/bin/mogilefsd [queryworker]
livetext 9982 0.0 1.1 17808 5756 pts/3 S+ Jul31 2:02 /usr/bin/mogilefsd [queryworker]
livetext 9983 0.0 1.8 16600 9232 pts/3 S+ Jul31 1:41 /usr/bin/mogilefsd [monitor]
livetext 9984 0.0 0.4 13260 2428 pts/3 S+ Jul31 0:05 /usr/bin/mogilefsd [reaper]
root 9998 0.0 1.3 27392 6888 pts/2 S+ Jul31 2:03 /usr/bin/perl -w /usr/bin/mogstored
root 10001 0.0 1.3 27392 6888 pts/2 S+ Jul31 2:07 /usr/bin/perl -w /usr/bin/mogstored
root 10002 0.0 1.3 27392 6888 pts/2 S+ Jul31 2:02 /usr/bin/perl -w /usr/bin/mogstored
root 15037 0.0 0.1 1820 560 pts/0 R+ 17:30 0:00 grep -E mogstored|mogilefsd
[root@livetext tmp]# ps aux | egrep "mogstored|mogilefsd"
root 20406 5.0 2.9 59232 14660 pts/2 Rl+ Jul31 145:00 /usr/bin/perl -w /usr/bin/mogstored
livetext 20411 0.6 2.0 16736 10144 pts/1 S+ Jul31 18:20 /usr/bin/perl /usr/bin/mogilefsd
livetext 20412 0.1 6.1 38792 31192 pts/1 S+ Jul31 3:44 /usr/bin/mogilefsd [replicate]
livetext 20413 0.0 1.7 17184 9000 pts/1 S+ Jul31 0:04 /usr/bin/mogilefsd [delete]
livetext 20414 0.0 3.0 22900 15296 pts/1 S+ Jul31 2:22 /usr/bin/mogilefsd [queryworker]
livetext 20415 0.0 3.0 22844 15320 pts/1 S+ Jul31 2:23 /usr/bin/mogilefsd [queryworker]
livetext 20416 0.0 3.0 22836 15272 pts/1 S+ Jul31 2:21 /usr/bin/mogilefsd [queryworker]
livetext 20417 0.0 3.0 22864 15292 pts/1 S+ Jul31 2:20 /usr/bin/mogilefsd [queryworker]
livetext 20418 0.0 3.0 22920 15328 pts/1 S+ Jul31 2:22 /usr/bin/mogilefsd [queryworker]
livetext 20419 0.0 3.0 22920 15312 pts/1 S+ Jul31 2:21 /usr/bin/mogilefsd [queryworker]
livetext 20420 0.0 3.0 22872 15300 pts/1 S+ Jul31 2:21 /usr/bin/mogilefsd [queryworker]
livetext 20421 0.0 3.0 22888 15340 pts/1 S+ Jul31 2:23 /usr/bin/mogilefsd [queryworker]
livetext 20422 0.0 3.0 22912 15312 pts/1 S+ Jul31 2:21 /usr/bin/mogilefsd [queryworker]
livetext 20423 0.0 3.0 22892 15328 pts/1 S+ Jul31 2:22 /usr/bin/mogilefsd [queryworker]
livetext 20424 0.0 2.1 18256 10804 pts/1 S+ Jul31 2:08 /usr/bin/mogilefsd [monitor]
livetext 20425 0.0 1.8 17264 9264 pts/1 S+ Jul31 0:07 /usr/bin/mogilefsd [reaper]
root 5051 0.0 0.1 3916 732 pts/0 S+ 17:31 0:00 egrep mogstored|mogilefsd
n3:/etc/init.d# ps aux | egrep "mogstored|mogilefsd"
root 11537 10.3 1.1 26952 5852 pts/1 S+ Jul31 295:23 /usr/bin/perl -w /usr/bin/mogstored
root 11538 0.0 1.1 26952 5852 pts/1 S+ Jul31 1:50 /usr/bin/perl -w /usr/bin/mogstored
root 11539 0.0 1.1 26952 5852 pts/1 S+ Jul31 1:50 /usr/bin/perl -w /usr/bin/mogstored
root 11540 0.0 1.1 26952 5852 pts/1 S+ Jul31 1:50 /usr/bin/perl -w /usr/bin/mogstored
livetext 11543 0.7 1.0 12744 5256 pts/2 S+ Jul31 21:05 /usr/bin/perl /usr/bin/mogilefsd
livetext 11544 0.0 0.3 13208 1856 pts/2 S+ Jul31 0:00 /usr/bin/mogilefsd [delete]
livetext 11545 0.0 1.1 17724 5772 pts/2 S+ Jul31 1:54 /usr/bin/mogilefsd [queryworker]
livetext 11546 0.0 1.1 17768 5736 pts/2 S+ Jul31 1:55 /usr/bin/mogilefsd [queryworker]
livetext 11547 0.0 1.1 17744 5660 pts/2 S+ Jul31 1:54 /usr/bin/mogilefsd [queryworker]
livetext 11548 0.0 1.1 17712 5768 pts/2 S+ Jul31 1:53 /usr/bin/mogilefsd [queryworker]
livetext 11549 0.0 1.1 17708 5808 pts/2 S+ Jul31 1:55 /usr/bin/mogilefsd [queryworker]
livetext 11550 0.0 1.1 17736 5752 pts/2 S+ Jul31 1:53 /usr/bin/mogilefsd [queryworker]
livetext 11551 0.0 1.1 17788 5640 pts/2 S+ Jul31 1:54 /usr/bin/mogilefsd [queryworker]
livetext 11552 0.0 1.1 17772 5764 pts/2 S+ Jul31 1:54 /usr/bin/mogilefsd [queryworker]
livetext 11553 0.0 1.1 17764 5780 pts/2 S+ Jul31 1:57 /usr/bin/mogilefsd [queryworker]
livetext 11554 0.0 1.1 17724 5736 pts/2 S+ Jul31 1:53 /usr/bin/mogilefsd [queryworker]
livetext 11555 0.0 1.7 16596 8768 pts/2 S+ Jul31 2:08 /usr/bin/mogilefsd [monitor]
livetext 11556 0.0 0.4 13260 2320 pts/2 S+ Jul31 0:00 /usr/bin/mogilefsd [reaper]
root 16141 0.0 0.1 1820 564 pts/0 R+ 12:33 0:00 grep -E mogstored|mogilefsd
</pre><p>
Mem diff for server n1:
</p>
<pre class="wiki">From start to 20th
/usr/bin/mogilefsd [delete] : 232/-7736
/usr/bin/mogilefsd [monitor] : 1244/-3028
/usr/bin/mogilefsd [queryworker] : 21032/-54748
/usr/bin/mogilefsd [reaper] : 260/-6008
/usr/bin/mogilefsd [replicate] : 14732/6740
/usr/bin/perl -w /usr/bin/mogstored : 125556/12384
/usr/bin/perl /usr/bin/mogilefsd : 0/-4364
>From 20th to 24th
/usr/bin/mogilefsd [delete] : 0/-8
/usr/bin/mogilefsd [monitor] : 212/-188
/usr/bin/mogilefsd [queryworker] : 4012/1976
/usr/bin/mogilefsd [reaper] : 0/-1320
/usr/bin/mogilefsd [replicate] : 892/44
/usr/bin/perl -w /usr/bin/mogstored : 0/2592
/usr/bin/perl /usr/bin/mogilefsd : 0/0
>From 24th to 39th
/usr/bin/mogilefsd [delete] : 232/36
/usr/bin/mogilefsd [monitor] : 944/1132
/usr/bin/mogilefsd [queryworker] : 16484/9860
/usr/bin/mogilefsd [reaper] : 252/84
/usr/bin/mogilefsd [replicate] : 3424/1608
/usr/bin/perl -w /usr/bin/mogstored : 0/2544
/usr/bin/perl /usr/bin/mogilefsd : 0/0
>From 39th to 48th
/usr/bin/mogilefsd [delete] : 0/-16
/usr/bin/mogilefsd [monitor] : 528/800
/usr/bin/mogilefsd [queryworker] : 8928/5156
/usr/bin/mogilefsd [reaper] : 0/36
/usr/bin/mogilefsd [replicate] : 1780/284
/usr/bin/perl -w /usr/bin/mogstored : 0/-1344
/usr/bin/perl /usr/bin/mogilefsd : 0/0
</pre><p>
Mem diff for server n2:
</p>
<pre class="wiki">From start to 20th
C:\tmp>perl diff.pl
/usr/bin/mogilefsd [delete] : 232/232
/usr/bin/mogilefsd [monitor] : 236/316
/usr/bin/mogilefsd [queryworker] : 26424/36848
/usr/bin/mogilefsd [reaper] : 264/260
/usr/bin/mogilefsd [replicate] : 15352/15556
/usr/bin/perl -w /usr/bin/mogstored : 37192/6584
/usr/bin/perl /usr/bin/mogilefsd : 132/84
>From 20th to 24th
/usr/bin/mogilefsd [delete] : 116/44
/usr/bin/mogilefsd [monitor] : 120/56
/usr/bin/mogilefsd [queryworker] : 4840/4948
/usr/bin/mogilefsd [reaper] : 132/52
/usr/bin/mogilefsd [replicate] : 892/952
/usr/bin/perl -w /usr/bin/mogstored : 0/0
/usr/bin/perl /usr/bin/mogilefsd : 0/0
>From 24th to 39th
/usr/bin/mogilefsd [delete] : 116/176
/usr/bin/mogilefsd [monitor] : 120/220
/usr/bin/mogilefsd [queryworker] : 20596/20388
/usr/bin/mogilefsd [reaper] : 132/192
/usr/bin/mogilefsd [replicate] : 3916/3888
/usr/bin/perl -w /usr/bin/mogstored : 0/0
/usr/bin/perl /usr/bin/mogilefsd : 0/0
>From 39th to 48th
/usr/bin/mogilefsd [delete] : 116/88
/usr/bin/mogilefsd [monitor] : 116/108
/usr/bin/mogilefsd [queryworker] : 10948/10956
/usr/bin/mogilefsd [reaper] : 132/104
/usr/bin/mogilefsd [replicate] : 2028/2036
/usr/bin/perl -w /usr/bin/mogstored : 0/0
/usr/bin/perl /usr/bin/mogilefsd : 0/0
</pre><p>
Mem diff for server n3:
</p>
<pre class="wiki">From start to 20th
/usr/bin/mogilefsd [delete] : 232/-7736
/usr/bin/mogilefsd [monitor] : 1248/-2800
/usr/bin/mogilefsd [queryworker] : 20924/-53464
/usr/bin/mogilefsd [reaper] : 260/-7104
/usr/bin/perl -w /usr/bin/mogstored : 64224/-7360
/usr/bin/perl /usr/bin/mogilefsd : 0/-4348
>From 20 to 24th
/usr/bin/mogilefsd [delete] : 0/-24
/usr/bin/mogilefsd [monitor] : 208/-304
/usr/bin/mogilefsd [queryworker] : 4064/1468
/usr/bin/mogilefsd [reaper] : 0/-12
/usr/bin/perl -w /usr/bin/mogstored : 0/-2224
/usr/bin/perl /usr/bin/mogilefsd : 0/0
>From 24 to 39th
/usr/bin/mogilefsd [delete] : 232/76
/usr/bin/mogilefsd [monitor] : 948/1340
/usr/bin/mogilefsd [queryworker] : 16432/11104
/usr/bin/mogilefsd [reaper] : 256/24
/usr/bin/perl -w /usr/bin/mogstored : 0/3920
/usr/bin/perl /usr/bin/mogilefsd : 0/0
>From 39 to 48th
/usr/bin/mogilefsd [delete] : 0/-76
/usr/bin/mogilefsd [monitor] : 524/16
/usr/bin/mogilefsd [queryworker] : 8580/2468
/usr/bin/mogilefsd [reaper] : 0/-224
/usr/bin/perl -w /usr/bin/mogstored : 0/-4432
/usr/bin/perl /usr/bin/mogilefsd : 0/0
</pre><h2>Conclusion</h2>
<p>
The noticeable memory leak is from the replicator and the query worker.
</p>
<p>
>From 20th to 48th:
</p>
<pre class="wiki">n1:
replicator: 892+44+3424+1608+1780+284 = 8032
queryworker: 4012+1976+16484+9860+8928+5156 = 46416
n2:
replicator: 892+952+3916+3888+2028+2036 = 13712
queryworker: 4840+4948+20596+20388+10948+10956 = 72676
n3:
queryworker: 4064+1468+16432+11104+8580+2468 = 44116
replicator leak avg: 10872k/per 28 hours = 388k/hour
queryworker leak avg: 54402k/per 28 hours = 1943k/hour
total : 2331k/hour
</pre><p>If the traffic to mogilefs is as high as the benchmark and
suppose we have 4G ram in each machine, 1G dedicated to OS and
something else: we have 3G, we would need to restart the replicator and
queryworker within 56 days. It's possible to do graceful restart for
both processes by restarting them at the point where they finish doing
important things by modifying /usr/bin/mogilefsd source.
</p>
<hr>
<p>As I mentioned above, tasks in mogilefsd can be easily restarted to
reclaim memory since it provides a mechanism to maintain the number of
processes of each task, any shutdowned task will be reforked
automatically. The way I modified the mogilefsd is by introducing a
counter to the while loops in each jobs, and when the counter reaches a
value, the while loop will be terminated making the process died and
restarted. See the patch [<a class="ext-link" title="https://xxxxxxxxxx" href="yyyyyyyyyy">zzzzzzzzz</a>].
</p>
<p>
The below test: replicator will be restarted within 1 hours,
queryworker will be restarted within 1000 queries (In the production,
we will increase these numbers).
</p>
<pre class="wiki">10am.
n1:~# ps aux | egrep "mogstored|mogilefsd"
root 1952 2.4 1.7 19200 8656 pts/1 R+ 09:34 0:48 /usr/bin/perl -w /usr/bin/mogstored
root 1975 0.0 1.7 19200 8656 pts/1 S+ 09:34 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 1976 0.0 1.7 19200 8656 pts/1 S+ 09:34 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 2071 0.0 1.7 19200 8656 pts/1 S+ 09:50 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 2072 0.0 1.7 19200 8656 pts/1 S+ 09:50 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 2073 0.0 1.7 19200 8656 pts/1 S+ 09:51 0:00 /usr/bin/perl -w /usr/bin/mogstored
livetext 2091 0.3 1.9 12748 9676 pts/2 S+ 09:56 0:02 /usr/bin/perl /usr/bin/mogilefsd
livetext 2092 0.0 2.3 14856 11756 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [replicate]
livetext 2093 0.0 1.8 12748 9624 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [delete]
livetext 2094 0.0 1.9 12748 9676 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2095 0.0 1.9 12872 9752 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2096 0.0 1.9 12876 9752 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2097 0.0 1.9 12748 9668 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2098 0.0 1.9 12748 9672 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2099 0.0 1.9 12748 9660 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2100 0.0 1.9 12748 9660 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2101 0.0 1.9 12748 9660 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2102 0.0 1.9 12876 9756 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2103 0.0 1.9 12880 9736 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2104 0.0 2.0 13768 10564 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [monitor]
livetext 2105 0.0 1.9 12748 9652 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [reaper]
11am.
n1:~# ps aux | egrep "mogstored|mogilefsd"
root 1952 5.3 1.8 20140 9424 pts/1 R+ 09:34 4:56 /usr/bin/perl -w /usr/bin/mogstored
root 1975 0.0 1.8 20140 9424 pts/1 S+ 09:34 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 1976 0.0 1.8 20140 9424 pts/1 S+ 09:34 0:03 /usr/bin/perl -w /usr/bin/mogstored
root 2071 0.0 1.8 20140 9424 pts/1 S+ 09:50 0:03 /usr/bin/perl -w /usr/bin/mogstored
root 2072 0.0 1.8 20140 9424 pts/1 S+ 09:50 0:03 /usr/bin/perl -w /usr/bin/mogstored
root 2073 0.0 1.8 20140 9424 pts/1 S+ 09:51 0:03 /usr/bin/perl -w /usr/bin/mogstored
livetext 2091 0.6 1.9 12748 9684 pts/2 S+ 09:56 0:26 /usr/bin/perl /usr/bin/mogilefsd
livetext 2093 0.0 1.8 12748 9628 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [delete]
livetext 2104 0.0 2.0 13768 10624 pts/2 S+ 09:56 0:02 /usr/bin/mogilefsd [monitor]
livetext 2105 0.0 1.9 12748 9660 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [reaper]
livetext 2162 0.0 1.9 13008 9852 pts/2 S+ 10:46 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2163 0.0 1.9 12968 9860 pts/2 S+ 10:46 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2164 0.0 1.9 12996 9856 pts/2 S+ 10:46 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2165 0.0 1.9 12984 9856 pts/2 S+ 10:46 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2166 0.0 1.9 12988 9856 pts/2 S+ 10:46 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2167 0.0 1.9 12988 9848 pts/2 S+ 10:46 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2168 0.0 1.9 12980 9860 pts/2 S+ 10:46 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2169 0.0 1.9 12872 9788 pts/2 S+ 10:46 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2170 0.0 1.9 13004 9856 pts/2 S+ 10:46 0:01 /usr/bin/mogilefsd [queryworker]
livetext 2171 0.0 1.9 12872 9788 pts/2 S+ 10:46 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2191 0.1 2.3 14840 11716 pts/2 S+ 11:00 0:00 /usr/bin/mogilefsd [replicate]
12pm.
n1:~# ps aux | egrep "mogstored|mogilefsd"
root 1952 5.7 1.8 20140 9424 pts/1 R+ 09:34 8:50 /usr/bin/perl -w /usr/bin/mogstored
root 1975 0.0 1.8 20140 9424 pts/1 S+ 09:34 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 1976 0.0 1.8 20140 9424 pts/1 S+ 09:34 0:05 /usr/bin/perl -w /usr/bin/mogstored
root 2071 0.0 1.8 20140 9424 pts/1 S+ 09:50 0:05 /usr/bin/perl -w /usr/bin/mogstored
root 2072 0.0 1.8 20140 9424 pts/1 S+ 09:50 0:05 /usr/bin/perl -w /usr/bin/mogstored
root 2073 0.0 1.8 20140 9424 pts/1 S+ 09:51 0:05 /usr/bin/perl -w /usr/bin/mogstored
livetext 2091 0.6 1.9 12748 9684 pts/2 S+ 09:56 0:55 /usr/bin/perl /usr/bin/mogilefsd
livetext 2093 0.0 1.8 12748 9640 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [delete]
livetext 2104 0.0 2.1 13872 10684 pts/2 S+ 09:56 0:04 /usr/bin/mogilefsd [monitor]
livetext 2105 0.0 1.9 12748 9672 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [reaper]
livetext 2285 0.0 1.9 12876 9776 pts/2 S+ 11:58 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2286 0.0 1.9 12992 9856 pts/2 S+ 11:58 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2287 0.0 1.9 12876 9776 pts/2 S+ 11:58 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2288 0.0 1.9 12976 9848 pts/2 S+ 11:58 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2289 0.0 1.9 12984 9844 pts/2 S+ 11:58 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2291 0.0 1.9 12876 9764 pts/2 S+ 11:58 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2292 0.0 1.9 12876 9768 pts/2 S+ 11:58 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2293 0.0 1.9 13000 9832 pts/2 S+ 11:58 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2294 0.0 1.9 12876 9780 pts/2 S+ 11:58 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2295 0.0 1.9 12876 9776 pts/2 S+ 11:58 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2302 0.0 2.2 14572 11532 pts/2 S+ 12:03 0:00 /usr/bin/mogilefsd [replicate]
1.30pm
n1:~# ps aux | egrep "mogstored|mogilefsd"
root 1952 6.2 1.8 20140 9424 pts/1 S+ 09:34 14:35 /usr/bin/perl -w /usr/bin/mogstored
root 1975 0.0 1.8 20140 9424 pts/1 S+ 09:34 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 1976 0.0 1.8 20140 9424 pts/1 S+ 09:34 0:08 /usr/bin/perl -w /usr/bin/mogstored
root 2071 0.0 1.8 20140 9424 pts/1 S+ 09:50 0:09 /usr/bin/perl -w /usr/bin/mogstored
root 2072 0.0 1.8 20140 9424 pts/1 S+ 09:50 0:08 /usr/bin/perl -w /usr/bin/mogstored
root 2073 0.0 1.8 20140 9424 pts/1 S+ 09:51 0:08 /usr/bin/perl -w /usr/bin/mogstored
livetext 2091 0.7 1.9 12748 9684 pts/2 S+ 09:56 1:32 /usr/bin/perl /usr/bin/mogilefsd
livetext 2093 0.0 1.9 12748 9656 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [delete]
livetext 2104 0.0 2.1 13976 10756 pts/2 S+ 09:56 0:07 /usr/bin/mogilefsd [monitor]
livetext 2105 0.0 1.9 12748 9688 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [reaper]
livetext 2382 0.0 2.3 14920 11840 pts/2 S+ 13:06 0:01 /usr/bin/mogilefsd [replicate]
livetext 2384 0.0 1.9 12992 9848 pts/2 S+ 13:08 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2385 0.0 1.9 12884 9788 pts/2 S+ 13:08 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2386 0.0 1.9 12980 9856 pts/2 S+ 13:08 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2388 0.0 1.9 12996 9860 pts/2 S+ 13:08 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2389 0.0 1.9 12876 9788 pts/2 S+ 13:08 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2390 0.0 1.9 12876 9784 pts/2 S+ 13:08 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2391 0.0 1.9 12884 9780 pts/2 S+ 13:08 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2392 0.0 1.9 12876 9784 pts/2 S+ 13:08 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2393 0.0 1.9 12996 9860 pts/2 S+ 13:08 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2394 0.0 1.9 12884 9780 pts/2 S+ 13:08 0:00 /usr/bin/mogilefsd [queryworker]
3pm
n1:~# ps aux | egrep "mogstored|mogilefsd"
root 1952 6.5 3.1 28028 16080 pts/1 R+ 09:34 22:18 /usr/bin/perl -w /usr/bin/mogstored
root 1975 0.0 3.1 28028 16080 pts/1 S+ 09:34 0:00 /usr/bin/perl -w /usr/bin/mogstored
root 1976 0.0 3.1 28028 16080 pts/1 S+ 09:34 0:13 /usr/bin/perl -w /usr/bin/mogstored
root 2071 0.0 3.1 28028 16080 pts/1 S+ 09:50 0:13 /usr/bin/perl -w /usr/bin/mogstored
root 2072 0.0 3.1 28028 16080 pts/1 S+ 09:50 0:12 /usr/bin/perl -w /usr/bin/mogstored
root 2073 0.0 3.1 28028 16080 pts/1 S+ 09:51 0:13 /usr/bin/perl -w /usr/bin/mogstored
livetext 2091 0.7 1.9 12748 9684 pts/2 S+ 09:56 2:25 /usr/bin/perl /usr/bin/mogilefsd
livetext 2093 0.0 1.9 12748 9676 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [delete]
livetext 2104 0.0 2.1 14080 10868 pts/2 S+ 09:56 0:11 /usr/bin/mogilefsd [monitor]
livetext 2105 0.0 1.9 12880 9708 pts/2 S+ 09:56 0:00 /usr/bin/mogilefsd [reaper]
livetext 2567 0.0 1.9 12980 9868 pts/2 S+ 14:48 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2568 0.0 1.9 12996 9876 pts/2 S+ 14:48 0:01 /usr/bin/mogilefsd [queryworker]
livetext 2569 0.0 1.9 12876 9804 pts/2 S+ 14:48 0:01 /usr/bin/mogilefsd [queryworker]
livetext 2570 0.0 1.9 12976 9864 pts/2 S+ 14:48 0:01 /usr/bin/mogilefsd [queryworker]
livetext 2572 0.0 1.9 12992 9864 pts/2 S+ 14:48 0:01 /usr/bin/mogilefsd [queryworker]
livetext 2573 0.0 1.9 13004 9876 pts/2 S+ 14:48 0:01 /usr/bin/mogilefsd [queryworker]
livetext 2576 0.0 1.9 12884 9800 pts/2 S+ 14:48 0:01 /usr/bin/mogilefsd [queryworker]
livetext 2579 0.0 1.9 12876 9800 pts/2 S+ 14:48 0:01 /usr/bin/mogilefsd [queryworker]
livetext 2580 0.0 1.9 12876 9808 pts/2 S+ 14:48 0:00 /usr/bin/mogilefsd [queryworker]
livetext 2581 0.0 1.9 13008 9872 pts/2 S+ 14:48 0:01 /usr/bin/mogilefsd [queryworker]
livetext 2665 0.0 1.9 12876 9736 pts/2 S+ 15:12 0:00 /usr/bin/mogilefsd [replicate]
</pre><h2>Conclusion</h2>
<p>The graceful restart can mitigate any memory leak problem in
mogilefsd. It's also safe because the shutdowning occurs after process
has complete its important executions. Thus, memory leak is now out of
our concern.
</p>
</div>
</div>
</body></html>