Analysis of the mogilefsd busywait bug

[Jared Klett]

	I just checked ProcManager.pm and David's patch is in there
(svn748).

	I checked the versions on my dependencies and they match as
well:

Danga::Socket is up to date (1.56).
IO::AIO is up to date (2.33).

	Nathan: how large are the files you're writing? I was just able
to reproduce by injecting a few files 900 MB in size.

cheers,

- Jared

-----Original Message-----
From: mogilefs-bounces at lists.danga.com
[mailto:mogilefs-bounces at lists.danga.com] On Behalf Of Nathan Schmidt
Sent: Wednesday, February 28, 2007 5:13 PM
To: mogilefs at lists.danga.com
Subject: Re: Analysis of the mogilefsd busywait bug


After applying the patch David Weekly suggested in http://
lists.danga.com/pipermail/mogilefs/2007-February/000762.html we've not
seen the high CPU usage problem after about a week of 5-20 writes/ sec.

2.6.19.1.pbwiki-grsec #1 SMP i686 GNU/Linux, IO::AIO v2.33,
Danga::Socket v1.56

Regards,
-Nathan / PBwiki



On Feb 28, 2007, at 1:45 PM, Brad Fitzpatrick wrote:

> I'm still very eager to solve this, but I've been traveling for past 
> 2+ weeks, away from primary dev environments.
>
> We need to solve this before we deploy this version of MogileFS into 
> LJ production, so don't worry -- it'll get fixed.  Hopefully this 
> week.
>
> - Brad
>
>
> On Wed, 28 Feb 2007, Jared Klett wrote:
>
>> 	I hate to be the squeaky wheel, but I'm still seeing the
mogilefsd 
>> taking 100% CPU aka the "busywait" bug after updating to
>> svn748 (the latest as of today).
>>
>> 	I reinitialized my entire MogileFS environment, injected one
file, 
>> and mogilefsd on one of the two tracker servers started spinning at 
>> 100% CPU.
>>
>> 	I provided some debug info in this thread:
>>
>> http://lists.danga.com/pipermail/mogilefs/2007-February/000792.html
>>
>> 	I checked lsof and strace output and it's pretty much the same
story 
>> I laid out in that post.
>>
>> 	is there any new info or resolution on this issue?
>>
>> cheers,
>>
>> - Jared
>>
>> -----Original Message-----
>> From: mogilefs-bounces at lists.danga.com 
>> [mailto:mogilefs-bounces at lists.danga.com] On Behalf Of Brad 
>> Fitzpatrick
>> Sent: Wednesday, February 14, 2007 3:23 AM
>> To: David Weekly
>> Cc: nathan at pbwiki.com; mogilefs at lists.danga.com; Brett G. Durrett
>> Subject: Re: Analysis of the mogilefsd busywait bug
>>
>> Committed in svn740.
>>
>> On Tue, 13 Feb 2007, David Weekly wrote: