Inconsistent output of 'mogadm domain list' encountered

Chaos Wang chaoslawful at gmail.com
Thu Mar 20 02:35:58 UTC 2008


Oops...I'm sorry I made a mistake. Yes, you're right, the patch
eliminated the caching problem.

Domain::of_dmid() is a internal method, right? So it won't be called
directly outside Domain and Class module. The situation I assumed in the
previous testing snippet won't appear at all in practice.

dormando wrote:
> That's really odd... I had been testing and reloading and it wasn't
> doing that for me at all.
>
> I see I did confuse the way of_dmid works when I placed the singleton
> cache clear... Thought the name2id check was above the singleton check
> (which calls reload_domains if the name map fails). The issue also
> wasn't happening again with my patch :\
>
> It's entirely possible I goofed my test setup, so I'll fix this later
> today. Would be nice to figure out what I did wrong with the testing.
>
> The changes to Class's caching issue should be good though. You test
> those?
>
> -Dormandou
>
> Chaos Wang wrote:
>> Emm...I'm afraid making modifications only in
>> MogileFS::Domain::reload_domains() won't be enough...
>>
>> The MogileFS::Domain::delete() method called
>> MogileFS::Domain::invalidate_cache() instead of reload_domains(), so if
>> you deleted a domain, then try to retrieve domain instance through
>> MogileFS::Domain::of_dmid() with the deleted domain's ID, the issue will
>> come up again.
>>
>> I have no idea about the reason of having cache invalidation code both
>> in reload_domains() and invalidate_cache(), but it seems to be a better
>> solution to keep that code only in invalidate_cache() and have
>> reload_domains() calling it at first.
>>
>> This issue don't need parallel environment to be revealed, here is the
>> corresponding testing snippet I appended in
>> mogilefs/server/t/domains-classes.t :
>>
>> MogileFS::Domain->create("bar");
>> my $tmp_dom=MogileFS::Domain->of_namespace("bar");
>> my $tmp_dmid=$tmp_dom->{dmid};
>> $tmp_dom->delete;
>> ok(!defined(MogileFS::Domain->of_dmid($tmp_dmid)), "no domain info
>> remained in cache after deletion");
>>
>> dormando wrote:
>>> Fixed both instances in trunk...
>>>
>>> They were trivial changes (although I put the cache clear in a
>>> different
>>> place), but I might not have totally understood the original goals of
>>> the singleton caches. Feel free to berate my fail :)
>>>
>>> Also haven't figured a decent test, since the issue tends to happen in
>>> parallel processes. Any ideas?
>>>
>>> -Dormando
>>>
>>> Frank Ng wrote:
>>>  
>>>> We see the same thing in our test mogilefs env.  i made a change of
>>>> mindevcount from 2 to 3 and ran mogadm class list a few times, it's
>>>> show
>>>> alternating results.
>>>>  
>>>> [root at xxx mogdata]# mogadm class list
>>>>   domain               class                mindevcount
>>>> -------------------- -------------------- -------------
>>>>  snf_test01           default                   2
>>>>  snf_test01           snf_hr                    2
>>>>  snf_test01           snf_lr                    3 <--
>>>>  
>>>> [root at xxx mogdata]# mogadm class list
>>>>  domain               class                mindevcount
>>>> -------------------- -------------------- -------------
>>>>  snf_test01           default                   2
>>>>  snf_test01           snf_hr                    2
>>>>  snf_test01           snf_lr                    2 <--
>>>>  
>>>> [root at xxx mogdata]# mogadm domain list
>>>>  domain               class                mindevcount
>>>> -------------------- -------------------- -------------
>>>>  snf_test01           default                   2
>>>>  snf_test01           snf_hr                    2
>>>>  snf_test01           snf_lr                    2 <--
>>>>  
>>>> [root at xxx mogdata]# mogadm domain list
>>>>  domain               class                mindevcount
>>>> -------------------- -------------------- -------------
>>>>  snf_test01           default                   2
>>>>  snf_test01           snf_hr                    2
>>>>  snf_test01           snf_lr                    3 <--
>>>>
>>>>  
>>>> On Wed, Mar 12, 2008 at 3:58 AM, Chaos Wang <chaoslawful at gmail.com
>>>> <mailto:chaoslawful at gmail.com>> wrote:
>>>>
>>>>     Hi,
>>>>
>>>>     I use the MogileFS code from official svn trunk to do some
>>>> test. But
>>>>     when I do some domain addition/deletion operations, the output of
>>>>     'mogadm domain list' becomes inconsistent between different calls.
>>>>     This phenomenon can be repeated by following steps:
>>>>
>>>>        1. mogadm domain add xxx
>>>>        2. mogadm domain delete xxx
>>>>        3. mogadm domain add yyy
>>>>        4. mogadm domain list (several times)
>>>>
>>>>     I thinks the problem is in the invalidate_cache() method in
>>>>     MogileFS::Domain module, because it forget to clean up the hash
>>>> map
>>>>     %singleton (of_dmid(), which is called in domains(), used this
>>>> hash
>>>>     map before trying %id2name map). After made the following
>>>> changes in
>>>>     MogileFS::Domain::invalidate_cache(), the phenomenon doesn't
>>>> show up
>>>>     again:
>>>>
>>>>     --code begin--
>>>>     sub invalidate_cache {
>>>>     ...
>>>>     %id2name=();
>>>>     %name2id=();
>>>>     %singleton=(); # XXX: this is newly added
>>>>     ...
>>>>     --code end--
>>>>
>>>>     And I have another question: does classes belong to certain
>>>> domain?
>>>>     If so, why delete a domain does not clean up the classes under it?
>>>>
>>>>
>>>>     
>>>
>>>   
>>
>
>



More information about the mogilefs mailing list