have a hard time injecting when doing replication or deleting

dormando dormando at rydia.net
Wed Jun 25 06:04:24 UTC 2008


Exactly how many jobs of each are you running?

Some jobs don't scale as well as others. Running more of them would
increase database load...

Are you properly monitoring your database server? Does it end up in
swap, are you high on IO usage already?

The mogilefs clients don't have a very forgiving timeout, so if a
tracker is in swap it'll be unlikely to ever finish it works. However
even if IO is loaded on a machine, the trackers usually respond in
decent time...

Are you running the trackers on the database? Out of CPU? Is your
database actually under any load?

There are no transactions in use in mogilefs... So your theory isn't
likely. The amount of stuff in the queue also doesn't have relation to
load. At least not in any mogilefs service I maintain... I can insert
20+ million fids into file_to_replicate and it'll work fine. It'll be
annoyed and that's a very stupid thing to do, but it won't overload
anything by nature of there being work to do.

What version of mogilefs are you running? 2.17? Latest trunk?

-Dormando

Komtanoo Pinpimai wrote:
> Hello,
> 
> My env is:
> 2 trackers running plenty of querywork,listener,monitor,reaper jobs.
> 1 tracker running only a few delete and replicate jobs.
> 
> I guess, I didn't have this problem in the old version of
> MogileFS(version 1??).
> What's happening is when a hard drive has gone bad and I mark it as dead,
> it will create tons of replication jobs. If I have 5 simultaneous
> replication jobs in a tracker,
> it will be really hard to inject one file into the system(with tracker
> busy). Or I have only 1 replication job
> and it's in customer busy time, injecting file usually fails.
> 
> What's annoying is no matter how many trackers or querywork/listener
> jobs I added,
> if there is a few simultaneous replicate jobs with tons of works in
> their queue,
> the whole trackers seem to be busy all the time. I also have a trying
> code, like trying to inject and sleep for 15 times,
> it still does not work very well in this situation.
> 
> Since these trackers share the same database, I'm trying to guess, it
> must have something to do with transaction,
> for example, the replication or deletion jobs might create some
> transactions that lock some tables preventing
> other jobs from injecting files. And when there are tons of
> replication/deletion jobs in the line, the whole the system
> will turn into readonly mode.
> 
> Have you ever experienced this and how do you deal with it ? Is there a
> way to tell mogilefs not to use transaction but still work _ pretty
> correct _?
> 
> -- 
> I'm going to stop checking email.
> Let's talk in my Hi5.



More information about the mogilefs mailing list