How to monitors [WAS Re: Can't fetch stats ]

Javier Frias jfrias at gmail.com
Wed Mar 26 22:37:26 UTC 2008


On Fri, Mar 21, 2008 at 1:56 AM, Robin H. Johnson <robbat2 at gentoo.org> wrote:
> On Fri, Mar 21, 2008 at 1:38 AM, Javier Frias <jfrias at gmail.com> wrote:
>
> >  Yeah, this is a similar query as i'm trying to implement, the script
>  >  running it would computer the difference from the last run, as well as
>  >  the time frame, so as to compute both the counts ( a worthwile yet not
>  >  really work draphing statistic, more  just a thing you check ), but
>  >  mostly what the growth rate is, as this I can use to plan out when in
>  >  the future i will need more storage nodes.
>  Graphing the 'device' table, as either:
>  MIN(mb_used/mb_total),AVG(mb_used/mb_total),MAX(mb_used/mb_total), or
>  the mb_used/mb_total for each device could be an interesting view (or
>  even both).
>
>
>  >  Another thing worth graphing is the replication stats, ie, files in
>  >  replication queue, etc etc.
>  Count, age-range would be interesting there.
>
>  Also a count of unreachable_fids.
>

I'm going to get some time this weekend to whip up some nagios checks
that do all of these, i will share back.

>
>  >  I handle it at the client level.  Aside from mogstored, we run an
>  >  image transformation proxy on each storage node, that can
>  >  upscale/downscale images. So when my client requests a file from
>  >  mogile, i do a get_paths, get the path, and then translate the port to
>  >  my image transformation proxy, if this fails, i try the next path, etc
>  >  etc. Since files are distributed, i woundlt be able to use my load
>  >  balancer on the storage daemons, since potentially, a file is not
>  >  guaranteed to be on the same path/device as in all the servers.
>  By the sounds of this, you are accessing the file locally on the
>  servers, instead of fetching it transforming, and storing?
>
>  Could you come up with a nice list of these things that are
>  MogileFS-specific from this thread, and summarize it back to the list?
>
Not sure i understand.

So i fetch a lot of data directly via the mogile client, my situation
is a bit different than most. 3/4 of the data i store in mogile are
images, 1/2 of which need to be scaled at some point. We used to store
all needed sizes in our previous storage solution. With mogile, we
came up with the strategy of storing 4 intermediate sizes (orig, and
then tree optimized ranges one for our largest size, one for our
medium size, and one for thumbnails ). We not only save in storage,
but also  have the flexibility to choose any size as we see fit.  We
import these sizes during our image acquisition pipeline.

We found that the conversion from these sizes was extremely low
overhead when done in the local disk ( and very low cpu , then again,
our storage nodes are pretty fast due to really good dell pricing :-D
) So aside from mogstored, we also run an apache instance with a small
mod_python handler that servers the mog_data directories, but can
apply transformations (size scaling ). So for images that need to be
transformed only, our client requests them from this apache instance
instead of using mogstored/perlbal. ( I know we could have written
this into a perlbal plugin, but we chose the path of least resistance
)


>
>

But my description of this system came about as a side thread, the
reason for this email was to figure out what things people monitored
in their mogile installations, and i got a pretty good answer on that,
hopefully I can contribute a set of scripts back for everyone to use.

>  --
>
>
> Robin Hugh Johnson
>  Gentoo Linux Developer & Infra Guy
>  E-Mail     : robbat2 at gentoo.org
>  GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85
>


More information about the mogilefs mailing list