mogilefs usage and performance

[Anatoli Stoyanovski]

> > If configured with 'memcached before get_paths':
> >
> > perlbal->mod_perl->memcached->perlbal->mogstored: 250req/sec
> >
> > mod_perl->memcached (no real file contents, just get x-reproxy): 1500
> > req/sec (memcached works fast, not an invention)
> >
> > I dont want to store actual file contents in memcached, so didn't test it.
> > So, (1500req/sec of mod_perl/memcached + 1170 req/sec mogstored) via
> > perlbal = 250req/sec overall performance for 10kb file downloading in
> > 10 concurrent connections.
>
> Paths don't usually change that often, so I just give them a five minute
> timeout. Very occasionally we'd see a blip, but it would correct itself.
> Also if you use perlbal's ability to X-Reproxy to a list of paths it'll
> almost always work even if one of the paths dies.

The problem is images, that users replaces via web-interface. If user
replaces images, we don't change url for this (not should be changed).
mogilefs will give new .fid url for any these images, but url
/images/somename.jpg will give 404 error for 5 minutes if we delete
old image from mogilefs, all site users will see it. But we can leave
it there, not worrying about image garbage, but it will confuse
publisher for up to 5 minutes, which isn't perfect too. So, the cache
timeout should be very small.

> So what you're saying is that:
>
> Just directly from mod_perl->memcached to fetch a path but not actually
> reproxying runs at 1500 requests/sec?

Yes

>
> How're you getting the requests/sec to mogstored? Directly querying it?

Yes. mod_perl on 81th port, just doing
$r->header_out('X-Reproxy-Url'=>$paths[0]);

In test conditions, it always returns constant url, which I use for
next benchmark 'ab' call.

> Then putting the whole chain together ends up with 250req/sec? Are you
> monitoring the box during your 'ab' tests and seeing if any one
> component starts nailing IO, page thrashing, or maxing out CPU?

Well... after 50 000 requests the system state (by top):
Cpu(s): 75.7% us,  8.9% sy,  0.0% ni, 11.9% id
load average: 2.05, 1.01, 0.44

perlbal 93% CPU
mogilefsd 17% (if memcached switched off)
apache/mod_perl: 5 processes with 7% per each
nginx (replacing mogstored in GETs): <5%

mysql runs other server.

no any disk activity since all logging switched off, the file is in RAM cache.

> Also, is your perlbal -> mod_perl step using backend keepalives? Not
> having that working adds a lot of overhead to perlbal.

  SET persist_client  = on
  SET persist_backend = on
  SET verify_backend  = off
  SET enable_reproxy  = true

I switched verify_backed off for test speed.

Perlbal was tuned as Mark Smith suggests, but it still eats CPU.
Maybe, I should search another web-accelerator software with
reproxy-like feature.