interpreting 'leaks' output - debugging memory leak

[John Berthels]

Hi folks,

I'm trying to chase down large spikes in memory usage (perlbal can
shoot rapidly up to ~5Gbyte, mostly swapped out, which I guess implies
a leak?) with perlbal in our environment. We're running 1.59 (the only
leak I see fixed in the changelog since then relates to the stats
plugin, which we don't have loaded). We are using the Urlmatch plugin.

The leak has only really been noticeable since we started reproxying
(using x-reproxy-url).

I've turned on TRACK_OBJECTS in Perlbal/Socket.pm and do see numbers
increasing when memory usage is rising.

As one example, here's a breakdown of items show in 'leaks' by type:

      8 Perlbal::BackendHTTP()
     67 Perlbal::BackendHTTP(R)
     24 Perlbal::ClientHTTPBase(R)
      1 Perlbal::ClientManage(R)
      1 Perlbal::ClientProxy()
      7 Perlbal::ClientProxy(R)
     10 Perlbal::ClientProxy(RW)
      2 Perlbal::TCPListener(R)
      1 Perlbal::UploadListener(R)

Of the 67 Backend objects, 63 are in state closed and look like this:

Perlbal::BackendHTTP(R): closed; uses=0; closed; has_attention=yes

(yes, all of the closed ones are uses=0, has_attention=yes).

We only have 6 possible back end servers (two apps servers, 4 mogilefs
servers which we reproxy to).

Is the number of ClientHTTPBase high? Shouldn't they be shortlived and
become either ClientHTTP or ClientProxy?


The code in Perlbal::ReproxyManager::note_backend_close appears to
want to clean up closed backends - so I'm guessing I should really
have no closed backends in 'leaks'.

Can anyone suggest a good next step in debugging this? Or should I
just upgrade and hope it goes away?

regards,

jb

PS. Does anyone have any tips on limiting the number of connections
from a specific IP? I don't see any code for it. I imagine I could
write a plugin, but maybe someone has done that already? Sorry if this
is a FAQ, I couldn't see anything more recent than 2005 on this (and
that suggested writing a simple plugin).