Perlbal proxy hangs

[Kevin Murphy]

I'm getting intermittent hangs on some perlbal instances that I'm using
for upload proxying in front of Apache.

After running for awhile, perlbal just stops responding. It's still
listening on port 80, but any requests just hang. The problem is worse
under heavy load.

Here's my config:

-----
CREATE POOL real
   POOL flickr_real ADD 127.0.0.1:81
LOAD Stats
CREATE SERVICE balancer
   SET listen          = 0.0.0.0:80
   SET role            = reverse_proxy
   SET pool            = real
   SET persist_client  = on
   SET persist_backend = on
   SET verify_backend  = on
   SET backend_persist_cache             = 5
   SET buffer_backend_connect            = 100k
   SET buffer_upload_threshold_size      = 100k
   SET buffer_upload_threshold_time      = 3
   SET buffer_uploads                    = on
   SET buffer_uploads_path               = /var/tmp/perlbal
   SET plugins                           = Stats

ENABLE balancer

# always good to keep an internal management port open:
CREATE SERVICE mgmt
   SET role   = management
   SET listen = 127.0.0.1:60000
ENABLE mgmt
SERVER pidfile = /var/run/perlbal.pid
-----

Versions:
Perlbal 1.7.0
Danga::Socket 1.59
IO::AIO 2.51
perl 5.8.6
RHEL 4
Linux kernel 2.6.9-67.0.7.ELsmp

strace on a hung process:
select(8, [7], NULL, NULL, NULL

GDB Backtrace on a hung process:

#0  0x007a87a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00883bf1 in ___newselect_nocancel () from /lib/tls/libc.so.6
#2  0x006da006 in poll_wait ()
    from /home/y/lib/perl5/site_perl/5.8/i686-linux-64int/auto/IO/AIO/AIO.so
#3  0x006de345 in XS_IO__AIO_flush ()
    from /home/y/lib/perl5/site_perl/5.8/i686-linux-64int/auto/IO/AIO/AIO.so
#4  0x080ae1b8 in Perl_pp_entersub ()
#5  0x080a7af8 in Perl_runops_standard ()
#6  0x08061bfa in S_call_body ()
#7  0x080619f2 in Perl_call_sv ()
#8  0x08064b8a in S_call_list_body ()
#9  0x080649a6 in Perl_call_list ()
#10 0x0805ffef in perl_destruct ()
#11 0x0805e5ae in main ()

Has anyone else seen this behavior?

--
Kevin Murphy <murphyk at yahoo-inc.com>