Patch and RFC: Perlbal OOM with buffered uploads and limited disk bandwidth. Also other upload weirdness

[jbj at forbidden.co.uk]

On Tue, May 15, 2007 at 07:31:29PM +0000, Brad Fitzpatrick wrote:
> Interesting, but how is that not handled by this existing code in
> ClientProxy.pm:
> 
>     # deal with chunked uploads
>     if (my $cus = $self->{chunked_upload_state}) {
>         $cus->on_readable($self);
> 
>         # if we got more than 1MB not flushed to disk,
>         # stop reading for a bit until disk catches up
>         if ($self->{read_ahead} > 1024*1024) {
>             $self->watch_read(0);
>         }
>         return;
>     }
> 
> That seems like it should do it already?

Unless I'm missing something, that will only help if
chunked_upload_state is not undef? (in our case it's just a huge block
of data being sent by an HTML form in a POST)

> If not, could you write a failing test for this, by starting with the
> existing buffered upload test, copying it to a new test, and making it
> just do large buffered to disk with forced delays in the Perlbal
> server at the right place (faking a slow disk), forcing the OOM?

Ok - I'll have a look at this. Any likely reason why it might OOM
after just 60MB? (servers have 2GB RAM, and plenty of it is free).

-jeremy