Patch: Perlbal HTTP 1.1 Chunked Backend Support (Was: Anyterm daemon won't work under perlbal)

[Jonty]

Hi All,

I found a fairly horrible memory leak in the last version of the patch,
you can find the new version here:

http://playground.audioscrobbler.com/jonty/perlbal_chunked_backend_support.diff

I suspect this memory leak also exists for chunked uploads, but I've not
had time to test yet.

Tests and doc for the above patch will be posted soon. I'd appreciate
people testing it in other setups if possible - we're running it on our
live cluster, but variety breeds bugs...

Cheers,
Jonty


On Mon, 2008-03-10 at 19:12 +0000, Jonty wrote:
> Evening,
> 
> Sorry for the horrendous delay, I've only just found time to fully test
> the rewritten version in production.
> 
> Anyway, this introduces two new per-service tunables:
> 
> SET backend_http_version = (1.0|1.1)
> 
> Specifies the HTTP version to be sent to the backend, as chunked
> encoding requires HTTP 1.1 and perlbal used 1.0 previously.
> Defaults to HTTP 1.0 for backward compatibility with old backends
> communicating with perlbal.
> Currently takes "1.1" or "1.0" as a value. Must be set to "1.1" to
> enable chunked requests to backends.
> 
> SET buffer_chunked_backend_response = (false|true)
> 
> Turns on buffering of the entire HTTP response from the backend for
> chunked responses in order to set the Content-Length header for the
> de-chunked response.
> Default false, as this can require a significant amount of memory for
> large responses and delays delivery of the response to the client.
> 
> Currently this does not support proxying chunked responses to chunked
> clients, however I may add this ability in the near future.
> 
> This also introduces a hook when backends are closed, making it simple
> to log the reason for the closure - this is invaluable when debugging
> connection problems to backends.
> 
> Hope this meets with approval.
> 
> Cheers,
> Jonty
> 
> 
> On Sat, 2008-03-08 at 19:59 -0800, Brad Fitzpatrick wrote:
> > On Tue, Feb 5, 2008 at 4:28 PM, Jonty <jonty.wareing at gmail.com> wrote:
> >         On Feb 5, 2008 5:53 PM, Daniel Risacher <magnus at alum.mit.edu>
> >         wrote:
> >         > Anyone feel like implementing chunked encoding on the
> >         backend?
> >         
> >         
> >         I implemented chunked encoding to backends about six months
> >         ago, and
> >         we've been using it in production solidly since.
> >         
> >         I've been meaning to clean up the patches and submit them back
> >         upstream (along with many, many others) but have had no time -
> >         I'm not
> >         entirely happy with them yet (there's a fair bit of code
> >         duplication
> >         with the chunkeduploadstate stuff).
> >         
> >         However I should have time to tidy up those patches this week
> >         if all
> >         goes well, so keep your eyes open.
> > 
> > Yeah, I'd also be interested!
> > 
> > - Brad
> > 
> >