Fwd: request - help with streaming flash video through perlbal

Fri Dec 7 10:01:56 UTC 2007

Hello all,

Sorry, I should have cc:ed this to the mailing lists.  It's my reply
to eric's message re: flash video streaming through perlbal.

Etienne

---------- Forwarded message ----------
From: EJH Pollard <ejhp at ejhp.net>
Date: Dec 4, 2007 11:05 PM
Subject: Re: request - help with streaming flash video through perlbal
To: Eric Lambrecht <eml at guba.com>
Cc: Matthew Somerville <matthew at mysociety.org>

Hi Eric,

Thanks for your email a few days ago - sorry I haven't been in touch
earlier.  This evening I sat down to try and make it work, but
unfortunately I got an error that I don't really know how to fix -
details below, after your original suggestion to give it some context.

On Nov 25, 2007 6:29 PM, Eric Lambrecht <eml at guba.com> wrote:
> > Is it possible to configure perlbal, patched as per
> > http://lists.danga.com/pipermail/mogilefs/2007-April/000862.html, to
> > behave in the manner described above?  If not, do you have any
> > suggestions for what I should do to the patch/config to make it work
> > like that?  We have a modest budget, and if it's a lot of work then I
>
> Hi Etienne -
>   I believe the patch I submitted will do what you want, but you're
> missing one component.
>
> To make things work with my patch, you need to first set up perlbal to
> reproxy all requests to some webserver you've got running on the same
> machine.
>
> Then let's assume your requests for flash files all look like this:
>
>         GET /play?id=123123&start=5675675
>
> Where 'id' is the id of the flash file, and 'start' is the byte offset
> for the FLV file.
>
> The '/play' cgi script on your webserver should map the value of 'id' to
> the location of your flash file in S3 (lets say that is
> 's3.amazon.com/123123'), and send these headers back to perlbal:
>
>         X-REPROXY-URL: http://s3.amazon.com/123123
>         X-REPROXY-FLV-START: 5675675
>
> perlbal will then close the connection to the webserver, open a
> connection to S3, jump to byte offset 5675675, send the 13 byte magic
> header to the client, then send the remainder of the flv file to the client.

I've applied the patch, and written the script to produce the headers
you described, but unfortunately perlbal is giving me 503 Service
Unavailable when I present it with those HTTP headers.  I'm sending
the following headers to perlbal (have verified this output by
debugging these lines with a preceding Content-type: text/plain):

X-REPROXY-URL: http://s3.amazonaws.com/parliament-flash-video/2249.flv
X-REPROXY-FLV-START: 3212

The second line is only there if I give the script a start param, of
course; the base case is just the X-REPROXY_URL line, which gives a
perfectly sane URL that works fine when I fetch it directly
(http://s3.amazonaws.com/parliament-flash-video/2260.flv).  Enabling
debug output on perlbal gives a whole bunch of stuff, copied at the
bottom of this email, but the strangest thing is the 404 error it
seems to be getting from Amazon.  Running lynx from the same machine
gets a 200 OK, and downloads fine onto the local disk.

So I'm a bit stuck!  Any ideas?

The perlbal config is:

# always good to keep an internal management port open:
CREATE SERVICE mgmt
  SET role   = management
  SET listen = 127.0.0.1:60000
ENABLE mgmt

CREATE POOL backend
  POOL backend ADD 127.0.0.1:81

CREATE SERVICE reproxy
  SET listen = 0.0.0.0:80
  SET role = reverse_proxy
  SET enable_reproxy = true
  SET pool = backend
ENABLE reproxy

I'm on perlbal 1.59, patched as per your patch from
http://lists.danga.com/pipermail/mogilefs/2007-April/000867.html .

Thanks,

Etienne

Debug output:

Got new conn: IO::Socket::INET=GLOB(0x86547f0) (127.0.0.1:49852) for
reverse_proxy
ClientProxy::event_read
  no headers.  reading.
Perlbal::Socket::read_headers(Perlbal::ClientProxy=ARRAY(0x80693e8)) is_res=0
  pre-parsed headers: [GET /redirect/2249.flv HTTP/1.0
Host: 127.0.0.1
Accept: text/html, text/plain, audio/mod, image/*, application/msword,
application/pdf, application/postscript, text/sgml, */*;q=0.01
Accept-Encoding: gzip, compress
Accept-Language: en
User-Agent: Lynx/2.8.5rel.1 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.8b]
  got valid headers
Backend Perlbal::BackendHTTP=ARRAY(0x806997c) is writeable!
Backend requesting client, got normal = 3.
write(Perlbal::BackendHTTP=ARRAY(0x806997c), <396>"GET
/redirect/224...") from (Perlbal::BackendHTTP,
/usr/lib/perl5/vendor_perl/5.8.8/Perlbal/BackendHTTP.pm, 292)
write(Perlbal::BackendHTTP=ARRAY(0x806997c), <15>"CODE(0x869457c)")
from (Perlbal::BackendHTTP,
/usr/lib/perl5/vendor_perl/5.8.8/Perlbal/BackendHTTP.pm, 302)
Backend Perlbal::BackendHTTP=ARRAY(0x806997c) is readable!
Perlbal::Socket::read_headers(Perlbal::BackendHTTP=ARRAY(0x806997c)) is_res=1
  pre-parsed headers: [HTTP/1.0 200 OK
Connection: close
X-REPROXY-URL: http://s3.amazonaws.com/parliament-flash-video/2249.flv
Date: Tue, 04 Dec 2007 22:55:17 GMT
Server: lighttpd/1.4.16]
  pushing back 1 bytes after header
  got valid headers
BackendHTTP: handle_response
Backend Perlbal::BackendHTTP=ARRAY(0x869496c) is writeable!
write(Perlbal::BackendHTTP=ARRAY(0x869496c), <90>"GET
/parliament-f...") from (Perlbal::ClientProxy,
/usr/lib/perl5/vendor_perl/5.8.8/Perlbal/ClientProxy.pm, 244)
Backend Perlbal::BackendHTTP=ARRAY(0x869496c) is writeable!
Use of uninitialized value in concatenation (.) or string at
/usr/lib/perl5/vendor_perl/5.8.8/Perlbal/Socket.pm line 74.
write(Perlbal::BackendHTTP=ARRAY(0x869496c), <undef>"") from
(Perlbal::BackendHTTP,
/usr/lib/perl5/vendor_perl/5.8.8/Perlbal/BackendHTTP.pm, 368)
Backend Perlbal::BackendHTTP=ARRAY(0x869496c) is readable!
Perlbal::Socket::read_headers(Perlbal::BackendHTTP=ARRAY(0x869496c)) is_res=1
  pre-parsed headers: [HTTP/1.1 404 Not Found
x-amz-request-id: 4B2E0B50A76F8FC4
x-amz-id-2: IEje1k7yXJ5J1jv2X+Glj5izkk0XRXunOJGmNiZZZHyJ8YpSVQNvJexbqINPk65A
Content-Type: application/xml
Date: Tue, 04 Dec 2007 22:55:17 GMT
Connection: close
Server: AmazonS3]
  pushing back 287 bytes after header
  got valid headers
BackendHTTP: handle_response
ClientHTTPBase::setup_keepalive(Perlbal::ClientProxy=ARRAY(0x80693e8))
  service's persist_client = 0
  doing connection: close
write(Perlbal::ClientProxy=ARRAY(0x80693e8), <117>"HTTP/1.0 503
Serv...") from (Perlbal::ClientHTTPBase,
/usr/lib/perl5/vendor_perl/5.8.8/Perlbal/ClientHTTPBase.pm, 734)
write(Perlbal::ClientProxy=ARRAY(0x80693e8), <35>"<h1>503 -
Service...") from (Perlbal::ClientHTTPBase,
/usr/lib/perl5/vendor_perl/5.8.8/Perlbal/ClientHTTPBase.pm, 735)
write(Perlbal::ClientProxy=ARRAY(0x80693e8), <15>"CODE(0x86950c8)")
from (Perlbal::ClientHTTPBase,
/usr/lib/perl5/vendor_perl/5.8.8/Perlbal/ClientHTTPBase.pm, 741)

-- 
Etienne Pollard
email at ejhp.net
+44 (0) 7946 415 996