Bizarre perlbal problem

[Brad Fitzpatrick]

Perlbal closes connections whenever it gets the slightly hint of somebody
being confused.   This was less true in the past, but the past 6 months or
so of development has been adding paranoia/robustness at the cost of
slight performance in some areas.


On Fri, 18 Nov 2005, Dormando wrote:

> Ok, all I had was speculation. This sounds more grounding.
>
> Now, you might know that I'm not a PHP haxx0r. You might say I barely
> know the first thing about the language; The fact that PHP seems to mix
> up some stuff per connection instead of per request was news to me. Our
> backend is also running a dozen different versions of PHP at the moment
> (which I'm working on fixing!), so it's possible there's a glitch or two
> involved here.
>
> I can reason the glitch happening sometimes and being related to PHP
> persisting some data either because of a glitch in an old version of PHP
> or because of a code bug in a backend I've never looked at myself. It
> doesn't feel like a widespread glitch though, given how few of our users
> are hit by it. If the first request to a keep-alive connection had all
> the login data for the rest of the requests, every one of our users
> would be logged out within a couple minutes.
>
> I'll go back to working with one of the programmers in hunting it down,
> but for the record; are there any other reasons why this might happen?
> Nothing related to connections blowing up or dying or getting confused?
> What happens when a bad content-length is fed back to perlbal on accident?
>
> (yes, we're checking X-Forwarded-For and such, I made sure of that at
> least).
>
> Thanks,
> -Alan
>
> Brad Fitzpatrick wrote:
> > I hear your concern, but I'm not worried about Perlbal mixing up requests
> > and responses.
> >
> > However, what I'm asking about is different:  not that each request can go
> > to different servers, but that PHP isn't caching any session info data on
> > the CONNECTION object, not the REQUEST object.
> >
> > So you got a TCP connection with a bunch of HTTP requests inside it:
> >
> >    <tcp>
> >       <http_req />
> >       <http_req />
> >       <http_req />
> >       <http_req />
> >       <http_req />
> >    </tcp>
> >
> > If the PHP is making the assumption that each http request is from the
> > same user (which is usually the case!) and caching on the TCP connection
> > object, then Perlbal could be confusing it, because Perlbal's persistent
> > connections to PHP aren't aligned with users at all.
> >
> >
> >
> > On Fri, 18 Nov 2005, Dormando wrote:
> >
> >
> >>I don't think so...
> >>
> >>Our backend is just direct connections to PHP running behind either
> >>apache1 or apache2. As long as the remote user's independent requests go
> >>down and come back from the same server it should work. IE; If user A
> >>sends a request down, it just needs to get the reply to that request
> >>back. If user A sends a request down, but gets the response headers from
> >>user B, we'd get what we're seeing now.
> >>
> >>There's no reliance on the same user hitting the same exact backend
> >>connection for multiple requests. Before we put the LB in there was no
> >>keepalive at all, and users would hit one of 80 servers at random.
> >>
> >>-Alan
> >>
> >>Brad Fitzpatrick wrote:
> >>
> >>>Does your backend's session management assuming that for a given
> >>>connection, the same user will always be on that connection?  What backend
> >>>do you use, btw?
> >>>
> >>>If so, that's your problem, since Perlbal mixes up requests and
> >>>connections.  (on purpose... it's a huge optimization)
> >>>
> >>>- Brad
> >>>
> >>>
> >>>On Fri, 18 Nov 2005, Dormando wrote:
> >>>
> >>>
> >>>
> >>>>(sorry, couldn't resist :P)
> >>>>
> >>>>We got HTTP 1.0 keepalives working (for the most part) late in the day
> >>>>yesterday. Overnight we witnessed a pretty bad glitch where our users
> >>>>would randomly get other user's site cookies and become logged in as
> >>>>someone else.
> >>>>
> >>>>It happened in a small percentage of users, but turning off the backend
> >>>>keepalives seems to have removed the issue. We're still investigating on
> >>>>our end, but I'm having a hard time even speculating how that happened;
> >>>>was perlbal sending back responses from clients other than the
> >>>>requestor? Any insight?
> >>>>
> >>>>Other things that came to mind:
> >>>> - It's possible sometimes we return an improper content-length.
> >>>> - If a client connection closes before reading any data back from its
> >>>>connection, does perlbal always junk the backend request before reusing it?
> >>>>
> >>>>I do have a weird routing setup in order to have two perlbal processes
> >>>>running on the same IP address, but since that's just on the frontend
> >>>>and only for incoming requests, I can't see how the responses would get
> >>>>switched... If they were, it'd happen a lot more often than with what we
> >>>>saw.
> >>>>
> >>>>Thanks,
> >>>>-Alan
> >>>>
> >>>>
> >>
> >>
>
>