Bizarre perlbal problem

[Dormando]

Ok, all I had was speculation. This sounds more grounding.

Now, you might know that I'm not a PHP haxx0r. You might say I barely 
know the first thing about the language; The fact that PHP seems to mix 
up some stuff per connection instead of per request was news to me. Our 
backend is also running a dozen different versions of PHP at the moment 
(which I'm working on fixing!), so it's possible there's a glitch or two 
involved here.

I can reason the glitch happening sometimes and being related to PHP 
persisting some data either because of a glitch in an old version of PHP 
or because of a code bug in a backend I've never looked at myself. It 
doesn't feel like a widespread glitch though, given how few of our users 
are hit by it. If the first request to a keep-alive connection had all 
the login data for the rest of the requests, every one of our users 
would be logged out within a couple minutes.

I'll go back to working with one of the programmers in hunting it down, 
but for the record; are there any other reasons why this might happen? 
Nothing related to connections blowing up or dying or getting confused? 
What happens when a bad content-length is fed back to perlbal on accident?

(yes, we're checking X-Forwarded-For and such, I made sure of that at 
least).

Thanks,
-Alan

Brad Fitzpatrick wrote:
> I hear your concern, but I'm not worried about Perlbal mixing up requests
> and responses.
> 
> However, what I'm asking about is different:  not that each request can go
> to different servers, but that PHP isn't caching any session info data on
> the CONNECTION object, not the REQUEST object.
> 
> So you got a TCP connection with a bunch of HTTP requests inside it:
> 
>    <tcp>
>       <http_req />
>       <http_req />
>       <http_req />
>       <http_req />
>       <http_req />
>    </tcp>
> 
> If the PHP is making the assumption that each http request is from the
> same user (which is usually the case!) and caching on the TCP connection
> object, then Perlbal could be confusing it, because Perlbal's persistent
> connections to PHP aren't aligned with users at all.
> 
> 
> 
> On Fri, 18 Nov 2005, Dormando wrote:
> 
> 
>>I don't think so...
>>
>>Our backend is just direct connections to PHP running behind either
>>apache1 or apache2. As long as the remote user's independent requests go
>>down and come back from the same server it should work. IE; If user A
>>sends a request down, it just needs to get the reply to that request
>>back. If user A sends a request down, but gets the response headers from
>>user B, we'd get what we're seeing now.
>>
>>There's no reliance on the same user hitting the same exact backend
>>connection for multiple requests. Before we put the LB in there was no
>>keepalive at all, and users would hit one of 80 servers at random.
>>
>>-Alan
>>
>>Brad Fitzpatrick wrote:
>>
>>>Does your backend's session management assuming that for a given
>>>connection, the same user will always be on that connection?  What backend
>>>do you use, btw?
>>>
>>>If so, that's your problem, since Perlbal mixes up requests and
>>>connections.  (on purpose... it's a huge optimization)
>>>
>>>- Brad
>>>
>>>
>>>On Fri, 18 Nov 2005, Dormando wrote:
>>>
>>>
>>>
>>>>(sorry, couldn't resist :P)
>>>>
>>>>We got HTTP 1.0 keepalives working (for the most part) late in the day
>>>>yesterday. Overnight we witnessed a pretty bad glitch where our users
>>>>would randomly get other user's site cookies and become logged in as
>>>>someone else.
>>>>
>>>>It happened in a small percentage of users, but turning off the backend
>>>>keepalives seems to have removed the issue. We're still investigating on
>>>>our end, but I'm having a hard time even speculating how that happened;
>>>>was perlbal sending back responses from clients other than the
>>>>requestor? Any insight?
>>>>
>>>>Other things that came to mind:
>>>> - It's possible sometimes we return an improper content-length.
>>>> - If a client connection closes before reading any data back from its
>>>>connection, does perlbal always junk the backend request before reusing it?
>>>>
>>>>I do have a weird routing setup in order to have two perlbal processes
>>>>running on the same IP address, but since that's just on the frontend
>>>>and only for incoming requests, I can't see how the responses would get
>>>>switched... If they were, it'd happen a lot more often than with what we
>>>>saw.
>>>>
>>>>Thanks,
>>>>-Alan
>>>>
>>>>
>>
>>