potential pipe-lining corruption fix

dormando dormando at rydia.net
Thu Sep 21 16:43:16 UTC 2006


Looks like that fixed it! We've had zero reports of page swapping / 
image swapping since applying the patch. Site's running much snappier 
during peak load with backend keepalives working.

I'll probably follow up with some information on exactly what kind of 
client is hitting the disconnect call in the patch... Not going to have 
time until next week.

-Alan / Dormando / whatever.

dormando wrote:
> I can't screw with the site past 4:30pm, so first thing tomorrow morning 
> I'll patch (hopefully just one of?) the LB's and try it out.
> 
> Thanks!
> -Alan
> 
> Brad Fitzpatrick wrote:
>> Today at the MogileFS summit Alan (from http://www.gaiaonline.com/)
>> mentioned that if he enables persistent backend connections, users get
>> mismatched requests to responses.... but not when he accesses it.  And it
>> takes awhile to happen during low load.
>>
>> All this pointed to errors handling malformed client requests, and sure
>> enough, I think this is it....
>>
>> We don't disconnect users who do pipelining.  I was able to sneak through
>> multiple requests to the backend and then two requests would come back,
>> but they'd be assigned to different users.
>>
>> This fix doesn't break the test suite, but before I commit, Alan --- can
>> you verify it fixes things for you?
>>
>>
>>
>> Index: lib/Perlbal/ClientProxy.pm
>> ===================================================================
>> --- lib/Perlbal/ClientProxy.pm  (revision 565)
>> +++ lib/Perlbal/ClientProxy.pm  (working copy)
>> @@ -584,13 +584,11 @@
>>      # (see: Danga::Socket::read)
>>      return $self->client_disconnected unless defined $bref;
>>
>> -    # if we got data that we weren't expecting, something's bogus with
>> -    # our state machine (internal error)
>> -    if (defined $remain && ! $remain) {
>> -        my $blen = length($$bref);
>> -        my $content = substr($$bref, 0, 80 < $blen ? 80 : $blen);
>> -        Carp::cluck("INTERNAL ERROR: event_read called on when we're 
>> expecting no more bytes.  len=$blen, content=[$content]\n");
>> -        $self->close;
>> +    # if they didn't declare a content body length and we just got a
>> +    # readable event that's not a disconnect, something's messed up.
>> +    # they're overflowing us.  disconnect!
>> +    if (! $remain) {
>> +        $self->close("over_wrote");
>>          return;
>>      }
>>
>>
>> - Brad


More information about the perlbal mailing list