Reentrancy problem forking children with the PHP client

[Kenner Stross]

Hi Maxwell,
 
Thanks for letting me hear from you -- I knew I wasn't crazy!!  Actually
I already knew that -- the behavior is so black-and-white in my case: If
I allow more than one child to fork, there are problems; If I restrict
it to just one child, no problems.  Clearly, something isn't working
correctly in the memcache php client, but I don't yet have the time to
put a test script together for Mikael.  One of these days...
 
Meanwhile, I'm living with the restriction that my daemon only forks one
child at a time.  Fortunately, I was able to make other performance
improvements so we can live with this restriction for a while, but
eventually I've got to come back to this problem.  Sorry I can't be of
more help to you.
 
I like your idea of building in an automatic check/retry.  It's not
ideal, of course, but it might help me when I go to the next stage of
truly needing multiple forked children, assuming the problem hasn't been
fixed by then.  Thanks!
 
Best of luck.  I'll let you know if I come up with anything.

-----Original Message-----
From: memcached-bounces at lists.danga.com
[mailto:memcached-bounces at lists.danga.com] On Behalf Of Maxwell Lamb
Sent: Friday, October 12, 2007 11:33 AM
To: memcached at lists.danga.com
Subject: Reentrancy problem forking children with the PHP client


Sorry, didn't realize it wouldn't find the context - see below.

Hi,

We've been experiencing the exact same issue, although our only
difference is that we're only opening the memcached connection once,
prior to forking using pctnl_fork.

I was wondering if you'd had any joy finding a resolution to this -
we're currently working on the basis of storing everything in memcached
within a one-keyed array, where the key is the same as the memcached
key, and the value the same as the value to be stored, and we're then
checking the key of the returned array against the requested key from
memcached, and if it's wrong, trying again. While this resolves the
issue insofar as it prevents us from having the wrong data in the wrong
place, it's obviously far from ideal due to the overhead of having to do
multiple requests in the case of failure, and of having to check a key
value after every get.

Anyway - even if you don't have a solution, it isn't just you having
this issue.

Thanks,

Maxwell

************************************************************************


Hi,

Since you actually fork the process there should be no possibility of
sharing memory between children. And since the connection is established
after the fork no sockets are shared, in which case you would probably
get data corruption errors instead of a whole object being swapped.
Also, the client doesn't use any globals and works fine under ZTS (the
thread safe PHP build).

If you have a short script that reproduces the problem, I'd be glad to
give it a try and see if I can hunt down the problem.

//Mikael


Kenner Stross wrote:
> There appears to be a re-entrancy issue with the PHP client.  Here's a
> brief recap of my situation:
> 
> I have a daemon process that is written in php (and is thus called
like
> this: "php -e myDaemon -- -paramOne -paramTwo &").  Once it is
launched,
> it follows standard daemon coding practices to set itself up as a
> detached session leader and wait for something to do.  Once there is
> something to do, it will fork some children to do the work.  Looks
like
> this:
> 
> myDaemon (the parent, which forks three times in this example)
> myDaemon (child 1)
> myDaemon (child 2)
> myDaemon (child 3)
> 
> The memcache client is not instantiated until the children have
forked,
> so that each child can have a memcache client that is separate from
and
> unrelated to the other children.  However, it appears that they are
all
> still intertwined at some deeper level, apparently as a result of
having
> forked out of the same php process space (I guess).
> 
> The symptoms are this: At various random points, a memcache "get" for
> object1 (whose type is class1) will return object2 (whose type is
> class2).  In other words, it is returning the wrong objects.  To me,
> this sure looks like a reentrancy problem, where one child is being
> swapped in by the processor at JUST THE RIGHT MOMENT to pick up the
> OTHER OBJECT from the OTHER CHILD that was just swapped out.
> 
> The confirmation that this is the problem is that when I set a limit
> such that the parent can never fork more than one child, the problem
> goes away.  Handled that way, memcache always returns the correct
> object.
> 
> Has anybody had similar experiences?  I'd love to hear what others may
> know about this issue and/or the architecture of the php client
> vis-a-vis forking.
> 
> Thanks,
> Kenner Stross
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20071012/c4e16a8d/attachment-0001.htm