Troubleshooting multiprocess communication

[dormando]

Hey all,

I'm completely sucking at finding a bug. Allow me to talk at you in 
hopes that someone knows the answer offhand.

MogileFS::Worker::Delete has the following code:

         # hit up the server and delete it
         # TODO: (optimization) use MogileFS->get_observed_state and 
don't try to
  delete things known to be down/etc
         my $sock = IO::Socket::INET->new(PeerAddr => $urlparts->[0],
                                          PeerPort => $urlparts->[1],
                                          Timeout => 2);
         unless ($sock) {
             # timeout or something, mark this device as down for now 
and move on
             $self->broadcast_host_unreachable($dev->hostid);
             $reschedule_fid->(60 * 60 * 2, "no_sock_to_hostid");
             next;
         }

(which I've now terribly pasted).

If a host times out, the deleter broadcasts to all workers that the host 
is unreachable. I think this is a little excessive, but it should be 
okay because:

MogileFS::Worker::Monitor should re-broadcast a 'reachable' state within 
the next ten seconds, if the host is actually up and the timeout was a 
fluke.

Except the delete job is never getting that message, and the procmanager 
code prevents the job monitors subsequent broadcasts from being sent to 
the deleter, since the status hasn't changed.

The symptom of this is any deletes destined for those hosts get cycled 
through file_to_delete_later and back again every 600 seconds.

Not 100% sure I'm looking in the right place. Given the timeout (600 
seconds) and that chunk of code, this is probably right? I should be 
able to verify this by sending !to commands to the delete job to say 
those hosts are back up, but I haven't gotten that to work yet.

Ideas?
-Dormando