Replication Craziness

[Russell Garrett]

We've been having some rather strange happenings with MogileFS (2.16) 
replication recently. We're seeing loads of files being replicated to 
silly devcounts (up to 27) - most of them on the same host.

Our cluster is mostly full, so replicated files are lingering in 
file_to_replicate due to them being sub-optimally replicated (all on one 
host), but when the replication job gets to them, it seems to replicate 
them again anyway (to yet another device on the same least-used host), 
when it should just pass until more disks become available.

I can't see any obvious problems, though, and it's really fiddly to 
debug on this live installation...

Cheers,

Russ Garrett
Last.fm Ltd.
russ at last.fm