hacktivity

Brad Fitzpatrick brad at danga.com
Thu Dec 28 17:27:18 UTC 2006


I've been going on a Mogile cleaning rampage the last day, and continuing
into today....

30 commits or so, all mostly related to:
   -- expanding the test suite
   -- removing all code from the "mogilefsd" file (package "Mgd::")
   -- making the code use the object-oriented APIs, rather than
      then old Mgd:: APIs (and expanding the OO APIs where necessary)

But also a few minor bug fixes, as well as the somewhat big bug that
Brett reported, where removing a host would kill parts of the server.
That's now fixed (and has tests for it).

The commits so far are below.

I'd love it if anybody else could contribute more tests, creating
different failure scenarios and making sure that MogileFS recovers
properly.  If you don't know how to write a test, just look at
t/00-startup.t (which is now much more than startup, should be
renamed), and copy/paste liberally.

Anybody have anything they want merged while I'm hacking on all this?
Unfortunately there's been so much churn, the patches might not apply.
But I'd handle rebasing the patch.

- Brad


------------------------------------------------------------------------
r550 | bradfitz | 2006-12-28 18:11:28 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Device.pm
   M /trunk/server/lib/MogileFS/Worker/Query.pm
   M /trunk/server/mogilefsd

move vivify_directories into MogileFS::Device


------------------------------------------------------------------------
r549 | bradfitz | 2006-12-28 17:59:30 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Device.pm
   M /trunk/server/lib/MogileFS/Host.pm
   M /trunk/server/lib/MogileFS/Worker/Delete.pm
   M /trunk/server/lib/MogileFS/Worker/Query.pm
   M /trunk/server/lib/MogileFS/Worker.pm
   M /trunk/server/mogilefsd

move all the observed_foo APIs into MogileFS::Host and MogileFS::Device, out of mogilefsd


------------------------------------------------------------------------
r548 | bradfitz | 2006-12-28 05:39:25 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Connection/Client.pm
   M /trunk/server/lib/MogileFS/Domain.pm
   M /trunk/server/lib/MogileFS/Worker/Query.pm
   M /trunk/server/mogilefsd

move all domain-related functions into MogileFS::Domain and out of mogilefsd


------------------------------------------------------------------------
r547 | bradfitz | 2006-12-28 05:34:38 +0000 (Thu, 28 Dec 2006) | 4 lines
Changed paths:
   M /trunk/server/MANIFEST
   A /trunk/server/lib/MogileFS/Domain.pm
   M /trunk/server/lib/MogileFS/Worker/Query.pm
   M /trunk/server/mogilefsd

move Mgd::domain_id() into new file, MogileFS::Domain (which isn't yet
an object, but just a namespace for future functions, object...)


------------------------------------------------------------------------
r546 | bradfitz | 2006-12-28 05:10:40 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/FID.pm
   M /trunk/server/lib/MogileFS/Worker/Query.pm
   M /trunk/server/mogilefsd

get rid of Mgd::key_filerow.  move it into a new ctor for MogileFS::FID objects


------------------------------------------------------------------------
r545 | bradfitz | 2006-12-28 04:57:06 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Util.pm
   M /trunk/server/lib/MogileFS/Worker/Delete.pm
   M /trunk/server/mogilefsd

move is_url out of mogilefsd and into MogileFS::Util, renamed url_parts


------------------------------------------------------------------------
r544 | bradfitz | 2006-12-28 04:52:40 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Class.pm
   M /trunk/server/lib/MogileFS/Worker/Query.pm
   M /trunk/server/mogilefsd

move Mgd::hostid_classes() and its one caller to MogileFS::Class->dmid_classes (wtf was with the original name?)


------------------------------------------------------------------------
r543 | bradfitz | 2006-12-28 04:49:02 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Worker/Query.pm

faster file check logic


------------------------------------------------------------------------
r542 | bradfitz | 2006-12-28 04:38:56 +0000 (Thu, 28 Dec 2006) | 4 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Class.pm
   M /trunk/server/lib/MogileFS/Worker/Query.pm
   M /trunk/server/mogilefsd

move class_id into MogileFS::Class



------------------------------------------------------------------------
r541 | bradfitz | 2006-12-28 04:27:13 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/HTTPFile.pm
   M /trunk/server/lib/MogileFS/Util.pm
   M /trunk/server/lib/MogileFS/Worker.pm
   M /trunk/server/mogilefsd

move wait_for_readability and wait_for_writeability into MogileFS::Util


------------------------------------------------------------------------
r540 | bradfitz | 2006-12-28 04:18:02 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Class.pm
   M /trunk/server/lib/MogileFS/Connection/Client.pm
   M /trunk/server/mogilefsd

move Mgd::get_mindevcounts() into MogileFS::Class


------------------------------------------------------------------------
r539 | bradfitz | 2006-12-28 04:14:05 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/FID.pm
   M /trunk/server/lib/MogileFS/Worker/Query.pm
   M /trunk/server/lib/MogileFS/Worker/Replicate.pm
   M /trunk/server/mogilefsd

more kill old APIs.  this time Mgd::update_fid_devcount();


------------------------------------------------------------------------
r538 | bradfitz | 2006-12-28 03:53:49 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Host.pm
   M /trunk/server/lib/MogileFS/Worker/Query.pm
   M /trunk/server/mogilefsd

more legacy API removal.  this time killing Mgd::host_id()


------------------------------------------------------------------------
r537 | bradfitz | 2006-12-28 03:42:25 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Host.pm
   M /trunk/server/lib/MogileFS/Worker/Replicate.pm
   M /trunk/server/mogilefsd

converting more code to OO API, killing old functions in Mgd:: in mogilefsd


------------------------------------------------------------------------
r536 | bradfitz | 2006-12-28 03:34:13 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/t/00-startup.t

remove sleep at end, and fix test count


------------------------------------------------------------------------
r535 | bradfitz | 2006-12-28 03:28:38 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Worker/Replicate.pm

variable renaming only, in prep for future patch


------------------------------------------------------------------------
r534 | bradfitz | 2006-12-28 03:26:42 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Device.pm

remove some non-OO Mgd:: code


------------------------------------------------------------------------
r533 | bradfitz | 2006-12-28 03:22:48 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/mogilefsd

this wasn't used


------------------------------------------------------------------------
r532 | bradfitz | 2006-12-28 03:21:23 +0000 (Thu, 28 Dec 2006) | 6 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Host.pm
   M /trunk/server/lib/MogileFS/Worker/Query.pm
   M /trunk/server/mogilefsd

more work on and usage of Mogile::Host.

plan is to slowly drain the mogilefsd file of all code, and especially
all non-OO code.


------------------------------------------------------------------------
r531 | bradfitz | 2006-12-28 02:41:49 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Worker/Monitor.pm

use keep alives from monitor


------------------------------------------------------------------------
r530 | bradfitz | 2006-12-28 02:13:30 +0000 (Thu, 28 Dec 2006) | 8 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Device.pm
   M /trunk/server/lib/MogileFS/Host.pm
   M /trunk/server/lib/MogileFS/Worker/Monitor.pm
   M /trunk/server/mogilefsd

make the monitor code even more object-oriented, this time focusing on
the MogileFS::Host object.

also fixes the bug as reported by Brett G. Durrett <brett at imvu.com>
that deleting a host could crash the server.



------------------------------------------------------------------------
r529 | bradfitz | 2006-12-28 01:32:07 +0000 (Thu, 28 Dec 2006) | 4 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Worker/Monitor.pm

update the monitor worker to be cleaner, smaller, and use the object-oriented apis.



------------------------------------------------------------------------
r528 | bradfitz | 2006-12-28 01:15:25 +0000 (Thu, 28 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Device.pm

class method to return all devices, as well as new accessor to see if device is marked down


------------------------------------------------------------------------
r527 | bradfitz | 2006-12-28 01:14:57 +0000 (Thu, 28 Dec 2006) | 7 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Connection/Worker.pm
   M /trunk/server/lib/MogileFS/ProcManager.pm
   M /trunk/server/lib/MogileFS/Util.pm
   M /trunk/server/mogilefsd

Mgd::error() from MogileFS::Util, used to be used for errors, fatal
things (uh, bad errors), as well as debug info.  with the introduction
of fatal(), it made sense to finally add debug() as well.

not many callers yet.  actually zero.


------------------------------------------------------------------------
r526 | bradfitz | 2006-12-27 23:45:14 +0000 (Wed, 27 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/t/00-startup.t

much expanded tests


------------------------------------------------------------------------
r525 | bradfitz | 2006-12-27 23:34:18 +0000 (Wed, 27 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Worker/Replicate.pm

don't send status updates, only in debug mode.


------------------------------------------------------------------------
r524 | bradfitz | 2006-12-27 22:26:45 +0000 (Wed, 27 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/Makefile.PL
   M /trunk/server/lib/MogileFS/Worker/Monitor.pm

require Danga::Socket 1.56 which resets epoll stuff for monitor worker.


------------------------------------------------------------------------
r523 | bradfitz | 2006-12-27 21:47:27 +0000 (Wed, 27 Dec 2006) | 5 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Worker/Monitor.pm

the real bug fix: can't "next" from a sub.  well, you can... perl LETS
YOU... but the flow control happens many callers above us, much to our
confusion.


------------------------------------------------------------------------
r522 | bradfitz | 2006-12-27 21:46:58 +0000 (Wed, 27 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/ProcManager.pm

die hard if a child process is getting a message from a child process.  this isn't a "warning" to be ignored.


------------------------------------------------------------------------
r521 | bradfitz | 2006-12-27 21:46:31 +0000 (Wed, 27 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Util.pm
   M /trunk/server/mogilefsd

add a more-several error() wrapper:  fatal.


------------------------------------------------------------------------
r520 | bradfitz | 2006-12-27 21:45:06 +0000 (Wed, 27 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/CHANGES.mogstored
   M /trunk/server/mogstored

* kill child processes (diskusage and iostat) when being killed.


------------------------------------------------------------------------
r519 | bradfitz | 2006-12-27 21:44:11 +0000 (Wed, 27 Dec 2006) | 3 lines
Changed paths:
   M /trunk/server/lib/MogileFS/Worker/Reaper.pm

unused




More information about the mogilefs mailing list