Suggested structure to backup multiple servers?
mart at degeneration.co.uk
Sun Apr 13 14:03:51 UTC 2008
Ed W wrote:
> Hi, Brackup looks like a good fit for my needs, but I need to backup
> quite a few identical servers (to S3) and so I'm looking to share the
> index files.
> How have other people solved this? Seems like the options are:
> - Backup from one machine accessing the filesystems remotely
> - Use some way to share the db files online (I guess sqlite deals with
> multiusers over a shared filesystem?)
> - Backup servers one by one and sync the dbs around to each one by one
> - Modify brackup to use mysql for the shared db files?
> Any comments from existing users?
I assume that by "index files" you're referring to the inventory
database that brackup keeps, which records what chunks are on the server.
This is an issue I've encountered to, as I'm pushing data from lots of
servers into one S3 account. Currently I'm just accepting that each
server will do a redundant upload of each chunk that would otherwise be
common across servers. The tricky part comes when you start cleaning up
chunks relating to old backups; I've just avoided doing this so far, but
if I were to run a garbage collection it would of course be necessary to
rebuild/delete the inventory databases on all sources because their
chunk lists will be incorrect.
I think it'd be useful to have a way for brackup to automatically
rebuild an inventory database from the target. Unfortunately even after
only running Brackup for a few weeks the chunks bucket already has so
much stuff in it that it takes a significant amount of time to list the
contents and is thus not something you can just do as a matter of course
before running a backup.
As you say, the ideal would be to have some mechanism for keeping the
inventory databases in sync. It would be useful to having the
brackup-target gc command store a list of chunks it's deleted in such a
way that other clients can fetch it and update their databases, though
I'm not sure how exactly you'd actually store that on the target; you
need to be able to say "tell me what's changed since <time of last
backup>", but S3's not really set up for that sort of thing.
More information about the brackup