Suggested structure to backup multiple servers?

[Martin Atkins]

Ed W wrote:
> Hi, Brackup looks like a good fit for my needs, but I need to backup 
> quite a few identical servers (to S3) and so I'm looking to share the 
> index files.
> 
> How have other people solved this?  Seems like the options are:
> 
> - Backup from one machine accessing the filesystems remotely
> - Use some way to share the db files online (I guess sqlite deals with 
> multiusers over a shared filesystem?)
> - Backup servers one by one and sync the dbs around to each one by one 
> (yuck)
> - Modify brackup to use mysql for the shared db files?
> 
> Any comments from existing users?
> 

I assume that by "index files" you're referring to the inventory 
database that brackup keeps, which records what chunks are on the server.

This is an issue I've encountered to, as I'm pushing data from lots of 
servers into one S3 account. Currently I'm just accepting that each 
server will do a redundant upload of each chunk that would otherwise be 
common across servers. The tricky part comes when you start cleaning up 
chunks relating to old backups; I've just avoided doing this so far, but 
if I were to run a garbage collection it would of course be necessary to 
rebuild/delete the inventory databases on all sources because their 
chunk lists will be incorrect.

I think it'd be useful to have a way for brackup to automatically 
rebuild an inventory database from the target. Unfortunately even after 
only running Brackup for a few weeks the chunks bucket already has so 
much stuff in it that it takes a significant amount of time to list the 
contents and is thus not something you can just do as a matter of course 
before running a backup.

As you say, the ideal would be to have some mechanism for keeping the 
inventory databases in sync. It would be useful to having the 
brackup-target gc command store a list of chunks it's deleted in such a 
way that other clients can fetch it and update their databases, though 
I'm not sure how exactly you'd actually store that on the target; you 
need to be able to say "tell me what's changed since <time of last 
backup>", but S3's not really set up for that sort of thing.