updated low-memory patch

[Kostas Chatzikokolakis]

Hello,

after the discussion about the problem with concurrent changes, I
realized something I didn't know: for files with more than one chunk,
brackup stores the raw digest of each chunk in the .brackup file (for
single-chunk files there's no need as it coincides with the file's
digest). This is currently not used at all, it is stored so that in
the future it is possible to reconstruct the inventory db from the
.brackup file (in my TODO list).

This is great because it allows a solution to the concurrency problem
without changing any algorithm or the .brackup file. On restore,
brackup can verify the raw digest of the chunks. If the full digest
doesn't match but all chunks' raw digests are verified, then it's not
a fatal situation. The full digest is probably wrong due to concurrent
updates, but we know that the data that were read are the same that
were restored, so we can only print a warning and continue. Failure
happens only for single-chunk files, if the full digest doesn't match.

For this to work, it is crucial that the raw digest of each chunk, and
the full digest for single-chunk files corresponds to the data that
were actually read. In the svn version, this does happen for the chunk
digest (since chunks are read in memory) but does not happen for the
full digest which is read from disk.

In my low-memory patch, the chunk digest might also be wrong since
there's no copy in memory. I updated my patch to correct this, now
when the data is sent to gpg a new digest is computed, and the updated
digest is used in the inventory db and the .brackup file. Then the
restore code checks the chunks' digest, as explained above. I added a
05-concurrent-changes.t test and also modified the do_backup test so
that it verifies the inventory db for digest incosistencies. Both pass
fine now.

I'll continue testing the patch and let you know how it goes. Btw,
since no algorithm or .brackup change was needed, the patched version
is fully compatible with the svn one.

Kostas


patch url: http://codereview.appspot.com/135046