concurrent changes patch

[Kostas Chatzikokolakis]

Hello,

now that the lowmem part of the patch has landed it's much easier to
merge the concurrent changes part. This makes brackup work correctly
when files are changed while brackup runs. The previous version had
already this problem, but now it's even worse cause chunks are not
loaded in memory so there's a higher chance for a chunk to be changed.

For simplicity I split the patch in 3 independent parts, all against the
current trunk.

test_inv_db.patch

This modifies Brackup::Test to check that the chunks in the inventory db
have the digest that the digest db claims they have. In case of
concurrent changes, the actual data might not correspond to the digest.

The current version should already pass this, but only because there is
no test to run backup under concurrent changes.


restore_verify_chunks.patch

This modifies Brackup::Restore to verify the digest of each individual
chunk apart from the digest of the whole file. If the file digest is
wrong but all chunks are verified, then it must be the case that the
file digest is "invalid" due to concurrent changes. Still the restored
data are the same as the data that were backed up, so brackup should
continue with only a warning (we were discussing this in previous mails
in the list).


concurrent.patch

Changes to make sure that we compute the digest of the actual stored
data. This means that the digest of a Brackup::PositionedChunk or
Brackup::File might change after the encryption, and be replaced by the
digest of the data sent to gpg.

Without encryption there is the problem that we need to compute the
*final* digest before starting uploading to the target. To do this, we
copy the chunk to a tempfile before uploading. This of course should
cause a performance hit.

This patch depend on the delete_temp_files.patch of my previous mail.
Also, the tests will fail unless restore_verify_chunks.patch is also merged.

Future work: the best way to avoid the hit IMHO, is to add a
'low_memory' configuration option. If 0 we can copy the chunk in memory
instead of a temp file (and the same can be done for composite chunks).
The same option can be used by targets that do not support direct upload
from a filehandle, to use a tempfile instead of memory if low_memory=1.
I could do these changes when the other patches are merged.


Cheers,
Kostas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: concurrent.patch
Type: text/x-patch
Size: 13082 bytes
Desc: not available
Url : http://lists.danga.com/pipermail/brackup/attachments/20100113/dab9e4f9/concurrent.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: restore_verify_chunks.patch
Type: text/x-patch
Size: 2552 bytes
Desc: not available
Url : http://lists.danga.com/pipermail/brackup/attachments/20100113/dab9e4f9/restore_verify_chunks.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_inv_db.patch
Type: text/x-patch
Size: 6439 bytes
Desc: not available
Url : http://lists.danga.com/pipermail/brackup/attachments/20100113/dab9e4f9/test_inv_db.bin