Pluggable dictionaries

[Kostas Chatzikokolakis]

That's really neat. I recently tried to backup 2M files and I ended up
rebooting the machine after brackup filled the whole swap file. I was
using the lowmem patch, so probably the problem was the memory for the
digestcache and inventorydb, plus the memory for the metafile (I saw
that you also fixed that).

The target is using constant memory in all cases, we're quite close.

Cheers
Kostas


Gavin Carr wrote:
> In merging Kostas' lowmem patches I realised that another place we're
> using a lot of ram in brackup is the DigestCache and InventoryDatabase
> handling. The storage backend for both (Dict::SQLite) is optimised for 
> speed at the expense of memory - it just slurps the entire database 
> into ram and then uses it from there.
> 
> So I've committed a change to make the dictionary type selectable for
> both the digest db (SOURCE 'digestdb_type') and the inventory database
> (TARGET 'inventorydb_type'), and added a Dict::SQLite2 module that
> uses conventional db queries/iterators instead of loading the db into 
> ram. Which is slower, but uses much less memory, so you can choose to 
> trade speed for memory if you want to.
> 
> This also allows other kinds of dictionary implementations to be 
> written. I'm benchmarking DBM and Redis implementations at the moment, 
> and will commit them as well if the performance looks interesting.
> 
> Cheers,
> Gavin
>