Kostas Chatzikokolakis' lowmem patches merged

[Kostas Chatzikokolakis]

Hi Gavin,

Thanks for merging the patch. I had started splitting it in parts, but
since you merged it already, it's even better :D I'll have a look to see
if everything looks ok and I'll test as much as possible. Some
clarifications:

About memory usage:

* the biggest memory drop should be when using encryption. In this case
brackup forks and reads many chunks in parallel.
(I'm not sure whether this will show up in the stats though. Gavin, your
stats are only about the parent process or include the memory of child
processes?)

* Without encryption brackup loads at most one chunk in memory (except
for composite chunks, but these are small) so there will still be a drop
in memory usage, but not that much.


About speed

* with encryption there should be also a small speedup, cause the
original version writes both the raw and the encrypted chunk to a
tempfile, the patched version writes only the encrypted chunk.

* without encryption there should be a performance hit because the
original version keeps everything in memory, the patched version writes
chunks to a tempfile (for concurrency issues).


Cheers,
Kostas




Gavin Carr wrote:
> I've merged Kostas Chatzikokolakis' lowmem patches from here:
> 
>   http://codereview.appspot.com/135046/show
> 
> into trunk, as they give a significant drop in memory usage without 
> any significant impact on performance. The basic idea is to pass around 
> content via filehandles instead of scalars where that makes sense.
> 
> I've tried to split the lowmem patches there out from the other stuff
> in that changeset (concurrent changes protection and additional tests),
> in order to keep the scope of the changes down, but they're still 
> pretty significant. (Kostas, it would be great if you had time to review 
> my changes to make sure I haven't done something stupid when I've pulled
> your changeset apart). I've also changed metafile storage to work the
> same way as in Kostas' patches.
> 
> All tests are passing, and I've been running these changes in production
> for a few days, but it would be great if some other people had time to 
> test trunk against various targets (Amazon is only lightly tested so far,
> and CloudFiles not at all) and could report their results. FWIW, I'm 
> seeing memory usage drops of 25-30% with Filesystem, Ftp and Sftp targets.
> 
> These changes add one additional dependency, on IO::InnerFile, from the
> IO::stringy distribution.
> 
> Cheers,
> Gavin
>