Compression: client or server?

[Brad Fitzpatrick]

So far the two big users of memcache (LiveJournal and Slashdot) are using
it to store text, and lots of text.

Text compresses very well.  Large chunks of text compress better, but
little text still compresses quite well.

So, memcache should support compression.  But, where to do it?

Any client can already do it without server knowledge, using one of the 16
opaque flags.  The Perl client could, say, check for items larger than 1k
and try to compress it using Compress::Zlib, and if compression helps, set
a bit meaning "I'm compressed!", and it'll uncompress it later.

Pro:  no server support.  compression CPU is distributed.
Con:  the server can't report stats about total usage vs. compressed
      usage, which would be neat.

Or, the server could do it:

Pro:  stats.
Con:  CPU drain.

Or, what if we augmented the protocol to allow a client to report both the
real size and the compressed size?

Pro:  CPU still on clients.  server can do stats.
Con:  protocol change (reverse compatible, though), and an extra
      4 bytes per compressed item, to store the other size.

Thoughts?


- Brad