Compression: client or server?

[Brad Fitzpatrick]

Ryan, let me know if you want this committed.  Or rather, send me an
updated patch with a ChangeLog/MC_VERSION increase (MC_VERSION only if you
want me to make a new release)

It should also be possible to do a ->set_compression(1 | 0) on the
memclient object, to selectively disable compression when storing images
or other uncompressable blobs into memcache.

Also, the PHP client shouldn't store the compressed version if it's
larger, or not signifigantly smaller.



On Wed, 13 Aug 2003, Justin Matlock wrote:

> Ugh; sorry.. I don't know what the heck that was... Not sleeping for two
> days'll do that.  :)
>
> ----- Original Message -----
> From: "Brad Fitzpatrick" <brad@danga.com>
> To: <memcached@lists.danga.com>
> Sent: Wednesday, August 13, 2003 5:38 PM
> Subject: Re: Compression: client or server?
>
>
> > I only like unified diffs (-u).  Please post those instead.
> >
> >
> > On Wed, 13 Aug 2003, Justin Matlock wrote:
> >
> > > Doh. Corrected patch.  gzipuncompress was failing in very rare cases
> (null
> > > on the end of the compressed data stream).
> > >
> > > If you don't want to patch, just remove the ",$len" from the end of the
> > > gzuncompress line. :)
> > >
> > > ----- Original Message -----
> > > From: "Justin Matlock" <jmat@shutdown.net>
> > > To: <memcached@lists.danga.com>
> > > Sent: Wednesday, August 13, 2003 5:24 PM
> > > Subject: Re: Compression: client or server?
> > >
> > >
> > > > You guys read my mind.  :)
> > > >
> > > > I've been doing this for a few days now...  in tests, it takes 400k
> text
> > > > (the linux words file), and takes it down to 100k.  The extra time it
> took
> > > > to compress (a few milliseconds) was insignificant compared to the
> speed
> > > > increase from the size reduction.  As far as CPU load goes -- I didn't
> > > > notice a major increase in CPU, but it's somewhat hard to measure with
> > > PHP.
> > > > My client machine is a Linux box on a AMD Athlon 700, 256mb RAM, so
> your
> > > > results may vary.
> > > >
> > > > Attached patch to make the PHP API use gzip compression; it also
> updates
> > > the
> > > > Documentation file about the new $options['compress'] configuration
> > > > variable, which allows complete disabling of compression, or setting a
> > > > minimum threshold to activate compression (I recommend 10240 from
> > > testing).
> > > >
> > > > Justin
> > > >
> > > >
> > > > ----- Original Message -----
> > > > From: "Brad Fitzpatrick" <brad@danga.com>
> > > > To: <memcached@lists.danga.com>
> > > > Sent: Wednesday, August 13, 2003 3:53 PM
> > > > Subject: Re: Compression: client or server?
> > > >
> > > >
> > > > > I agree.
> > > > >
> > > > > The issue then is stats, though.  It'd be interesting to know how
> much
> > > > > physical vs. logical data is in memcache.  Based on that, you could
> > > adjust
> > > > > the compression threshold on the clients.  I imagine adding an
> option to
> > > > > the Perl MemCachedClient API that says, "Compress after this size".
> > > > >
> > > > > At lunch just now Evan and I realized another point to add it to the
> > > > > server, though:  what if somebody had more memcache servers than web
> > > > > nodes?  But we don't buy this argument, because in our experience
> you
> > > > > can't just _buy_ a memcache machine.  We were unable to find a
> system or
> > > > > parts vendor who could get us a machine with a crap processor and
> tons
> > > of
> > > > > RAM.
> > > > >
> > > > > So, yeah:  compression definitely client-side, but the server should
> do
> > > > > stats by the client sending more data about how much it shrank it
> to.
> > > See
> > > > > any problem with that?
> > > > >
> > > > >
> > > > > On Wed, 13 Aug 2003, Lisa Marie Seelye wrote:
> > > > >
> > > > > > On Wed, 2003-08-13 at 14:44, Brad Fitzpatrick wrote:
> > > > > > > Thoughts?
> > > > > >
> > > > > > What CPU resources are required to compress the data? How much of
> a
> > > lag
> > > > > > will it introduce to the System?
> > > > > >
> > > > > > The best way to go about storing compressed data is for the client
> to
> > > > > > _send the data in a compressed format_ and then know well enough
> to
> > > > > > decompress it when they get it back.
> > > > > >
> > > > > > The server should store whatever the client sends them and don't
> mess
> > > > > > with it -- put the burden on the user, not the server.
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > -Lisa
> > > > > > <Vix ulla tam iniqua pax, quin bello vel aequissimo sit potior>
> > > > > >
> > > > >
> > > >
> > >
> >
>