Compression: client or server?
Justin Matlock
jmat@shutdown.net
Wed, 13 Aug 2003 17:43:35 -0400
This is a multi-part message in MIME format.
------=_NextPart_000_006E_01C361C2.6B6D22D0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Ugh; sorry.. I don't know what the heck that was... Not sleeping for two
days'll do that. :)
----- Original Message -----
From: "Brad Fitzpatrick" <brad@danga.com>
To: <memcached@lists.danga.com>
Sent: Wednesday, August 13, 2003 5:38 PM
Subject: Re: Compression: client or server?
> I only like unified diffs (-u). Please post those instead.
>
>
> On Wed, 13 Aug 2003, Justin Matlock wrote:
>
> > Doh. Corrected patch. gzipuncompress was failing in very rare cases
(null
> > on the end of the compressed data stream).
> >
> > If you don't want to patch, just remove the ",$len" from the end of the
> > gzuncompress line. :)
> >
> > ----- Original Message -----
> > From: "Justin Matlock" <jmat@shutdown.net>
> > To: <memcached@lists.danga.com>
> > Sent: Wednesday, August 13, 2003 5:24 PM
> > Subject: Re: Compression: client or server?
> >
> >
> > > You guys read my mind. :)
> > >
> > > I've been doing this for a few days now... in tests, it takes 400k
text
> > > (the linux words file), and takes it down to 100k. The extra time it
took
> > > to compress (a few milliseconds) was insignificant compared to the
speed
> > > increase from the size reduction. As far as CPU load goes -- I didn't
> > > notice a major increase in CPU, but it's somewhat hard to measure with
> > PHP.
> > > My client machine is a Linux box on a AMD Athlon 700, 256mb RAM, so
your
> > > results may vary.
> > >
> > > Attached patch to make the PHP API use gzip compression; it also
updates
> > the
> > > Documentation file about the new $options['compress'] configuration
> > > variable, which allows complete disabling of compression, or setting a
> > > minimum threshold to activate compression (I recommend 10240 from
> > testing).
> > >
> > > Justin
> > >
> > >
> > > ----- Original Message -----
> > > From: "Brad Fitzpatrick" <brad@danga.com>
> > > To: <memcached@lists.danga.com>
> > > Sent: Wednesday, August 13, 2003 3:53 PM
> > > Subject: Re: Compression: client or server?
> > >
> > >
> > > > I agree.
> > > >
> > > > The issue then is stats, though. It'd be interesting to know how
much
> > > > physical vs. logical data is in memcache. Based on that, you could
> > adjust
> > > > the compression threshold on the clients. I imagine adding an
option to
> > > > the Perl MemCachedClient API that says, "Compress after this size".
> > > >
> > > > At lunch just now Evan and I realized another point to add it to the
> > > > server, though: what if somebody had more memcache servers than web
> > > > nodes? But we don't buy this argument, because in our experience
you
> > > > can't just _buy_ a memcache machine. We were unable to find a
system or
> > > > parts vendor who could get us a machine with a crap processor and
tons
> > of
> > > > RAM.
> > > >
> > > > So, yeah: compression definitely client-side, but the server should
do
> > > > stats by the client sending more data about how much it shrank it
to.
> > See
> > > > any problem with that?
> > > >
> > > >
> > > > On Wed, 13 Aug 2003, Lisa Marie Seelye wrote:
> > > >
> > > > > On Wed, 2003-08-13 at 14:44, Brad Fitzpatrick wrote:
> > > > > > Thoughts?
> > > > >
> > > > > What CPU resources are required to compress the data? How much of
a
> > lag
> > > > > will it introduce to the System?
> > > > >
> > > > > The best way to go about storing compressed data is for the client
to
> > > > > _send the data in a compressed format_ and then know well enough
to
> > > > > decompress it when they get it back.
> > > > >
> > > > > The server should store whatever the client sends them and don't
mess
> > > > > with it -- put the burden on the user, not the server.
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > -Lisa
> > > > > <Vix ulla tam iniqua pax, quin bello vel aequissimo sit potior>
> > > > >
> > > >
> > >
> >
>
------=_NextPart_000_006E_01C361C2.6B6D22D0
Content-Type: application/octet-stream;
name="php-memcached-1.0.9.gzip.patch"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
filename="php-memcached-1.0.9.gzip.patch"
--- php-memcached-1.0.9/MemCachedClient.inc.php Tue Aug 12 20:09:53 2003=0A=
+++ php-memcached-1.0.9.gzip/MemCachedClient.inc.php Wed Aug 13 17:39:02 =
2003=0A=
@@ -90,6 +90,11 @@=0A=
* @var string=0A=
*/=0A=
var $errstr;=0A=
+ /**=0A=
+ * size of val to force compression; 0 turns off=0A=
+ * @ var int=0A=
+ */=0A=
+ var $compress;=0A=
=0A=
=0A=
/**=0A=
@@ -116,6 +121,7 @@=0A=
{=0A=
$this->set_servers($options["servers"]);=0A=
$this->debug =3D $options["debug"];=0A=
+ $this->compress =3D $options["compress"];=0A=
$this->cache_sock =3D array();=0A=
}=0A=
=0A=
@@ -783,7 +789,6 @@=0A=
return trim($retval);=0A=
}=0A=
=0A=
-=0A=
/**=0A=
* sends the command to the server=0A=
* Possible errors set are:=0A=
@@ -837,6 +842,13 @@=0A=
$flags |=3D 1;=0A=
}=0A=
=0A=
+ if (($this->compress > 0) && (strlen($val) > $this->compress)) {=0A=
+ $this->_debug("_set(): compressing data. size in:".strlen($val));=0A=
+ $val=3Dgzcompress($val);=0A=
+ $this->_debug("_set(): done compressing data. size =
out:".strlen($val));=0A=
+ $flags |=3D 2;=0A=
+ }=0A=
+=0A=
$len =3D strlen($val);=0A=
if (!is_int($exptime))=0A=
$exptime =3D 0;=0A=
@@ -1034,6 +1046,8 @@=0A=
=0A=
if(strlen($val[$sk]) !=3D $len_array[$sk])=0A=
continue;=0A=
+ if(@$flags_array[$sk] & 2)=0A=
+ $val[$sk] =3D gzuncompress($val[$sk]);=0A=
=0A=
if(@$flags_array[$sk] & 1)=0A=
$val[$sk] =3D unserialize($val[$sk]);=0A=
------=_NextPart_000_006E_01C361C2.6B6D22D0--