[patch] Compression for Perl API

Fri, 15 Aug 2003 11:00:59 -0700 (PDT)

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.
  Send mail to mime@docserver.cac.washington.edu for more info.

---1774023665-1090092404-1060970459=:7242
Content-Type: TEXT/PLAIN; charset=US-ASCII

I've had this on my todo list for some time, but after it was implemented
in the PHP API, I couldn't let Perl get behind. :P

A diff to MemCachedClient.pm is attached.  It uses Compress::Zlib to do
gzip compression on inserts and decompression on gets.  There is currently
a constant minimum compression gain of 20%.  That is, if it does the GZIP
on write and it saves less than 20%, it will insert the uncompressed
version in an effort to speed up later gets.  This number should probably
be tweaked later (and maybe bet set by the user?), but for now this seems
reasonable.

A new 'compress_threshold' key can now be passed to the constructor to
determine the minimum size threshold before a value is compressed.
Since compression is disabled by default, this key also serves as the
global on/off switch.  To change its value later there is also a
set_compress_threshold() method.

For instances when compression should be temporarily disabled regardless
of data size (such as inserting compressed images), there is a new
enable_compress() method which takes a true or false value.

Curious as to how compression affected speed, I wrote up a little test
inserting a set of test data repeatedly.  Below is the output of 3 runs
showing the number of seconds get/set actions took with and without
compression:

compress: no
sets: => 5.33751511573792
gets: => 6.15450966358185
compress: yes
sets: => 5.281329870224
gets: => 8.92959928512573

compress: no
sets: => 5.56796407699585
gets: => 6.13490855693817
compress: yes
sets: => 5.11762177944183
gets: => 8.82273375988007

compress: no
sets: => 5.89584612846375
gets: => 6.15535402297974
compress: yes
sets: => 5.19342517852783
gets: => 8.98201024532318

After writing this mail, I checked and saw that Brad committed this patch
just before he left.  As a result, I've put the new Perl client live at
test.livejournal.org and I'm watching to make sure everything continues to
run smoothly.

--
Brad Whitaker
whitaker@danga.com
---1774023665-1090092404-1060970459=:7242
Content-Type: TEXT/plain; name="MemCachedClient.pm.diff"
Content-Transfer-Encoding: BASE64
Content-ID: <Pine.LNX.4.55.0308151100580.7242@danga.com>
Content-Description: 
Content-Disposition: attachment; filename="MemCachedClient.pm.diff"

bWFpbiAtPiB3Y210b29scyAgICAgICAgICBjZ2ktYmluL01lbUNhY2hlZENs
aWVudC5wbQ0KLS0tIGN2cy93Y210b29scy9tZW1jYWNoZWQvYXBpL3Blcmwv
TWVtQ2FjaGVkQ2xpZW50LnBtCTIwMDMtMDctMjggMDk6Mzg6MzIuMDAwMDAw
MDAwIC0wNzAwDQorKysgY2dpLWJpbi9NZW1DYWNoZWRDbGllbnQucG0JMjAw
My0wOC0xNSAwOTo1OTo0MS4wMDAwMDAwMDAgLTA3MDANCkBAIC0xMSw5ICsx
MSwyMCBAQA0KIA0KIHBhY2thZ2UgTWVtQ2FjaGVkQ2xpZW50Ow0KIA0KLXVz
ZSB2YXJzIHF3KCRWRVJTSU9OKTsNCisjIGZsYWcgZGVmaW5pdGlvbnMNCit1
c2UgY29uc3RhbnQgRl9TVE9SQUJMRSA9PiAxOw0KK3VzZSBjb25zdGFudCBG
X0NPTVBSRVNTID0+IDI7DQorDQorIyBzaXplIHNhdmluZ3MgcmVxdWlyZWQg
YmVmb3JlIHNhdmluZyBjb21wcmVzc2VkIHZhbHVlDQordXNlIGNvbnN0YW50
IENPTVBSRVNTX1NBVklOR1MgPT4gMC4yMDsgIyBwZXJjZW50DQorDQordXNl
IHZhcnMgcXcoJFZFUlNJT04gJEhBVkVfWkxJQik7DQogJFZFUlNJT04gPSAi
MS4wLjYiOw0KIA0KK0JFR0lOIHsNCisgICAgJEhBVkVfWkxJQiA9IGV2YWwg
InVzZSBDb21wcmVzczo6WmxpYiAoKTsgMTsiOw0KK30NCisNCiBteSAlaG9z
dF9kZWFkOyAgICMgaG9zdCAtPiB1bml4dGltZSBtYXJrZWQgZGVhZCB1bnRp
bA0KIG15ICVjYWNoZV9zb2NrOyAgIyBob3N0IC0+IHNvY2tldA0KIA0KQEAg
LTI2LDYgKzM3LDkgQEANCiAgICAgJHNlbGYtPnNldF9zZXJ2ZXJzKCRhcmdz
LT57J3NlcnZlcnMnfSk7DQogICAgICRzZWxmLT57J2RlYnVnJ30gPSAkYXJn
cy0+eydkZWJ1Zyd9Ow0KICAgICAkc2VsZi0+eydzdGF0cyd9ID0ge307DQor
ICAgICRzZWxmLT57J2NvbXByZXNzX3RocmVzaG9sZCd9ID0gJGFyZ3MtPnsn
Y29tcHJlc3NfdGhyZXNob2xkJ307DQorICAgICRzZWxmLT57J2NvbXByZXNz
X2VuYWJsZSd9ICAgID0gMTsNCisNCiAgICAgcmV0dXJuICRzZWxmOw0KIH0N
CiANCkBAIC00Myw2ICs1NywxNiBAQA0KICAgICAkc2VsZi0+eydkZWJ1Zyd9
ID0gJGRiZzsNCiB9DQogDQorc3ViIHNldF9jb21wcmVzc190aHJlc2hvbGQg
ew0KKyAgICBteSAoJHNlbGYsICR0aHJlc2gpID0gQF87DQorICAgICRzZWxm
LT57J2NvbXByZXNzX3RocmVzaG9sZCd9ID0gJHRocmVzaDsNCit9DQorDQor
c3ViIGVuYWJsZV9jb21wcmVzcyB7DQorICAgIG15ICgkc2VsZiwgJGVuYWJs
ZSkgPSBAXzsNCisgICAgJHNlbGYtPnsnY29tcHJlc3NfZW5hYmxlJ30gPSAk
ZW5hYmxlOw0KK30NCisNCiBzdWIgZm9yZ2V0X2RlYWRfaG9zdHMgew0KICAg
ICAlaG9zdF9kZWFkID0gKCk7DQogfQ0KQEAgLTEzMSwxNSArMTU1LDM0IEBA
DQogICAgIHJldHVybiAwIHVubGVzcyAkc2VsZi0+eydhY3RpdmUnfTsNCiAg
ICAgbXkgJHNvY2sgPSAkc2VsZi0+Z2V0X3NvY2soJGtleSk7DQogICAgIHJl
dHVybiAwIHVubGVzcyAkc29jazsNCisNCisgICAgdXNlIGJ5dGVzOyAjIHJl
dHVybiBieXRlcyBmcm9tIGxlbmd0aCgpDQorDQogICAgICRzZWxmLT57J3N0
YXRzJ30tPnskY21kbmFtZX0rKzsNCiAgICAgbXkgJGZsYWdzID0gMDsNCiAg
ICAgJGtleSA9IHJlZiAka2V5IGVxICJBUlJBWSIgPyAka2V5LT5bMV0gOiAk
a2V5Ow0KLSAgICBteSAkcmF3X3ZhbCA9ICR2YWw7DQorICAgIG15ICRyYXdf
dmFsID0gJHNlbGYtPnsnZGVidWcnfSA/ICR2YWwgOiB1bmRlZjsNCiAgICAg
aWYgKHJlZiAkdmFsKSB7DQogICAgICAgICAkdmFsID0gU3RvcmFibGU6OmZy
ZWV6ZSgkdmFsKTsNCi0gICAgICAgICRmbGFncyB8PSAxOw0KKyAgICAgICAg
JGZsYWdzIHw9IEZfU1RPUkFCTEU7DQogICAgIH0NCisNCiAgICAgbXkgJGxl
biA9IGxlbmd0aCgkdmFsKTsNCisNCisgICAgaWYgKCRIQVZFX1pMSUIgJiYg
JHNlbGYtPnsnY29tcHJlc3NfdGhyZXNob2xkJ30gJiYgJHNlbGYtPnsnY29t
cHJlc3NfZW5hYmxlJ30gJiYNCisgICAgICAgICRsZW4gPj0gJHNlbGYtPnsn
Y29tcHJlc3NfdGhyZXNob2xkJ30pIHsNCisNCisgICAgICAgIG15ICRjX3Zh
bCA9IENvbXByZXNzOjpabGliOjptZW1HemlwKCR2YWwpOw0KKyAgICAgICAg
bXkgJGNfbGVuID0gbGVuZ3RoKCRjX3ZhbCk7DQorDQorICAgICAgICAjIGRv
IHdlIHdhbnQgdG8ga2VlcCBpdD8NCisgICAgICAgIGlmICgkY19sZW4gPCAk
bGVuKigxIC0gQ09NUFJFU1NfU0FWSU5HUykpIHsNCisgICAgICAgICAgICAk
dmFsID0gJGNfdmFsOw0KKyAgICAgICAgICAgICRsZW4gPSAkY19sZW47DQor
ICAgICAgICAgICAgJGZsYWdzIHw9IEZfQ09NUFJFU1M7DQorICAgICAgICB9
DQorICAgIH0NCisNCiAgICAgJGV4cHRpbWUgPSBpbnQoJGV4cHRpbWUgfHwg
MCk7DQogICAgIG15ICRjbWQgPSAiJGNtZG5hbWUgJGtleSAkZmxhZ3MgJGV4
cHRpbWUgJGxlblxyXG4kdmFsXHJcbiI7DQogICAgICRzb2NrLT5wcmludCgk
Y21kKTsNCkBAIC0yMjYsNiArMjY5LDggQEANCiAgICAgbXkgJHNvY2sgPSBz
aGlmdDsNCiAgICAgbXkgJG91dHJlZiA9IHNoaWZ0Ow0KIA0KKyAgICB1c2Ug
Ynl0ZXM7ICMgcmV0dXJuIGJ5dGVzIGZyb20gbGVuZ3RoKCkNCisNCiAgICAg
bXkgJWZsYWdzOw0KICAgICBteSAldmFsOw0KICAgICBteSAlbGVuOyAgICMg
a2V5IC0+IGludGVuZGVkIGxlbmd0aA0KQEAgLTI1OCw3ICszMDMsOCBAQA0K
ICAgICAgICAgICAgIGZvcmVhY2ggKGtleXMgJWxlbikgew0KICAgICAgICAg
ICAgICAgICBuZXh0IHVubGVzcyBleGlzdHMgJHZhbHskX307DQogICAgICAg
ICAgICAgICAgIG5leHQgdW5sZXNzIGxlbmd0aCgkdmFseyRffSkgPT0gJGxl
bnskX307DQotICAgICAgICAgICAgICAgICR2YWx7JF99ID0gU3RvcmFibGU6
OnRoYXcoJHZhbHskX30pIGlmICRmbGFnc3skX30gJiAxOw0KKyAgICAgICAg
ICAgICAgICAkdmFseyRffSA9IENvbXByZXNzOjpabGliOjptZW1HdW56aXAo
JHZhbHskX30pIGlmICRIQVZFX1pMSUIgJiYgJGZsYWdzeyRffSAmIEZfQ09N
UFJFU1M7DQorICAgICAgICAgICAgICAgICR2YWx7JF99ID0gU3RvcmFibGU6
OnRoYXcoJHZhbHskX30pIGlmICRmbGFnc3skX30gJiBGX1NUT1JBQkxFOw0K
ICAgICAgICAgICAgICAgICAkb3V0cmVmLT57JF99ID0gJHZhbHskX307DQog
ICAgICAgICAgICAgfQ0KICAgICAgICAgICAgIHJldHVybiAxOw0KQEAgLTI5
Miw4ICszMzgsMTEgQEANCiAgICAgJ3NlcnZlcnMnID0+IFsgIjEwLjAuMC4x
NToxMTIxMSIsICIxMC4wLjAuMTU6MTEyMTIiLCANCiAgICAgICAgICAgICAg
ICAgICAgIjEwLjAuMC4xNzoxMTIxMSIsIFsgIjEwLjAuMC4xNzoxMTIxMSIs
IDMgXSBdLA0KICAgICAnZGVidWcnID0+IDAsDQorICAgICdjb21wcmVzc190
aHJlc2hvbGQnID0+IDEwXzAwMCwNCiAgIH07DQogICAkbWVtYy0+c2V0X3Nl
cnZlcnMoJGFycmF5X3JlZik7DQorICAkbWVtYy0+c2V0X2NvbXByZXNzX3Ro
cmVzaG9sZCgxMF8wMDApOw0KKyAgJG1lbWMtPmVuYWJsZV9jb21wcmVzcygw
KTsNCiANCiAgICRtZW1jLT5zZXQoIm15X2tleSIsICJTb21lIHZhbHVlIik7
DQogICAkbWVtYy0+c2V0KCJvYmplY3Rfa2V5IiwgeyAnY29tcGxleCcgPT4g
WyAib2JqZWN0IiwgMiwgNCBdfSk7DQpAQCAtMzI4LDYgKzM3NywxMCBAQA0K
IGFzIHBvc3NpYmxlLCBhcyB0aGlzIG1vZHVsZSBjdXJyZW50bHkgYWxsb2Nh
dGVzIG1lbW9yeSBmb3IgYnVja2V0DQogZGlzdHJpYnV0aW9uIHByb3BvcnRp
b25hbCB0byB0aGUgdG90YWwgaG9zdCB3ZWlnaHRzLg0KIA0KK1VzZSBDPGNv
bXByZXNzX3RocmVzaG9sZD4gdG8gc2V0IGEgY29tcHJlc3Npb24gdGhyZXNo
b2xkLCBpbiBieXRlcy4NCitWYWx1ZXMgbGFyZ2VyIHRoYW4gdGhpcyB0aHJl
c2hvbGQgd2lsbCBiZSBjb21wcmVzc2VkIGJ5IEM8c2V0PiBhbmQNCitkZWNv
bXByZXNzZWQgYnkgQzxnZXQ+Lg0KKw0KIFRoZSBvdGhlciB1c2VmdWwga2V5
IGlzIEM8ZGVidWc+LCB3aGljaCB3aGVuIHNldCB0byB0cnVlIHdpbGwgcHJv
ZHVjZQ0KIGRpYWdub3N0aWNzIG9uIFNUREVSUi4NCiANCkBAIC0zNDMsNiAr
Mzk2LDE5IEBADQogVGhlIGZvcm1hdCBpcyBhbiBhcnJheXJlZiBvZiBpZGVu
dGljYWwgZm9ybSBhcyBkZXNjcmliZWQgaW4gdGhlIEM8bmV3Pg0KIGNvbnN0
cnVjdG9yLg0KIA0KKz1pdGVtIEM8c2V0X2RlYnVnPg0KKw0KK1NldHMgdGhl
IEM8ZGVidWc+IGZsYWcuICBTZWUgQzxuZXc+IGNvbnN0cnVjdG9yIGZvciBt
b3JlIGluZm9ybWF0aW9uLg0KKw0KKz1pdGVtIEM8c2V0X2NvbXByZXNzX3Ro
cmVzaG9sZD4NCisNCitTZXRzIHRoZSBjb21wcmVzc2lvbiB0aHJlc2hvbGQu
IFNlZSBDPG5ldz4gY29uc3RydWN0b3IgZm9yIG1vcmUgaW5mb3JtYXRpb24u
DQorDQorPWl0ZW0gQzxlbmFibGVfY29tcHJlc3M+DQorDQorVGVtcG9yYXJp
bHkgZW5hYmxlIG9yIGRpc2FibGUgY29tcHJlc3Npb24uICBIYXMgbm8gZWZm
ZWN0IGlmIEM8Y29tcHJlc3NfdGhyZXNob2xkPg0KK2lzbid0IHNldCwgYnV0
IGhhcyBhbiBvdmVycmlkaW5nIGVmZmVjdCBpZiBpdCBpcy4NCisNCiA9aXRl
bSBDPGdldD4NCiANCiBteSAkdmFsID0gJG1lbS0+Z2V0KCRrZXkpOw0K

---1774023665-1090092404-1060970459=:7242--