PHP pecl/memcache with udp, consistent hashing, non-blocking-io and pipelining

[Mikael Johansson]

Hi,

Support for libevent would be nice but since the client wouldn't have
more than N sockets active at any one time, while the server have a
potential N*M sockets open (where N is the number of servers and M is
the number of potential clients (eg. webservers * max-processes)) and it
requires extra libraries it is not prio 1.

I'm also not currently aware as to how this might affect the current
architecture, as I understand it libevent uses callbacks to provide
event notification? It might require much #ifdef's to support both
select() and libevent schemes.

Memcached sends udp responses in 1400 byte chunks including a header and
pecl/memcache glues them together before returning to the caller;
however with large multi-packet responses the possibility of dropped or
out-of-sequence packets increases which makes pecl/memcache back off udp
for a few seconds and revert to tcp. Have you had problemes with
responses larger than 20kb, ie. "E_NOTICE: udp packet loss, expected .."
messages?

//Mikael

Hongyu wrote:
> How about using libevent for non-blocking io instead of select()?
> libevent is more effective than select().
> 
> From my test maxium size of single udp datagram can transfer
> successfully is about 20K(not 64k) for local network.
> Item which value size is bigger than 20k will split into several
> datagrams, pecl/memcache extension has to
> re-assemble them. Situation becomes even worse when some of split
> datagrams lost during transmittion.
> 
> Regards,
> 
> Que Hongyu
> 
> On 5/31/07, *Mikael Johansson* <mikael at synd.info
> <mailto:mikael at synd.info>> wrote:
> 
>     Hi,
> 
>     I've had some questions about the future of pecl/memcache, so I thought
>     I might as well make some sort of announcement. The new NON_BLOCKING_IO
>     branch represents the next major milestone and contains some rather big
>     infrastructure changes; as such it should not be considered production
>     ready (although it runs the unit-tests without fail.)
> 
>     Help with testing and performance tuning/benchmarking would be much
>     appreciated, especially in larger setups while using UDP. New features
>     include:
> 
>     * Non-blocking io where requests are queued onto servers and processed
>     in a select() loop; enables increased throughput since request
>     construction (serialize, compress, ..) and request parsing
>     (deserialize,
>     decompress, ..) is now done in parallel with network send/recv. One
>     consequence is that performance should now increase as you add more
>     servers to the pool (before you might have seen a slight decrease.)
>     * Request pipelining, requests are processed as they are built and as
>     responses are received, senders don't wait for readers and vice versa.
>     * UDP support, currently only get requests use it per default but it
>     will be trivial to support UDP for other type of requests in the
>     future.
>     * Multi-key (and pipelined) set/delete/increment/decrement (about 10x
>     the performance of single-key requests when doing 50-keys per request)
>     * Consistent hashing using crc32("host:port"), as opposed to using md5
>     or sha1, motivation being that crc32 is readily available and fast. Has
>     any standard been reached on this issue?
> 
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 250 bytes
Desc: OpenPGP digital signature
Url : http://lists.danga.com/pipermail/memcached/attachments/20070601/de9e1246/signature.pgp