memcached persistent load balancing with an f5 big-ip

[Andrew Miehs]

Hi Matthew,

I don't know which F5s you are running, but you seem to have a lot
of free cpu time if you can afford to run iRules.

You probably find that this EATs cpu time - especially if you have a
lot of requests to the memcached servers. These packets can suddenly
no longer be switched on layer4 and need to go through the CPU.

Your next problem will be with the 'persistence'.

What happens when one of your memcached servers goes down?
The new requests get handed out to those remaining - based on the hash.
When your broken server comes back up, the new machines still get the
traffic due to the connection being 'persistant' and your fixed
server gets no traffic.

With us its HTTP, but similar principal....

What we did (we only have '2' servers)... roughly from memory

pool cache1 {
     min active member 1
     member 192.168.0.1:8000 priority 5
     member 192.168.0.2:8000
}

pool cache1 {
     min active member 1
     member 192.168.0.2:8000 priority 5
     member 192.168.0.1:8000
}


virtual vs_10.0.0.1 {
     pool cache1
}

virtual vs_10.0.0.2 {
     pool cache2
}


---
The configured the servers to use cache1 or cache2 based on whether they 
were odd or even...

Cheers

Andrew



Matthew Kent wrote:
> Thought I'd share something handy I cooked up based on examples:
> 
> If you have a f5 big-ip load balancer in your network (running some of
> the more recent software) you can use an iRule to distribute data to a
> pool of memcached servers based on a crc32 of the incoming key. The
> payload will be reexamined for each get/set request, allowing you to
> leave the connection open indefinitely.
> 
> Although most (all?) of the client apis seem to handle multiple
> memcached servers and them being up/down, this seemed like a more
> graceful approach to provide 1 unified ip for my configs plus solve some
> issues with different apis (apr-util, ruby, php) hashing data between a
> 3 server group differently, making it annoying to share.
> 
> Not in production yet but seems fine in testing. 
> 
> Oh and it does assume your memcached servers have an equal amount of
> ram. I haven't looked into how to implement weighting as I don't need it
> presently.
> 
> 
> ------------------------------------------------------------------------
> 
> # memcached persistent load balancing based on keys passed
> # how fun is this? :)
> when CLIENT_ACCEPTED {
>   # debug
> #  log local0. "memcached pool: client accepted"
>   TCP::collect 
> }
> when CLIENT_DATA {
>   # memcached protocol is nice and simple
>   # <command name> <key> <flags> <exptime> <bytes>\r\n
>   set key [string trim [getfield [TCP::payload] " " 2]]
>   # debug
> #  log local0. "memcached pool: raw client data: '[TCP::payload]' key: '$key'"
>   # f5 says they crc32 the key passed
>   persist hash $key
>   TCP::release
>   TCP::collect
> }
> when LB_SELECTED {
>   # debug
> #  log local0. "memcached pool: connecting to server [IP::client_addr]:[TCP::client_port] --> [LB::server addr]:[LB::server port]"
> } 
> when SERVER_CONNECTED {
>   TCP::collect
>   # debug
> #  log local0. "memcached pool: server connected"
> }
> when SERVER_DATA {
>   # debug
> #  log local0. "memcached pool: server data invoked"
>   TCP::release
>   LB::detach
> }