ip_conntrack problem with memcached

[NTPT]

This is because kernel try to track  all your memcached connections - ie watch all tcp/ip connection that you made to your memcache. and because conntrack table have a limited size ,  and it must hold a connection information for relative long time, when this table get overloaded, packets are dropped . 


there are general tree possible solutions 

1: Do not use Ip_conntrack feature of the kernel . ie if you do not need to perform a nat or some fancy packet mangling, SPI or something on your box,   do not load (modporbe or  insmod ) ip_conntrack module to the kernal .

2: Increase a size of the conntrack table like this 

echo "yoursize" > /proc/sys/net/ipv4/ip_conntrack_max 

where "yoursize"  is a  number of tcp/ip connections to be tracked. something about 32000 - 128000.   Work well, but eat up memory and CPU time, with is a big disadvantage on busy sites  (and if you get conntrack_table full message, you are a busy site  ) because it lower the amounth of available memory for memchached too.


3: Compile your kernel   in netfilter section  with "RAW table" support and "NOTRACK"  target for iptables. With this feature you can (via  iptables ) easily disable connection tracking for connection to memchached  while leaving other connection tracking mechanism for nat, spi  and other etc. so you box can do a nat., púacket mangling or SPI , but ip conntrack mechanism in kernal wil not be burdened with memcached related traffic.  For future detail consult iptables manual 
   
the command wil be something like this  (without warranty - for detail man iptables)

iptables -t raw -I PREROUTING -i interface to  your memchached clients -s ip_addrsss of your memcached clients -d ip_address of your memcached server --dport port of your memcached daemon  -j NOTRACK 

iptables -t raw -I OUTPUT -i interface to  your memchached clients -s ip_addrsss of your memcached box --sport port of your memcached daemon   -j NOTRACK 

first rule is for not track incoming traffic to memcached , second is for not tracking of outgoing traffic from memcached. 


Note thad disablink connection tracking for memcached generated trafic may be generally a very good idea because conntrac mechanism of the kernal eat up memory and cpu time and may be it can improve overall performance of memcached over tcp/ip . real benchmark needed .

















>  ------------ Původní zpráva ------------
>  Od: Hansflug (at) freenet <hansflug at freenet.de>
>  Předmět: ip_conntrack problem with memcached
>  Datum: 13.11.2006 09:39:10
>  ----------------------------------------
>  Hello,
>  
>  i have a Problem with my Connection to memcached.
>  in peaktime the kernel log show that i have a ip_conntrack problem ?
>  i use php5 and memcachelib for save db results. 
>  
>  What tuning i can do for my problem  ????
>  
>  here information about my memcache daemon.....
>  the server no swapping or have no high load. but i have many
>  connections !!
>  
>  server:
>  2 CPU but memcache use only one....
>  2GB RAM
>  load average: 0.08, 0.09, 0.09
>  Cpu(s):  1.3% us,  2.5% sy,  0.0% ni, 94.3% id,  0.0% wa,  0.3% hi,
>  1.5% si
>  Mem:   2077052k total,  1943424k used,   133628k free,   100344k buffers
>  Swap:  3148720k total,        0k used,  3148720k free,   285896k cached
>  
>  
>  kernel.log
>  
>  kernel: ip_conntrack: table full, dropping packet.
>  last message repeated 9 times
>  kernel: printk: 3815 messages suppressed.
>  ...................
>  ./memcached-tool localhost:11516 display
>   6         B      0 s       0     yes
>    7         B      0 s       0     yes
>    8     256 B  44405 s       8     yes
>    9     512 B  48126 s      12     yes
>   10      1 kB  51820 s       1     yes
>   11      2 kB  38461 s     130     yes
>   12      4 kB  42341 s     117     yes
>   13      8 kB  39478 s     229     yes
>   14     16 kB  59783 s       2     yes
>   15     32 kB 833888 s       1      no
>   16         B      0 s       0     yes
>   17         B      0 s       0     yes
>  
>  startparameter
>  /usr/bin/memcached -m 500 -p 11516 -u root -c 4500
>  
>  statsinfo
>  telnet localhost 11516
>  Trying 127.0.0.1...
>  Connected to localhost.
>  Escape character is '^]'.
>  stats
>  STAT pid 9639
>  STAT uptime 1532210
>  STAT time 1163406727
>  STAT version 1.1.12
>  STAT rusage_user 16376.415406
>  STAT rusage_system 28586.777149
>  STAT curr_items 184292
>  STAT total_items 125758879
>  STAT bytes 343284616
>  STAT curr_connections 547
>  STAT total_connections 43200011
>  STAT connection_structures 3253
>  STAT cmd_get 346041673
>  STAT cmd_set 125758879
>  STAT get_hits 304561195
>  STAT get_misses 41480478
>  STAT bytes_read 90234189974
>  STAT bytes_written 402886887964
>  STAT limit_maxbytes 524288000
>  
>  
>  
>  
>  
>