Heavy network loading...
Steven Grimm
sgrimm at facebook.com
Tue Sep 5 23:04:19 UTC 2006
That sendmsg is from ping, not memcached, right?
-Steve
Brad Fitzpatrick wrote:
> Which memcached version?
>
> (wondering because of sendmsg)
>
>
> On Tue, 5 Sep 2006, Tim Yardley wrote:
>
>
>> All;
>>
>> Since a lot of us here run machines with high traffic loads, I hope this is
>> relatively on-topic. I have an interesting situation in which a box is
>> "heavily" loaded with network traffic on a bridged interface. The loading
>> isn't directly related to memcached, but I figured someone here might have
>> run into this before based on stress testing memcache or just from high load
>> sites.
>>
>> During the heavy load I get the following situation:
>>
>> # ping -n google.com
>> PING google.com (64.233.167.99) 56(84) bytes of data.
>> ping: sendmsg: Operation not permitted
>> ping: sendmsg: Operation not permitted
>> ping: sendmsg: Operation not permitted
>> ping: sendmsg: Operation not permitted
>> ping: sendmsg: Operation not permitted
>> ping: sendmsg: Operation not permitted
>> 64 bytes from 64.233.167.99: icmp_seq=6 ttl=242 time=1023 ms
>> 64 bytes from 64.233.167.99: icmp_seq=8 ttl=242 time=22.8 ms
>> 64 bytes from 64.233.167.99: icmp_seq=9 ttl=242 time=22.0 ms
>> ping: sendmsg: Operation not permitted
>>
>> Snip of the strace:
>>
>> --------
>>
>> sendmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(0),
>> sin_addr=inet_addr("64.233.167.99")},
>> msg_iov(1)=[{"\10\0\211\321\225h\0m/\373\375D\270\25\10\0\10\t\n\v\f"...,
>> 64}], msg_controllen=0, msg_flags=0}, MSG_CONFIRM) = -1 EPERM (Operation not
>> permitted)
>> recvmsg(3, 0xbfe96470, MSG_ERRQUEUE|MSG_DONTWAIT) = -1 EAGAIN (Resource
>> temporarily unavailable)
>> dup(2) = 4
>> fcntl64(4, F_GETFL) = 0x2 (flags O_RDWR)
>> fstat64(4, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 5), ...}) = 0
>> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
>> 0xb7e44000
>> _llseek(4, 0, 0xbfe96540, SEEK_CUR) = -1 ESPIPE (Illegal seek)
>> write(4, "ping: sendmsg: Operation not per"..., 39) = 39
>> close(4) = 0
>> munmap(0xb7e44000, 4096) = 0
>> recvmsg(3, 0xbfe96710, 0) = -1 EAGAIN (Resource temporarily
>> unavailable)
>> gettimeofday({1157495600, 551997}, NULL) = 0
>> gettimeofday({1157495600, 552181}, NULL) = 0
>> sendmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(0),
>> sin_addr=inet_addr("64.233.167.99")},
>> msg_iov(1)=[{"\10\0Ky\225h\0n0\373\375D\365l\10\0\10\t\n\v\f\r\16\17"...,
>> 64}], msg_controllen=0, msg_flags=0}, 0) = 64
>> recvmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(22759),
>> sin_addr=inet_addr("64.233.167.99")},
>> msg_iov(1)=[{"E\0\0T\0m@\0\363\1\0165@\351\247cD\264L\6\0\0Sy\225h\0"...,
>> 192}], msg_controllen=20, {cmsg_len=20, cmsg_level=SOL_SOCKET,
>> cmsg_type=0x1d /* SCM_??? */, ...}, msg_flags=0}, 0) = 84
>> write(1, "64 bytes from 64.233.167.99: icm"..., 63) = 63
>> gettimeofday({1157495600, 580525}, NULL) = 0
>> poll([{fd=3, events=POLLIN|POLLERR, revents=POLLIN}], 1, 972) = 1
>> recvmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(46027),
>> sin_addr=inet_addr("68.180.76.6")},
>> msg_iov(1)=[{"E\300\0XaI\0\0@\1\367\'D\264L\6D\264L\6\3\1\34\303\0\0"...,
>> 192}], msg_controllen=20, {cmsg_len=20, cmsg_level=SOL_SOCKET,
>> cmsg_type=0x1d /* SCM_??? */, ...}, msg_flags=0}, MSG_DONTWAIT) = 88
>> gettimeofday({1157495600, 851056}, NULL) = 0
>>
>> ----------------
>>
>> Sysctl.conf:
>> net.ipv4.conf.default.rp_filter = 1
>> net.ipv4.conf.all.rp_filter = 1
>> kernel.sysrq=1
>> net.ipv4.ip_forward=1
>> kernel.sem=250 32000 100 128
>> # Adjust where the gc will leave arp table alone
>> net.ipv4.neigh.default.gc_thresh1=256
>> net.ipv4.neigh.default.gc_thresh2=1024
>> net.ipv4.neigh.default.gc_thresh3=2048
>> # Increase TCP
>> net.ipv4.neigh.default.proxy_qlen = 96
>> net.ipv4.neigh.default.unres_qlen = 6
>> # Increase size of socket buffers
>> net.ipv4.tcp_rmem = 4096 98304 349520
>> net.ipv4.tcp_wmem = 4096 65535 262142
>> net.ipv4.tcp_mem = 98304 262142 393216
>> # Bump up TCP socket queue mechanism to help with syn floods
>> net.ipv4.tcp_max_syn_backlog = 2048
>> # Drop it so lack of FIN times out quicker
>> net.ipv4.tcp_fin_timeout = 30
>> # Increase number of incoming connections backlog
>> net.core.somaxconn = 512
>> # Bump optmem_max up
>> net.core.optmem_max = 20480
>> # Increase number of incoming connections backlog
>> net.core.netdev_max_backlog = 1024
>> # Bump up default r/wmem to max
>> net.core.rmem_default = 262141
>> net.core.wmem_default = 262141
>> # Bump up max r/wmem
>> net.core.rmem_max = 262141
>> net.core.wmem_max = 262141
>> # Increase size of file handles and inode cache
>> fs.file-max = 209708
>>
>> -----------------
>>
>> Some proc relevance (eth1 and eth2 are the devices pushing the traffic):
>>
>> # cat /proc/interrupts
>> CPU0 CPU1
>> 0: 350401618 0 IO-APIC-edge timer
>> 9: 0 0 IO-APIC-level acpi
>> 14: 15736294 0 IO-APIC-edge libata
>> 15: 0 0 IO-APIC-edge libata
>> 169: 0 0 IO-APIC-level uhci_hcd:usb2
>> 177: 1557943765 0 IO-APIC-level uhci_hcd:usb3, eth0, eth2
>> 185: 41 0 IO-APIC-level HDA Intel, uhci_hcd:usb4
>> 193: 0 0 IO-APIC-level uhci_hcd:usb1, ehci_hcd:usb5
>> 201: 1316369138 0 IO-APIC-level eth1
>> NMI: 0 0
>> LOC: 349727026 349727025
>> ERR: 0
>> MIS: 0
>>
>> # cat /proc/loadavg
>> 1.33 0.90 1.16 2/159 28533
>>
>> # cat /proc/version
>> Linux version 2.6.12-gentoo-r10 (root at meepmeep) (gcc version 3.3.5-20050130
>> (Gentoo 3.3.5.20050130-r1, ssp-3.3.5.20050130-1, pie-8.7.7.1)) #4 SMP Sun
>> Mar 26 23:00:57 CST 2006
>>
>> Anyone have any ideas as to what this might be? Interrupt overload? Any
>> way to relieve this situation? I've searched around but haven't found much
>> definitive. Linux kernel tuning (2.6.x) websites would be appreciated if
>> any of you know of good ones.
>>
>> Any help is definitely appreciated. Sysctl tweaks are very much welcome as
>> well. If this is too off-topic for the list, feel free to just reply
>> directly. Thanks in advance.
>>
>> /tmy
>>
>>
>>
More information about the memcached
mailing list