Heavy network loading...
Tim Yardley
liquid at haveheart.com
Wed Sep 6 00:33:29 UTC 2006
Brad gave me the okay to continue this thread, so here it goes.
As Brad pointed out, this is partially off-topic, but it may pose an issue
in the future if this ends up being sendmsg related (since memcached was
recently converted to use sendmsg). Below is an exert of an email sent to
Brad:
I have never seen an EPERM returned in this situation, if anything, one
would think I would see something like NETIF_RX_BUSY or NETIF_TX_BUSY. I
was planning on leveraging memcached to deal with some of the load on this
system (decreasing the amount of network traffic by using memcached to hold
post-processed data), but it can't reliably connect to the local memcached
node due to this problem.
If anything, I figured someone might share their sysctl.conf settings
(tcp/fs/buffer/etc tuning) that may help alleviate a situation like this.
With all the traffic livejournal pushes, digg pushes, facebook, etc... I
figure someone's done quite a bit of tweaking.
/tmy
-----Original Message-----
From: memcached-bounces at lists.danga.com
[mailto:memcached-bounces at lists.danga.com] On Behalf Of Brad Fitzpatrick
Sent: Tuesday, September 05, 2006 6:30 PM
To: Tim Yardley
Cc: memcached at lists.danga.com
Subject: RE: Heavy network loading...
Yeah, I can read the strace. I know it's not memcached.
I ask which version of memcached because new versions use sendmsg and old
versions don't. If it's a new version of memcached using sendmsg, perhaps
you're competing for iovec structures in the kernel. etc etc.
So the memcached version could matter. Or the output of the stats
command. It's at least interesting. Otherwise this thread is totally
off-topic.
On Tue, 5 Sep 2006, Tim Yardley wrote:
> Steve;
>
> Yeah, that is from ping not memcache (since it is easier to show that
way).
> Anything accessing the network is generating the message under the load
> though.
>
> Brad.. As noted above, the sendmsg isnt from memcache in this case.
>
> /tmy
>
> -----Original Message-----
> From: memcached-bounces at lists.danga.com
> [mailto:memcached-bounces at lists.danga.com] On Behalf Of Steven Grimm
> Sent: Tuesday, September 05, 2006 6:04 PM
> To: Brad Fitzpatrick
> Cc: memcached at lists.danga.com; Tim Yardley
> Subject: Re: Heavy network loading...
>
> That sendmsg is from ping, not memcached, right?
>
> -Steve
>
>
> Brad Fitzpatrick wrote:
> > Which memcached version?
> >
> > (wondering because of sendmsg)
> >
> >
> > On Tue, 5 Sep 2006, Tim Yardley wrote:
> >
> >
> >> All;
> >>
> >> Since a lot of us here run machines with high traffic loads, I hope
this
> is
> >> relatively on-topic. I have an interesting situation in which a box is
> >> "heavily" loaded with network traffic on a bridged interface. The
> loading
> >> isn't directly related to memcached, but I figured someone here might
> have
> >> run into this before based on stress testing memcache or just from high
> load
> >> sites.
> >>
> >> During the heavy load I get the following situation:
> >>
> >> # ping -n google.com
> >> PING google.com (64.233.167.99) 56(84) bytes of data.
> >> ping: sendmsg: Operation not permitted
> >> ping: sendmsg: Operation not permitted
> >> ping: sendmsg: Operation not permitted
> >> ping: sendmsg: Operation not permitted
> >> ping: sendmsg: Operation not permitted
> >> ping: sendmsg: Operation not permitted
> >> 64 bytes from 64.233.167.99: icmp_seq=6 ttl=242 time=1023 ms
> >> 64 bytes from 64.233.167.99: icmp_seq=8 ttl=242 time=22.8 ms
> >> 64 bytes from 64.233.167.99: icmp_seq=9 ttl=242 time=22.0 ms
> >> ping: sendmsg: Operation not permitted
> >>
> >> Snip of the strace:
> >>
> >> --------
> >>
> >> sendmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(0),
> >> sin_addr=inet_addr("64.233.167.99")},
> >>
msg_iov(1)=[{"\10\0\211\321\225h\0m/\373\375D\270\25\10\0\10\t\n\v\f"...,
> >> 64}], msg_controllen=0, msg_flags=0}, MSG_CONFIRM) = -1 EPERM
(Operation
> not
> >> permitted)
> >> recvmsg(3, 0xbfe96470, MSG_ERRQUEUE|MSG_DONTWAIT) = -1 EAGAIN (Resource
> >> temporarily unavailable)
> >> dup(2) = 4
> >> fcntl64(4, F_GETFL) = 0x2 (flags O_RDWR)
> >> fstat64(4, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 5), ...}) = 0
> >> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0)
> =
> >> 0xb7e44000
> >> _llseek(4, 0, 0xbfe96540, SEEK_CUR) = -1 ESPIPE (Illegal seek)
> >> write(4, "ping: sendmsg: Operation not per"..., 39) = 39
> >> close(4) = 0
> >> munmap(0xb7e44000, 4096) = 0
> >> recvmsg(3, 0xbfe96710, 0) = -1 EAGAIN (Resource
temporarily
> >> unavailable)
> >> gettimeofday({1157495600, 551997}, NULL) = 0
> >> gettimeofday({1157495600, 552181}, NULL) = 0
> >> sendmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(0),
> >> sin_addr=inet_addr("64.233.167.99")},
> >>
msg_iov(1)=[{"\10\0Ky\225h\0n0\373\375D\365l\10\0\10\t\n\v\f\r\16\17"...,
> >> 64}], msg_controllen=0, msg_flags=0}, 0) = 64
> >> recvmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(22759),
> >> sin_addr=inet_addr("64.233.167.99")},
> >>
msg_iov(1)=[{"E\0\0T\0m@\0\363\1\0165@\351\247cD\264L\6\0\0Sy\225h\0"...,
> >> 192}], msg_controllen=20, {cmsg_len=20, cmsg_level=SOL_SOCKET,
> >> cmsg_type=0x1d /* SCM_??? */, ...}, msg_flags=0}, 0) = 84
> >> write(1, "64 bytes from 64.233.167.99: icm"..., 63) = 63
> >> gettimeofday({1157495600, 580525}, NULL) = 0
> >> poll([{fd=3, events=POLLIN|POLLERR, revents=POLLIN}], 1, 972) = 1
> >> recvmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(46027),
> >> sin_addr=inet_addr("68.180.76.6")},
> >>
msg_iov(1)=[{"E\300\0XaI\0\0@\1\367\'D\264L\6D\264L\6\3\1\34\303\0\0"...,
> >> 192}], msg_controllen=20, {cmsg_len=20, cmsg_level=SOL_SOCKET,
> >> cmsg_type=0x1d /* SCM_??? */, ...}, msg_flags=0}, MSG_DONTWAIT) = 88
> >> gettimeofday({1157495600, 851056}, NULL) = 0
> >>
> >> ----------------
> >>
> >> Sysctl.conf:
> >> net.ipv4.conf.default.rp_filter = 1
> >> net.ipv4.conf.all.rp_filter = 1
> >> kernel.sysrq=1
> >> net.ipv4.ip_forward=1
> >> kernel.sem=250 32000 100 128
> >> # Adjust where the gc will leave arp table alone
> >> net.ipv4.neigh.default.gc_thresh1=256
> >> net.ipv4.neigh.default.gc_thresh2=1024
> >> net.ipv4.neigh.default.gc_thresh3=2048
> >> # Increase TCP
> >> net.ipv4.neigh.default.proxy_qlen = 96
> >> net.ipv4.neigh.default.unres_qlen = 6
> >> # Increase size of socket buffers
> >> net.ipv4.tcp_rmem = 4096 98304 349520
> >> net.ipv4.tcp_wmem = 4096 65535 262142
> >> net.ipv4.tcp_mem = 98304 262142 393216
> >> # Bump up TCP socket queue mechanism to help with syn floods
> >> net.ipv4.tcp_max_syn_backlog = 2048
> >> # Drop it so lack of FIN times out quicker
> >> net.ipv4.tcp_fin_timeout = 30
> >> # Increase number of incoming connections backlog
> >> net.core.somaxconn = 512
> >> # Bump optmem_max up
> >> net.core.optmem_max = 20480
> >> # Increase number of incoming connections backlog
> >> net.core.netdev_max_backlog = 1024
> >> # Bump up default r/wmem to max
> >> net.core.rmem_default = 262141
> >> net.core.wmem_default = 262141
> >> # Bump up max r/wmem
> >> net.core.rmem_max = 262141
> >> net.core.wmem_max = 262141
> >> # Increase size of file handles and inode cache
> >> fs.file-max = 209708
> >>
> >> -----------------
> >>
> >> Some proc relevance (eth1 and eth2 are the devices pushing the
traffic):
> >>
> >> # cat /proc/interrupts
> >> CPU0 CPU1
> >> 0: 350401618 0 IO-APIC-edge timer
> >> 9: 0 0 IO-APIC-level acpi
> >> 14: 15736294 0 IO-APIC-edge libata
> >> 15: 0 0 IO-APIC-edge libata
> >> 169: 0 0 IO-APIC-level uhci_hcd:usb2
> >> 177: 1557943765 0 IO-APIC-level uhci_hcd:usb3, eth0, eth2
> >> 185: 41 0 IO-APIC-level HDA Intel, uhci_hcd:usb4
> >> 193: 0 0 IO-APIC-level uhci_hcd:usb1,
ehci_hcd:usb5
> >> 201: 1316369138 0 IO-APIC-level eth1
> >> NMI: 0 0
> >> LOC: 349727026 349727025
> >> ERR: 0
> >> MIS: 0
> >>
> >> # cat /proc/loadavg
> >> 1.33 0.90 1.16 2/159 28533
> >>
> >> # cat /proc/version
> >> Linux version 2.6.12-gentoo-r10 (root at meepmeep) (gcc version
> 3.3.5-20050130
> >> (Gentoo 3.3.5.20050130-r1, ssp-3.3.5.20050130-1, pie-8.7.7.1)) #4 SMP
Sun
> >> Mar 26 23:00:57 CST 2006
> >>
> >> Anyone have any ideas as to what this might be? Interrupt overload?
Any
> >> way to relieve this situation? I've searched around but haven't found
> much
> >> definitive. Linux kernel tuning (2.6.x) websites would be appreciated
if
> >> any of you know of good ones.
> >>
> >> Any help is definitely appreciated. Sysctl tweaks are very much
welcome
> as
> >> well. If this is too off-topic for the list, feel free to just reply
> >> directly. Thanks in advance.
> >>
> >> /tmy
> >>
> >>
> >>
>
>
More information about the memcached
mailing list