Intermittent memcached pauses
Chris Hondl
chris at imvu.com
Tue Nov 7 22:11:37 UTC 2006
We've had memcached deployed at IMVU for about 9 months now. It works
great, scales easily, never goes down, and totally saved us as traffic on
our site went up and up.
We are now seeing a problem where one of our memcached hosts periodically
stops handling requests for several seconds. This particular host has a
traffic pattern with about 30k get requests per second, a small number of
set requests, all data values stored are small (booleans), and 90+% of the
gets return a cache miss.
Watching the machine it looks like it is actually memcached that is
experiencing the delays and using all of the CPU. We tried disabling all
non-critical resources and still saw the pauses. When we do see CPU get
used up for 3-5 seconds, typically the number of processes waiting for time
goes to 2 or 3 and if you look at top, the memcached process CPU utilization
goes from ~30% to 99+%.
Has anyone else seen behavior like this?
Chris
Normal:
top - 13:00:00 up 123 days, 14:02, 4 users, load average: 0.36, 0.52, 0.43
Tasks: 51 total, 3 running, 48 sleeping, 0 stopped, 0 zombie
Cpu(s): 22.3% us, 5.0% sy, 0.0% ni, 68.8% id, 0.0% wa, 0.0% hi, 4.0% si
Mem: 3969768k total, 260392k used, 3709376k free, 22176k buffers
Swap: 6144852k total, 0k used, 6144852k free, 60380k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
* 6465 root 16 0 108m 81m 1380 R 28.0 2.1 7:25.40 memcached*
6829 root 16 0 1876 928 1672 R 0.7 0.0 0:00.11 top
1 root 16 0 1504 512 1352 S 0.0 0.0 0:13.55 init
Problem:
top - 13:00:42 up 123 days, 14:03, 4 users, load average: 0.31, 0.49, 0.42
Tasks: 52 total, 2 running, 50 sleeping, 0 stopped, 0 zombie
Cpu(s): 98.3% us, 0.3% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 1.3% si
Mem: 3969768k total, 265136k used, 3704632k free, 22176k buffers
Swap: 6144852k total, 0k used, 6144852k free, 60388k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
* 6465 root 20 0 110m 82m 1380 R 99.6 2.1 7:40.27 memcached*
1 root 16 0 1504 512 1352 S 0.0 0.0 0:13.55 init
2 root 34 19 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/0
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
2 0 0 3713424 22176 60332 0 0 0 0 0 0 22 9
69 0
1 0 0 3713296 22176 60332 0 0 0 0 0 0 23 9
68 0
2 0 0 3713360 22176 60332 0 0 0 0 0 0 22 10
68 0
1 0 0 3713312 22176 60332 0 0 0 0 0 0 24 10
66 0
1 0 0 3713312 22176 60332 0 0 0 0 0 0 22 12
66 0
1 0 0 3713320 22176 60332 0 0 0 0 0 0 23 11
66 0
1 0 0 3713320 22176 60332 0 0 0 25 0 0 23 8
69 0
2 0 0 3713336 22176 60332 0 0 0 17 0 0 22 10
68 0
1 0 0 3713400 22176 60332 0 0 0 0 0 0 25 10
65 0
1 0 0 3713400 22176 60336 0 0 0 0 0 0 24 10
66 0
1 0 0 3713400 22176 60336 0 0 0 0 0 0 22 8
70 0
1 0 0 3713336 22176 60336 0 0 0 0 0 0 24 8
68 0
1 0 0 3713272 22176 60336 0 0 0 24 0 0 22 11
67 0
1 0 0 3713272 22176 60336 0 0 0 0 0 0 24 10
66 0
1 0 0 3713208 22176 60336 0 0 0 0 0 0 23 9
68 0
1 0 0 3713208 22176 60336 0 0 0 0 0 0 20 10
70 0
* 2 0 0 3711032 22176 60336 0 0 0 0 0 0 73 2
25 0
2 0 0 3705720 22176 60340 0 0 0 132 0 0 98 2
0 0
2 0 0 3703544 22176 60344 0 0 0 0 0 0 99 1
0 0
2 0 0 3706488 22176 60344 0 0 0 0 0 0 97 3
0 0
2 0 0 3711416 22176 60348 0 0 0 0 0 0 98 2
0 0
2 0 0 3709624 22176 60348 0 0 0 0 0 0 99 1
0 0
2 0 0 3708848 22176 60348 0 0 0 72 0 0 98 2
0 0
2 0 0 3712048 22176 60348 0 0 0 0 0 0 87 3
10 0*
1 0 0 3712048 22176 60348 0 0 0 0 0 0 16 2
82 0
1 0 0 3712048 22176 60348 0 0 0 0 0 0 15 1
84 0
1 0 0 3712048 22176 60348 0 0 0 0 0 0 15 3
82 0
1 0 0 3712048 22176 60348 0 0 0 54 0 0 15 2
83 0
1 0 0 3712048 22176 60348 0 0 0 0 0 0 15 2
83 0
1 0 0 3712056 22176 60348 0 0 0 0 0 0 15 2
83 0
1 0 0 3712056 22176 60348 0 0 0 0 0 0 17 2
81 0
1 0 0 3712056 22176 60348 0 0 0 0 0 0 14 3
83 0
1 0 0 3712056 22176 60348 0 0 0 12 0 0 16 2
82 0
--
http://avatars.imvu.com/chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20061107/439bf6f2/attachment.htm
More information about the memcached
mailing list