1.2.3 Segfaulting on RHEL 5 (x86_64) with -P option & threads enabled

[Brendan W. McAdams]

Kind of weird: we're using the EPEL build [memcached-1.2.3-4.el5], but I
also built from scratch and get the same issue.  On 4 brand new machines
(Dell Poweredge 1950s, RHEL 5, 8 gigs RAM, 2xQuad-Core Intel(R) Xeon(R) CPU
L5320  @ 1.86GHz), we get segfaults after memcached writes the PID file:

memcached[5634]: segfault at 0000000000007889 rip 0000003583c03223 rsp
00007fff63cd5698 error 4
memcached[5651]: segfault at 0000000000007889 rip 0000003583c03223 rsp
00007fff24b1e4d8 error 4
memcached[5734]: segfault at 0000000000007889 rip 0000003583c03223 rsp
00007ffff51ae7a8 error 4


I played with options taking things in and out, and it's definitely the PID
(-P) option:the file writes, contains a PID but it looks like write after
it's written memcached fails.  If I remove the -P option, the server starts
up cleanly and runs fine.  Also played with memory allocation and max
connections, but it's definitely the PID option...


Here's the relevant Strace output:


mmap(NULL, 528384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x2aaaaacc4000
rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN, {u32=487064496, u64=487064496}}) =
0
open("/var/run/memcached/memcached.pid", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x2aaaaad45000
write(7, "10070\n", 6)                  = 6
close(7)                                = 0
munmap(0x2aaaaad45000, 4096)            = 0
pipe([7, 8])                            = 0
epoll_ctl(4, EPOLL_CTL_ADD, 7, {EPOLLIN, {u32=487064560, u64=487064560}}) =
0
pipe([9, 10])                           = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
epoll_create(1024)                      = 11
fcntl(11, F_SETFD, FD_CLOEXEC)          = 0
socketpair(PF_FILE, SOCK_STREAM, 0, [12, 13]) = 0
fcntl(12, F_SETFD, FD_CLOEXEC)          = 0
fcntl(13, F_SETFD, FD_CLOEXEC)          = 0
epoll_ctl(11, EPOLL_CTL_ADD, 9, {EPOLLIN, {u32=487107392, u64=487107392}}) =
0
pipe([14, 15])                          = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 10070 detached

Just as I was about to send this I had a thought, and tried something else:
IF I build without --enable-threads -P works fine.  I haven't played with
1.2.2 yet as I'm trying to get these machines online and into production for
a release, but any insight into what's doing this would be great.  These
machines are basically stock RedHat 5, with lighttpd pulled from EPEL.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20071001/e14b625a/attachment.html