<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Send the core dumps. You have --enable-threads correct?<div><br class="webkit-block-placeholder"></div><div><div> pthread_mutex_lock(&cache_lock);</div><div> ret = do_item_stats_sizes(bytes);</div><div> pthread_mutex_unlock(&cache_lock);</div><div><br class="webkit-block-placeholder"></div><div>It's wrapped in pthread mutex, you do know its going to lock the entire cache by calling this correct?</div><div><br class="webkit-block-placeholder"></div><div>I remember reading sometime ago that by doing this (dump script) it would lock entire cache.</div><div><br class="webkit-block-placeholder"></div><div>-Chris</div><div><br class="webkit-block-placeholder"></div><div><div>On Nov 16, 2007, at 7:52 PM, Jeremy LaTrasse wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span style="border-collapse:collapse"><div>We're running memcached 1.2.2 with libevent 1.3 on OpenSolaris 11.</div><div><br></div><div>We recently changed some of the ways that our application interacts with memcached, and then very suddenly afterward started experiencing core dumps across our 32 instances of memcached, seemingly arbitrarily. </div><div><br></div><div>We captured the core files and discovered that they were all generated when a request for 'stats sizes' was issued by our monitoring processes.</div><div><br></div><div>One of the engineers here postulates the following: </div><blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:40px;border-top-style:none;border-right-style:none;border-bottom-style:none;border-left-style:none;border-width:initial;border-color:initial;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px"> <br>SEGFAULT on line 341 of items.c:<br><br> /* build the histogram */<br> memset(histogram, 0, (size_t)num_buckets * sizeof(int ));<br> for (i = 0; i < LARGEST_ID; i++) {<br> item *iter = heads[i];<br> while (iter) {<br> int ntotal = ITEM_ntotal(iter); <br> int bucket = ntotal / 32;<br> if ((ntotal % 32) != 0) bucket++;<br> if (bucket < num_buckets) histogram[bucket]++; <br> iter = iter->next;<br> }<br> }<br><br>That's:<br><br> int ntotal = ITEM_ntotal(iter);<br></blockquote><div><br></div><blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:40px;border-top-style:none;border-right-style:none;border-bottom-style:none;border-left-style:none;border-width:initial;border-color:initial;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px"> Given the huge amount of transactions we're doing, we're probably hitting a race condition around moving items from one bucket to the other. Perhaps a mutex lock is not being set properly<br></blockquote><div><br> </div><div>For the time being we've disabled the 'stats sizes' request from our monitoring processes to preclude this situation.</div><div><br></div><div>I could not find this to be a known issue in previous messages on this list, but I am certain that someone will end up in this scenario. </div><div><br></div><div>I can send the core files or gdb output to anyone interested in addressing this.</div><font color="#888888"><div><br></div><div>Jeremy LaTrasse</div><div>Operations</div><div>Twitter</div></font> </span></blockquote></div><br></div></body></html>