memcached 1.2.2 core dump

[dormando]

Hey,

Good news!

The stats commands aren't thread safe in 1.2.2.

Since r589 in SVN they have been thread safe.

I'd ask if you folks could attempt using trunk on one server and see if
that fixes your issue. We'll have 1.2.4 out the door soon anyway, but
trunk could use more testing.

-Dormando

Jeremy LaTrasse wrote:
> We're running memcached 1.2.2 with libevent 1.3 on OpenSolaris 11.
> 
> We recently changed some of the ways that our application interacts with
> memcached, and then very suddenly afterward started experiencing core
> dumps across our 32 instances of memcached, seemingly arbitrarily.
> 
> We captured the core files and discovered that they were all generated
> when a request for 'stats sizes' was issued by our monitoring processes.
> 
> One of the engineers here postulates the following:
> 
> 
>     SEGFAULT on line 341 of items.c:
> 
>         /* build the histogram */
>         memset(histogram, 0, (size_t)num_buckets * sizeof(int ));
>         for (i = 0; i < LARGEST_ID; i++) {
>             item *iter = heads[i];
>             while (iter) {
>                 int ntotal = ITEM_ntotal(iter); 
>                 int bucket = ntotal / 32;
>                 if ((ntotal % 32) != 0) bucket++;
>                 if (bucket < num_buckets) histogram[bucket]++;
>                 iter = iter->next;
>             }
>         }
> 
>     That's:
> 
>                 int ntotal = ITEM_ntotal(iter);
> 
> 
>     Given the huge amount of transactions we're doing, we're probably
>     hitting a race condition around moving items from one bucket to the
>     other.  Perhaps a mutex lock is not being set properly
> 
> 
> For the time being we've disabled the 'stats sizes' request from our
> monitoring processes to preclude this situation.
> 
> I could not find this to be a known issue in previous messages on this
> list, but I am certain that someone will end up in this scenario.
> 
> I can send the core files or gdb output to anyone interested in
> addressing this.
> 
> Jeremy LaTrasse
> Operations
> Twitter