"SERVER_ERROR out of memory" without -M

[Brad Fitzpatrick]

This is turning into a FAQ, so I just committed this:

http://www.danga.com/memcached/dist/patches/2006-01-01-prealloc-slabs.txt

It'll be in the next release.

It'll allocate one slab of each type on start-up.  Maybe/arguably a bit of
a waste for some workloads, but not really.  In the end I'm finally
convinced it's worth it, with everybody getting bit by it.

- Brad


On Sun, 1 Jan 2006, Adrian Ziemkowski wrote:

> We don't specify -M on our servers, so we shouldn't receive the
> "SERVER_ERROR out of memory" error, correct?
>
> I'm having the problem that our 128b slabs are filling up and instead of
> kicking out an old item, I'm getting the "SERVER_ERROR out of memory"
> message.  My understanding based on the documentation and the code, is
> that it would simply kick off the first item at the end of the list that
> isn't being accessed, but that doesn't appear to be the case here.
>
> To verify this, I tried to manually create an item of just one byte in
> one of the exhausted pools (this works fine in the other pools):
>
> set testing 0 0 1
> ERROR
> set testing 0 0 1
> SERVER_ERROR out of memory
> set testing 0 0 1
> ERROR
> set testing 0 0 1
> SERVER_ERROR out of memory
>
> What's interesting is how it isn't every single item, but instead every
> other.   Any ideas why I'd be alternating between them?
>
> The stats on one of the memcached instances (wouldn't it be useful if
> stats included an argv line?):
>
> stats
> STAT pid 1041
> STAT uptime 4179949
> STAT time 1136166523
> STAT version 1.1.11
> STAT rusage_user 3195:890000
> STAT rusage_system 18467:130000
> STAT curr_items 615754
> STAT total_items 10758554
> STAT bytes 1522770483
> STAT curr_connections 15
> STAT total_connections 32001085
> STAT connection_structures 320
> STAT cmd_get 153620631
> STAT cmd_set 10757775
> STAT get_hits 124470604
> STAT get_misses 29150027
> STAT bytes_read 27982254387
> STAT bytes_written 128254256777
> STAT limit_maxbytes 2147483648
> END
>
>
>
> stats items
> STAT items:7:number 212992
> STAT items:7:age 884240
> STAT items:8:number 8192
> STAT items:8:age 1279441
> STAT items:9:number 147456
> STAT items:9:age 662595
> STAT items:10:number 88064
> STAT items:10:age 770302
> STAT items:11:number 29183
> STAT items:11:age 768640
> STAT items:12:number 44799
> STAT items:12:age 877318
> STAT items:13:number 53120
> STAT items:13:age 756741
> STAT items:14:number 10432
> STAT items:14:age 932763
> STAT items:15:number 17184
> STAT items:15:age 1009452
> STAT items:16:number 2384
> STAT items:16:age 626366
> STAT items:17:number 1392
> STAT items:17:age 896439
> STAT items:18:number 444
> STAT items:18:age 888790
> STAT items:19:number 60
> STAT items:19:age 894845
> STAT items:20:number 51
> STAT items:20:age 445419
> END
>
> stats slabs
> STAT 7:chunk_size 128
> STAT 7:chunks_per_page 8192
> STAT 7:total_pages 26
> STAT 7:total_chunks 212992
> STAT 7:used_chunks 212992
> STAT 7:free_chunks 0
> STAT 7:free_chunks_end 0
> STAT 8:chunk_size 256
> STAT 8:chunks_per_page 4096
> STAT 8:total_pages 2
> STAT 8:total_chunks 8192
> STAT 8:used_chunks 8192
> STAT 8:free_chunks 0
> STAT 8:free_chunks_end 0
> STAT 9:chunk_size 512
> STAT 9:chunks_per_page 2048
> STAT 9:total_pages 72
> STAT 9:total_chunks 147456
> STAT 9:used_chunks 147456
> STAT 9:free_chunks 0
> STAT 9:free_chunks_end 0
> STAT 10:chunk_size 1024
> STAT 10:chunks_per_page 1024
> STAT 10:total_pages 86
> STAT 10:total_chunks 88064
> STAT 10:used_chunks 88064
> STAT 10:free_chunks 0
> STAT 10:free_chunks_end 0
> STAT 11:chunk_size 2048
> STAT 11:chunks_per_page 512
> STAT 11:total_pages 57
> STAT 11:total_chunks 29184
> STAT 11:used_chunks 29183
> STAT 11:free_chunks 1
> STAT 11:free_chunks_end 0
> STAT 12:chunk_size 4096
> STAT 12:chunks_per_page 256
> STAT 12:total_pages 175
> STAT 12:total_chunks 44800
> STAT 12:used_chunks 44799
> STAT 12:free_chunks 1
> STAT 12:free_chunks_end 0
> STAT 13:chunk_size 8192
> STAT 13:chunks_per_page 128
> STAT 13:total_pages 415
> STAT 13:total_chunks 53120
> STAT 13:used_chunks 53119
> STAT 13:free_chunks 1
> STAT 13:free_chunks_end 0
> STAT 14:chunk_size 16384
> STAT 14:chunks_per_page 64
> STAT 14:total_pages 163
> STAT 14:total_chunks 10432
> STAT 14:used_chunks 10432
> STAT 14:free_chunks 0
> STAT 14:free_chunks_end 0
> STAT 15:chunk_size 32768
> STAT 15:chunks_per_page 32
> STAT 15:total_pages 537
> STAT 15:total_chunks 17184
> STAT 15:used_chunks 17184
> STAT 15:free_chunks 0
> STAT 15:free_chunks_end 0
> STAT 16:chunk_size 65536
> STAT 16:chunks_per_page 16
> STAT 16:total_pages 149
> STAT 16:total_chunks 2384
> STAT 16:used_chunks 2384
> STAT 16:free_chunks 0
> STAT 16:free_chunks_end 0
> STAT 17:chunk_size 131072
> STAT 17:chunks_per_page 8
> STAT 17:total_pages 174
> STAT 17:total_chunks 1392
> STAT 17:used_chunks 1392
> STAT 17:free_chunks 0
> STAT 17:free_chunks_end 0
> STAT 18:chunk_size 262144
> STAT 18:chunks_per_page 4
> STAT 18:total_pages 111
> STAT 18:total_chunks 444
> STAT 18:used_chunks 444
> STAT 18:free_chunks 0
> STAT 18:free_chunks_end 0
> STAT 19:chunk_size 524288
> STAT 19:chunks_per_page 2
> STAT 19:total_pages 30
> STAT 19:total_chunks 60
> STAT 19:used_chunks 60
> STAT 19:free_chunks 0
> STAT 19:free_chunks_end 0
> STAT 20:chunk_size 1048576
> STAT 20:chunks_per_page 1
> STAT 20:total_pages 51
> STAT 20:total_chunks 51
> STAT 20:used_chunks 51
> STAT 20:free_chunks 0
> STAT 20:free_chunks_end 0
> STAT active_slabs 14
> STAT total_malloced 2147483648
> END
>
>
> --
> Adrian Ziemkowski
> Application Development Manager
> Revelex Corporation
>