Large memory support
Jason Titus
jtitus@postini.com
Wed, 18 Feb 2004 23:39:38 -0800
Seems like there is much confusion on the zie of a unsigned int w/ =
regard to 32 bit/64 bit systems. But we found that on RHEL 3 AMD64, w/ =
gcc 3.2.3 20030502 the unsigned ints are 32 bits (the same as RHEL 3 =
x86). Other folks seem to have stumbled upon the same info:
"> I see you have a background in environments where you move between =
16-=20
> and 32-bit machines. Guess what, in Linux the major movement is=20
> between 32- and 64-bit machines, and "unsigned int" is consistent,=20
> whereas "unsigned long" isn't (long is 32 bits on 32-bit machines, 64=20
> bits on 64-bit machines.)=20
"
http://www.ussg.iu.edu/hypermail/linux/kernel/0112.1/0186.html
I'd propose making it clearer and using size_t for the memory size =
related values. From Sun's web site =
(http://developers.sun.com/solaris/articles/solarisupgrade/64bit/Convert.=
html) :
"
Derived Types
Using the system derived types helps make code 32-bit and 64-bit safe, =
since the derived types themselves must be safe for both the ILP32 and =
LP64 data models. In general, using derived types to allow for change is =
good programming practice. Should the data model change in the future, =
or when porting to a different platform, only the system derived types =
need to change rather than the application.
The system include files <sys/types.h> and <inttypes.h>, which contain =
constants, macros, and derived types that are helpful in making =
applications 32-bit and 64-bit safe.
<sys/types.h>
An application source file that includes <sys/types.h> makes the =
definitions of _LP64 and _ILP32 available through inclusion of =
<sys/isa_defs.h>. This header also contains a number of basic derived =
types that should be used whenever appropriate. In particular, the =
following are of special interest:
Type Purpose
clock_t - Represents the system times in clock ticks.=A0
dev_t - Used for device numbers.=A0=A0
off_t - Used for file sizes and offsets.=A0
ptrdiff_t - The signed integral type for the result of subtracting two =
pointers.=A0
size_t - The size, in bytes, of objects in memory.=A0
ssize_t - Used by functions that return a count of bytes or an error =
indication.
time_t - Used for time in seconds.=A0
"
Here is a first pass diff:
[linux]$ diff memcached-bigmem-1.1.10/ memcached-1.1.10
diff memcached-bigmem-1.1.10/memcached.c memcached-1.1.10/memcached.c
339c339
< pos +=3D sprintf(pos, "STAT limit_maxbytes %llu\r\n", =
settings.maxbytes);
---
> pos +=3D sprintf(pos, "STAT limit_maxbytes %u\r\n", =
settings.maxbytes);
1282c1282
< settings.maxbytes =3D (size_t) atoi(optarg)* (size_t) =
1024* (size_t) 1024;
---
> settings.maxbytes =3D atoi(optarg)*1024*1024;
diff memcached-bigmem-1.1.10/memcached.h memcached-1.1.10/memcached.h
13c13
< size_t curr_bytes;
---
> unsigned long long curr_bytes;
27c27
< size_t maxbytes;
---
> unsigned int maxbytes;
150c150
< void slabs_init(size_t limit);
---
> void slabs_init(unsigned int limit);
154c154
< unsigned int slabs_clsid(size_t size);
---
> unsigned int slabs_clsid(unsigned int size);
157c157
< void *slabs_alloc(size_t size);
---
> void *slabs_alloc(unsigned int size);
160c160
< void slabs_free(void *ptr, size_t size);
---
> void slabs_free(void *ptr, unsigned int size);
diff memcached-bigmem-1.1.10/slabs.c memcached-1.1.10/slabs.c
52,53c52,53
< static size_t mem_limit =3D 0;
< static size_t mem_malloced =3D 0;
---
> static unsigned int mem_limit =3D 0;
> static unsigned int mem_malloced =3D 0;
55c55
< unsigned int slabs_clsid(size_t size) {
---
> unsigned int slabs_clsid(unsigned int size) {
70c70
< void slabs_init(size_t limit) {
---
> void slabs_init(unsigned int limit) {
91c91
< size_t new_size =3D p->list_size ? p->list_size * 2 : 16;
---
> unsigned int new_size =3D p->list_size ? p->list_size * 2 : =
16;
123c123
< void *slabs_alloc(size_t size) {
---
> void *slabs_alloc(unsigned int size) {
163c163
< void slabs_free(void *ptr, size_t size) {
---
> void slabs_free(void *ptr, unsigned int size) {
Not sure if all of the changes are necessary, but combined they seem to =
work. I just changed all of the areas that seemed to deal with memory =
limits.
Jason
-----Original Message-----
From: Brad Fitzpatrick [mailto:brad@danga.com]
Sent: Wed 2/18/2004 9:52 PM
To: Jason Titus
Cc: memcached@lists.danga.com
Subject: RE: Large memory support
=20
Jason,
I have access to an Opteron right now, but I haven't done extensive
testing on it past building it. I'll try and work on that.
What would help, though, is if you could get me:
-- a test script to fill up memcached with 5GB of data. (or hell, this =
is
like 10 lines, so I could do it)
-- a patch to change things to size_t (or void* or whatever's =
appropriate)
Then I'll test it before/after and review other changes and get it
committed. I don't want to commit without personally testing, though.
So unsigned int is really just 32-bits on 64 bit archs? Can anybody =
point
me at a reference to what types map to on different archs?
- Brad
On Wed, 18 Feb 2004, Jason Titus wrote:
> Actually, this was on RedHat Enterprise Linux 64 on a dual Opteron =
system. No matter what I entered as a memory size, it listed 2GB as =
maxbytes in STATS. When I added more records after it reached 1.9GB it =
would start tossing things.
>
> When I replaced the unsigned ints with unsigned long longs, it would =
grow past 2GB.
>
> Seems like switching to size_t would be the best path. What do you =
think?
>
> Jason
>
> p.s. - Our goal is to have a ~5GB cache with an entire database. We =
want to disable expiration (i.e. nothing getting removed out of the =
cache) and have the cache be the definitive read only source. It would =
have a syncing process to keep it up to date with any changes.
>
> -----Original Message-----
> From: Brad Fitzpatrick [mailto:brad@danga.com]
> Sent: Wed 2/18/2004 9:37 PM
> To: Jason Titus
> Cc: memcached@lists.danga.com
> Subject: Re: Large memory support
>
> Jason,
>
> I've built and run memcached on 64-bit machines.
>
> On a 32-bit machine you won't be able to store more memory than your
> operating system is giving you address space for. (Not sure if
> your OS is giving you a 2G/2G, 3G/1G, or 3.5G/0.5G split)
>
> I assume you're on 32-bit? Otherwise, isn't an "unsigned long" just a
> uint64 on 64-bi machines? Or should we be using void* somewhere we're
> using unsigned int?
>
> Let me know your OS, arch, and the contents of "stats maps" (if on =
Linux):
>
> $ telnet localhost 11211
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> stats maps
> 08048000-08050000 r-xp 00000000 03:03 2167117 =
/usr/local/bin/memcached-1.1.10pre2
> 08050000-08051000 rw-p 00008000 03:03 2167117 =
/usr/local/bin/memcached-1.1.10pre2
> 08051000-09c02000 rwxp 00000000 00:00 0
> 40000000-40011000 r-xp 00000000 03:03 165006 /lib/ld-2.3.1.so
> 40011000-40012000 rw-p 00011000 03:03 165006 /lib/ld-2.3.1.so
> 40017000-4011f000 r-xp 00000000 03:03 165009 /lib/libc-2.3.1.so
> 4011f000-40125000 rw-p 00107000 03:03 165009 /lib/libc-2.3.1.so
> 40125000-40128000 rw-p 00000000 00:00 0
> 40128000-40131000 r-xp 00000000 03:03 165014 =
/lib/libnss_compat-2.3.1.so
> 40131000-40132000 rw-p 00009000 03:03 165014 =
/lib/libnss_compat-2.3.1.so
> 40132000-40142000 r-xp 00000000 03:03 165013 /lib/libnsl-2.3.1.so
> 40142000-40143000 rw-p 00010000 03:03 165013 /lib/libnsl-2.3.1.so
> 40143000-80946000 rw-p 00000000 00:00 0
> bfffc000-c0000000 rwxp ffffd000 00:00 0
> END
>
>
> - Brad
>
>
> On Wed, 18 Feb 2004, Jason Titus wrote:
>
> > Looks like we just stumbled onto some 32 bit limits in memcached. =
Just
> > wondering how you guys want to deal with supporting >2GB per cache. =
We
> > have experimented with switching some of the unsigned ints into =
unsigned
> > long longs and it seemed to work (>4GB seemed ok. No thorough =
testing
> > yet though). A cleaner solution might be adjusting the memory =
related
> > variables to size_t or some such so that they work well on 32 and 64 =
bit
> > systems.
> >
> > What makes the most sense to you folks?
> >
> > Jason
> >
> >
>
>