Re: Changing Kernel 3g / 1g memory split

[Chetan Nanda <chetannanda@xxxxxxxxx>]

On Thu, Oct 8, 2009 at 2:23 PM, Rajat Jain <Rajat.Jain@xxxxxxxxxxxx> wrote:

Hi,

>>> 3) And finally, in order to complete this change and to ensure my
> >>> applications are awrae of it and are doing the needful to restrict
> >>> themselves in only 2G, do I also need to make changes in
> >> the glibc or
> >>> the gcc?
> >>
> >> No, you don't need changes to glibc or gcc.
> >>
> >
> > I think he will need to. In the ELF user space executables
> in most of
> > the cases, the address at which each synbol (function / data) will
> > go is
> > generally fixed at compile time. So I think mosty of the
> > applications in
> > user space will need a recomile with a compiler that is
> aware of the
> > new
> > 2G/2G split.
> >
>
> Interesting. I've previously used 2G/2G split (some years
> ago, before
> 64b was prevalent) quite a bit and never needed to recompile
> existing
> binaries and libraries on the distro, and never had issues. Perhaps
> the default addresses are quite low (well below 2G)?

I think so. See below:

[rajat@linux-server module]$ cat t.c
int main()
{
       for(;;);
       return 0;
}
[rajat@linux-server module]$ gcc t.c
[rajat@linux-server module]$ ./a.out &
[1] 18189
[rajat@linux-server module]$ readelf -S a.out | grep "test\|data\|bss"
 [14] .rodata           PROGBITS        0804842c 00042c 000008 00   A
0   0  4
 [22] .data             PROGBITS        08049528 000528 00000c 00  WA
0   0  4
 [23] .bss              NOBITS          08049534 000534 000004 00  WA
0   0  4

The above the virtual addresses of the code and data sections (which are
well under 2G).

[rajat@linux-server module]$
[rajat@linux-server module]$ cat /proc/18189/maps |grep "a.out"
08048000-08049000 r-xp 00000000 00:13 1594602
/netapp_filer/users/rajat/linux/module/a.out
08049000-0804a000 rw-p 00000000 00:13 1594602
/netapp_filer/users/rajat/linux/module/a.out

The same memory pages have been mapped into user application as evident
above. Also, all the symbols in the application are well below 2G:

[rajat@linux-server module]$ readelf -s a.out

Symbol table '.dynsym' contains 5 entries:
  Num:    Value  Size Type    Bind   Vis      Ndx Name
    0: 00000000     0 NOTYPE  LOCAL  DEFAULT  UND
    1: 00000000   239 FUNC    GLOBAL DEFAULT  UND
__libc_start_main@xxxxxxxxx (2)
    2: 08048430     4 OBJECT  GLOBAL DEFAULT   14 _IO_stdin_used
    3: 00000000     0 NOTYPE  WEAK   DEFAULT  UND _Jv_RegisterClasses
    4: 00000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__

Symbol table '.symtab' contains 69 entries:
  Num:    Value  Size Type    Bind   Vis      Ndx Name
    0: 00000000     0 NOTYPE  LOCAL  DEFAULT  UND
    1: 08048114     0 SECTION LOCAL  DEFAULT    1
    2: 08048128     0 SECTION LOCAL  DEFAULT    2
    3: 08048148     0 SECTION LOCAL  DEFAULT    3
    4: 08048170     0 SECTION LOCAL  DEFAULT    4
    5: 080481c0     0 SECTION LOCAL  DEFAULT    5
    6: 0804821a     0 SECTION LOCAL  DEFAULT    6
    7: 08048224     0 SECTION LOCAL  DEFAULT    7
    8: 08048244     0 SECTION LOCAL  DEFAULT    8
    9: 0804824c     0 SECTION LOCAL  DEFAULT    9
   10: 08048254     0 SECTION LOCAL  DEFAULT   10
   11: 0804826c     0 SECTION LOCAL  DEFAULT   11
   12: 0804828c     0 SECTION LOCAL  DEFAULT   12
   13: 08048410     0 SECTION LOCAL  DEFAULT   13
   14: 0804842c     0 SECTION LOCAL  DEFAULT   14
   15: 08048434     0 SECTION LOCAL  DEFAULT   15
   16: 08049438     0 SECTION LOCAL  DEFAULT   16
   17: 08049440     0 SECTION LOCAL  DEFAULT   17
   18: 08049448     0 SECTION LOCAL  DEFAULT   18
   19: 0804944c     0 SECTION LOCAL  DEFAULT   19
   20: 08049514     0 SECTION LOCAL  DEFAULT   20
   21: 08049518     0 SECTION LOCAL  DEFAULT   21
   22: 08049528     0 SECTION LOCAL  DEFAULT   22
   23: 08049534     0 SECTION LOCAL  DEFAULT   23
   24: 00000000     0 SECTION LOCAL  DEFAULT   24
   25: 00000000     0 SECTION LOCAL  DEFAULT   25
   26: 00000000     0 SECTION LOCAL  DEFAULT   26
   27: 00000000     0 SECTION LOCAL  DEFAULT   27
   28: 080482b0     0 FUNC    LOCAL  DEFAULT   12 call_gmon_start
   29: 00000000     0 FILE    LOCAL  DEFAULT  ABS crtstuff.c
   30: 08049438     0 OBJECT  LOCAL  DEFAULT   16 __CTOR_LIST__
   31: 08049440     0 OBJECT  LOCAL  DEFAULT   17 __DTOR_LIST__
   32: 08049448     0 OBJECT  LOCAL  DEFAULT   18 __JCR_LIST__
   33: 08049530     0 OBJECT  LOCAL  DEFAULT   22 p.0
   34: 08049534     1 OBJECT  LOCAL  DEFAULT   23 completed.1
   35: 080482d4     0 FUNC    LOCAL  DEFAULT   12 __do_global_dtors_aux
   36: 08048308     0 FUNC    LOCAL  DEFAULT   12 frame_dummy
   37: 00000000     0 FILE    LOCAL  DEFAULT  ABS crtstuff.c
   38: 0804943c     0 OBJECT  LOCAL  DEFAULT   16 __CTOR_END__
   39: 08049444     0 OBJECT  LOCAL  DEFAULT   17 __DTOR_END__
   40: 08048434     0 OBJECT  LOCAL  DEFAULT   15 __FRAME_END__
   41: 08049448     0 OBJECT  LOCAL  DEFAULT   18 __JCR_END__
   42: 080483ec     0 FUNC    LOCAL  DEFAULT   12 __do_global_ctors_aux
   43: 00000000     0 FILE    LOCAL  DEFAULT  ABS t.c
   44: 0804944c     0 OBJECT  GLOBAL DEFAULT   19 _DYNAMIC
   45: 0804842c     4 OBJECT  GLOBAL DEFAULT   14 _fp_hw
   46: 08049438     0 NOTYPE  GLOBAL HIDDEN  ABS __fini_array_end
   47: 0804952c     0 OBJECT  GLOBAL HIDDEN   22 __dso_handle
   48: 080483a8    66 FUNC    GLOBAL DEFAULT   12 __libc_csu_fini
   49: 08048254     0 FUNC    GLOBAL DEFAULT   10 _init
   50: 0804828c     0 FUNC    GLOBAL DEFAULT   12 _start
   51: 08049438     0 NOTYPE  GLOBAL HIDDEN  ABS __fini_array_start
   52: 08048354    82 FUNC    GLOBAL DEFAULT   12 __libc_csu_init
   53: 08049534     0 NOTYPE  GLOBAL DEFAULT  ABS __bss_start
   54: 08048334    30 FUNC    GLOBAL DEFAULT   12 main
   55: 00000000   239 FUNC    GLOBAL DEFAULT  UND
__libc_start_main@@GLIBC_
   56: 08049438     0 NOTYPE  GLOBAL HIDDEN  ABS __init_array_end
   57: 08049528     0 NOTYPE  WEAK   DEFAULT   22 data_start
   58: 08048410     0 FUNC    GLOBAL DEFAULT   13 _fini
   59: 08049438     0 NOTYPE  GLOBAL HIDDEN  ABS __preinit_array_end
   60: 08049534     0 NOTYPE  GLOBAL DEFAULT  ABS _edata
   61: 08049518     0 OBJECT  GLOBAL DEFAULT   21 _GLOBAL_OFFSET_TABLE_
   62: 08049538     0 NOTYPE  GLOBAL DEFAULT  ABS _end
   63: 08049438     0 NOTYPE  GLOBAL HIDDEN  ABS __init_array_start
   64: 08048430     4 OBJECT  GLOBAL DEFAULT   14 _IO_stdin_used
   65: 08049528     0 NOTYPE  GLOBAL DEFAULT   22 __data_start
   66: 00000000     0 NOTYPE  WEAK   DEFAULT  UND _Jv_RegisterClasses
   67: 08049438     0 NOTYPE  GLOBAL HIDDEN  ABS __preinit_array_start
   68: 00000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__

Even the libc and dynamic loader have been mapped below 2G:

[rajat@linux-server module]$ cat /proc/18189/maps
0062d000-00642000 r-xp 00000000 fd:00 4432591    /lib/ld-2.3.4.so
00642000-00643000 r--p 00014000 fd:00 4432591    /lib/ld-2.3.4.so
00643000-00644000 rw-p 00015000 fd:00 4432591    /lib/ld-2.3.4.so
0064a000-0076d000 r-xp 00000000 fd:00 4432592    /lib/tls/libc-2.3.4.so
0076d000-0076e000 r--p 00123000 fd:00 4432592    /lib/tls/libc-2.3.4.so
0076e000-00771000 rw-p 00124000 fd:00 4432592    /lib/tls/libc-2.3.4.so
00771000-00773000 rw-p 00771000 00:00 0
08048000-08049000 r-xp 00000000 00:13 1594602
/netapp_filer/users/rajat/linux/module/.nfs001854ea00000002
08049000-0804a000 rw-p 00000000 00:13 1594602
/netapp_filer/users/rajat/linux/module/.nfs001854ea00000002
b7fe2000-b7fe3000 rw-p b7fe2000 00:00 0
bffd2000-c0000000 rw-p bffd2000 00:00 0
ffffe000-fffff000 ---p 00000000 00:00 0

Only the last three entries are suspicious however, but since their size
is zero, I assume there are special zero sized mapping never used by the
application.

the one-page anonymous memory region from 0xffffe000 contains the vsyscall page of the process .

bffd2000-c0000000 user mode stack in 2G/2G split it  user space stack will start at 0x80000000
'b7fe2000-b7fe3000' not sure about this probably a region used by library.

Also point to note here that two type of memory region layout being implemented in kernel and controlled by '/proc /sys/vm/legacy_va_layout'
If this variable is set to 0 then dynamic libraries will mapped to higher regions just after user mode stack.

So probably in your system it has been set to '0' (can confirm using cat /proc /sys/vm/legacy_va_layout)
 
 

> Also, how does
> ASLR ( http://en.wikipedia.org/wiki/
> Address_space_layout_randomization ) play into this?
>

No idea. Is this currently implemented into the toolchain / binutils /
ld?

AFAIK, Address space randomization is kernel feature  in which starting address of user mode stack address is not fixed at 0xc0000000 (in 3G/1G split) but calculated randomly by subtracting some random value from 0xc0000000, correct me if am wrong

Thanks,

Rajat

--
To unsubscribe from this list: send an email with
"unsubscribe kernelnewbies" to ecartis@xxxxxxxxxxxx
Please read the FAQ at http://kernelnewbies.org/FAQ

Thanks,
Chetan Nanda