On Sat, Jan 23, 2021 at 02:59:28PM -0800, Fangrui Song wrote: > On 2020-08-28, Josh Poimboeuf wrote: > > On Fri, Aug 28, 2020 at 12:21:13PM +0200, Miroslav Benes wrote: > > > > Hi there! I was trying to find a super easy way to address this, so I > > > > thought the best thing would be if there were a compiler or linker > > > > switch to just eliminate any duplicate symbols at compile time for > > > > vmlinux. I filed this question on the binutils bugzilla looking to see > > > > if there were existing flags that might do this, but H.J. Lu went ahead > > > > and created a new one "-z unique", that seems to do what we would need > > > > it to do. > > > > > > > > https://sourceware.org/bugzilla/show_bug.cgi?id=26391 > > > > > > > > When I use this option, it renames any duplicate symbols with an > > > > extension - for example duplicatefunc.1 or duplicatefunc.2. You could > > > > either match on the full unique name of the specific binary you are > > > > trying to patch, or you match the base name and use the extension to > > > > determine original position. Do you think this solution would work? > > > > > > Yes, I think so (thanks, Joe, for testing!). > > > > > > It looks cleaner to me than the options above, but it may just be a matter > > > of taste. Anyway, I'd go with full name matching, because -z unique-symbol > > > would allow us to remove sympos altogether, which is appealing. > > > > > > > If > > > > so, I can modify livepatch to refuse to patch on duplicated symbols if > > > > CONFIG_FG_KASLR and when this option is merged into the tool chain I > > > > can add it to KBUILD_LDFLAGS when CONFIG_FG_KASLR and livepatching > > > > should work in all cases. > > > > > > Ok. > > > > > > Josh, Petr, would this work for you too? > > > > Sounds good to me. Kristen, thanks for finding a solution! > > (I am not subscribed. I came here via https://sourceware.org/bugzilla/show_bug.cgi?id=26391 (ld -z unique-symbol)) > > > This works great after randomization because it always receives the > > current address at runtime rather than relying on any kind of > > buildtime address. The issue with with the live-patching code's > > algorithm for resolving duplicate symbol names. If they request a > > symbol by name from the kernel and there are 3 symbols with the same > > name, they use the symbol's position in the built binary image to > > select the correct symbol. > > If a.o, b.o and c.o define local symbol 'foo'. > By position, do you mean that > > * the live-patching code uses something like (findall("foo")[0], findall("foo")[1], findall("foo")[2]) ? Yes, it depends on their order in the symbol table of the linked binary (vmlinux). > * shuffling a.o/b.o/c.o will make the returned triple different Yes, though it's actually functions that get shuffled. > Local symbols are not required to be unique. Instead of patching the toolchain, > have you thought about making the live-patching code smarter? It's a possibility (more on that below). > (Depend on the duplicates, such a linker option can increase the link time/binary size considerably Have you tried it on vmlinux? Just wondering what the time/size impact would be in real-world numbers. Duplicate symbols make up a very small percentage of all symbols in the kernel, so I would think the binary size change (to the strtab?) would be insignificant? > AND I don't know in what other cases such an option will be useful) I believe some other kernel components (tracing, kprobes, bpf) have the same problem as livepatch with respect to disambiguating duplicate symbols, for the purposes of tracing/debugging. So I'm thinking it would be a nice overall improvement to the kernel. > For the following example, > > https://sourceware.org/bugzilla/show_bug.cgi?id=26822 > > # RUN: split-file %s %t > # RUN: gcc -c %t/a.s -o %t/a.o > # RUN: gcc -c %t/b.s -o %t/b.o > # RUN: gcc -c %t/c.s -o %t/c.o > # RUN: ld-new %t/a.o %t/b.o %t/c.o -z unique-symbol -o %t.exe > #--- a.s > a: a.1: a.2: nop > #--- b.s > a: nop > #--- c.s > a: nop > > readelf -Ws output: > > Symbol table '.symtab' contains 13 entries: > Num: Value Size Type Bind Vis Ndx Name > 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND 1: > 0000000000000000 0 FILE LOCAL DEFAULT ABS a.o > 2: 0000000000401000 0 NOTYPE LOCAL DEFAULT 1 a > 3: 0000000000401000 0 NOTYPE LOCAL DEFAULT 1 a.1 > 4: 0000000000401000 0 NOTYPE LOCAL DEFAULT 1 a.2 > 5: 0000000000000000 0 FILE LOCAL DEFAULT ABS b.o > 6: 0000000000401001 0 NOTYPE LOCAL DEFAULT 1 a.1 > 7: 0000000000000000 0 FILE LOCAL DEFAULT ABS c.o > 8: 0000000000401002 0 NOTYPE LOCAL DEFAULT 1 a.2 > 9: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND _start > 10: 0000000000402000 0 NOTYPE GLOBAL DEFAULT 1 __bss_start > 11: 0000000000402000 0 NOTYPE GLOBAL DEFAULT 1 _edata > 12: 0000000000402000 0 NOTYPE GLOBAL DEFAULT 1 _end > > Note that you have STT_FILE SHN_ABS symbols. > If the compiler does not produce them, they will be synthesized by GNU ld. > > https://sourceware.org/bugzilla/show_bug.cgi?id=26822 > ld.bfd copies non-STT_SECTION local symbols from input object files. If an > object file does not have STT_FILE symbols (no .file directive) but has > non-STT_SECTION local symbols, ld.bfd synthesizes a STT_FILE symbol Right, I see what you're getting at. As far as I can tell, there are potentially two ways for fgkaslr to handle this: a) shuffle files, not functions. i.e. keep the functions' order intact within the STT_FILE group, shuffling the file groups themselves. (NOTE: this may have an additional benefit of improving i-cache performance, compared to the current fgkaslr implementation.) or b) shuffle functions, keeping track of what file they belonged to. Maybe Kristen could comment on the feasibility of either of these options. I believe the STT_FILE symbols are not currently available to the kernel at runtime. They would need to be made available to both fgkaslr and livepatch code. Overall "ld -z unique-symbol" would be much easier from a kernel standpoint, and would benefit multiple components as I mentioned above. > The filenames are usually base names, so "a.o" and "a.o" in two directories will > be indistinguishable. The live-patching code can possibly work around this by > not changing the relative order of the two "a.o". Right, there are some file:func duplicates so this case would indeed need to be handled somehow. $ readelf -s --wide vmlinux |awk '$4 == "FILE" {file=$8; next} $4 == "FUNC" {printf "%s:%s\n", file, $8}' |sort |uniq -d bus.c:new_id_store core.c:cmask_show core.c:edge_show core.c:event_show core.c:inv_show core.c:paravirt_read_msr core.c:paravirt_read_msr_safe core.c:type_show core.c:umask_show hid-core.c:hid_exit hid-core.c:hid_init inode.c:init_once inode.c:remove_one msr.c:msr_init proc.c:c_next proc.c:c_start proc.c:c_stop raw.c:dst_output raw.c:raw_ioctl route.c:dst_discard super.c:init_once udp.c:udp_lib_close udp.c:udp_lib_hash udp.c:udplite_getfrag udplite.c:udp_lib_close udplite.c:udp_lib_hash udplite.c:udplite_sk_init -- Josh
