Hi Luis,
On 7/1/22 23:23, Luis Chamberlain wrote:
> On Fri, Jul 01, 2022 at 08:40:02PM +0200, Helge Deller wrote:
>> In the kernel image vmlinux.lds.S linker scripts the .altinstructions
>> and __bug_table sections are 32- or 64-bit aligned because they hold 32-
>> and/or 64-bit values.
>>
>> But for modules the module.lds.S linker script doesn't define a default
>> alignment yet, so the linker chooses the default byte alignment, which
>> then leads to unnecessary unaligned memory accesses at runtime.
>>
>> Usually such unaligned accesses are unnoticed, because either the
>> hardware (as on x86 CPUs) or in-kernel exception handlers (e.g. on hppa
>> or sparc) emulate and fix them up at runtime.
>>
>> On hppa the 32-bit unalignment exception handler was temporarily broken
>> due another bad commit, and as such wrong values were returned on
>> unaligned accesses to the altinstructions table.
>
> OK so some bad commit broke something which caused bad alignment access
> on altinstructions... But why on modules?!
>
> I am not aware of modules using alternatives, given that alternatives
> are hacks to help with bootup. For modules we can use other things
> like jump labels, static keys.
IMHO altinstructions isn't a hack.
They are much simpler and easier to use for static replacements.
jump labels and static keys are much more komplex, but of course they
give the possibility to switch back and forth if you need it.
But let's keep this discussion aside...
I checked a few other architectures, and here is what I found.
I dropped unimportant sections/lines.
Linux amdahl 4.19.0-20-arm64 #1 SMP Debian 4.19.235-1 (2022-03-17) aarch64 GNU/Linux
deller@amdahl:/lib/modules/4.19.0-19-arm64/kernel/block$ objdump -h bfq.ko
bfq.ko: file format elf64-littleaarch64
Sections:
Idx Name Size VMA LMA File off Algn
6 .altinstructions 000000b4 0000000000000000 0000000000000000 000090a4 2**0
CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
13 __jump_table 00000018 0000000000000000 0000000000000000 0000d358 2**3
CONTENTS, ALLOC, LOAD, RELOC, DATA
15 __bug_table 00000018 0000000000000000 0000000000000000 0000dcf8 2**2
CONTENTS, ALLOC, LOAD, RELOC, DATA
-> aarch64 uses altinstructions in modules as well.
-> alignment of altinstructions is wrong (but offset suggests it gets address right).
-> jump_table/bug_table -> Ok.
----
Linux abel 4.19.0-20-armmp-lpae #1 SMP Debian 4.19.235-1 (2022-03-17) armv7l GNU/Linux
deller@abel:/lib/modules/4.19.0-20-armmp-lpae/kernel/block$ objdump -h bfq.ko
bfq.ko: file format elf32-littlearm
Sections:
Idx Name Size VMA LMA File off Algn
9 __mcount_loc 000002ac 00000000 00000000 00009bf4 2**2
CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
11 __jump_table 0000000c 00000000 00000000 0000b320 2**3
CONTENTS, ALLOC, LOAD, RELOC, DATA
-> arm looks good.
----
Linux plummer 4.19.0-20-powerpc64le #1 SMP Debian 4.19.235-1 (2022-03-17) ppc64le GNU/Linux
deller@plummer:/lib/modules/4.19.0-20-powerpc64le/kernel/block$ objdump -h bfq.ko
bfq.ko: file format elf64-powerpcle
Sections:
Idx Name Size VMA LMA File off Algn
9 __mcount_loc 00000530 0000000000000000 0000000000000000 0000bc68 2**0
CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
12 __jump_table 00000018 0000000000000000 0000000000000000 000109d8 2**3
CONTENTS, ALLOC, LOAD, RELOC, DATA
16 __bug_table 00000030 0000000000000000 0000000000000000 000115a0 2**0
CONTENTS, ALLOC, LOAD, RELOC, DATA
-> ppc64le has wrong alignment for mcount_loc and bug_table (but file offs suggests it's correct).
----
Linux zelenka 4.19.0-20-s390x #1 SMP Debian 4.19.235-1 (2022-03-17) s390x GNU/Linux
deller@zelenka:/lib/modules/4.19.0-20-s390x/kernel/block$ objdump -h bfq.ko
bfq.ko: file format elf64-s390
Sections:
Idx Name Size VMA LMA File off Algn
3 .altinstr_replacement 00000038 0000000000000000 0000000000000000 0000a440 2**0
CONTENTS, ALLOC, LOAD, READONLY, CODE
8 .altinstructions 000000a8 0000000000000000 0000000000000000 0000b04e 2**0
CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
10 __mcount_loc 00000538 0000000000000000 0000000000000000 0000b1b0 2**3
CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
13 __jump_table 00000018 0000000000000000 0000000000000000 0000c8e0 2**3
CONTENTS, ALLOC, LOAD, RELOC, DATA
17 __bug_table 00000018 0000000000000000 0000000000000000 0000d280 2**0
CONTENTS, ALLOC, LOAD, RELOC, DATA
-> s390x uses altinstructions in modules.
-> alignment should be fixed for altinstructions and bug_table
> So I don't understand still how this happened yet.
Happened what?
Even on x86 there is a call to apply_alternatives() in module_finalize() in
arch/x86/kernel/module.c.
I didn't found alternatives in amd64 modules in kernel 4.19 though...
>> This then led to
>> undefined behaviour because wrong kernel addresses were patched and we
>> suddenly faced lots of unrelated bugs, as can be seen in this mail
>> thread:
>> https://lore.kernel.org/all/07d91863-dacc-a503-aa2b-05c3b92a1e39@xxxxxxxx/T/#mab602dfa32be5e229d5e192ab012af196d04d75d
>>
>> This patch adds the missing natural alignment for kernel modules to
>> avoid unnecessary (hard- or software-based) fixups.
>
> Is it correct to infer that issue you found through a bad commit was
> then through code inspection after the bad commit made the kernel do
> something stupid with unaligned access to some module altinstructions
> section ? Ie, that should not have happened.
Right. Without the bad commit I would not have noticed the problem.
> I'd like to determine if this is a stable fix, a regression, etc. And
> this is not yet clear.
I fully understand that it's a hard to decide if it should go to stable!
It's not critical or required to go to stable series now.
My suggestion:
Add it to current head, wait for 1-2 releases, and *if required* we can
push it backwards at any time later.
Helge
> Luis
>
>>
>> Signed-off-by: Helge Deller <deller@xxxxxx>
>> ---
>> scripts/module.lds.S | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> --
>> v2: updated commit message
>>
>> diff --git a/scripts/module.lds.S b/scripts/module.lds.S
>> index 1d0e1e4dc3d2..3a3aa2354ed8 100644
>> --- a/scripts/module.lds.S
>> +++ b/scripts/module.lds.S
>> @@ -27,6 +27,8 @@ SECTIONS {
>> .ctors 0 : ALIGN(8) { *(SORT(.ctors.*)) *(.ctors) }
>> .init_array 0 : ALIGN(8) { *(SORT(.init_array.*)) *(.init_array) }
>>
>> + .altinstructions 0 : ALIGN(8) { KEEP(*(.altinstructions)) }
>> + __bug_table 0 : ALIGN(8) { KEEP(*(__bug_table)) }
>> __jump_table 0 : ALIGN(8) { KEEP(*(__jump_table)) }
>>
>> __patchable_function_entries : { *(__patchable_function_entries) }
