On Mon, Jun 26, 2023 at 7:15 AM Conor Dooley <conor.dooley@xxxxxxxxxxxxx> wrote:
>
> Hey Evan,
>
> Some minor nitpickery comments & one actual one.
>
> On Fri, Jun 23, 2023 at 03:20:15PM -0700, Evan Green wrote:
> > Rather than deferring misaligned access speed determinations to a vendor
>
> Could you pick a consistent word to use? You've got "unaligned",
> "misaligned" and "noalign" sprinkled through out the series.
>
> > function, let's probe them and find out how fast they are. If we
> > determine that a misaligned word access is faster than N byte accesses,
> > mark the hardware's misaligned access as "fast".
> >
> > Fix the documentation as well to reflect this bar. Previously the only
> > SoC that returned "fast" was the THead C906. The change to the
> > documentation is more a clarification, since the C906 is fast in the
> > sense of the corrected documentation.
> >
> > Signed-off-by: Evan Green <evan@xxxxxxxxxxxx>
> > ---
> >
> > Documentation/riscv/hwprobe.rst | 8 +--
> > arch/riscv/include/asm/cpufeature.h | 2 +
> > arch/riscv/kernel/Makefile | 1 +
> > arch/riscv/kernel/copy-noalign.S | 71 ++++++++++++++++++++++++++
> > arch/riscv/kernel/copy-noalign.h | 13 +++++
> > arch/riscv/kernel/cpufeature.c | 78 +++++++++++++++++++++++++++++
> > arch/riscv/kernel/smpboot.c | 2 +
> > 7 files changed, 171 insertions(+), 4 deletions(-)
> > create mode 100644 arch/riscv/kernel/copy-noalign.S
> > create mode 100644 arch/riscv/kernel/copy-noalign.h
> >
> > diff --git a/Documentation/riscv/hwprobe.rst b/Documentation/riscv/hwprobe.rst
> > index 19165ebd82ba..710325751766 100644
> > --- a/Documentation/riscv/hwprobe.rst
> > +++ b/Documentation/riscv/hwprobe.rst
> > @@ -88,12 +88,12 @@ The following keys are defined:
> > always extremely slow.
> >
> > * :c:macro:`RISCV_HWPROBE_MISALIGNED_SLOW`: Misaligned accesses are supported
> > - in hardware, but are slower than the cooresponding aligned accesses
> > - sequences.
>
> Nice, fixed the typo by removing the offender ;)
>
> > + in hardware, but are slower than N byte accesses, where N is the native
> > + word size.
> >
> > * :c:macro:`RISCV_HWPROBE_MISALIGNED_FAST`: Misaligned accesses are supported
> > - in hardware and are faster than the cooresponding aligned accesses
> > - sequences.
> > + in hardware and are faster than N byte accesses, where N is the native
> > + word size.
> >
> > * :c:macro:`RISCV_HWPROBE_MISALIGNED_UNSUPPORTED`: Misaligned accesses are
> > not supported at all and will generate a misaligned address fault.
> > diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h
> > index 23fed53b8815..b8e917176616 100644
> > --- a/arch/riscv/include/asm/cpufeature.h
> > +++ b/arch/riscv/include/asm/cpufeature.h
> > @@ -30,4 +30,6 @@ DECLARE_PER_CPU(long, misaligned_access_speed);
> > /* Per-cpu ISA extensions. */
> > extern struct riscv_isainfo hart_isa[NR_CPUS];
> >
> > +void check_misaligned_access(int cpu);
> > +
> > #endif
> > diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile
> > index a42951911067..f934d7ab7840 100644
> > --- a/arch/riscv/kernel/Makefile
> > +++ b/arch/riscv/kernel/Makefile
> > @@ -34,6 +34,7 @@ extra-y += vmlinux.lds
> > obj-y += head.o
> > obj-y += soc.o
> > obj-$(CONFIG_RISCV_ALTERNATIVE) += alternative.o
> > +obj-y += copy-noalign.o
> > obj-y += cpu.o
> > obj-y += cpufeature.o
> > obj-y += entry.o
> > diff --git a/arch/riscv/kernel/copy-noalign.S b/arch/riscv/kernel/copy-noalign.S
> > new file mode 100644
> > index 000000000000..3807fc2324b2
> > --- /dev/null
> > +++ b/arch/riscv/kernel/copy-noalign.S
> > @@ -0,0 +1,71 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/* Copyright (C) 2023 Rivos Inc. */
> > +
> > +#include <linux/linkage.h>
> > +#include <asm/asm.h>
> > +
> > + .text
> > +
> > +/* void __copy_words_unaligned(void *, const void *, size_t) */
> > +/* Performs a memcpy without aligning buffers, using word loads and stores. */
> > +/* Note: The size is truncated to a multiple of 8 * SZREG */
> > +ENTRY(__copy_words_unaligned)
> > + andi a4, a2, ~((8*SZREG)-1)
> > + beqz a4, 2f
> > + add a3, a1, a4
> > +1:
> > + REG_L a4, 0(a1)
> > + REG_L a5, SZREG(a1)
> > + REG_L a6, 2*SZREG(a1)
> > + REG_L a7, 3*SZREG(a1)
> > + REG_L t0, 4*SZREG(a1)
> > + REG_L t1, 5*SZREG(a1)
> > + REG_L t2, 6*SZREG(a1)
> > + REG_L t3, 7*SZREG(a1)
> > + REG_S a4, 0(a0)
> > + REG_S a5, SZREG(a0)
> > + REG_S a6, 2*SZREG(a0)
> > + REG_S a7, 3*SZREG(a0)
> > + REG_S t0, 4*SZREG(a0)
> > + REG_S t1, 5*SZREG(a0)
> > + REG_S t2, 6*SZREG(a0)
> > + REG_S t3, 7*SZREG(a0)
> > + addi a0, a0, 8*SZREG
> > + addi a1, a1, 8*SZREG
> > + bltu a1, a3, 1b
> > +
> > +2:
> > + ret
> > +END(__copy_words_unaligned)
> > +
> > +/* void __copy_bytes_unaligned(void *, const void *, size_t) */
> > +/* Performs a memcpy without aligning buffers, using only byte accesses. */
> > +/* Note: The size is truncated to a multiple of 8 */
> > +ENTRY(__copy_bytes_unaligned)
> > + andi a4, a2, ~(8-1)
> > + beqz a4, 2f
> > + add a3, a1, a4
>
> Could you align operands for ASM please, to make this a little easier to
> read?
>
> > +1:
> > + lb a4, 0(a1)
> > + lb a5, 1(a1)
> > + lb a6, 2(a1)
> > + lb a7, 3(a1)
> > + lb t0, 4(a1)
> > + lb t1, 5(a1)
> > + lb t2, 6(a1)
> > + lb t3, 7(a1)
> > + sb a4, 0(a0)
> > + sb a5, 1(a0)
> > + sb a6, 2(a0)
> > + sb a7, 3(a0)
> > + sb t0, 4(a0)
> > + sb t1, 5(a0)
> > + sb t2, 6(a0)
> > + sb t3, 7(a0)
> > + addi a0, a0, 8
> > + addi a1, a1, 8
> > + bltu a1, a3, 1b
> > +
> > +2:
> > + ret
> > +END(__copy_bytes_unaligned)
> > diff --git a/arch/riscv/kernel/copy-noalign.h b/arch/riscv/kernel/copy-noalign.h
> > new file mode 100644
> > index 000000000000..99fbb9c763e0
> > --- /dev/null
> > +++ b/arch/riscv/kernel/copy-noalign.h
> > @@ -0,0 +1,13 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * Copyright (C) 2023 Rivos, Inc.
> > + */
> > +#ifndef __RISCV_KERNEL_COPY_NOALIGN_H
> > +#define __RISCV_KERNEL_COPY_NOALIGN_H
> > +
> > +#include <linux/types.h>
> > +
> > +void __copy_words_unaligned(void *dst, const void *src, size_t size);
> > +void __copy_bytes_unaligned(void *dst, const void *src, size_t size);
> > +
> > +#endif /* __RISCV_KERNEL_COPY_NOALIGN_H */
> > diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c
> > index bdcf460ea53d..3f7200dcc00c 100644
> > --- a/arch/riscv/kernel/cpufeature.c
> > +++ b/arch/riscv/kernel/cpufeature.c
> > @@ -19,11 +19,21 @@
> > #include <asm/cacheflush.h>
> > #include <asm/cpufeature.h>
> > #include <asm/hwcap.h>
> > +#include <asm/hwprobe.h>
> > #include <asm/patch.h>
> > #include <asm/processor.h>
> > #include <asm/vector.h>
> >
> > +#include "copy-noalign.h"
> > +
> > #define NUM_ALPHA_EXTS ('z' - 'a' + 1)
> > +#define MISALIGNED_ACCESS_JIFFIES_LG2 1
> > +#define MISALIGNED_BUFFER_SIZE 0x4000
> > +#define MISALIGNED_COPY_SIZE ((MISALIGNED_BUFFER_SIZE / 2) - 0x80)
> > +
>
> I think this blank line is misplaced, it should go after NUM_ALPHA_EXTS
> instead of (or as well as) here.
>
> > +#define MISALIGNED_COPY_MBS(_count) \
> > + ((HZ * (_count) * MISALIGNED_COPY_SIZE) >> \
> > + (20 + MISALIGNED_ACCESS_JIFFIES_LG2))
> >
> > unsigned long elf_hwcap __read_mostly;
> >
> > @@ -396,6 +406,74 @@ unsigned long riscv_get_elf_hwcap(void)
> > return hwcap;
> > }
> >
> > +void check_misaligned_access(int cpu)
> > +{
> > + unsigned long j0, j1;
> > + struct page *page;
> > + void *dst;
> > + void *src;
> > + long word_copies = 0;
> > + long byte_copies = 0;
> > + long speed = RISCV_HWPROBE_MISALIGNED_SLOW;
>
> Is this not a change from current behaviour, that may actually lead to
> incorrect reporting. Presently, only T-Head stuff sets a speed, so
> hwprobe falls back to UNKNOWN for everything else. With this, we will
> get slow set, for anything failing the test.
> Slow is defined as "Misaligned accesses are supported in hardware, but
> are slower than the cooresponding aligned accesses sequences (sic)", but
> you have no way of knowing, based on the test you are performing, whether
> the hardware supports it or if it is emulated by firmware.
> Perhaps that is not relevant to userspace, but wanted to know your
> thoughts.
>
Hm, that's true. EMULATED was an easy one when we were planning to get
this info from the DT. It also might be an easy one in the future, if
we get an SBI call that allows the kernel to take over misaligned trap
handling. We'd then be able to do a misaligned access and see if our
trap handler got called.
One option is to leave the value alone if we fail the FAST test
(rather than changing it from UNKNOWN to SLOW). This isn't great
though, as it effectively makes UNKNOWN synonymous with SLOW, but in a
way where usermode can't tell the difference between "I truly don't
know" and "I tried the fast test and it failed".
The alternative, as it is now, may mislabel some emulated systems as
slow until the new SBI call shows up. I'm not sure how bad this is in
practice. We could add a subsequent performance bar below which we
guess "emulated". This probably matches what usermode will use that
value for anyway (a synonym for "very slow"), but it's basically the
same problem with reversed polarity (we mislabel some slow systems as
emulated). I'm open to suggestions!
-Evan
> > +
> > + page = alloc_pages(GFP_NOWAIT, get_order(MISALIGNED_BUFFER_SIZE));
> > + if (!page) {
> > + pr_warn("Can't alloc pages to measure memcpy performance");
> > + return;
> > + }
> > +
> > + /* Make a misaligned destination buffer. */
> > + dst = (void *)((unsigned long)page_address(page) | 0x1);
> > + /* Misalign src as well, but differently (off by 1 + 2 = 3). */
> > + src = dst + (MISALIGNED_BUFFER_SIZE / 2);
> > + src += 2;
> > + /* Do a warmup. */
> > + __copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE);
> > + preempt_disable();
> > + j0 = jiffies;
> > + while ((j1 = jiffies) == j0)
> > + cpu_relax();
> > +
> > + while (time_before(jiffies,
> > + j1 + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) {
>
> Does this not fit in 100 chars?
>
> > +
> > + __copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE);
> > + word_copies += 1;
> > + }
> > +
> > + __copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE);
> > + j0 = jiffies;
> > + while ((j1 = jiffies) == j0)
> > + cpu_relax();
> > +
> > + while (time_before(jiffies,
> > + j1 + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) {
>
> Ditto here, no?
>
> Cheers,
> Conor.
