On Wed, Feb 03, 2021 at 12:36:17PM +0100, Ard Biesheuvel wrote: > Given how kernel mode NEON code disables preemption (to ensure that the > FP/SIMD register state is protected without having to context switch it), > we need to take care not to let those algorithms operate on unbounded > input data, or we may end up with excessive scheduling blackouts on > CONFIG_PREEMPT kernels. > > This is currently handled by the cond_yield_neon macros, which check the > preempt count and the TIF_NEED_RESCHED flag from assembler code, and call > into kernel_neon_end()+kernel_neon_begin(), triggering a reschedule. > This works as expected, but is a bit messy, given how much of the state > preserve/restore code in the algorithm needs to be duplicated, as well as > causing the need to manage the stack frame explicitly. All of this is better > handled by the compiler, especially now that we have enabled features such > as the shadow call stack and BTI, and are working to improve call stack > validation. > > In some cases, yielding is not necessary at all: algoritms that implement > skciphers and use the skcipher walk API will be invoked at page granularity, > which is granular enough for our purpose. > > In other cases, it is better to simply return early from the assembler > routine if a reschedule is pending, and let the C code handle with this, by > retrying the call until it completes. This removes any voluntary schedule() > calls from the call stack, making the code much easier to reason about in > the context of stack validation, rcu_tasks synchronization, etc. > > Practical note: assuming there are no objections to these changes, it may > be the most convenient to take patch #1 into the arm64 tree for v5.12, > and postpone the rest for merging via the crypto tree. (Note that this > series was created against the cryptodev tree, and so the arm64 maintainers > are also welcome to take the whole set if it applies cleanly to the arm64 > tree) > > Will: if you stick #1 on a separate branch, please base it on v5.11-rc1 > > Changes since v1: > - use sub+cbz instead of cmp+b.eq to avoid clobbering the flags in cond_yield > (patch #1) > > Cc: Dave Martin <dave.martin@xxxxxxx> > Cc: Eric Biggers <ebiggers@xxxxxxxxxx> > > Ard Biesheuvel (9): > arm64: assembler: add cond_yield macro > crypto: arm64/sha1-ce - simplify NEON yield > crypto: arm64/sha2-ce - simplify NEON yield > crypto: arm64/sha3-ce - simplify NEON yield > crypto: arm64/sha512-ce - simplify NEON yield > crypto: arm64/aes-neonbs - remove NEON yield calls > crypto: arm64/aes-ce-mac - simplify NEON yield > crypto: arm64/crc-t10dif - move NEON yield to C code > arm64: assembler: remove conditional NEON yield macros > > arch/arm64/crypto/aes-glue.c | 21 +++-- > arch/arm64/crypto/aes-modes.S | 52 +++++-------- > arch/arm64/crypto/aes-neonbs-core.S | 8 +- > arch/arm64/crypto/crct10dif-ce-core.S | 43 +++-------- > arch/arm64/crypto/crct10dif-ce-glue.c | 30 ++++++-- > arch/arm64/crypto/sha1-ce-core.S | 47 ++++-------- > arch/arm64/crypto/sha1-ce-glue.c | 22 +++--- > arch/arm64/crypto/sha2-ce-core.S | 38 ++++----- > arch/arm64/crypto/sha2-ce-glue.c | 22 +++--- > arch/arm64/crypto/sha3-ce-core.S | 81 ++++++++------------ > arch/arm64/crypto/sha3-ce-glue.c | 14 ++-- > arch/arm64/crypto/sha512-ce-core.S | 29 ++----- > arch/arm64/crypto/sha512-ce-glue.c | 53 +++++++------ > arch/arm64/include/asm/assembler.h | 78 +++---------------- > 14 files changed, 209 insertions(+), 329 deletions(-) Patches 2-8 applied. Thanks. -- Email: Herbert Xu <herbert@xxxxxxxxxxxxxxxxxxx> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
