On Sun, 21 Jan 2024 16:19:11 PST (-0800), ebiggers@xxxxxxxxxx wrote:
This patchset, which applies to v6.8-rc1, adds cryptographic algorithm
implementations accelerated using the RISC-V vector crypto extensions
(https://github.com/riscv/riscv-crypto/releases/download/v1.0.0/riscv-crypto-spec-vector.pdf)
and RISC-V vector extension
(https://github.com/riscv/riscv-v-spec/releases/download/v1.0/riscv-v-spec-1.0.pdf).
The following algorithms are included: AES in ECB, CBC, CTR, and XTS modes;
ChaCha20; GHASH; SHA-2; SM3; and SM4.
In general, the assembly code requires a 64-bit RISC-V CPU with VLEN >= 128,
little endian byte order, and vector unaligned access support. The ECB, CTR,
XTS, and ChaCha20 code is designed to naturally scale up to larger VLEN values.
Building the assembly code requires tip-of-tree binutils (future 2.42) or
tip-of-tree clang (future 18.x). All algorithms pass testing in QEMU, using
CONFIG_CRYPTO_MANAGER_EXTRA_TESTS=y. Much of the assembly code is derived from
OpenSSL code that was added by https://github.com/openssl/openssl/pull/21923.
It's been cleaned up for integration with the kernel, e.g. reducing code
duplication, eliminating use of .inst and perlasm, and fixing a few bugs.
This patchset incorporates the work of multiple people, including Jerry Shih,
Heiko Stuebner, Christoph Müllner, Phoebe Chen, Charalampos Mitrodimas, and
myself. This patchset went through several versions from Heiko (last version
https://lore.kernel.org/linux-crypto/20230711153743.1970625-1-heiko@xxxxxxxxx),
then several versions from Jerry (last version:
https://lore.kernel.org/linux-crypto/20231231152743.6304-1-jerry.shih@xxxxxxxxxx),
then finally several versions from me. Thanks to everyone who has contributed
to this patchset or its prerequisites. Since v6.8-rc1, all prerequisite kernel
patches are upstream. I think this is now ready, and I'd like for it to be
applied for 6.9, either to the crypto or riscv tree (at maintainers' choice).
I'm OK taking this through the RISC-V tree if folks want, I've gone and
queued it up for my tester. Just LMK if it should go somewhere else,
otherwise it'll end up on for-next in a day or so (I'm trying to get LTO
builds added right now, so it might take a bit).
Below is the changelog for my versions of the patchset. For the changelog of
the older versions, see the above links.
Changed in v3:
- Fixed a bug in the AES-XTS implementation where it assumed the CPU
always set vl to the maximum possible value. This was okay for
QEMU, but the vector spec allows CPUs to have different behavior.
- Increased the LMUL for AES-ECB to 8, as the registers are available.
- Fixed some license text that I had mistakenly changed when doing a
find-and-replace of code.
- Addressed a checkpatch warning by not including filename in file.
- Rename some labels.
- Constify a variable.
Changed in v2:
- Merged the AES modules together to prevent a build error.
- Only unregister AES algorithms that were registered.
- Corrected walksize properties to match the LMUL used by asm code.
- Simplified the CTR and XTS glue code slightly.
- Minor cleanups.
Changed in v1:
- Refer to my cover letter
https://lore.kernel.org/linux-crypto/20240102064743.220490-1-ebiggers@xxxxxxxxxx/
Eric Biggers (1):
RISC-V: add TOOLCHAIN_HAS_VECTOR_CRYPTO
Heiko Stuebner (2):
RISC-V: add helper function to read the vector VLEN
RISC-V: hook new crypto subdir into build-system
Jerry Shih (7):
crypto: riscv - add vector crypto accelerated AES-{ECB,CBC,CTR,XTS}
crypto: riscv - add vector crypto accelerated ChaCha20
crypto: riscv - add vector crypto accelerated GHASH
crypto: riscv - add vector crypto accelerated SHA-{256,224}
crypto: riscv - add vector crypto accelerated SHA-{512,384}
crypto: riscv - add vector crypto accelerated SM3
crypto: riscv - add vector crypto accelerated SM4
arch/riscv/Kbuild | 1 +
arch/riscv/Kconfig | 7 +
arch/riscv/crypto/Kconfig | 93 +++
arch/riscv/crypto/Makefile | 23 +
arch/riscv/crypto/aes-macros.S | 156 +++++
arch/riscv/crypto/aes-riscv64-glue.c | 550 ++++++++++++++++++
.../crypto/aes-riscv64-zvkned-zvbb-zvkg.S | 312 ++++++++++
arch/riscv/crypto/aes-riscv64-zvkned-zvkb.S | 146 +++++
arch/riscv/crypto/aes-riscv64-zvkned.S | 180 ++++++
arch/riscv/crypto/chacha-riscv64-glue.c | 101 ++++
arch/riscv/crypto/chacha-riscv64-zvkb.S | 294 ++++++++++
arch/riscv/crypto/ghash-riscv64-glue.c | 168 ++++++
arch/riscv/crypto/ghash-riscv64-zvkg.S | 72 +++
arch/riscv/crypto/sha256-riscv64-glue.c | 137 +++++
.../sha256-riscv64-zvknha_or_zvknhb-zvkb.S | 225 +++++++
arch/riscv/crypto/sha512-riscv64-glue.c | 133 +++++
.../riscv/crypto/sha512-riscv64-zvknhb-zvkb.S | 203 +++++++
arch/riscv/crypto/sm3-riscv64-glue.c | 112 ++++
arch/riscv/crypto/sm3-riscv64-zvksh-zvkb.S | 123 ++++
arch/riscv/crypto/sm4-riscv64-glue.c | 107 ++++
arch/riscv/crypto/sm4-riscv64-zvksed-zvkb.S | 117 ++++
arch/riscv/include/asm/vector.h | 11 +
crypto/Kconfig | 3 +
23 files changed, 3274 insertions(+)
create mode 100644 arch/riscv/crypto/Kconfig
create mode 100644 arch/riscv/crypto/Makefile
create mode 100644 arch/riscv/crypto/aes-macros.S
create mode 100644 arch/riscv/crypto/aes-riscv64-glue.c
create mode 100644 arch/riscv/crypto/aes-riscv64-zvkned-zvbb-zvkg.S
create mode 100644 arch/riscv/crypto/aes-riscv64-zvkned-zvkb.S
create mode 100644 arch/riscv/crypto/aes-riscv64-zvkned.S
create mode 100644 arch/riscv/crypto/chacha-riscv64-glue.c
create mode 100644 arch/riscv/crypto/chacha-riscv64-zvkb.S
create mode 100644 arch/riscv/crypto/ghash-riscv64-glue.c
create mode 100644 arch/riscv/crypto/ghash-riscv64-zvkg.S
create mode 100644 arch/riscv/crypto/sha256-riscv64-glue.c
create mode 100644 arch/riscv/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S
create mode 100644 arch/riscv/crypto/sha512-riscv64-glue.c
create mode 100644 arch/riscv/crypto/sha512-riscv64-zvknhb-zvkb.S
create mode 100644 arch/riscv/crypto/sm3-riscv64-glue.c
create mode 100644 arch/riscv/crypto/sm3-riscv64-zvksh-zvkb.S
create mode 100644 arch/riscv/crypto/sm4-riscv64-glue.c
create mode 100644 arch/riscv/crypto/sm4-riscv64-zvksed-zvkb.S
base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d
