On 17.06.2018 11:40, Ard Biesheuvel wrote:
> On 17 June 2018 at 11:30, Ard Biesheuvel <ard.biesheuvel@xxxxxxxxxx> wrote:
>> On 17 June 2018 at 00:40, Stefan Agner <stefan@xxxxxxxx> wrote:
>>> Hi Eric,
>>>
>>> On 14.02.2018 19:42, Eric Biggers wrote:
>>>> Add an ARM NEON-accelerated implementation of Speck-XTS. It operates on
>>>> 128-byte chunks at a time, i.e. 8 blocks for Speck128 or 16 blocks for
>>>> Speck64. Each 128-byte chunk goes through XTS preprocessing, then is
>>>> encrypted/decrypted (doing one cipher round for all the blocks, then the
>>>> next round, etc.), then goes through XTS postprocessing.
>>>>
>>>> The performance depends on the processor but can be about 3 times faster
>>>> than the generic code. For example, on an ARMv7 processor we observe
>>>> the following performance with Speck128/256-XTS:
>>>>
>>>> xts-speck128-neon: Encryption 107.9 MB/s, Decryption 108.1 MB/s
>>>> xts(speck128-generic): Encryption 32.1 MB/s, Decryption 36.6 MB/s
>>>>
>>>> In comparison to AES-256-XTS without the Cryptography Extensions:
>>>>
>>>> xts-aes-neonbs: Encryption 41.2 MB/s, Decryption 36.7 MB/s
>>>> xts(aes-asm): Encryption 31.7 MB/s, Decryption 30.8 MB/s
>>>> xts(aes-generic): Encryption 21.2 MB/s, Decryption 20.9 MB/s
>>>>
>>>> Speck64/128-XTS is even faster:
>>>>
>>>> xts-speck64-neon: Encryption 138.6 MB/s, Decryption 139.1 MB/s
>>>>
>>>> Note that as with the generic code, only the Speck128 and Speck64
>>>> variants are supported. Also, for now only the XTS mode of operation is
>>>> supported, to target the disk and file encryption use cases. The NEON
>>>> code also only handles the portion of the data that is evenly divisible
>>>> into 128-byte chunks, with any remainder handled by a C fallback. Of
>>>> course, other modes of operation could be added later if needed, and/or
>>>> the NEON code could be updated to handle other buffer sizes.
>>>>
>>>> The XTS specification is only defined for AES which has a 128-bit block
>>>> size, so for the GF(2^64) math needed for Speck64-XTS we use the
>>>> reducing polynomial 'x^64 + x^4 + x^3 + x + 1' given by the original XEX
>>>> paper. Of course, when possible users should use Speck128-XTS, but even
>>>> that may be too slow on some processors; Speck64-XTS can be faster.
>>>>
>>>> Signed-off-by: Eric Biggers <ebiggers@xxxxxxxxxx>
>>>> ---
>>>> arch/arm/crypto/Kconfig | 6 +
>>>> arch/arm/crypto/Makefile | 2 +
>>>> arch/arm/crypto/speck-neon-core.S | 432 ++++++++++++++++++++++++++++++
>>>> arch/arm/crypto/speck-neon-glue.c | 288 ++++++++++++++++++++
>>>> 4 files changed, 728 insertions(+)
>>>> create mode 100644 arch/arm/crypto/speck-neon-core.S
>>>> create mode 100644 arch/arm/crypto/speck-neon-glue.c
>>>>
>>>> diff --git a/arch/arm/crypto/Kconfig b/arch/arm/crypto/Kconfig
>>>> index b8e69fe282b8..925d1364727a 100644
>>>> --- a/arch/arm/crypto/Kconfig
>>>> +++ b/arch/arm/crypto/Kconfig
>>>> @@ -121,4 +121,10 @@ config CRYPTO_CHACHA20_NEON
>>>> select CRYPTO_BLKCIPHER
>>>> select CRYPTO_CHACHA20
>>>>
>>>> +config CRYPTO_SPECK_NEON
>>>> + tristate "NEON accelerated Speck cipher algorithms"
>>>> + depends on KERNEL_MODE_NEON
>>>> + select CRYPTO_BLKCIPHER
>>>> + select CRYPTO_SPECK
>>>> +
>>>> endif
>>>> diff --git a/arch/arm/crypto/Makefile b/arch/arm/crypto/Makefile
>>>> index 30ef8e291271..a758107c5525 100644
>>>> --- a/arch/arm/crypto/Makefile
>>>> +++ b/arch/arm/crypto/Makefile
>>>> @@ -10,6 +10,7 @@ obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o
>>>> obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o
>>>> obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o
>>>> obj-$(CONFIG_CRYPTO_CHACHA20_NEON) += chacha20-neon.o
>>>> +obj-$(CONFIG_CRYPTO_SPECK_NEON) += speck-neon.o
>>>>
>>>> ce-obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o
>>>> ce-obj-$(CONFIG_CRYPTO_SHA1_ARM_CE) += sha1-arm-ce.o
>>>> @@ -53,6 +54,7 @@ ghash-arm-ce-y := ghash-ce-core.o ghash-ce-glue.o
>>>> crct10dif-arm-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o
>>>> crc32-arm-ce-y:= crc32-ce-core.o crc32-ce-glue.o
>>>> chacha20-neon-y := chacha20-neon-core.o chacha20-neon-glue.o
>>>> +speck-neon-y := speck-neon-core.o speck-neon-glue.o
>>>>
>>>> quiet_cmd_perl = PERL $@
>>>> cmd_perl = $(PERL) $(<) > $(@)
>>>> diff --git a/arch/arm/crypto/speck-neon-core.S
>>>> b/arch/arm/crypto/speck-neon-core.S
>>>> new file mode 100644
>>>> index 000000000000..3c1e203e53b9
>>>> --- /dev/null
>>>> +++ b/arch/arm/crypto/speck-neon-core.S
>>>> @@ -0,0 +1,432 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * NEON-accelerated implementation of Speck128-XTS and Speck64-XTS
>>>> + *
>>>> + * Copyright (c) 2018 Google, Inc
>>>> + *
>>>> + * Author: Eric Biggers <ebiggers@xxxxxxxxxx>
>>>> + */
>>>> +
>>>> +#include <linux/linkage.h>
>>>> +
>>>> + .text
>>>> + .fpu neon
>>>> +
>>>> + // arguments
>>>> + ROUND_KEYS .req r0 // const {u64,u32} *round_keys
>>>> + NROUNDS .req r1 // int nrounds
>>>> + DST .req r2 // void *dst
>>>> + SRC .req r3 // const void *src
>>>> + NBYTES .req r4 // unsigned int nbytes
>>>> + TWEAK .req r5 // void *tweak
>>>> +
>>>> + // registers which hold the data being encrypted/decrypted
>>>> + X0 .req q0
>>>> + X0_L .req d0
>>>> + X0_H .req d1
>>>> + Y0 .req q1
>>>> + Y0_H .req d3
>>>> + X1 .req q2
>>>> + X1_L .req d4
>>>> + X1_H .req d5
>>>> + Y1 .req q3
>>>> + Y1_H .req d7
>>>> + X2 .req q4
>>>> + X2_L .req d8
>>>> + X2_H .req d9
>>>> + Y2 .req q5
>>>> + Y2_H .req d11
>>>> + X3 .req q6
>>>> + X3_L .req d12
>>>> + X3_H .req d13
>>>> + Y3 .req q7
>>>> + Y3_H .req d15
>>>> +
>>>> + // the round key, duplicated in all lanes
>>>> + ROUND_KEY .req q8
>>>> + ROUND_KEY_L .req d16
>>>> + ROUND_KEY_H .req d17
>>>> +
>>>> + // index vector for vtbl-based 8-bit rotates
>>>> + ROTATE_TABLE .req d18
>>>> +
>>>> + // multiplication table for updating XTS tweaks
>>>> + GF128MUL_TABLE .req d19
>>>> + GF64MUL_TABLE .req d19
>>>> +
>>>> + // current XTS tweak value(s)
>>>> + TWEAKV .req q10
>>>> + TWEAKV_L .req d20
>>>> + TWEAKV_H .req d21
>>>> +
>>>> + TMP0 .req q12
>>>> + TMP0_L .req d24
>>>> + TMP0_H .req d25
>>>> + TMP1 .req q13
>>>> + TMP2 .req q14
>>>> + TMP3 .req q15
>>>> +
>>>> + .align 4
>>>> +.Lror64_8_table:
>>>> + .byte 1, 2, 3, 4, 5, 6, 7, 0
>>>> +.Lror32_8_table:
>>>> + .byte 1, 2, 3, 0, 5, 6, 7, 4
>>>> +.Lrol64_8_table:
>>>> + .byte 7, 0, 1, 2, 3, 4, 5, 6
>>>> +.Lrol32_8_table:
>>>> + .byte 3, 0, 1, 2, 7, 4, 5, 6
>>>> +.Lgf128mul_table:
>>>> + .byte 0, 0x87
>>>> + .fill 14
>>>> +.Lgf64mul_table:
>>>> + .byte 0, 0x1b, (0x1b << 1), (0x1b << 1) ^ 0x1b
>>>> + .fill 12
>>>> +
>>>> +/*
>>>> + * _speck_round_128bytes() - Speck encryption round on 128 bytes at a time
>>>> + *
>>>> + * Do one Speck encryption round on the 128 bytes (8 blocks for
>>>> Speck128, 16 for
>>>> + * Speck64) stored in X0-X3 and Y0-Y3, using the round key stored in all lanes
>>>> + * of ROUND_KEY. 'n' is the lane size: 64 for Speck128, or 32 for Speck64.
>>>> + *
>>>> + * The 8-bit rotates are implemented using vtbl instead of vshr + vsli because
>>>> + * the vtbl approach is faster on some processors and the same speed on others.
>>>> + */
>>>> +.macro _speck_round_128bytes n
>>>> +
>>>> + // x = ror(x, 8)
>>>> + vtbl.8 X0_L, {X0_L}, ROTATE_TABLE
>>>> + vtbl.8 X0_H, {X0_H}, ROTATE_TABLE
>>>> + vtbl.8 X1_L, {X1_L}, ROTATE_TABLE
>>>> + vtbl.8 X1_H, {X1_H}, ROTATE_TABLE
>>>> + vtbl.8 X2_L, {X2_L}, ROTATE_TABLE
>>>> + vtbl.8 X2_H, {X2_H}, ROTATE_TABLE
>>>> + vtbl.8 X3_L, {X3_L}, ROTATE_TABLE
>>>> + vtbl.8 X3_H, {X3_H}, ROTATE_TABLE
>>>> +
>>>> + // x += y
>>>> + vadd.u\n X0, Y0
>>>> + vadd.u\n X1, Y1
>>>> + vadd.u\n X2, Y2
>>>> + vadd.u\n X3, Y3
>>>> +
>>>> + // x ^= k
>>>> + veor X0, ROUND_KEY
>>>> + veor X1, ROUND_KEY
>>>> + veor X2, ROUND_KEY
>>>> + veor X3, ROUND_KEY
>>>> +
>>>> + // y = rol(y, 3)
>>>> + vshl.u\n TMP0, Y0, #3
>>>> + vshl.u\n TMP1, Y1, #3
>>>> + vshl.u\n TMP2, Y2, #3
>>>> + vshl.u\n TMP3, Y3, #3
>>>> + vsri.u\n TMP0, Y0, #(\n - 3)
>>>> + vsri.u\n TMP1, Y1, #(\n - 3)
>>>> + vsri.u\n TMP2, Y2, #(\n - 3)
>>>> + vsri.u\n TMP3, Y3, #(\n - 3)
>>>> +
>>>> + // y ^= x
>>>> + veor Y0, TMP0, X0
>>>> + veor Y1, TMP1, X1
>>>> + veor Y2, TMP2, X2
>>>> + veor Y3, TMP3, X3
>>>> +.endm
>>>> +
>>>> +/*
>>>> + * _speck_unround_128bytes() - Speck decryption round on 128 bytes at a time
>>>> + *
>>>> + * This is the inverse of _speck_round_128bytes().
>>>> + */
>>>> +.macro _speck_unround_128bytes n
>>>> +
>>>> + // y ^= x
>>>> + veor TMP0, Y0, X0
>>>> + veor TMP1, Y1, X1
>>>> + veor TMP2, Y2, X2
>>>> + veor TMP3, Y3, X3
>>>> +
>>>> + // y = ror(y, 3)
>>>> + vshr.u\n Y0, TMP0, #3
>>>> + vshr.u\n Y1, TMP1, #3
>>>> + vshr.u\n Y2, TMP2, #3
>>>> + vshr.u\n Y3, TMP3, #3
>>>> + vsli.u\n Y0, TMP0, #(\n - 3)
>>>> + vsli.u\n Y1, TMP1, #(\n - 3)
>>>> + vsli.u\n Y2, TMP2, #(\n - 3)
>>>> + vsli.u\n Y3, TMP3, #(\n - 3)
>>>> +
>>>> + // x ^= k
>>>> + veor X0, ROUND_KEY
>>>> + veor X1, ROUND_KEY
>>>> + veor X2, ROUND_KEY
>>>> + veor X3, ROUND_KEY
>>>> +
>>>> + // x -= y
>>>> + vsub.u\n X0, Y0
>>>> + vsub.u\n X1, Y1
>>>> + vsub.u\n X2, Y2
>>>> + vsub.u\n X3, Y3
>>>> +
>>>> + // x = rol(x, 8);
>>>> + vtbl.8 X0_L, {X0_L}, ROTATE_TABLE
>>>> + vtbl.8 X0_H, {X0_H}, ROTATE_TABLE
>>>> + vtbl.8 X1_L, {X1_L}, ROTATE_TABLE
>>>> + vtbl.8 X1_H, {X1_H}, ROTATE_TABLE
>>>> + vtbl.8 X2_L, {X2_L}, ROTATE_TABLE
>>>> + vtbl.8 X2_H, {X2_H}, ROTATE_TABLE
>>>> + vtbl.8 X3_L, {X3_L}, ROTATE_TABLE
>>>> + vtbl.8 X3_H, {X3_H}, ROTATE_TABLE
>>>> +.endm
>>>> +
>>>> +.macro _xts128_precrypt_one dst_reg, tweak_buf, tmp
>>>> +
>>>> + // Load the next source block
>>>> + vld1.8 {\dst_reg}, [SRC]!
>>>> +
>>>> + // Save the current tweak in the tweak buffer
>>>> + vst1.8 {TWEAKV}, [\tweak_buf:128]!
>>>> +
>>>> + // XOR the next source block with the current tweak
>>>> + veor \dst_reg, TWEAKV
>>>> +
>>>> + /*
>>>> + * Calculate the next tweak by multiplying the current one by x,
>>>> + * modulo p(x) = x^128 + x^7 + x^2 + x + 1.
>>>> + */
>>>> + vshr.u64 \tmp, TWEAKV, #63
>>>> + vshl.u64 TWEAKV, #1
>>>> + veor TWEAKV_H, \tmp\()_L
>>>> + vtbl.8 \tmp\()_H, {GF128MUL_TABLE}, \tmp\()_H
>>>> + veor TWEAKV_L, \tmp\()_H
>>>> +.endm
>>>> +
>>>> +.macro _xts64_precrypt_two dst_reg, tweak_buf, tmp
>>>> +
>>>> + // Load the next two source blocks
>>>> + vld1.8 {\dst_reg}, [SRC]!
>>>> +
>>>> + // Save the current two tweaks in the tweak buffer
>>>> + vst1.8 {TWEAKV}, [\tweak_buf:128]!
>>>> +
>>>> + // XOR the next two source blocks with the current two tweaks
>>>> + veor \dst_reg, TWEAKV
>>>> +
>>>> + /*
>>>> + * Calculate the next two tweaks by multiplying the current ones by x^2,
>>>> + * modulo p(x) = x^64 + x^4 + x^3 + x + 1.
>>>> + */
>>>> + vshr.u64 \tmp, TWEAKV, #62
>>>> + vshl.u64 TWEAKV, #2
>>>> + vtbl.8 \tmp\()_L, {GF64MUL_TABLE}, \tmp\()_L
>>>> + vtbl.8 \tmp\()_H, {GF64MUL_TABLE}, \tmp\()_H
>>>> + veor TWEAKV, \tmp
>>>> +.endm
>>>> +
>>>> +/*
>>>> + * _speck_xts_crypt() - Speck-XTS encryption/decryption
>>>> + *
>>>> + * Encrypt or decrypt NBYTES bytes of data from the SRC buffer to the
>>>> DST buffer
>>>> + * using Speck-XTS, specifically the variant with a block size of
>>>> '2n' and round
>>>> + * count given by NROUNDS. The expanded round keys are given in
>>>> ROUND_KEYS, and
>>>> + * the current XTS tweak value is given in TWEAK. It's assumed that
>>>> NBYTES is a
>>>> + * nonzero multiple of 128.
>>>> + */
>>>> +.macro _speck_xts_crypt n, decrypting
>>>> + push {r4-r7}
>>>> + mov r7, sp
>>>> +
>>>> + /*
>>>> + * The first four parameters were passed in registers r0-r3. Load the
>>>> + * additional parameters, which were passed on the stack.
>>>> + */
>>>> + ldr NBYTES, [sp, #16]
>>>> + ldr TWEAK, [sp, #20]
>>>> +
>>>> + /*
>>>> + * If decrypting, modify the ROUND_KEYS parameter to point to the last
>>>> + * round key rather than the first, since for decryption the round keys
>>>> + * are used in reverse order.
>>>> + */
>>>> +.if \decrypting
>>>> +.if \n == 64
>>>> + add ROUND_KEYS, ROUND_KEYS, NROUNDS, lsl #3
>>>> + sub ROUND_KEYS, #8
>>>> +.else
>>>> + add ROUND_KEYS, ROUND_KEYS, NROUNDS, lsl #2
>>>> + sub ROUND_KEYS, #4
>>>> +.endif
>>>> +.endif
>>>> +
>>>> + // Load the index vector for vtbl-based 8-bit rotates
>>>> +.if \decrypting
>>>> + ldr r12, =.Lrol\n\()_8_table
>>>> +.else
>>>> + ldr r12, =.Lror\n\()_8_table
>>>> +.endif
>>>> + vld1.8 {ROTATE_TABLE}, [r12:64]
>>>> +
>>>> + // One-time XTS preparation
>>>> +
>>>> + /*
>>>> + * Allocate stack space to store 128 bytes worth of tweaks. For
>>>> + * performance, this space is aligned to a 16-byte boundary so that we
>>>> + * can use the load/store instructions that declare 16-byte alignment.
>>>> + */
>>>> + sub sp, #128
>>>> + bic sp, #0xf
>>>
>>>
>>> This fails here when building with CONFIG_THUMB2_KERNEL=y
>>>
>>> AS arch/arm/crypto/speck-neon-core.o
>>>
>>> arch/arm/crypto/speck-neon-core.S: Assembler messages:
>>>
>>> arch/arm/crypto/speck-neon-core.S:419: Error: r13 not allowed here --
>>> `bic sp,#0xf'
>>> arch/arm/crypto/speck-neon-core.S:423: Error: r13 not allowed here --
>>> `bic sp,#0xf'
>>> arch/arm/crypto/speck-neon-core.S:427: Error: r13 not allowed here --
>>> `bic sp,#0xf'
>>> arch/arm/crypto/speck-neon-core.S:431: Error: r13 not allowed here --
>>> `bic sp,#0xf'
>>>
>>> In a quick hack this change seems to address it:
>>>
>>>
>>> - sub sp, #128
>>> - bic sp, #0xf
>>> + mov r6, sp
>>> + sub r6, #128
>>> + bic r6, #0xf
>>> + mov sp, r6
>>>
>>> But there is probably a better solution to address this.
>>>
>>
>> Given that there is no NEON on M class cores, I recommend we put something like
>>
>> THUMB(bx pc)
>> THUMB(nop.w)
>> THUMB(.arm)
>>
>> at the beginning and be done with it.
>
> I mean nop.n or just nop, of course, and we may need a '.align 2' at
> the beginning as well.
Wouldn't it be preferable to have it assemble it in Thumb2 too? It seems
that bic sp,#0xf is the only issue...
--
Stefan
