> padding = 3 means "no padding" indicating that the data for signature is already padded. That's why the data size (flen) is 256 (hashed data padded to the rsa key size of 2048 bits, I guess). If you are using OpenSSL 1.1.1, this could be due to PSS padding in which case current implementation passes pre-padded data for raw signature to the callback. AFAIK, pkcs11-helper only handles PKCS1 padding (CKM_RSA_PKCS) though pkcs11 standard does support raw signatures. https://github.com/OpenSC/pkcs11-helper/blob/0e2ae10ef9611beef92457171e8c78d8e936dfca/lib/pkcs11h-openssl.c#L570 if (padding != RSA_PKCS1_PADDING) { rv = CKR_MECHANISM_INVALID; goto cleanup; }
