Why do I get the following error `wrong signature length` when I try to validate a signed file using the c++ OpenSSL 3.1 library?

["Christian F. Gonzalez Di Antonio" <christiangda@xxxxxxxxx>]

I posted this on https://stackoverflow.com/questions/78604338/why-do-i-get-the-following-error-wrong-signature-length-when-i-try-to-validate

I'm writing an c++ program LicenseValidator -> https://github.com/christiangda/LicenseValidator to validate a hypothetical program license using OpenSSL 3.1 Library, and when I tried to validate the licensed content I got the following error:

Failed to verify license
008C1AF901000000:error:02000077:rsa routines:ossl_rsa_verify:wrong signature length:crypto/rsa/rsa_sign.c:338:
008C1AF901000000:error:1C880004:Provider routines:rsa_verify:RSA lib:providers/implementations/signature/rsa_sig.c:785:

I would appreciate any help or guidance on what I am doing wrong.

I am not at all an expert in the c/c++ programming language and this is the first time I have tried to use the OpenSSL library.

Of course, I've used GitHub Copilot, gemini, and chatgpt to write and understand the repository code. The chalenge is about the examples I found on internet, the majority of them are about OpenSSL v1 and the v3 is very different, so was hard to understand the migration.

The README.md file has the instructions to create all the necessary keys, etc, references I used and the instructions to compile it using cmake.

The core function is LicenseValidator/src/License.cpp:

bool verifyLicense(const unsigned char *licenseContent, const unsigned char *licenseSignature, const std::string pubkey)
{
    EVP_PKEY *pkey = loadRsaPemPubKey(pubkey);
    if (pkey == NULL)
    {
        std::cerr << "Failed to load public key" << std::endl;
        ERR_print_errors_fp(stdout);
        return false;
    }

    EVP_PKEY_CTX *ctx = EVP_PKEY_CTX_new(pkey, NULL);
    if (ctx == NULL)
    {
        std::cerr << "Failed to create EVP_PKEY_CTX" << std::endl;
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    if (EVP_PKEY_verify_init(ctx) <= 0)
    {
        std::cerr << "Failed to initialize EVP_PKEY_CTX" << std::endl;
        EVP_PKEY_CTX_free(ctx);
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    // PKCS1 padding scheme
    if (EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_PADDING) <= 0)
    {
        std::cerr << "Failed to set RSA padding" << std::endl;
        EVP_PKEY_CTX_free(ctx);
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    // SHA256 digest
    if (EVP_PKEY_CTX_set_signature_md(ctx, EVP_sha256()) <= 0)
    {
        std::cerr << "Failed to set signature MD" << std::endl;
        EVP_PKEY_CTX_free(ctx);
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    if (EVP_PKEY_verify(ctx, licenseSignature, sizeof(licenseSignature), licenseContent, sizeof(licenseContent)) <= 0)
    {
        std::cerr << "Failed to verify license" << std::endl;
        EVP_PKEY_CTX_free(ctx);
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    EVP_PKEY_CTX_free(ctx);
    EVP_PKEY_free(pkey);

    return true;
}

Some guidance about how to solve the error I got.

--

Saludos,
Christian