Provider: cannot sign with ed25519-like key

["'Bernd Ritter' via openssl-users" <openssl-users@xxxxxxxxxxx>]

Hello everyone,

I am writing a custom provider which uses a different OID for ED25519 
but internally uses this cipher. I'm now stuck on the signing for weeks.

The private and public key look identical except for the OID when 
compared to a ED25519-key created by the default provider. This is the 
logging output I see, when I run into the problem with

The used md_ctx is built up in the 
OSSL_FUNC_signature_digest_sign_init_fn. Shortened for readability:

####################################################################
static int digest_sign_init(void *ctx, const char *mdname, void 
*provkey, const OSSL_PARAM *params)
{
  sig_ctx *sigctx = (sig_ctx*) ctx;

  EVP_PKEY *pkey = (EVP_PKEY*) provkey;
  EVP_PKEY_CTX* pkey_ctx = EVP_PKEY_CTX_new_from_pkey(NULL, pkey, 
"provider=default");

  EVP_MD_CTX* dflt_md_ctx = EVP_MD_CTX_new();
  EVP_MD_CTX_set_pkey_ctx(dflt_md_ctx, pkey_ctx);

  // These two are checked in their return code but omitted here
  EVP_DigestSignInit_ex(dflt_md_ctx, NULL, NULL, NULL, NULL, pkey, 
params));
  EVP_PKEY_CTX_ctrl_str(pkey_ctx, "instance", "ed25519ph");

  sigctx->md_ctx = dflt_md_ctx;

  return 1;
}
####################################################################

The digest-sign function is implemented as 
OSSL_FUNC_signature_digest_sign_update_fn digest_sign_update:

####################################################################
static int digest_sign_update(void *ctx, const unsigned char *data, 
size_t datalen)
{
  DBG("%s - digest sign update. datalen=%lu\n", __FILE__, datalen);
  sig_ctx *sigctx = (sig_ctx*) ctx;

  // Just to see if we have a valid key
  EVP_PKEY* pkey = 
EVP_PKEY_CTX_get0_pkey(EVP_MD_CTX_get_pkey_ctx(sigctx->md_ctx));
  DBG("%s - Public Key Context: %s, size:%d\n", __FILE__, 
EVP_PKEY_get0_description(pkey), EVP_PKEY_get_size(pkey));

  const int rc = EVP_DigestSignUpdate(sigctx->md_ctx, data, datalen);
  if (rc != 1) {
    DBG("%s - digest sign update failed rc=%d. %s\n", __FILE__, rc, 
ERR_error_string(ERR_get_error(), NULL));
  }

  return rc;
}
####################################################################

So this is now the resulting log messages:

####################################################################
operating switch: 12 (1=DIGEST, 2=CIPHER, 5=RAND, 10=KEYMGMT, 12=SIG, 
20=ENC, 21=DEC, 22=STOR)
/src/ed25519ph_sign_x509.c - d
/src/ed25519ph_sign_x509.c - digest sign update. datalen=29
/src/ed25519ph_sign_x509.c - Public Key Context: OpenSSL ED25519 
implementation, size:64
/src/ed25519ph_sign_x509.c - digest sign update failed rc=0. 
error:030000A7:digital envelope routines::unknown max size
Error signing raw input data
Public Key operation error
804B6635217A0000:error:030C0101:digital envelope 
routines:EVP_DigestSignUpdate:called a function you should not 
call:crypto/evp/m_sigver.c:420:
####################################################################

What does "unknown max size" mean in this context? The (custom) public 
key can be worked on with EVP_PKEY_print_public so it seems correct-(ish)?

####################################################################
ED25519 Public-Key:
pub:
    86:f1:df:8f:7c:d1:51:b1:8d:b8:bf:4c:d7:e3:28:
    b3:1c:d6:1a:66:2a:00:2b:e8:34:18:bb:c4:e8:82:
    f5:d1
####################################################################

Hope anyone can give me some hint or direction, as said above I am 
trying to figure this out for some weeks now.

This message has been posted to OpenSSL github discussion board as well, 
it has nicer formatting if you like: 
https://github.com/openssl/openssl/discussions/25050

All the best,
Bernd

--
Bernd Ritter
Senior Linux Developer
Tel.: +49 175 534 4534
Mail: ritter@xxxxxxxxxxxxx

B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt, HRB 3537

--
You received this message because you are subscribed to the Google Groups "openssl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openssl-users+unsubscribe@xxxxxxxxxxx.
To view this discussion on the web visit https://groups.google.com/a/openssl.org/d/msgid/openssl-users/3c1d1e07-f502-4d1d-bf4a-0a0a790fa9ec%40b1-systems.de.