There must be some things done in your provider and in the application (or OpenSSL configuration) to make this work seamlessly. 1. The provider must properly fail attempts to export the private key. I.e., it must never export a public key when it is asked to export a full keypair.
That is already implemented in the provider. If OSSL_KEYMGMT_SELECT_PRIVATE_KEY-bit is set in the selection_mask the provider export function returns 0.
2. The default property query must deprioritize your provider. I.e., "?provider!=yourprovider"
That did the trick. If I specify the statement as mentioned in the SSL_CTX_new_ex() function as propq argument the TLS connection is established and the signature is generated from the key stored in my secure element. But it did not work if I set it in my default query with EVP_set_default_properties().
3. When your application wants to use the key from your provider it needs to load it via a store uri.
That was also already implemented.
With this above everything should work correctly.
Thank you very much for pointing my in the right direction. Best regards, Timo
