Re: CAPI engine seems to break server validation

[Brett Stahlman <brettstahlman@xxxxxxxxx>]

On Mon, Oct 26, 2020 at 9:35 AM Jakob Bohm <jb-openssl@xxxxxxxxxx> wrote:

On 2020-10-24 16:09, Brett Stahlman wrote:
> Jakob,
> I don't really understand why the engine *needs* to do PSS. Neither of
> the badssl certificates seem to use it for signatures. (I'm assuming the
> fact that a cert was signed with RSA-PSS would show up in the Windows
> certificate viewer...) If you could give a short summary of the problem
> as you understand it, perhaps it would help me narrow in on a
> workaround. I'd be happy with even an ugly patch at this point. Given
> that server verification works fine with a ca-bundle file, I wonder
> whether it would be possible to have the capi engine handle only the
> client authentication. As you understand it, would the problem breaking
> server verification also preclude client authentication with the capi
> engine?
>

 From the content of your mails, I inferred that whatever you tried to
do caused OpenSSL to attempt to generate PSS signatures, but failing to
pass that job to the CAPI engine.  I was commenting on how that might be
made to work.

The failure was occurring during server cert verification. Which was surprising because I'd provided a ca-bundle.crt file to be used for server authentication, and CAPI doesn't seem to get involved directly in server authentication anyways. But apparently, the presence of the CAPI engine was preventing openssl from validating the server cert the way it normally would. The error occurred in do_sigver_init (crypto/evp/m_sigver.c) at the `ERR_raise` below:

```

        /*
         * If we couldn't find a keymgmt at all try legacy.
         * TODO(3.0): Once all legacy algorithms (SM2, HMAC etc) have provider
         * based implementations this fallback shouldn't be necessary. Either
         * we have an ENGINE based implementation (in which case we should have
         * already fallen back in the test above here), or we don't have the
         * provider based implementation loaded (in which case this is an
         * application config error)
         */
        if (locpctx->keymgmt == NULL)
            goto legacy;
        ERR_raise(ERR_LIB_EVP, EVP_R_INITIALIZATION_ERROR);
        goto err;

```

I was able to get past the error simply by commenting the conditional several lines higher in the function:

//if (evp_pkey_ctx_is_legacy(locpctx))
    goto legacy;

For some reason, the call to evp_pkey_export_to_provider() fails when the CAPI engine is in use, but forcing the jump to legacy bypasses the test altogether and permits me to use the (working) legacy logic for server cert validation. I need to debug further to understand why evp_pkey_export_to_provider() is failing, but the simple change shown above allows two-way authentication to work with the client.badssl.com test site: client auth uses a cert in my Personal "MY" store; server auth uses the CA certs in ca-bundle.cer.

Any light anyone can shed on the history of the legacy fallback logic would be greatly appreciated...

Thanks,

Brett S.

> On Fri, Oct 23, 2020 at 11:34 AM Jakob Bohm via openssl-users
> <openssl-users@xxxxxxxxxxx <mailto:openssl-users@xxxxxxxxxxx>> wrote:
>
>     On 2020-10-23 15:45, Matt Caswell wrote:
>      >
>      > On 23/10/2020 14:10, Brett Stahlman wrote:
>      >> It seems that the CAPI engine is breaking the server
>     verification somehow.
>      >> Note that the only reason I'm using the ca-bundle.crt is that I
>     couldn't
>      >> figure out how to get CAPI to load the Windows "ROOT" certificate
>      >> store, which contains the requisite CA certs. Ideally, server
>      >> authentication would use the CA certs in the Windows "ROOT"
>     store, and
>      >> client authentication would use the certs in the Windows "MY"
>     store, but
>      >> CAPI doesn't appear to be loading either one.
>      > This is probably the following issue:
>      >
>      > https://github.com/openssl/openssl/issues/8872
>      >
>      > Matt
>     Looking at the brutal wontfixing of that bug, maybe reconsider if the
>     existing engine interface can do PSS by simply having the CAPI/CAPIng
>     engine export the generic PKEY type for PSS-capable RSA keys.  Also,
>     maybe use a compatible stronger CAPI "provider" (their engines) to do
>     stronger hashes etc.
>
>   


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded