Re: PKCS7_decrypt vs RSA OAEP padding

Eliot Lear <lear@xxxxxxxxxxxxxxxxxxx> · Mon, 12 Apr 2021 19:24:42 +0200



    Hi Michal,
    CMS has limited backward compatibility with PKCS#7.  This is
      discussed in RFC 5652,
      and includes some suggestions as to how to some issues that might
      crop up.   At least the old draft of SCEP very specifically does
      NOT specify CMS, but that might not have stopped someone from
      using it.  These docs tend to be quite sloppy.  For example, I
      know one that uses both PEM and DER encodings on the wire.  Drives
      me batty.
    For enrolment this is particularly embarrassing, considering its
      sensitivity.  As a sysadmin, you might try wrapping this stuff 
in
      PHP and just trying both the pkcs_decrypt and cms_decrypt calls
      and seeing if either work.
    Eliot

    
    On 12.04.21 14:13, Michal Moravec
      wrote:

    
      Hello,
        

        I am a system administrator trying to integrate
          two pieces of software using the SCEP protocol (more on that
          later in the More Context section) .
        Integration was working but one side (SaaS
          product) changed the algorithms it uses for signing and
          encryption within the PKCS7 message.
        After that I am unable to decrypt the PKCS7
          messages on my end and get the error: "ERROR
          139701710696896:error:06065064:digital envelope
          routines:EVP_DecryptFinal_ex:bad
          decrypt:../crypto/evp/evp_enc.c:597:

          message_static_functions.c:221: decryption failed"
        

        From my current state of investigation it looks
          highly likely there is a problem with PKCS7_decrypt method
          which might be unable decrypt the PKCS7/CMS payload if the RSA
          is used together with OAEP padding.
        

        1. What is a diffrences between pkcs7 https://github.com/openssl/openssl/tree/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/pkcs7 and
          CMS 
https://github.com/openssl/openssl/tree/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/cms implementations?
        

        If I understand correctly CMS is just newer name
          for PKCS7 adopted by newer RFCs?
        Do these two implementations overlap?
        Or are there some kind of major diffrence like
          PKCS7 module being there for compatibility with older code and
          CMS is basically replacement to use for the future?
        

        2. Is PKCS7_decrypt method in 
https://github.com/openssl/openssl/blob/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/pkcs7/pk7_smime.c capable
          of decrypting the PKCS7 payload whebf OAEP padding is used
          together with RSA to encrypt it? If it is indeed not capable
          of doing that would it be a bug or desired behavior?
        

        If found a few pointers:
        

        - https://stackoverflow.com/questions/56941480/how-to-set-padding-oaep-for-pkcs7-decrypt-function-using-openssl <-
          Developer is unable to use PKCS7_decrypt with RSA OAEP and
          forced to switch to CMS_decrypt method.
        - https://stackoverflow.com/questions/34304570/how-to-resolve-the-evp-decryptfinal-ex-bad-decrypt-during-file-decryption <-
          Padding being a source problem of EVP_DecryptFinal_ex:bad
          decrypt error message.
        

        3. How to replace PKCS7_decrypt method CMS_decrypt
          as easily as possible?
        

          I have no hard proof of the PKCS7_decrypt being
            the culprit here so I started working on simple C program to
            reproduce the issue. 
          Problem is my C skills are very rusty and I have
            no experience working with OpenSSL C library.
        
        
          int PKCS7_decrypt(PKCS7 *p7,       
           
             EVP_PKEY *pkey, X509 *cert, BIO *data,                    
                         int flags);
          
            int CMS_decrypt    (CMS_ContentInfo 
*cms,
              EVP_PKEY *pkey, X509 *cert, BIO *dcont, BIO *out, unsigned
              int flags);
          
        
        Methods are very similiar but input data format it
          different. It there an easyway to convert PKCS7 struct into
          CMS_ContentInfo struct?
        

        Original usage of PKCS7_decrypt here -> https://github.com/openxpki/libscep/blob/4067eae283ce0b3025d414e9d3b6af30def8c093/src/message_static_functions.c 
        

          # More Context
        
        
          My goal is to obtain signed client certificate
            using the SCEP protocol.
          

          SCEP server: OpenXPKI 
              https://github.com/openxpki/openxpki (Perl) with
            libscep https://github.com/openxpki/libscep library
            (C). libscep used the OpenSSL.
          SCEP client: VMware Workspace ONE UEM SaaS
            (WSO). Black box. From the error codes client produces I
            strongly suspect they use OpenSSL within .Net code.

            
            Originally WSO SCEP client used these three algorithms for
            PKCS7 operation: SHA-1, RSA (PKCS 1.5 padding or no padding
            <- can't tell which) and 3DES-CBC.
          They made some sort of change (unannouced) and
            since the last SaaS upgdate WSO client uses SHA-2 (256bit),
            RSA with OAEP padding and AES-256-CBC.

            Compare the attached PKCS7 messages (libscep_wso*.p7b). I
            use https://lapo.it/asn1js to
            decode them easily.
          After the change I am unable to decrypt the
            PKIOperation message on the server side.
          

          I suspect the RSA with OAEP is the cause of the
            problem because using different client with SHA-256,
            AES-256-CBC and RSA decryption works without any problem
            (libscep_sscep_ok.p7b).
          

          Best Regards,
          Michal Moravec
          

Attachment:
OpenPGP_signature

Description: OpenPGP digital signature