Hello,
I believe that I have narrowed the problem down to one specific version of OpenSSL. Version 1.1.0b works as expected while OpenSSL 1.1.0c does not.
I have currently only verified this using PKCS7_verify and CMS_verify since I have no CLI at hand for these versions.
The changelog for 1.1.0c describes the following change for CMS:
*) CMS Null dereference
Applications parsing invalid CMS structures can crash with a NULL pointer
dereference. This is caused by a bug in the handling of the ASN.1 CHOICE
type in OpenSSL 1.1.0 which can result in a NULL value being passed to the
structure callback if an attempt is made to free certain invalid encodings.
Only CHOICE structures using a callback which do not handle NULL value are
affected.
This issue was reported to OpenSSL by Tyler Nighswander of ForAllSecure.
(CVE-2016-7053)
[Stephen Henson]