Summary: The OpenSSL 1.1.1g test suite contains at least two bugs:
TestBug#1: Test suite fails if local network has no IPv6, error message
(non-
verbose) doesn't say that's the issue. [ Testing IPv6 makes sense,
rejecting
regression tests on inadequate machines is important to avoid silent
failures,
but not telling testers that a test precondition failed or which ones
is bad.
]
TestBug#2: Test suite uses ambiguous wording to report the index of
failed test.
Should have said "Failed test indexes: 2" or "Failed test: #2" (latter
needs to repeat "#" for each index listed)
On 21/04/2020 19:34, Claus Assmann wrote:
On Tue, Apr 21, 2020, Benjamin Kaduk via openssl-users wrote:
On Tue, Apr 21, 2020 at 07:22:38PM +0200, Claus Assmann wrote:
../test/recipes/80-test_ssl_old.t ..................
Dubious, test returned 1 (wstat 256, 0x100)
Please run again with `make V=1 TESTS=test_ssl_old test` and post the relevant parts of the output?
Thanks for the reply, below is the output, It seems it only fails
because the host doesn't support IPv6?
make depend && make _tests
( cd test; mkdir -p test-runs; SRCTOP=../. BLDTOP=../. RESULT_D=test-runs PERL="/usr/bin/perl" EXE_EXT= OPENSSL_ENGINES=`cd .././engines 2>/dev/null && pwd` OPENSSL_DEBUG_MEMORY=on /usr/bin/perl .././test/run_tests.pl test_ssl_old )
../test/recipes/80-test_ssl_old.t ..
1..6
# Subtest: test_ss
1..17
...
0:error:0200E016:system library:setsockopt:Invalid argument:crypto/bio/b_sock2.c:255:
0:error:2008B088:BIO routines:BIO_listen:listen v6 only:crypto/bio/b_sock2.c:256:
Doing handshakes=1 bytes=256
TLSv1.3, cipher (NONE) (NONE)
../../util/shlib_wrap.sh ../ssltest_old -s_key keyU.ss -s_cert certU.ss -c_key keyU.ss -c_cert certU.ss -ipv6 => 1
not ok 13 - test TLS via IPv6
# Failed test 'test TLS via IPv6'
# at ../test/recipes/80-test_ssl_old.t line 390.
# Looks like you failed 1 test of 13.
not ok 2 - standard SSL tests
# Failed test 'standard SSL tests'
# at /home/ca/pd/security/openssl-1.1.1g/test/../util/perl/OpenSSL/Test.pm line 1212.
...
# Looks like you failed 1 test of 6.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/6 subtests
Test Summary Report
-------------------
../test/recipes/80-test_ssl_old.t (Wstat: 256 Tests: 6 Failed: 1)
Failed test: 2
Non-zero exit status: 1
Files=1, Tests=6, 12 wallclock secs ( 0.04 usr 0.06 sys + 1.77 cusr 9.78 csys = 11.65 CPU)
Result: FAIL
*** Error 1 in . (Makefile:217 '_tests')
*** Error 1 in /home/ca/pd/security/openssl-1.1.1g (Makefile:205 'tests')
Enjoy
Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. http://www.wisemo.com
Transformervej 29, 2860 Soborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded
