I think this could be an issue with the system's /dev/urandom or entropy, as I've observed similar infinite loops in BN_prime when I changed OpenSSL code to always return the same sequence of bytes from its PRNG (for testing purposes). It could also be a genuine bug in OpenSSL, or both. I'll let others comment on that.
On Thu, Jun 18, 2020 at 9:47 AM Ronny Meeus <ronny.meeus@xxxxxxxxx> wrote:
Hello
we are in the process of upgrading our openssl to version 1.1.1g.
On one of our architectures (Cavium MIPS, running kernel 4.9) we have
an issue in the ssh-keygen tool. It keeps on consuming 100% CPU of 1
core.
On other architectures we do not see the issue at all.
I instrumented the openssl library with some traces and observed that
it keeps on looping in the "probable prime" function.
At the end of the function the "BN_num_bits" check is done and if the
return value is not equal to "bits" it basically starts all over
again.
}
if (!BN_add_word(rnd, delta))
return 0;
if (BN_num_bits(rnd) != bits) {
printf("%s BN_num_bits %d %d\n", __FUNCTION__, BN_num_bits(rnd), bits);
goto again;
}
bn_check_top(rnd);
return 1;
}
I added the print function and the result of the print is as follows:
probable_prime BN_num_bits 1473 1536
This trace keeps on going forever and the values never change.
Any idea what could be the underlying root-cause?
Many thanks and best regards,
Ronny
