Re: BIO_read fails when performing DTLS handshake

[Matthew <matthew.goulart@xxxxxxxxx>]

Hi Michael,

Of course I should have provided a version: 1.1.1c, on linux-x64 (dotnet core 2.2)

Just a quick question while I prepare a clearer, more contextual and concise email about my problem;

Is it normal for an SSL's readBio to be empty right after DTLSv1_listen?

I noticed that after calling DTLSv1_listen, and it returning 1, the SSL's readBio's bytes pending = 0. Would this not force the client to send a second hello with cookie? Or does DTLSv1_listen perform the clientHello processing and then let SSL_accept send the response?

On 2019-08-31 10:51 a.m., Michael Wojcik wrote:

From: openssl-users [mailto:openssl-users-bounces@xxxxxxxxxxx] On Behalf Of Matthew
Sent: Friday, August 30, 2019 23:06

Welcome to the list. When posting, please remember to tell us what version of OpenSSL you're using, and what platform you're on.

Since you're talking about C#, I'll assume the platform is Windows. And I'll assume you're working with OpenSSL 1.1.1c, because that would be the sensible thing to do. But it would be better if I didn't have to make either assumption.

I stepped through both the working unit test and the non-working one in order to find
differences in the result. What I have found is that, in ssl3_read_n, the call to
BIO_read (line 300 in rec_layer_s3.c) returns -1.
ret = BIO_read(s->rbio, pkt + len + left, max - left);

At this line, pkt is a char[8], len and left = 0 and max = 16717

I don't think pkt is a char[8]. It's defined at the top of ssl3_read_n as unsigne char *pkt.

And it had better not be a char[8], since 1) plain char and unsigned char are not the same type, and 2) if max - left is 16717, then you have potential for a massive buffer overflow.

I'm curious as to why the "data" argument is not a pointer to a buffer, but rather
the result of an addition. Maybe my C isnt strong enough...

It's a pointer into a buffer (specifically, in this case, a pointer to the start of a buffer). In C, adding an integer type to a pointer results in a pointer value. That is, in fact, basic C.

(At least you're only reading C, not writing it. I have in recent days seen C code posted by people who really need to put the language down and back away slowly. C should not be used by people who don't know the language very well.)

Going even further down the stack, I finally end up at the bottom:
static int mem_read(BIO *b, char *out, int outl)
...
} else if (bm->length == 0) {
    ret = b->num;
    if (ret != 0)
        BIO_set_retry_read(b);
}
return ret;

At this point, ret = -1...

So b->num == -1 when you arrived here with bm->length == 0.

The num field is initialized to -1 when a memory BIO is initialized (mem_init in bss_mem.c). And the length is 0, which means there's no data in the BIO.

I don't know (without reading through your code, which I don't have time to do right now) why you're using a memory BIO, or how you've initialized it. It looks like you've simply never put any data into it.

--
Michael Wojcik
Distinguished Engineer, Micro Focus