On Wed, Oct 27, 2021 at 08:47:10AM -0700, Jakub Kicinski wrote:
> On Wed, 27 Oct 2021 17:38:27 +0200 Karsten Graul wrote:
> > What we found out was that applications called sendmsg() with large data
> > buffers using blocking sockets. This led to the described situation, were the
> > solution was to early return to user space even if not all data were sent yet.
> > Userspace applications should not have a problem with the fact that sendmsg()
> > returns a smaller byte count than requested.
> >
> > Reverting this patch would bring back the stalled connection problem.
>
> I'm not sure. The man page for send says:
>
> When the message does not fit into the send buffer of the socket,
> send() normally blocks, unless the socket has been placed in nonblock‐
> ing I/O mode. In nonblocking mode it would fail with the error EAGAIN
> or EWOULDBLOCK in this case.
>
> dunno if that's required by POSIX or just a best practice.
The man page describes the common cases about the socket API behavior,
but depends on the implement.
For example, the connect(2) implement of TCP, it would never block, but
also provides EAGAIN errors for UNIX domain sockets.
EAGAIN For nonblocking UNIX domain sockets, the socket is
nonblocking, and the connection cannot be completed
immediately. For other socket families, there are
insufficient entries in the routing cache.
In my opinion, if we are going to replace TCP with SMC, these userspace
socket API should behavior as same, and don't break the userspace
applications like netperf. It could be better to block here when sending
message without enough buffer.
In our benchmarks and E2E tests (Redis, MySQL, etc.), it is acceptable to
block here. Because the userspace applications usually block in the loop
until the data send out. If it blocks, the scheduler can handle it.
