On 6/11/2019 9:21 AM, Bernard Metzler wrote:
Hi all, If enabled for siw, during receive operation, a crc32c over header and data is being generated and checked. So far, siw was generating that CRC from the content of the just written target buffer. What kept me busy last weekend were spurious CRC errors, if running qperf. I finally found the application is constantly writing the target buffer while data are placed concurrently, which sometimes races with the CRC computation for that buffer, and yields a broken CRC.
Well, that's a clear bug in the application, assuming siw has not yet delivered a send completion for the operation using the buffer. This is a basic Verbs API contract.
siw uses skb_copy_bits() to move the data. I now added an extra round of skb walking via __skb_checksum() in front of it, which resolves the issue. Unfortunately, performance significantly drops with that (some 30% or worse compared to generating the CRC from a linear buffer). To preserve performance for kernel clients, I propose checksumming the data before the copy only for user land applications, and leave it as is for kernel clients. I am not aware of kernel clients which are constantly reading/writing a target buffer to detect it has been written.
This, too, is an invalid application. The RDMA provider is free to write the target buffer at any time. Many implementations take full advantage of this by placing received RDMA Writes in memory prior to validating their packets' checksum(s). If there is a mismatch, retries are initiated. The realtime contents of the buffers, prior to a completion, are undefined. Tom.
I also checked other kernel code using skb_copy_bits(), which also needs to checksum the received data. Those code (such as nvme tcp) also does the CRC on the linear buffer after data receive. The best solution might be to fold the CRC into the skb_copy_bits() function itself. That being something we might propose later? Thoughts? Many thanks, Bernard.
