On Wednesday, March 03/18/20, 2020 at 09:49:07 -0700, Sagi Grimberg wrote:
>
> >>Thanks Krishna,
> >>
> >>I assume that this makes the issue go away?
> >>--
> >>diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> >>index 11e10fe1760f..cc93e1949b2c 100644
> >>--- a/drivers/nvme/host/tcp.c
> >>+++ b/drivers/nvme/host/tcp.c
> >>@@ -889,7 +889,7 @@ static int nvme_tcp_try_send_data(struct
> >>nvme_tcp_request *req)
> >> flags |= MSG_MORE;
> >>
> >> /* can't zcopy slab pages */
> >>- if (unlikely(PageSlab(page))) {
> >>+ if (unlikely(PageSlab(page)) || queue->data_digest) {
> >> ret = sock_no_sendpage(queue->sock, page,
> >>offset, len,
> >> flags);
> >> } else {
> >>--
> >
> >Unfortunately, issue is still occuring with this patch also.
> >
> >Looks like the integrity of the data buffer right after the CRC
> >computation(data digest) is what causing this issue, despite the
> >buffer being sent via sendpage or no_sendpage.
>
> I assume this happens with iSCSI as well? There is nothing special
> we are doing with respect to digest.
I don't see this issue with iscsi-tcp.
May be blk-mq is causing this issue? I assume iscsi-tcp does not have
blk_mq support yet upstream to verify with blk_mq enabled.
I tried on Ubuntu 19.10(which is based on Linux kernel 5.3), note that
RHEL does not support DataDigest.
The reason that I'm seeing this issue only with NVMe(tcp/softiwarp) &
iSER(softiwarp) is becuase of NVMeF&ISER using blk-mq?
Anyhow, I see the content of the page is being updated by upper layers
while the tranport driver is computing CRC on that page content and
this needs a fix.
one could very easily recreate this issue running the below simple program over
NVMe/TCP.
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
int main() {
int i;
char* line1 = "123";
FILE* fp;
while(1) {
fp = fopen("/mnt/tmp.txt", "w");
setvbuf(fp, NULL, _IONBF, 0);
for (i=0; i<100000; i++)
if ((fwrite(line1, 1, strlen(line1), fp) !=
strlen(line1)))
exit(1);
if (fclose(fp) != 0)
exit(1);
}
return 0;
}
