Hi Sagi,
Hi Raju,
We are running nvmf link-test (toggling the link with some delay) to ensure that the connections are restored and I/O resumes on link-up. This test used to work until the commit e818a5b. But with this commit included, we see I/O errors as the link goes down, also when the link comes up, the nvmf disks are remounted as read-only and doesn't allow I/O run.
Is this a manual remount? or just after a successful reconnect?
As per commit message, this seems to be the expected behaviour. Could you please confirm if what we are seeing is expected?
The behavior is fail all inflight requests immediately when a link
failure is detected. Continuation of service is expected from a
multipath layer.
When it worked for you before, what was your link toggle delay?
Same behaviour is seen with other vendors too.
Here is the commit msg:
*commit e818a5b487fea20494b0e48548c1085634abdc0d
Author: Sagi Grimberg <sagi@xxxxxxxxxxx>
Date: Mon Jun 5 20:35:56 2017 +0300
nvme-rdma: fast fail incoming requests while we reconnect
When we encounter an transport/controller errors, error recovery
kicks in which performs:
1. stops io/admin queues
2. moves transport queues out of LIVE state
3. fast fail pending io
4. schedule periodic reconnects.
But we also need to fast fail incoming IO taht enters after we
already scheduled. Given that our queue is not LIVE anymore, simply
restart the request queues to fail in .queue_rq
I/O errors:
[Tue Apr 3 19:14:55 2018] print_req_error: I/O error, dev nvme2n1, sector 1108136
[Tue Apr 3 19:14:55 2018] Aborting journal on device nvme2n1-8.
[Tue Apr 3 19:14:55 2018] print_req_error: I/O error, dev nvme2n1, sector 1052688
[Tue Apr 3 19:14:55 2018] Buffer I/O error on dev nvme2n1, logical block 131586, lost sync page write
[Tue Apr 3 19:14:55 2018] JBD2: Error -5 detected when updating journal superblock for nvme2n1-8.
and IO fails to resume as the devices are remounted as read-only...
This looks like a journal write failed which was failed when link
failure occurred.
Can you verify if mount --remount makes this go away?
If this is the case, then yes, its expected. The driver will fail
inflight IO at some point, today it does this immediately, we could add
another timer to fail inflight IO which would allow to reconnect
before, but it would require some work.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
