|
Just to give a short feedback - everything is fine now:
- via ceph-ansible we got some tcmu-runner / ceph-iscsi development versions - our ISCSI alua setup was a mess (it was a mixture of explicit and implicit alua while only implicit alua is supported at the moment) - our multipath devices showed the same priorities for all of our pathes (instead of 50 / 10 - 10 - 10 priorities)
Fix:
- shutdown compelte iscsi traffic - iscsiadm logout / multipath -F (removes all devices) - update ceph-iscsi & tcmu runner to stable versions - reinitialize iscsi devices: login & multipath
Now it looks like it should with only implicit alua mode and the correct priorities on our multipath devices ;)
Thanks @Mike for your help!
Von: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> im Auftrag von Kilian Ries <mail@xxxxxxxxxxxxxx>
Gesendet: Dienstag, 22. Oktober 2019 23:38:51 An: Mike Christie; ceph-users@xxxxxxxxxxxxxx Betreff: Re: TCMU Runner: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown - Each LUN is exported to multiple clients (at the same time) - yes, IO is done to the LUNs (read and write); (oVirt runs VMs on each of the LUNs)
Ok, i'll update this tomorrow with the logs you asked for ... Von: Mike Christie <mchristi@xxxxxxxxxx>
Gesendet: Dienstag, 22. Oktober 2019 19:43:40 An: Kilian Ries; ceph-users@xxxxxxxxxxxxxx Betreff: Re: TCMU Runner: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown On 10/22/2019 03:20 AM, Kilian Ries wrote:
> Hi, > > > i'm running a ceph cluster with 4x ISCSI exporter nodes and oVirt on the > client side. In the tcmu-runner logs i the the following happening every > few seconds: > > Are you exporting a LUN to one client or multiple clients at the same time? > > tcmu-runner-1.4.0-106.gd17d24e.el7.x86_64 Are you doing any IO to the iscsi LUN? If not, then we normally saw this with a older version. It would start at dm-multipath initialization and then just continue forever. Your package looks like it has the fix: commit dd7dd51c6cafa8bbcd3ca0eef31fb378b27ff499 Author: Mike Christie <mchristi@xxxxxxxxxx> Date: Mon Jan 14 17:06:27 2019 -0600 Allow some commands to run while taking lock so we should not be seeing it. Could you turn on tcmu-runner debugging? Open the file: /etc/tcmu/tcmu.conf and set: log_level = 5 Do this while you are hitting this bug. I only need a couple seconds so I can see what commands are being sent. |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
