Issue is resolved now. After verifying that all esx hosts are configured for MRU, i took a closer look on the paths on each host. `gwcli` reported lun in question was owned by gateway A, but one esx host used the path to gateway B for I/O. I reconfigured that particular host and it’s now using the correct path to gateway A. Logs are clean now and I/O on that Datastore is back to normal. This was probably caused by an outage of one of our gateways last week (the physical host, not the daemon), where the iSCSI Daemon didn’t shut down cleanly. One last question though: >From my understanding the "Dynamic Discovery“ just creates the „Static Discovery“ Targets for all available gateways. Is it also responsible for telling the client, which path to use (aka which gateway is the owner of a LUN)? --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Am 13.12.2022 um 13:21 schrieb Xiubo Li <xiubli@xxxxxxxxxx>: On 13/12/2022 18:57, Stolte, Felix wrote: Hi Xiubo, Thx for pointing me into the right direction. All involved esx host seem to use the correct policy. I am going to detach the LUN on each host one by one until i found the host causing the problem. >From the logs it means the client was switching the path in turn. BTW, what's policy are you using ? Thanks - Xiubo Regards Felix --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Am 12.12.2022 um 13:03 schrieb Xiubo Li <xiubli@xxxxxxxxxx><mailto:xiubli@xxxxxxxxxx>: Hi Stolte, For the VMware config could you refer to : https://docs.ceph.com/en/latest/rbd/iscsi-initiator-esx/ ? What's the "Path Selection Policy with ALUA" you are using ? The ceph-iscsi couldn't implement the real AA, so if you use the RR I think it will be like this. - Xiubo On 12/12/2022 17:45, Stolte, Felix wrote: Hi guys, we are using ceph-iscsi to provide block storage for Microsoft Exchange and vmware vsphere. Ceph docs state that you need to configure Windows iSCSI Initatior for fail-over-only but there is no such point for vmware. In my tcmu-runner logs on both ceph-iscsi gateways I see the following: 2022-12-12 10:36:06.978 33789 [WARN] tcmu_notify_lock_lost:222 rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1 2022-12-12 10:36:06.993 33789 [INFO] alua_implicit_transition:570 rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation. 2022-12-12 10:36:08.064 33789 [WARN] tcmu_rbd_lock:762 rbd/mailbox.vmdk_junet_sata: Acquired exclusive lock. 2022-12-12 10:36:09.067 33789 [WARN] tcmu_notify_lock_lost:222 rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1 2022-12-12 10:36:09.071 33789 [INFO] alua_implicit_transition:570 rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation. 2022-12-12 10:36:10.109 33789 [WARN] tcmu_rbd_lock:762 rbd/mailbox.vmdk_junet_sata: Acquired exclusive lock. 2022-12-12 10:36:11.104 33789 [WARN] tcmu_notify_lock_lost:222 rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1 2022-12-12 10:36:11.106 33789 [INFO] alua_implicit_transition:570 rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation. At the same time there are these log entries in ceph.audit.logs: 2022-12-12T10:36:06.731621+0100 mon.mon-k2-1 (mon.1) 3407851 : audit [INF] from='client.? 10.100.8.55:0/2392201639' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "10 .100.8.56:0/1598475844"}]: dispatch 2022-12-12T10:36:06.731913+0100 mon.mon-e2-1 (mon.0) 783726 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "10.100.8.56:0/1598475844"}] : dispatch 2022-12-12T10:36:06.905082+0100 mon.mon-e2-1 (mon.0) 783727 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "10.100.8.56:0/1598475844"} ]': finished Can someone explaint to me, what is happening? Why are the gateways blacklisting each other? All involved daemons are running Version 16.2.10. ceph-iscsi gateways are running on Ubuntu 20.04 with ceph-isci package from the Ubuntu repo (all other packers came directly from ceph.com <http://ceph.com/><http://ceph.com/>) regards Felix --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |