Re: ceph-iscsi lock ping pong

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I am curious about what is happening with your iscsi configuration
Is this a new iscsi config or something that has just cropped up ?
 
We are using/have been using vmware for 5+ years with iscsi
We are using the kernel iscsi vs tcmu
 
We are running ALUA and all datastores are setup as RR
We routinely reboot the iscsi gateways - during patching and updates and the storage migrates to and from all servers without issue
We usually wait about 10 minutes before a gateway restart, so there is not an outage 
 
It has been extremely stable for us
 
Thanks Joe
 


>>> Xiubo Li <xiubli@xxxxxxxxxx> 12/13/2022 4:21 AM >>>

On 13/12/2022 18:57, Stolte, Felix wrote:
> Hi Xiubo,
>
> Thx for pointing me into the right direction. All involved esx host 
> seem to use the correct policy. I am going to detach the LUN on each 
> host one by one until i found the host causing the problem.
>
>From the logs it means the client was switching the path in turn.

BTW, what's policy are you using ?

Thanks

- Xiubo

> Regards Felix
> ---------------------------------------------------------------------------------------------
> ---------------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
> ---------------------------------------------------------------------------------------------
> ---------------------------------------------------------------------------------------------
>
>> Am 12.12.2022 um 13:03 schrieb Xiubo Li <xiubli@xxxxxxxxxx>:
>>
>> Hi Stolte,
>>
>> For the VMware config could you refer to : 
>> https://docs.ceph.com/en/latest/rbd/iscsi-initiator-esx/ ?
>>
>> What's the "Path Selection Policy with ALUA" you are using ? The 
>> ceph-iscsi couldn't implement the real AA, so if you use the RR I 
>> think it will be like this.
>>
>> - Xiubo
>>
>> On 12/12/2022 17:45, Stolte, Felix wrote:
>>> Hi guys,
>>>
>>> we are using ceph-iscsi to provide block storage for Microsoft Exchange and vmware vsphere. Ceph docs state that you need to configure Windows iSCSI Initatior for fail-over-only but there is no such point for vmware. In my tcmu-runner logs on both ceph-iscsi gateways I see the following:
>>>
>>> 2022-12-12 10:36:06.978 33789 [WARN] tcmu_notify_lock_lost:222 rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
>>> 2022-12-12 10:36:06.993 33789 [INFO] alua_implicit_transition:570 rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.
>>> 2022-12-12 10:36:08.064 33789 [WARN] tcmu_rbd_lock:762 rbd/mailbox.vmdk_junet_sata: Acquired exclusive lock.
>>> 2022-12-12 10:36:09.067 33789 [WARN] tcmu_notify_lock_lost:222 rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
>>> 2022-12-12 10:36:09.071 33789 [INFO] alua_implicit_transition:570 rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.
>>> 2022-12-12 10:36:10.109 33789 [WARN] tcmu_rbd_lock:762 rbd/mailbox.vmdk_junet_sata: Acquired exclusive lock.
>>> 2022-12-12 10:36:11.104 33789 [WARN] tcmu_notify_lock_lost:222 rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
>>> 2022-12-12 10:36:11.106 33789 [INFO] alua_implicit_transition:570 rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.
>>>
>>> At the same time there are these log entries in ceph.audit.logs:
>>> 2022-12-12T10:36:06.731621+0100 mon.mon-k2-1 (mon.1) 3407851 : audit [INF] from='client.? 10.100.8.55:0/2392201639' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "10
>>> .100.8.56:0/1598475844"}]: dispatch
>>> 2022-12-12T10:36:06.731913+0100 mon.mon-e2-1 (mon.0) 783726 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "10.100.8.56:0/1598475844"}]
>>> : dispatch
>>> 2022-12-12T10:36:06.905082+0100 mon.mon-e2-1 (mon.0) 783727 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "10.100.8.56:0/1598475844"}
>>> ]': finished
>>>
>>> Can someone explaint to me, what is happening? Why are the gateways blacklisting each other? All involved daemons are running Version 16.2.10. ceph-iscsi gateways are running on Ubuntu 20.04 with ceph-isci package from the Ubuntu repo (all other packers came directly from ceph.com<http://ceph.com/>)
>>>
>>>
>>> regards Felix
>>>
>>> ---------------------------------------------------------------------------------------------
>>> ---------------------------------------------------------------------------------------------
>>> Forschungszentrum Juelich GmbH
>>> 52425 Juelich
>>> Sitz der Gesellschaft: Juelich
>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>> Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
>>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>>> Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
>>> ---------------------------------------------------------------------------------------------
>>> ---------------------------------------------------------------------------------------------
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list --ceph-users@xxxxxxx
>>> To unsubscribe send an email toceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux