On 15/12/2022 02:46, Joe Comeau wrote:
That's correct - we use the kernel target not tcmu-runner
Okay.
There are some difference for the configurations between kernel target
and the ceph-iscsi target.
Thanks,
- Xiubo
>>> Xiubo Li <xiubli@xxxxxxxxxx> 12/13/2022 6:02 PM >>>
On 14/12/2022 06:54, Joe Comeau wrote:
> I am curious about what is happening with your iscsi configuration
> Is this a new iscsi config or something that has just cropped up ?
>
> We are using/have been using vmware for 5+ years with iscsi
> We are using the kernel iscsi vs tcmu
>
Do you mean you are using kernel target, not the ceph-iscsi/tcmu-runner
in user space, right ?
> We are running ALUA and all datastores are setup as RR
> We routinely reboot the iscsi gateways - during patching and updates
and the storage migrates to and from all servers without issue
> We usually wait about 10 minutes before a gateway restart, so there
is not an outage
>
> It has been extremely stable for us
>
> Thanks Joe
>
>
>
>>>> Xiubo Li <xiubli@xxxxxxxxxx> 12/13/2022 4:21 AM >>>
> On 13/12/2022 18:57, Stolte, Felix wrote:
>> Hi Xiubo,
>>
>> Thx for pointing me into the right direction. All involved esx host
>> seem to use the correct policy. I am going to detach the LUN on each
>> host one by one until i found the host causing the problem.
>>
> From the logs it means the client was switching the path in turn.
>
> BTW, what's policy are you using ?
>
> Thanks
>
> - Xiubo
>
>> Regards Felix
>>
---------------------------------------------------------------------------------------------
>>
---------------------------------------------------------------------------------------------
>> Forschungszentrum Juelich GmbH
>> 52425 Juelich
>> Sitz der Gesellschaft: Juelich
>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>> Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>> Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
>>
---------------------------------------------------------------------------------------------
>>
---------------------------------------------------------------------------------------------
>>
>>> Am 12.12.2022 um 13:03 schrieb Xiubo Li <xiubli@xxxxxxxxxx>:
>>>
>>> Hi Stolte,
>>>
>>> For the VMware config could you refer to :
>>> https://docs.ceph.com/en/latest/rbd/iscsi-initiator-esx/
<https://docs.ceph.com/en/latest/rbd/iscsi-initiator-esx/> ?
>>>
>>> What's the "Path Selection Policy with ALUA" you are using ? The
>>> ceph-iscsi couldn't implement the real AA, so if you use the RR I
>>> think it will be like this.
>>>
>>> - Xiubo
>>>
>>> On 12/12/2022 17:45, Stolte, Felix wrote:
>>>> Hi guys,
>>>>
>>>> we are using ceph-iscsi to provide block storage for Microsoft
Exchange and vmware vsphere. Ceph docs state that you need to
configure Windows iSCSI Initatior for fail-over-only but there is no
such point for vmware. In my tcmu-runner logs on both ceph-iscsi
gateways I see the following:
>>>>
>>>> 2022-12-12 10:36:06.978 33789 [WARN] tcmu_notify_lock_lost:222
rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
>>>> 2022-12-12 10:36:06.993 33789 [INFO] alua_implicit_transition:570
rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.
>>>> 2022-12-12 10:36:08.064 33789 [WARN] tcmu_rbd_lock:762
rbd/mailbox.vmdk_junet_sata: Acquired exclusive lock.
>>>> 2022-12-12 10:36:09.067 33789 [WARN] tcmu_notify_lock_lost:222
rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
>>>> 2022-12-12 10:36:09.071 33789 [INFO] alua_implicit_transition:570
rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.
>>>> 2022-12-12 10:36:10.109 33789 [WARN] tcmu_rbd_lock:762
rbd/mailbox.vmdk_junet_sata: Acquired exclusive lock.
>>>> 2022-12-12 10:36:11.104 33789 [WARN] tcmu_notify_lock_lost:222
rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
>>>> 2022-12-12 10:36:11.106 33789 [INFO] alua_implicit_transition:570
rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.
>>>>
>>>> At the same time there are these log entries in ceph.audit.logs:
>>>> 2022-12-12T10:36:06.731621+0100 mon.mon-k2-1 (mon.1) 3407851 :
audit [INF] from='client.? 10.100.8.55:0/2392201639'
entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop":
"add", "addr": "10
>>>> .100.8.56:0/1598475844"}]: dispatch
>>>> 2022-12-12T10:36:06.731913+0100 mon.mon-e2-1 (mon.0) 783726 :
audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":
"osd blocklist", "blocklistop": "add", "addr":
"10.100.8.56:0/1598475844"}]
>>>> : dispatch
>>>> 2022-12-12T10:36:06.905082+0100 mon.mon-e2-1 (mon.0) 783727 :
audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":
"osd blocklist", "blocklistop": "add", "addr": "10.100.8.56:0/1598475844"}
>>>> ]': finished
>>>>
>>>> Can someone explaint to me, what is happening? Why are the
gateways blacklisting each other? All involved daemons are running
Version 16.2.10. ceph-iscsi gateways are running on Ubuntu 20.04 with
ceph-isci package from the Ubuntu repo (all other packers came
directly from ceph.com<http://ceph.com/ <http://ceph.com/>>)
>>>>
>>>>
>>>> regards Felix
>>>>
>>>>
---------------------------------------------------------------------------------------------
>>>>
---------------------------------------------------------------------------------------------
>>>> Forschungszentrum Juelich GmbH
>>>> 52425 Juelich
>>>> Sitz der Gesellschaft: Juelich
>>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>>> Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
>>>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>>>> Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
>>>>
---------------------------------------------------------------------------------------------
>>>>
---------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list --ceph-users@xxxxxxx
>>>> To unsubscribe send an email toceph-users-leave@xxxxxxx
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
