I've opened a bug report https://tracker.ceph.com/issues/61589, which unfortunately received no attention. I fixed the issue by manually setting directory ownership for /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash and /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash/posted to 167:167, which in my system is the user ID crash process uses inside the crash container. /Z On Mon, 5 Jun 2023 at 11:24, Zakhar Kirpichenko <zakhar@xxxxxxxxx> wrote: > Any other thoughts on this, please? Should I file a bug report? > > /Z > > On Fri, 2 Jun 2023 at 06:11, Zakhar Kirpichenko <zakhar@xxxxxxxxx> wrote: > >> Thanks, Josh. The cluster is managed by cephadm. >> >> On Thu, 1 Jun 2023, 23:07 Josh Baergen, <jbaergen@xxxxxxxxxxxxxxxx> >> wrote: >> >>> Hi Zakhar, >>> >>> I'm going to guess that it's a permissions issue arising from >>> https://github.com/ceph/ceph/pull/48804, which was included in 16.2.13. >>> You may need to change the directory permissions, assuming that you manage >>> the directories yourself. If this is managed by cephadm or something like >>> that, then that seems like some sort of missing migration in the upgrade. >>> >>> Josh >>> >>> On Thu, Jun 1, 2023 at 12:34 PM Zakhar Kirpichenko <zakhar@xxxxxxxxx> >>> wrote: >>> >>>> Hi, >>>> >>>> I'm having an issue with crash daemons on Pacific 16.2.13 hosts. >>>> ceph-crash >>>> throws the following error on all hosts: >>>> >>>> ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; >>>> please create >>>> ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; >>>> please create >>>> ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; >>>> please create >>>> >>>> ceph-crash runs in docker, the container has the directory mounted: -v >>>> >>>> /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z >>>> >>>> The mount works correctly: >>>> >>>> 18:26 [root@ceph02 >>>> /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# ls >>>> -al crash/posted/ >>>> total 8 >>>> drwx------ 2 nobody nogroup 4096 May 6 2021 . >>>> drwx------ 3 nobody nogroup 4096 May 6 2021 .. >>>> >>>> 18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# >>>> touch crash/posted/a >>>> >>>> 18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# >>>> docker exec -it c0cd2b8022d8 bash >>>> >>>> [root@ceph02 /]# ls -al /var/lib/ceph/crash/posted/ >>>> total 8 >>>> drwx------ 2 nobody nobody 4096 Jun 1 18:26 . >>>> drwx------ 3 nobody nobody 4096 May 6 2021 .. >>>> -rw-r--r-- 1 root root 0 Jun 1 18:26 a >>>> >>>> I.e. the directory actually exists and is correctly mounted in the crash >>>> container, yet ceph-crash says it doesn't exist. How can I convince it >>>> that the directory is there? >>>> >>>> Best regards, >>>> Zakhar >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>>> >>> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |