Re: Cephadm bootstrap sets incorrect user & group id for ceph user

Sebastian Wagner <sewagner@xxxxxxxxxx> · Thu, 26 Aug 2021 14:00:36 +0200



    Hi Massim,
    cephadm looks for the IDs in the container:
    https://github.com/ceph/ceph/blob/master/src/cephadm/cephadm#L4499
    so as long as the host's (in this case the VM) IDs stay the same
      thorough the lifecycle, everything works. This seems to be an edge
      case for me. Would you be interested in adding a check to cephadm?
      Maybe to 

    
    https://github.com/ceph/ceph/blob/b7b73d5daa020643d0d4b1bba0d6fb6b091b005b/src/cephadm/cephadm#L5798
    

    ?

    
    Am 26.08.21 um 13:33 schrieb Massim
      Knaapen:

    
          Hi,
          

          I got this cluster setup on 3VMs for testing purposes.
            Every time one of the boxes rebooted the OSDs daemons
            failed.
          They worked fine, as long as I didn't reboot....
          

          Then I noticed that the UID and GID for the Ceph user
            were set incorrectly.
          If I understand correctly: the `ceph) user needs to have
            a UID and GID set to 167.
          In the Docker container this was correct, but on the host
            it was a different story.
          

          The fix was quite easy of course:
          $ usermod -u 167 ceph
          $ groupmod -u 167 ceph
          

          But the errors are not that clear (only found some
            reference about this in the OSD service log, see below).
          

          If Ceph needs to have this user set to this specific ID.
            Maybe it would be nice to add this to the Troubleshooting
            section of the docs?
          

          This is the output of the OSD service log (in case
            someone tries to search for this error):
          `Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.133+0000 7f025ef4a080  0 set uid:gid to
            167:167 (ceph:ceph)                                        
                                               

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.133+0000 7f025ef4a080  0 ceph version
            16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific
            (stable), process ceph-osd, pid 8      

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.133+0000 7f025ef4a080  0 pidfile_write:
            ignore empty --pid-file                                    
                                               

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.137+0000 7f025ef4a080  1
            bdev(0x5646bc068800 /var/lib/ceph/osd/ceph-0/block) open
            path /var/lib/ceph/osd/ceph-0/block                  

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.137+0000 7f025ef4a080  1
            bdev(0x5646bc068800 /var/lib/ceph/osd/ceph-0/block) open
            size 214744170496 (0x31ffc00000, 200 GiB) block_size>

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.137+0000 7f025ef4a080  1
            bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes
            cache_size 1073741824 meta 0.45 kv 0.45 data 0.06        

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.137+0000 7f025ef4a080  1
            bdev(0x5646bc068c00 /var/lib/ceph/osd/ceph-0/block) open
            path /var/lib/ceph/osd/ceph-0/block                  

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.137+0000 7f025ef4a080  1
            bdev(0x5646bc068c00 /var/lib/ceph/osd/ceph-0/block) open
            size 214744170496 (0x31ffc00000, 200 GiB) block_size>

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.137+0000 7f025ef4a080  1 bluefs
            add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block
            size 200 GiB                            

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.137+0000 7f025ef4a080  1
            bdev(0x5646bc068c00 /var/lib/ceph/osd/ceph-0/block) close  
                                                              

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.161+0000 7f025ef4a080  1
            bdev(0x5646bc068800 /var/lib/ceph/osd/ceph-0/block) close  
                                                              

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.425+0000 7f025ef4a080  0 starting osd.0
            osd_data /var/lib/ceph/osd/ceph-0
            /var/lib/ceph/osd/ceph-0/journal                            
            

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.425+0000 7f025ef4a080 -1 Falling back to
            public interface                                            
                                             

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.429+0000 7f025ef4a080  0 load: jerasure
            load: lrc load: isa                                        
                                               

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.437+0000 7f025ef4a080  1
            bdev(0x5646bc03c400 /var/lib/ceph/osd/ceph-0/block) open
            path /var/lib/ceph/osd/ceph-0/block                  

            Aug 24 17:00:09 alpha0001 bash[7301]: debug
            2021-08-24T17:00:09.437+0000 7f025ef4a080 -1
            bdev(0x5646bc03c400 /var/lib/ceph/osd/ceph-0/block) open
            open got: (13) Permission denied  `
        
      
      _______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx

    
Attachment:
OpenPGP_signature

Description: OpenPGP digital signature
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx