Re: 2 fast allocations != 4 num_osds

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 22/08/2021 00.42, Torkil Svensgaard wrote:
Hi

Any suggestions as to the cause of this error? The device list seems fine, a mix of already active OSDs and 4 empty, available drives.

There were 2 orphaned LVs on the db device. After I removed those the 4 available devices came up as OSDs. There were plenty of space on the db device though, so unsure how the orphans came to block OSD creation.

Mvh.

Torkil

RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb -e NODE_NAME=dcn-ceph-03 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -v /var/run/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984:/var/run/ceph:z -v /var/log/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984:/var/log/ceph:z -v /var/lib/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984/selinux:/sys/fs/selinux:ro -v /tmp/ceph-tmpee_fkmi9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpeapr11q1:/var/lib/ceph/bootstrap-osd/ceph.keyring:z docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb lvm batch --no-auto /dev/sdaa /dev/sdab /dev/sdac /dev/sdh /dev/sdk /dev/sdo /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --db-devices /dev/nvme0n1 --block-db-size 67000000000 --yes --no-systemd
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 582, in _apply_all_services
     if self._apply_service(spec):
   File "/usr/share/ceph/mgr/cephadm/serve.py", line 639, in _apply_service
     self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec))
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec
     ret = create_from_spec_one(self.prepare_drivegroup(drive_group))
  File "/usr/share/ceph/mgr/cephadm/utils.py", line 79, in forall_hosts_wrapper
     return CephadmOrchestrator.instance._worker_pool.map(do_work, vals)
   File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map
     return self._map_async(func, iterable, mapstar, chunksize).get()
   File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get
     raise self._value
   File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker
     result = (True, func(*args, **kwds))
   File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar
     return list(map(*args))
   File "/usr/share/ceph/mgr/cephadm/utils.py", line 73, in do_work
     return f(*arg)
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one
     replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host
     code, '\n'.join(err)))
RuntimeError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb -e NODE_NAME=dcn-ceph-03 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -v /var/run/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984:/var/run/ceph:z -v /var/log/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984:/var/log/ceph:z -v /var/lib/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984/selinux:/sys/fs/selinux:ro -v /tmp/ceph-tmpee_fkmi9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpeapr11q1:/var/lib/ceph/bootstrap-osd/ceph.keyring:z docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb lvm batch --no-auto /dev/sdaa /dev/sdab /dev/sdac /dev/sdh /dev/sdk /dev/sdo /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --db-devices /dev/nvme0n1 --block-db-size 67000000000 --yes --no-systemd
/usr/bin/docker: stderr --> passed data devices: 11 physical, 0 LVM
/usr/bin/docker: stderr --> relative data size: 1.0
/usr/bin/docker: stderr --> passed block_db devices: 1 physical, 0 LVM
/usr/bin/docker: stderr --> 2 fast allocations != 4 num_osds
Traceback (most recent call last):
  File "/var/lib/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in <module>
     main()
  File "/var/lib/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main
     r = ctx.func(ctx)
  File "/var/lib/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1653, in _infer_fsid
     return func(ctx)
  File "/var/lib/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1737, in _infer_image
     return func(ctx)
  File "/var/lib/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4599, in command_ceph_volume
     out, err, code = call_throws(ctx, c.run_cmd())
  File "/var/lib/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws
     raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb -e NODE_NAME=dcn-ceph-03 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -v /var/run/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984:/var/run/ceph:z -v /var/log/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984:/var/log/ceph:z -v /var/lib/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/3b7736c6-00e4-11ec-a3c5-3cecef467984/selinux:/sys/fs/selinux:ro -v /tmp/ceph-tmpee_fkmi9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpeapr11q1:/var/lib/ceph/bootstrap-osd/ceph.keyring:z docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb lvm batch --no-auto /dev/sdaa /dev/sdab /dev/sdac /dev/sdh /dev/sdk /dev/sdo /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --db-devices /dev/nvme0n1 --block-db-size 67000000000 --yes --no-systemd

This is the OSD spec which works just fine for for the two other similar hosts in the cluster:

service_type: osd
service_id: slow
placement:
   host_pattern: '*'
data_devices:
   rotational: 1
#  size: '10000G:'
db_devices:
   rotational: 0
   size: '1000G:'
db_slots: 22
block_db_size: 67000000000

Thanks,

Torkil
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux