Re: Recreating a purged OSD fails

Paul Emmerich <paul.emmerich@xxxxxxxx> · Wed, 27 Jun 2018 08:57:55 +0200

You are running into https://tracker.ceph.com/issues/24423
I've fixed it here: https://github.com/ceph/ceph/pull/22585

The fix has already been backported and will be in 13.2.1

Paul

2018-06-27 8:40 GMT+02:00 Steffen Winther Sørensen <stefws@xxxxxxxxx>:
List,
Had a failed disk behind an OSD in a Mimic Cluster 13.2.0, so I tried following the doc on removal of an OSD.

I did:

# ceph osd crush reweight osd.19 0
waited for rebalancing to finish and cont.:
# ceph osd out 19
# systemctl stop ceph-osd@19
# ceph osd purge 19 --yes-i-really-mean-it

verified that osd.19 was out of map w/ ceph osd tree

Still found this tmpfs mounted though to my surprise:
tmpfs                    7.8G   48K  7.8G   1% /var/lib/ceph/osd/ceph-19

Replaced the failed drive and then attempted:

# ceph-volume lvm zap /dev/sdh
# ceph-volume lvm create --osd-id 19 --data /dev/sdh
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring 
/var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring 
/var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 5352d594-aa19-4147-a884-ca
2c5775aa1b
Running command: /usr/sbin/vgcreate --force --yes ceph-a2ebf47b-fa4a-43ce-b087-1
2dbafb5796e /dev/sdh
 stderr: WARNING: Device for PV CdiFOZ-n89Z-G5EF-JBBV-GFfU-bDRV-VJQHho not found
 or rejected by a filter.
 stderr: WARNING: Device for PV CdiFOZ-n89Z-G5EF-JBBV-GFfU-bDRV-VJQHho not found
 or rejected by a filter.
 stderr: /dev/ceph-a6541e3f-0a7f-4268-823c-668c515b5edc/osd-block-efae9323-b934-
408e-a4f9-1e1f62d88f2d: read failed after 0 of 4096 at 0: Input/output error
  /dev/ceph-a6541e3f-0a7f-4268-823c-668c515b5edc/osd-block-efae9323-b934-408e-a4
f9-1e1f62d88f2d: read failed after 0 of 4096 at 146775408640: Input/output error
  /dev/ceph-a6541e3f-0a7f-4268-823c-668c515b5edc/osd-block-efae9323-b934-408e-a4
f9-1e1f62d88f2d: read failed after 0 of 4096 at 146775465984: Input/output error
 stderr: /dev/ceph-a6541e3f-0a7f-4268-823c-668c515b5edc/osd-block-efae9323-b934-
408e-a4f9-1e1f62d88f2d: read failed after 0 of 4096 at 4096: Input/output error
 stderr: WARNING: Device for PV CdiFOZ-n89Z-G5EF-JBBV-GFfU-bDRV-VJQHho not found
 or rejected by a filter.
 stdout: Physical volume "/dev/sdh" successfully created.
 stdout: Volume group "ceph-a2ebf47b-fa4a-43ce-b087-12dbafb5796e" successfully created
Running command: /usr/sbin/lvcreate --yes -l 100%FREE -n osd-block-5352d594-aa19
-4147-a884-ca2c5775aa1b ceph-a2ebf47b-fa4a-43ce-b087-12dbafb5796e
 stdout: Logical volume "osd-block-5352d594-aa19-4147-a884-ca2c5775aa1b" created
.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-19
Running command: /bin/chown -R ceph:ceph /dev/dm-9
Running command: /bin/ln -s /dev/ceph-a2ebf47b-fa4a-43ce-b087-12dbafb5796e/osd-b
lock-5352d594-aa19-4147-a884-ca2c5775aa1b /var/lib/ceph/osd/ceph-19/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring 
/var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-19/activate.monmap
 stderr: got monmap epoch 1
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-19/keyring --create-k
eyring --name osd.19 --add-key AQBY1TBbN8I+HxAAMHGWKLgJugmtzdqllQh5sA==
 stdout: creating /var/lib/ceph/osd/ceph-19/keyring
 stdout: added entity osd.19 auth auth(auid = 18446744073709551615 key=AQBY1TBbN
8I+HxAAMHGWKLgJugmtzdqllQh5sA== with 0 caps)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-19/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-19/
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs
 -i 19 --monmap /var/lib/ceph/osd/ceph-19/activate.monmap --keyfile - --osd-data
 /var/lib/ceph/osd/ceph-19/ --osd-uuid 5352d594-aa19-4147-a884-ca2c5775aa1b --se
tuser ceph --setgroup ceph
--> ceph-volume lvm prepare successful for: /dev/sdh
Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /de
v/ceph-a2ebf47b-fa4a-43ce-b087-12dbafb5796e/osd-block-5352d594-aa19-4147-a884-ca
2c5775aa1b --path /var/lib/ceph/osd/ceph-19
Running command: /bin/ln -snf /dev/ceph-a2ebf47b-fa4a-43ce-b087-12dbafb5796e/osd
-block-5352d594-aa19-4147-a884-ca2c5775aa1b /var/lib/ceph/osd/ceph-19/block
Running command: /bin/chown -R ceph:ceph /dev/dm-9
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-19
Running command: /bin/systemctl enable ceph-volume@lvm-19-5352d594-aa19-4147-a88
4-ca2c5775aa1b
 stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-v
olume@lvm-19-5352d594-aa19-4147-a884-ca2c5775aa1b.service to /usr/lib/systemd/sy
stem/ceph-volume@.service.
Running command: /bin/systemctl start ceph-osd@19
--> ceph-volume lvm activate successful for osd ID: 19
--> ceph-volume lvm create successful for: /dev/sdh

verified that osd.19 was in the map with:
# ceph osd tree
ID CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF 
-1       3.20398 root default                           
-9       0.80099     host n1                            
18   hdd 0.13350         osd.18     up  1.00000 1.00000 
19   hdd 0.13350         osd.19   down        0 1.00000 
20   hdd 0.13350         osd.20     up  1.00000 1.00000 
21   hdd 0.13350         osd.21     up  1.00000 1.00000 
22   hdd 0.13350         osd.22     up  1.00000 1.00000 
23   hdd 0.13350         osd.23     up  1.00000 1.00000 

Only it fails to launch
# systemctl start ceph-osd@19
# systemctl status ceph-osd@19
â ceph-osd@19.service - Ceph object storage daemon osd.19
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; disabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: signal) since Mon 2018-06-25 13:44:35 CEST; 3s ago
  Process: 2046453 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=killed, signal=ABRT)
  Process: 2046447 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)
 Main PID: 2046453 (code=killed, signal=ABRT)

Jun 25 13:44:35 n1.sprawl.dk ceph-osd[2046453]: 8: (OSD::handle_osd_map(MOSDMap*)+0x1020) [0x56353eac71f0]
Jun 25 13:44:35 n1.sprawl.dk ceph-osd[2046453]: 9: (OSD::_dispatch(Message*)+0xa1) [0x56353eac9d21]
Jun 25 13:44:35 n1.sprawl.dk ceph-osd[2046453]: 10: (OSD::ms_dispatch(Message*)+0x56) [0x56353eaca066]
Jun 25 13:44:35 n1.sprawl.dk ceph-osd[2046453]: 11: (DispatchQueue::entry()+0xb5a) [0x7f302acce74a]
Jun 25 13:44:35 n1.sprawl.dk ceph-osd[2046453]: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f302ad6ef2d]
Jun 25 13:44:35 n1.sprawl.dk ceph-osd[2046453]: 13: (()+0x7e25) [0x7f30277b0e25]
Jun 25 13:44:35 n1.sprawl.dk ceph-osd[2046453]: 14: (clone()+0x6d) [0x7f30268a1bad]
Jun 25 13:44:35 n1.sprawl.dk ceph-osd[2046453]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jun 25 13:44:35 n1.sprawl.dk systemd[1]: Unit ceph-osd@19.service entered failed state.
Jun 25 13:44:35 n1.sprawl.dk systemd[1]: ceph-osd@19.service failed.

osd.19 log show:

--- begin dump of recent events ---
     0> 2018-06-25 13:48:47.139 7fc6b91c5700 -1 *** Caught signal (Aborted) **
 in thread 7fc6b91c5700 thread_name:ms_dispatch

 ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)
 1: (()+0x8e1870) [0x55da2ff6e870]
 2: (()+0xf6d0) [0x7fc6c97ba6d0]
 3: (gsignal()+0x37) [0x7fc6c87db277]
 4: (abort()+0x148) [0x7fc6c87dc968]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x25d) [0x7fc6ccc5a69d]
 6: (()+0x286727) [0x7fc6ccc5a727]
 7: (OSDService::get_map(unsigned int)+0x4a) [0x55da2faa3dda]
 8: (OSD::handle_osd_map(MOSDMap*)+0x1020) [0x55da2fa511f0]
 9: (OSD::_dispatch(Message*)+0xa1) [0x55da2fa53d21]
 10: (OSD::ms_dispatch(Message*)+0x56) [0x55da2fa54066]
 11: (DispatchQueue::entry()+0xb5a) [0x7fc6cccd074a]
 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fc6ccd70f2d]
 13: (()+0x7e25) [0x7fc6c97b2e25]
 14: (clone()+0x6d) [0x7fc6c88a3bad]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Any hints would be appreciated, TIA!

/Steffen

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com