If you followed the standard setup,
each OSD is it's own disk + filesystem. /var/lib/ceph/osd/ceph-2
is in use, as the mount point for the OSD.2 filesystem. Double
check by examining the output of the `mount` command.
I get the same error when I try to rename a directory that's used
as a mount point.
Try `umount /var/lib/ceph/osd/ceph-2` instead of the mv and rm.
The fuser command is telling you that the kernel has a filesystem
mounted in that directory. Nothing else appears to be using it,
so the umount should complete successfully.
Also, you should fix that time skew on mon.ceph-node5. The
mailing list archives have several posts with good answers.
On 6/15/2013 2:14 AM, Da Chun wrote:
Hi all,
On Ubuntu 13.04 with ceph 0.61.3.
I want to remove osd.2 from my cluster. The following steps were performed:
root@ceph-node6:~# ceph osd out osd.2
marked out osd.2.
root@ceph-node6:~# ceph -w
health HEALTH_WARN clock skew
detected on mon.ceph-node5
monmap e1: 3 mons at
{ceph-node4=172.18.46.34:6789/0,ceph-node5=172.18.46.35:6789/0,ceph-node6=172.18.46.36:6789/0},
election epoch 124, quorum 0,1,2
ceph-node4,ceph-node5,ceph-node6
osdmap e414: 6 osds: 5 up, 5 in
pgmap v10540: 456 pgs: 456
active+clean; 12171 MB data, 24325 MB used, 50360 MB / 74685
MB avail
mdsmap e102: 1/1/1 up
{0=ceph-node4=up:active}
2013-06-15 16:55:22.096059 mon.0
[INF] pgmap v10540: 456 pgs: 456 active+clean; 12171 MB
data, 24325 MB used, 50360 MB / 74685 MB avail
^C
root@ceph-node6:~# stop ceph-osd id=2
ceph-osd stop/waiting
root@ceph-node6:~# ceph osd crush
remove osd.2
removed item id 2 name 'osd.2' from
crush map
root@ceph-node6:~# ceph auth del
osd.2
updated
root@ceph-node6:~# ceph osd rm 2
removed osd.2
root@ceph-node6:~# mv
/var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2.bak
mv: cannot move
‘/var/lib/ceph/osd/ceph-2’ to
‘/var/lib/ceph/osd/ceph-2.bak’: Device or resource busy
Everything was working OK until the last step to remove the
osd.2 directory /var/lib/ceph/osd/ceph-2.
root@ceph-node6:~# fuser -v
/var/lib/ceph/osd/ceph-2
USER PID
ACCESS COMMAND
/var/lib/ceph/osd/ceph-2:
root kernel
mount /var/lib/ceph/osd/ceph-2 ////////////////// What
does this mean?
root@ceph-node6:~# lsof +D
/var/lib/ceph/osd/ceph-2
root@ceph-node6:~#
I restarted the system, and found that the osd.2 daemon was
still running:
root@ceph-node6:~# ps aux | grep osd
root 1264 1.4 12.3 550940
125732 ? Ssl 16:41 0:20 /usr/bin/ceph-osd
--cluster=ceph -i 2 -f
root 2876 0.0 0.0 4440 628
? Ss 16:44 0:00 /bin/sh -e -c /usr/bin/ceph-osd
--cluster="${cluster:-ceph}" -i "$id" -f /bin/sh
root 2877 4.9 18.2 613780
185676 ? Sl 16:44 1:04 /usr/bin/ceph-osd
--cluster=ceph -i 5 -f
I have to take this workaround:
root@ceph-node6:~# rm -rf
/var/lib/ceph/osd/ceph-2
rm: cannot remove
‘/var/lib/ceph/osd/ceph-2’: Device or resource busy
root@ceph-node6:~# ls
/var/lib/ceph/osd/ceph-2
root@ceph-node6:~# shutdown -r now
....
root@ceph-node6:~# ps aux | grep osd
root 1416 0.0 0.0 4440 628
? Ss 17:10 0:00 /bin/sh -e -c /usr/bin/ceph-osd
--cluster="${cluster:-ceph}" -i "$id" -f /bin/sh
root 1417 8.9 5.8 468052 59868
? Sl 17:10 0:02 /usr/bin/ceph-osd --cluster=ceph
-i 5 -f
root@ceph-node6:~# rm -r
/var/lib/ceph/osd/ceph-2
root@ceph-node6:~#
Any idea? HELP!
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
