Hi, I cannot add a new OSD to a current Ceph cluster. It just hangs, here is the debug log: ceph-osd -d --debug-ms=20 --debug-osd=20 --debug-filestore=31 -i 10 --osd-journal=/mnt/ceph/journal_vg_sda/journal0 --mkfs --mkjournal --mkkey 2014-07-09 10:50:28.934959 7f80f6a737a0 0 ceph version 0.72.1 (4d923861868f6a15dcb33fef7f50f674997322de), process ceph-osd, pid 12435 2014-07-09 10:50:28.938613 7f80f6a737a0 10 -- :/0 ready :/0 2014-07-09 10:50:28.938686 7f80f6a737a0 1 -- :/0 messenger.start 2014-07-09 10:50:28.938738 7f80f6a737a0 10 -- :/12435 shutdown :/12435 2014-07-09 10:50:28.938741 7f80f6a737a0 1 -- :/12435 mark_down_all 2014-07-09 10:50:28.938749 7f80f6a737a0 10 -- :/12435 wait: waiting for dispatch queue 2014-07-09 10:50:28.938755 7f80f0d2e700 10 -- :/12435 reaper_entry start 2014-07-09 10:50:28.938774 7f80f0d2e700 10 -- :/12435 reaper 2014-07-09 10:50:28.938777 7f80f0d2e700 10 -- :/12435 reaper done 2014-07-09 10:50:28.938784 7f80f6a737a0 10 -- :/12435 wait: dispatch queue is stopped 2014-07-09 10:50:28.938788 7f80f6a737a0 20 -- :/12435 wait: stopping reaper thread 2014-07-09 10:50:28.938801 7f80f0d2e700 10 -- :/12435 reaper_entry done 2014-07-09 10:50:28.938845 7f80f6a737a0 20 -- :/12435 wait: stopped reaper thread 2014-07-09 10:50:28.938858 7f80f6a737a0 10 -- :/12435 wait: closing pipes 2014-07-09 10:50:28.938861 7f80f6a737a0 10 -- :/12435 reaper 2014-07-09 10:50:28.938866 7f80f6a737a0 10 -- :/12435 reaper done 2014-07-09 10:50:28.938868 7f80f6a737a0 10 -- :/12435 wait: waiting for pipes to close 2014-07-09 10:50:28.938873 7f80f6a737a0 10 -- :/12435 wait: done. 2014-07-09 10:50:28.938876 7f80f6a737a0 1 -- :/12435 shutdown complete. 2014-07-09 10:50:28.939099 7f80f6a737a0 1 filestore(/var/lib/ceph/osd/ceph-10) mkfs in /var/lib/ceph/osd/ceph-10 2014-07-09 10:50:28.939191 7f80f6a737a0 1 filestore(/var/lib/ceph/osd/ceph-10) mkfs generated fsid 4f0664b1-62dd-44d5-9cf6-d50697883b8e 2014-07-09 10:50:28.940649 7f80f6a737a0 10 filestore(/var/lib/ceph/osd/ceph-10) mkfs fsid is 4f0664b1-62dd-44d5-9cf6-d50697883b8e 2014-07-09 10:50:28.946029 7f80f6a737a0 1 filestore(/var/lib/ceph/osd/ceph-10) leveldb db exists/created 2014-07-09 10:50:28.946079 7f80f6a737a0 10 filestore(/var/lib/ceph/osd/ceph-10) open_journal at /mnt/ceph/journal_vg_sda/journal0 2014-07-09 10:50:28.946214 7f80f6a737a0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-07-09 10:50:28.948884 7f80f6a737a0 1 journal _open /mnt/ceph/journal_vg_sda/journal0 fd 9: 4294967296 bytes, block size 4096 bytes, directio = 1, aio = 0 2014-07-09 10:50:28.951046 7f80f6a737a0 0 filestore(/var/lib/ceph/osd/ceph-10) mkjournal created journal on /mnt/ceph/journal_vg_sda/journal0 2014-07-09 10:50:28.951065 7f80f6a737a0 1 filestore(/var/lib/ceph/osd/ceph-10) mkfs done in /var/lib/ceph/osd/ceph-10 2014-07-09 10:50:28.951085 7f80f6a737a0 5 filestore(/var/lib/ceph/osd/ceph-10) basedir /var/lib/ceph/osd/ceph-10 journal /mnt/ceph/journal_vg_sda/journal0 2014-07-09 10:50:28.951113 7f80f6a737a0 10 filestore(/var/lib/ceph/osd/ceph-10) mount fsid is 4f0664b1-62dd-44d5-9cf6-d50697883b8e 2014-07-09 10:50:28.951151 7f80f6a737a0 1 filestore(/var/lib/ceph/osd/ceph-10) mount detected xfs 2014-07-09 10:50:28.951157 7f80f6a737a0 1 filestore(/var/lib/ceph/osd/ceph-10) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2014-07-09 10:50:29.009207 7f80f6a737a0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: FIEMAP ioctl is supported and appears to work 2014-07-09 10:50:29.009226 7f80f6a737a0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2014-07-09 10:50:29.036036 7f80f6a737a0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: syscall(SYS_syncfs, fd) fully supported 2014-07-09 10:50:29.036401 7f80f6a737a0 5 filestore(/var/lib/ceph/osd/ceph-10) mount op_seq is 1 I see a few files are created in de ceph dir: . ./fsid ./store_version ./superblock ./current ./current/commit_op_seq ./current/omap ./current/omap/LOG ./current/omap/LOCK ./current/omap/CURRENT ./current/omap/000005.log ./current/omap/MANIFEST-000004 ./current/nosnap When I do a tcpdump no traffic is going to the monitor hosts (no idea if this is normal). This is ceph 0.72.1 on CentOS. Any help is appreciated. Thx, Robert
 |