I have 7 servers, each containing 60 x 6TB drives in jbod mode. When I first started, I only activated a couple drives on 3 nodes as Ceph OSDs. Yesterday, I went to expand to the remaining nodes as well as prepare and activate all the drives.
ceph-disk prepare worked just fine. However, ceph-disk activate-all managed to only activate 33 drives and failed on the rest. This is consistent all 7 nodes (existing and newly installed). At the end of the day, I have 33 Ceph OSDs activated per server and can't activate any more. I did have to bump up the pg_num and pgp_num on the pool in order to accommodate the drives that did activate. I don't know if having a low pg number during the mass influx of OSDs caused an issue or not within the pool. I don't think so because I can only set the pg_num to a maximum value determined by the number of known OSDs. But maybe you have to expand slowly, increase pg's, expand osds, increase pgs in a slow fashion. I certainly have not seen anything to suggest a magic "33/node limit", and I've seen references to servers with up to 72 Ceph OSDs on them.
I then attempted to activate individual ceph osd's and got the same set of errors. I even wiped a drive, re-ran `ceph-disk prepare` and `ceph-disk activate` to have it fail in the same way.
status:
```
root@ljb01:/home/ceph/rain-cluster# ceph status
cluster 4ebe7995-6a33-42be-bd4d-20f51d02ae45
health HEALTH_OK
monmap e5: 5 mons at {hail02-r01-06=172.29.4.153:6789/0,hail02-r01-08=172.29.4.155:6789/0,rain02-r01-01=172.29.4.148:6789/0,rain02-r01-03=172.29.4.150:6789/0,rain02-r01-04=172.29.4.151:6789/0}
election epoch 12, quorum 0,1,2,3,4 rain02-r01-01,rain02-r01-03,rain02-r01-04,hail02-r01-06,hail02-r01-08
osdmap e1116: 420 osds: 232 up, 232 in
flags sortbitwise
pgmap v397198: 10872 pgs, 14 pools, 101 MB data, 8456 objects
38666 MB used, 1264 TB / 1264 TB avail
10872 active+clean
```
Here is what I get when I run ceph-disk prepare on a blank drive:
```
root@rain02-r01-01:/etc/ceph# ceph-disk prepare /dev/sdbh1
The operation has completed successfully.
The operation has completed successfully.
meta-data="" isize=2048 agcount=6, agsize=268435455 blks
= sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=1463819665, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
The operation has completed successfully.
root@rain02-r01-01:/etc/ceph# parted /dev/sdh print
Model: ATA HUS726060ALA640 (scsi)
Disk /dev/sdh: 6001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
2 1049kB 5369MB 5368MB ceph journal
1 5370MB 6001GB 5996GB xfs ceph data
```
And finally the errors from attempting to activate the drive.
```
root@rain02-r01-01:/etc/ceph# ceph-disk activate /dev/sdbh1
got monmap epoch 5
2016-02-12 12:53:43.340526 7f149bc71940 -1 journal FileJournal::_open: unable to setup io_context (0) Success
2016-02-12 12:53:43.340748 7f1493f83700 -1 journal io_submit to 0~4096 got (22) Invalid argument
2016-02-12 12:53:43.341186 7f149bc71940 -1 filestore(/var/lib/ceph/tmp/mnt.KRphD_) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
os/FileJournal.cc: In function 'int FileJournal::write_aio_bl(off64_t&, ceph::bufferlist&, uint64_t)' thread 7f1493f83700 time 2016-02-12 12:53:43.341355
os/FileJournal.cc: 1469: FAILED assert(0 == "io_submit got unexpected error")
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f149b767f2b]
2: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned long)+0x5ad) [0x7f149b5fe27d]
3: (FileJournal::do_aio_write(ceph::buffer::list&)+0x263) [0x7f149b602e63]
4: (FileJournal::write_thread_entry()+0x4e4) [0x7f149b607394]
5: (FileJournal::Writer::entry()+0xd) [0x7f149b44bddd]
6: (()+0x8182) [0x7f1499d87182]
7: (clone()+0x6d) [0x7f14980ce47d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2016-02-12 12:53:43.345434 7f1493f83700 -1 os/FileJournal.cc: In function 'int FileJournal::write_aio_bl(off64_t&, ceph::bufferlist&, uint64_t)' thread 7f1493f83700 time 2016-02-12 12:53:43.341355
os/FileJournal.cc: 1469: FAILED assert(0 == "io_submit got unexpected error")
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f149b767f2b]
2: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned long)+0x5ad) [0x7f149b5fe27d]
3: (FileJournal::do_aio_write(ceph::buffer::list&)+0x263) [0x7f149b602e63]
4: (FileJournal::write_thread_entry()+0x4e4) [0x7f149b607394]
5: (FileJournal::Writer::entry()+0xd) [0x7f149b44bddd]
6: (()+0x8182) [0x7f1499d87182]
7: (clone()+0x6d) [0x7f14980ce47d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-4> 2016-02-12 12:53:43.340526 7f149bc71940 -1 journal FileJournal::_open: unable to setup io_context (0) Success
-3> 2016-02-12 12:53:43.340748 7f1493f83700 -1 journal io_submit to 0~4096 got (22) Invalid argument
-1> 2016-02-12 12:53:43.341186 7f149bc71940 -1 filestore(/var/lib/ceph/tmp/mnt.KRphD_) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
0> 2016-02-12 12:53:43.345434 7f1493f83700 -1 os/FileJournal.cc: In function 'int FileJournal::write_aio_bl(off64_t&, ceph::bufferlist&, uint64_t)' thread 7f1493f83700 time 2016-02-12 12:53:43.3
41355
os/FileJournal.cc: 1469: FAILED assert(0 == "io_submit got unexpected error”)
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f149b767f2b]
2: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned long)+0x5ad) [0x7f149b5fe27d]
3: (FileJournal::do_aio_write(ceph::buffer::list&)+0x263) [0x7f149b602e63]
4: (FileJournal::write_thread_entry()+0x4e4) [0x7f149b607394]
5: (FileJournal::Writer::entry()+0xd) [0x7f149b44bddd]
6: (()+0x8182) [0x7f1499d87182]
7: (clone()+0x6d) [0x7f14980ce47d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion'
*** Caught signal (Aborted) **
in thread 7f1493f83700
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (()+0x7d02ca) [0x7f149b67b2ca]
2: (()+0x10340) [0x7f1499d8f340]
3: (gsignal()+0x39) [0x7f149800acc9]
4: (abort()+0x148) [0x7f149800e0d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f1498915535]
6: (()+0x5e6d6) [0x7f14989136d6]
7: (()+0x5e703) [0x7f1498913703]
8: (()+0x5e922) [0x7f1498913922]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x7f149b768118]
10: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned long)+0x5ad) [0x7f149b5fe27d]
11: (FileJournal::do_aio_write(ceph::buffer::list&)+0x263) [0x7f149b602e63]
12: (FileJournal::write_thread_entry()+0x4e4) [0x7f149b607394]
13: (FileJournal::Writer::entry()+0xd) [0x7f149b44bddd]
14: (()+0x8182) [0x7f1499d87182]
15: (clone()+0x6d) [0x7f14980ce47d]
2016-02-12 12:53:43.348498 7f1493f83700 -1 *** Caught signal (Aborted) **
in thread 7f1493f83700
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (()+0x7d02ca) [0x7f149b67b2ca]
2: (()+0x10340) [0x7f1499d8f340]
3: (gsignal()+0x39) [0x7f149800acc9]
4: (abort()+0x148) [0x7f149800e0d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f1498915535]
6: (()+0x5e6d6) [0x7f14989136d6]
7: (()+0x5e703) [0x7f1498913703]
8: (()+0x5e922) [0x7f1498913922]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x7f149b768118]
10: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned long)+0x5ad) [0x7f149b5fe27d]
11: (FileJournal::do_aio_write(ceph::buffer::list&)+0x263) [0x7f149b602e63]
12: (FileJournal::write_thread_entry()+0x4e4) [0x7f149b607394]
13: (FileJournal::Writer::entry()+0xd) [0x7f149b44bddd]
14: (()+0x8182) [0x7f1499d87182]
15: (clone()+0x6d) [0x7f14980ce47d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
0> 2016-02-12 12:53:43.348498 7f1493f83700 -1 *** Caught signal (Aborted) **
in thread 7f1493f83700
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
1: (()+0x7d02ca) [0x7f149b67b2ca]
2: (()+0x10340) [0x7f1499d8f340]
3: (gsignal()+0x39) [0x7f149800acc9]
4: (abort()+0x148) [0x7f149800e0d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f1498915535]
6: (()+0x5e6d6) [0x7f14989136d6]
7: (()+0x5e703) [0x7f1498913703]
8: (()+0x5e922) [0x7f1498913922]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x7f149b768118]
10: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned long)+0x5ad) [0x7f149b5fe27d]
11: (FileJournal::do_aio_write(ceph::buffer::list&)+0x263) [0x7f149b602e63] 12: (FileJournal::write_thread_entry()+0x4e4) [0x7f149b607394] 13: (FileJournal::Writer::entry()+0xd) [0x7f149b44bddd]
14: (()+0x8182) [0x7f1499d87182]
15: (clone()+0x6d) [0x7f14980ce47d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ERROR:ceph-disk:Failed to activate
Traceback (most recent call last):
File "/usr/sbin/ceph-disk", line 3576, in <module>
main(sys.argv[1:])
File "/usr/sbin/ceph-disk", line 3532, in main
main_catch(args.func, args)
File "/usr/sbin/ceph-disk", line 3554, in main_catch
func(args)
File "/usr/sbin/ceph-disk", line 2424, in main_activate
dmcrypt_key_dir=args.dmcrypt_key_dir,
File "/usr/sbin/ceph-disk", line 2197, in mount_activate
(osd_id, cluster) = activate(path, activate_key_template, init)
File "/usr/sbin/ceph-disk", line 2360, in activate
keyring=keyring,
File "/usr/sbin/ceph-disk", line 1950, in mkfs
'--setgroup', get_ceph_user(), File "/usr/sbin/ceph-disk", line 349, in command_check_call return subprocess.check_call(arguments)
File "/usr/lib/python2.7/subprocess.py", line 540, in check_call raise CalledProcessError(retcode, cmd)subprocess.CalledProcessError: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '165', '--monmap', '/var/lib/ceph/tmp/mnt.KRphD_/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.KRphD_', '--osd-journal', '/var/lib/ceph/tmp/mnt.K
root@rain02-r01-01:/etc/ceph# ls -l /var/lib/ceph/tmp/
total 0
-rw-r--r-- 1 root root 0 Feb 12 12:58 ceph-disk.activate.lock
-rw-r--r-- 1 root root 0 Feb 12 12:58 ceph-disk.prepare.lock
```
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
