> Op 11 januari 2017 om 12:24 schreef Jayaram R <jaylinuxgeek@xxxxxxxxx>: > > > Hello, > > > > We from Nokia are validating bluestore on 3 node cluster with EC 2+1 > > > > While upgrading our cluster from Kraken 11.0.2 to 11.1.1 with bluesotre , > the cluster affected more than half of the OSDs went down. > Yes, see the Release Notes: http://docs.ceph.com/docs/master/release-notes/#v11-1-1-release-candidate "The new BlueStore backend now has a change in the on-disk format, from the previous release candidate 11.1.0 and there might possibly be a change before the final release is cut" Those OSDs are not able to start due to the changed on-disk format. Wido > > > $ceph -s > cluster cb55baa8-d5a5-442e-9aae-3fd83553824e > health HEALTH_ERR > 792 pgs are stuck inactive for more than 300 seconds > 792 pgs stale > 792 pgs stuck stale > 8/12 in osds are down > monmap e2: 3 mons at {PL0-CN1= > 10.50.5.16:6789/0,PL0-CN2=10.50.5.17:6789/0,PL0-CN3=10.50.5.18:6789/0} > election epoch 28, quorum 0,1,2 PL0-CN1,PL0-CN2,PL0-CN3 > mgr active: PL0-CN2 standbys: PL0-CN1, PL0-CN3 > osdmap e191: 15 osds: 4 up, 12 in; 856 remapped pgs > flags sortbitwise,require_jewel_osds,require_kraken_osds > pgmap v508: 1088 pgs, 2 pools, 0 bytes data, 0 objects > 157 MB used, 33531 GB / 33531 GB avail > 792 stale+active+clean > 296 active+clean > > OSD logs.. > -------------- > > ~~~ > 2017-01-11 12:03:38.740504 7f7741b13940 0 pidfile_write: ignore empty > --pid-file > 2017-01-11 12:03:38.758541 7f7741b13940 -1 WARNING: the following dangerous > and experimental features are enabled: bluestore,rocksdb > 2017-01-11 12:03:38.767324 7f7741b13940 0 load: jerasure load: lrc load: > isa > 2017-01-11 12:03:38.767791 7f7741b13940 1 > bluestore(/var/lib/ceph/osd/ceph-5) mount path /var/lib/ceph/osd/ceph-5 > 2017-01-11 12:03:38.769697 7f7741b13940 1 bluefs add_block_device bdev 1 > path /var/lib/ceph/osd/ceph-5/block.db size 65536 kB > 2017-01-11 12:03:38.770443 7f7741b13940 1 bluefs add_block_device bdev 2 > path /var/lib/ceph/osd/ceph-5/block size 2794 GB > 2017-01-11 12:03:38.770961 7f7741b13940 1 bluefs add_block_device bdev 0 > path /var/lib/ceph/osd/ceph-5/block.wal size 128 MB > 2017-01-11 12:03:38.771176 7f7741b13940 1 bluefs mount > 2017-01-11 12:03:38.790311 7f7741b13940 0 set rocksdb option compression > = kNoCompression > 2017-01-11 12:03:38.790320 7f7741b13940 0 set rocksdb option > max_write_buffer_number = 4 > 2017-01-11 12:03:38.790323 7f7741b13940 0 set rocksdb option > min_write_buffer_number_to_merge = 1 > 2017-01-11 12:03:38.790328 7f7741b13940 0 set rocksdb option > recycle_log_file_num = 4 > 2017-01-11 12:03:38.790332 7f7741b13940 0 set rocksdb option > write_buffer_size = 268435456 > 2017-01-11 12:03:38.790354 7f7741b13940 0 set rocksdb option compression > = kNoCompression > 2017-01-11 12:03:38.790356 7f7741b13940 0 set rocksdb option > max_write_buffer_number = 4 > 2017-01-11 12:03:38.790358 7f7741b13940 0 set rocksdb option > min_write_buffer_number_to_merge = 1 > 2017-01-11 12:03:38.790360 7f7741b13940 0 set rocksdb option > recycle_log_file_num = 4 > 2017-01-11 12:03:38.790362 7f7741b13940 0 set rocksdb option > write_buffer_size = 268435456 > 2017-01-11 12:03:38.790493 7f7741b13940 4 rocksdb: RocksDB version: 5.0.0 > > <snip> > > 2017-01-11 12:03:38.839442 7f7741b13940 4 rocksdb: DB pointer > 0x7f774cef0b00 > 2017-01-11 12:03:38.839470 7f7741b13940 1 > bluestore(/var/lib/ceph/osd/ceph-5) _open_db opened rocksdb path db options > compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456 > 2017-01-11 12:03:38.840407 7f7741b13940 -1 > bluestore(/var/lib/ceph/osd/ceph-5) warning: bluestore_min_alloc_size 65536 > > min_min_alloc_size 4096, may impact performance. > 2017-01-11 12:03:38.840429 7f7741b13940 1 freelist init > 2017-01-11 12:03:39.339645 7f7741b13940 -1 osd.5 0 OSD::init() : unable to > read osd superblock > 2017-01-11 12:03:39.339659 7f7741b13940 1 > bluestore(/var/lib/ceph/osd/ceph-5) umount > 2017-01-11 12:03:39.454719 7f7741b13940 1 freelist shutdown > 2017-01-11 12:03:39.454960 7f7741b13940 1 bluefs umount > 2017-01-11 12:03:40.278242 7f7741b13940 -1 ESC[0;31m ** ERROR: osd init > failed: (22) Invalid argumentESC[0m > 2017-01-11 12:04:00.487006 7f9c3bebc940 -1 WARNING: the following dangerous > and experimental features are enabled: bluestore,rocksdb > 2017-01-11 12:04:00.487011 7f9c3bebc940 0 set uid:gid to 167:167 > (ceph:ceph) > 2017-01-11 12:04:00.487021 7f9c3bebc940 0 ceph version 11.1.1 > (87597971b371d7f497d7eabad3545d72d18dd755), process ceph-osd, pid 11983 > 2017-01-11 12:04:00.487058 7f9c3bebc940 -1 WARNING: experimental feature > 'bluestore' is enabled > > .~~~~~ > > Our Findings:-- > > 1) Cluster installed from scratch with 11.1.1 works fine , having issues > with upgrade . > > 2) OSD's not getting activated after upgrade which cause osd process failed > to read the superblock. > > > > Please provide suggestions/feedback to unblock this issue. > > > Thanks > > Jayaram > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |