Can you set 'debug_osd = 20' in ceph.conf and restart the osd again, and post the corrupt log. I doubt it's problem related to "0 byte osdmap" decode problem? 2016-04-16 12:14 GMT+08:00 hjcho616 <hjcho616@xxxxxxxxx>: > I've been successfully running cephfs on my Debian Jessies for a while and > one day after power outage, MDS wasn't happy. MDS crashing after it was > done loading, increasing the memory utilization quite a bit. I was running > infernalis 9.2.0 and did successful upgrade from Hammer before... so I > thought I may have hit a bug and decided to try 9.2.1. > > In 9.2.1, it was not happy that my journal didn't have permission for user > ceph. So corrected it. Then all of my OSDs are no longer starting. > Failing with similar messages as below. I upgraded to Jewel, as I didn't > see too much more complexitiy to upgrade from Infernalis and am still seeing > these errors. > > 2016-04-15 22:47:04.897500 7f65fbbb0800 0 set uid:gid to 1001:1001 > (ceph:ceph) > 2016-04-15 22:47:04.897635 7f65fbbb0800 0 ceph version 10.1.2 > (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 1284 > 2016-04-15 22:47:04.900585 7f65fbbb0800 0 pidfile_write: ignore empty > --pid-file > 2016-04-15 22:47:05.467530 7f65fbbb0800 0 > filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) > 2016-04-15 22:47:05.477912 7f65fbbb0800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP > ioctl is disabled via 'filestore fiemap' config option > 2016-04-15 22:47:05.477999 7f65fbbb0800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: > SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option > 2016-04-15 22:47:05.478091 7f65fbbb0800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice is > supported > 2016-04-15 22:47:05.494593 7f65fbbb0800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) > syscall fully supported (by glibc and kernel) > 2016-04-15 22:47:05.494785 7f65fbbb0800 0 > xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is > disabled by conf > 2016-04-15 22:47:05.596738 7f65fbbb0800 1 leveldb: Recovering log #20899 > 2016-04-15 22:47:05.825914 7f65fbbb0800 1 leveldb: Delete type=0 #20899 > > 2016-04-15 22:47:05.826089 7f65fbbb0800 1 leveldb: Delete type=3 #20898 > > 2016-04-15 22:47:05.900058 7f65fbbb0800 0 > filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: > checkpoint is not enabled > 2016-04-15 22:47:06.377878 7f65fbbb0800 1 journal _open > /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 > bytes, directio = 1, aio = 1 > 2016-04-15 22:47:06.381738 7f65fbbb0800 1 journal _open > /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 > bytes, directio = 1, aio = 1 > 2016-04-15 22:47:06.384954 7f65fbbb0800 1 > filestore(/var/lib/ceph/osd/ceph-3) upgrade > 2016-04-15 22:47:06.415851 7f65fbbb0800 0 <cls> > cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan > 2016-04-15 22:47:06.419654 7f65fbbb0800 0 <cls> cls/hello/cls_hello.cc:305: > loading cls_hello > 2016-04-15 22:47:06.498512 7f65fbbb0800 -1 osd/OSD.h: In function 'OSDMapRef > OSDService::get_map(epoch_t)' thread 7f65fbbb0800 time 2016-04-15 > 22:47:06.494680 > osd/OSD.h: 885: FAILED assert(ret) > > ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x82) [0x7f65fb6364f2] > 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] > 3: (OSD::init()+0x1862) [0x7f65faf6ba52] > 4: (main()+0x2b05) [0x7f65faed1735] > 5: (__libc_start_main()+0xf5) [0x7f65f7a67b45] > 6: (()+0x337197) [0x7f65faf1c197] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > --- begin dump of recent events --- > -78> 2016-04-15 22:47:04.873688 7f65fbbb0800 5 asok(0x7f660689a000) > register_command perfcounters_dump hook 0x7f66067e2030 > -77> 2016-04-15 22:47:04.873771 7f65fbbb0800 5 asok(0x7f660689a000) > register_command 1 hook 0x7f66067e2030 > -76> 2016-04-15 22:47:04.873804 7f65fbbb0800 5 asok(0x7f660689a000) > register_command perf dump hook 0x7f66067e2030 > -75> 2016-04-15 22:47:04.873834 7f65fbbb0800 5 asok(0x7f660689a000) > register_command perfcounters_schema hook 0x7f66067e2030 > -76> 2016-04-15 22:47:04.873804 7f65fbbb0800 5 asok(0x7f660689a000) > register_command perf dump hook 0x7f66067e2030 > -75> 2016-04-15 22:47:04.873834 7f65fbbb0800 5 asok(0x7f660689a000) > register_command perfcounters_schema hook 0x7f66067e2030 > -74> 2016-04-15 22:47:04.873860 7f65fbbb0800 5 asok(0x7f660689a000) > register_command 2 hook 0x7f66067e2030 > -73> 2016-04-15 22:47:04.873886 7f65fbbb0800 5 asok(0x7f660689a000) > register_command perf schema hook 0x7f66067e2030 > -72> 2016-04-15 22:47:04.873916 7f65fbbb0800 5 asok(0x7f660689a000) > register_command perf reset hook 0x7f66067e2030 > -71> 2016-04-15 22:47:04.873943 7f65fbbb0800 5 asok(0x7f660689a000) > register_command config show hook 0x7f66067e2030 > -70> 2016-04-15 22:47:04.873974 7f65fbbb0800 5 asok(0x7f660689a000) > register_command config set hook 0x7f66067e2030 > -69> 2016-04-15 22:47:04.874000 7f65fbbb0800 5 asok(0x7f660689a000) > register_command config get hook 0x7f66067e2030 > -68> 2016-04-15 22:47:04.874029 7f65fbbb0800 5 asok(0x7f660689a000) > register_command config diff hook 0x7f66067e2030 > -67> 2016-04-15 22:47:04.874055 7f65fbbb0800 5 asok(0x7f660689a000) > register_command log flush hook 0x7f66067e2030 > -66> 2016-04-15 22:47:04.874082 7f65fbbb0800 5 asok(0x7f660689a000) > register_command log dump hook 0x7f66067e2030 > -65> 2016-04-15 22:47:04.874109 7f65fbbb0800 5 asok(0x7f660689a000) > register_command log reopen hook 0x7f66067e2030 > -64> 2016-04-15 22:47:04.897500 7f65fbbb0800 0 set uid:gid to 1001:1001 > (ceph:ceph) > -63> 2016-04-15 22:47:04.897635 7f65fbbb0800 0 ceph version 10.1.2 > (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 1284 > -62> 2016-04-15 22:47:04.900224 7f65fbbb0800 1 -- 192.168.1.31:0/0 > learned my addr 192.168.1.31:0/0 > -61> 2016-04-15 22:47:04.900286 7f65fbbb0800 1 accepter.accepter.bind > my_inst.addr is 192.168.1.31:6802/1284 need_addr=0 > -60> 2016-04-15 22:47:04.900350 7f65fbbb0800 1 -- 192.168.2.31:0/0 > learned my addr 192.168.2.31:0/0 > -59> 2016-04-15 22:47:04.900375 7f65fbbb0800 1 accepter.accepter.bind > my_inst.addr is 192.168.2.31:6802/1284 need_addr=0 > -58> 2016-04-15 22:47:04.900443 7f65fbbb0800 1 -- 192.168.2.31:0/0 > learned my addr 192.168.2.31:0/0 > -57> 2016-04-15 22:47:04.900475 7f65fbbb0800 1 accepter.accepter.bind > my_inst.addr is 192.168.2.31:6803/1284 need_addr=0 > -56> 2016-04-15 22:47:04.900538 7f65fbbb0800 1 -- 192.168.1.31:0/0 > learned my addr 192.168.1.31:0/0 > -55> 2016-04-15 22:47:04.900562 7f65fbbb0800 1 accepter.accepter.bind > my_inst.addr is 192.168.1.31:6803/1284 need_addr=0 > -54> 2016-04-15 22:47:04.900585 7f65fbbb0800 0 pidfile_write: ignore > empty --pid-file > -53> 2016-04-15 22:47:04.909743 7f65fbbb0800 5 asok(0x7f660689a000) init > /var/run/ceph/ceph-osd.3.asok > -52> 2016-04-15 22:47:04.909792 7f65fbbb0800 5 asok(0x7f660689a000) > bind_and_listen /var/run/ceph/ceph-osd.3.asok > -51> 2016-04-15 22:47:04.909891 7f65fbbb0800 5 asok(0x7f660689a000) > register_command 0 hook 0x7f66067de0d8 > -50> 2016-04-15 22:47:04.909928 7f65fbbb0800 5 asok(0x7f660689a000) > register_command version hook 0x7f66067de0d8 > -49> 2016-04-15 22:47:04.909955 7f65fbbb0800 5 asok(0x7f660689a000) > register_command git_version hook 0x7f66067de0d8 > -48> 2016-04-15 22:47:04.909988 7f65fbbb0800 5 asok(0x7f660689a000) > register_command help hook 0x7f66067e21e0 > -47> 2016-04-15 22:47:04.910015 7f65fbbb0800 5 asok(0x7f660689a000) > register_command get_command_descriptions hook 0x7f66067e21f0 > -46> 2016-04-15 22:47:04.910205 7f65f43c9700 5 asok(0x7f660689a000) > entry start > -45> 2016-04-15 22:47:04.910330 7f65fbbb0800 10 monclient(hunting): > build_initial_monmap > -44> 2016-04-15 22:47:04.939070 7f65fbbb0800 5 adding auth protocol: > cephx > -43> 2016-04-15 22:47:04.939118 7f65fbbb0800 5 adding auth protocol: > cephx > -42> 2016-04-15 22:47:04.939986 7f65fbbb0800 5 asok(0x7f660689a000) > register_command objecter_requests hook 0x7f66067e22b0 > -41> 2016-04-15 22:47:04.940256 7f65fbbb0800 1 -- 192.168.1.31:6802/1284 > messenger.start > -40> 2016-04-15 22:47:04.940413 7f65fbbb0800 1 -- :/0 messenger.start > -39> 2016-04-15 22:47:04.940557 7f65fbbb0800 1 -- 192.168.1.31:6803/1284 > messenger.start > -38> 2016-04-15 22:47:04.940686 7f65fbbb0800 1 -- 192.168.2.31:6803/1284 > messenger.start > -37> 2016-04-15 22:47:04.940798 7f65fbbb0800 1 -- 192.168.2.31:6802/1284 > messenger.start > -36> 2016-04-15 22:47:04.940899 7f65fbbb0800 1 -- :/0 messenger.start > -35> 2016-04-15 22:47:04.941223 7f65fbbb0800 2 osd.3 0 mounting > /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal > -34> 2016-04-15 22:47:05.467530 7f65fbbb0800 0 > filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) > -33> 2016-04-15 22:47:05.477912 7f65fbbb0800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP > ioctl is disabled via 'filestore fiemap' config option > -32> 2016-04-15 22:47:05.477999 7f65fbbb0800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: > SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option > -31> 2016-04-15 22:47:05.478091 7f65fbbb0800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: splice is > supported > -30> 2016-04-15 22:47:05.494593 7f65fbbb0800 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) > syscall fully supported (by glibc and kernel) > -29> 2016-04-15 22:47:05.494785 7f65fbbb0800 0 > xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is > disabled by conf > -28> 2016-04-15 22:47:05.596738 7f65fbbb0800 1 leveldb: Recovering log > #20899 > -27> 2016-04-15 22:47:05.825914 7f65fbbb0800 1 leveldb: Delete type=0 > #20899 > > -26> 2016-04-15 22:47:05.826089 7f65fbbb0800 1 leveldb: Delete type=3 > #20898 > > -25> 2016-04-15 22:47:05.900058 7f65fbbb0800 0 > filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: > checkpoint is not enabled > -24> 2016-04-15 22:47:06.377715 7f65fbbb0800 2 journal open > /var/lib/ceph/osd/ceph-3/journal fsid 4f86a418-6c67-4cb4-83a1-6c123c890036 > fs_op_seq 9829589 > -23> 2016-04-15 22:47:06.377878 7f65fbbb0800 1 journal _open > /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 > bytes, directio = 1, aio = 1 > -22> 2016-04-15 22:47:06.379811 7f65fbbb0800 2 journal open advancing > committed_seq 9829584 to fs op_seq 9829589 > -21> 2016-04-15 22:47:06.380757 7f65fbbb0800 2 journal read_entry > 2537717760 : seq 9829585 29509 bytes > -20> 2016-04-15 22:47:06.380996 7f65fbbb0800 2 journal read_entry > 2537750528 : seq 9829586 8134 bytes > -19> 2016-04-15 22:47:06.381091 7f65fbbb0800 2 journal read_entry > 2537762816 : seq 9829587 3064 bytes > -18> 2016-04-15 22:47:06.381155 7f65fbbb0800 2 journal read_entry > 2537766912 : seq 9829588 7647 bytes > -17> 2016-04-15 22:47:06.381219 7f65fbbb0800 2 journal read_entry > 2537775104 : seq 9829589 4737 bytes > -16> 2016-04-15 22:47:06.381257 7f65fbbb0800 2 journal No further valid > entries found, journal is most likely valid > -15> 2016-04-15 22:47:06.381287 7f65fbbb0800 2 journal No further valid > entries found, journal is most likely valid > -14> 2016-04-15 22:47:06.381302 7f65fbbb0800 3 journal journal_replay: > end of journal, done. > -13> 2016-04-15 22:47:06.381738 7f65fbbb0800 1 journal _open > /var/lib/ceph/osd/ceph-3/journal fd 18: 14998831104 bytes, block size 4096 > bytes, directio = 1, aio = 1 > -12> 2016-04-15 22:47:06.384954 7f65fbbb0800 1 > filestore(/var/lib/ceph/osd/ceph-3) upgrade > -11> 2016-04-15 22:47:06.385071 7f65fbbb0800 2 osd.3 0 boot > -10> 2016-04-15 22:47:06.415253 7f65fbbb0800 1 <cls> > cls/statelog/cls_statelog.cc:306: Loaded log class! > -9> 2016-04-15 22:47:06.415851 7f65fbbb0800 0 <cls> > cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan > -8> 2016-04-15 22:47:06.418172 7f65fbbb0800 1 <cls> > cls/version/cls_version.cc:228: Loaded version class! > -7> 2016-04-15 22:47:06.419654 7f65fbbb0800 0 <cls> > cls/hello/cls_hello.cc:305: loading cls_hello > -6> 2016-04-15 22:47:06.426520 7f65fbbb0800 1 <cls> > cls/refcount/cls_refcount.cc:232: Loaded refcount class! > -5> 2016-04-15 22:47:06.427217 7f65fbbb0800 1 <cls> > cls/user/cls_user.cc:375: Loaded user class! > -4> 2016-04-15 22:47:06.428364 7f65fbbb0800 1 <cls> > cls/replica_log/cls_replica_log.cc:141: Loaded replica log class! > -3> 2016-04-15 22:47:06.428970 7f65fbbb0800 1 <cls> > cls/timeindex/cls_timeindex.cc:259: Loaded timeindex class! > -2> 2016-04-15 22:47:06.430177 7f65fbbb0800 1 <cls> > cls/log/cls_log.cc:317: Loaded log class! > -1> 2016-04-15 22:47:06.438063 7f65fbbb0800 1 <cls> > cls/rgw/cls_rgw.cc:3206: Loaded rgw class! > 0> 2016-04-15 22:47:06.498512 7f65fbbb0800 -1 osd/OSD.h: In function > 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f65fbbb0800 time 2016-04-15 > 22:47:06.494680 > osd/OSD.h: 885: FAILED assert(ret) > > ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x82) [0x7f65fb6364f2] > 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] > 3: (OSD::init()+0x1862) [0x7f65faf6ba52] > 4: (main()+0x2b05) [0x7f65faed1735] > 5: (__libc_start_main()+0xf5) [0x7f65f7a67b45] > 6: (()+0x337197) [0x7f65faf1c197] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_mirror > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > 1/ 5 compressor > 1/ 5 newstore > 1/ 5 bluestore > 1/ 5 bluefs > 1/ 3 bdev > 1/ 5 kstore > 4/ 5 rocksdb > 4/ 5 leveldb > 1/ 5 kinetic > 1/ 5 fuse > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.3.log > --- end dump of recent events --- > 2016-04-15 22:47:06.509080 7f65fbbb0800 -1 *** Caught signal (Aborted) ** > in thread 7f65fbbb0800 thread_name:ceph-osd > > ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) > 1: (()+0x949117) [0x7f65fb52e117] > 2: (()+0xf8d0) [0x7f65f9a318d0] > 3: (gsignal()+0x37) [0x7f65f7a7b067] > 4: (abort()+0x148) [0x7f65f7a7c448] > 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x256) [0x7f65fb6366c6] > 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] > 7: (OSD::init()+0x1862) [0x7f65faf6ba52] > 8: (main()+0x2b05) [0x7f65faed1735] > 9: (__libc_start_main()+0xf5) [0x7f65f7a67b45] > 10: (()+0x337197) [0x7f65faf1c197] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > --- begin dump of recent events --- > 0> 2016-04-15 22:47:06.509080 7f65fbbb0800 -1 *** Caught signal > (Aborted) ** > in thread 7f65fbbb0800 thread_name:ceph-osd > > ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4) > 1: (()+0x949117) [0x7f65fb52e117] > 2: (()+0xf8d0) [0x7f65f9a318d0] > 3: (gsignal()+0x37) [0x7f65f7a7b067] > 4: (abort()+0x148) [0x7f65f7a7c448] > 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x256) [0x7f65fb6366c6] > 6: (OSDService::get_map(unsigned int)+0x3d) [0x7f65fafbd83d] > 7: (OSD::init()+0x1862) [0x7f65faf6ba52] > 8: (main()+0x2b05) [0x7f65faed1735] > 9: (__libc_start_main()+0xf5) [0x7f65f7a67b45] > 10: (()+0x337197) [0x7f65faf1c197] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_mirror > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > 1/ 5 compressor > 1/ 5 newstore > 1/ 5 bluestore > 1/ 5 bluefs > 1/ 3 bdev > 1/ 5 kstore > 4/ 5 rocksdb > 4/ 5 leveldb > 1/ 5 kinetic > 1/ 5 fuse > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.3.log > --- end dump of recent events --- > > What can I try to get this OSD back online? I saw some similar issues on > google but I wasn't sure if that was actually the same issue. > > If I run in to MDS issue after resolving this, I'll send out another one. =) > Thanks all! > > Regards, > Hong > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Thank you! HuangJun _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |