Re: mimic: 3/4 OSDs crashed on "bluefs enospc"

[Igor Fedotov <ifedotov@xxxxxxx>]

So you should call repair which rebalances (i.e. allocates additional 
space) BlueFS space. Hence allowing OSD to start.

Thanks,

Igor


On 10/1/2018 7:22 PM, Igor Fedotov wrote:
> Not exactly. The rebalancing from this kv_sync_thread still might be 
> deferred due to the nature of this thread (haven't 100% sure though).
>
> Here is my PR showing the idea (still untested and perhaps unfinished!!!)
>
> https://github.com/ceph/ceph/pull/24353
>
>
> Igor
>
>
> On 10/1/2018 7:07 PM, Sergey Malinin wrote:
> > Can you please confirm whether I got this right:
> >
> > --- BlueStore.cc.bak    2018-10-01 18:54:45.096836419 +0300
> > +++ BlueStore.cc    2018-10-01 19:01:35.937623861 +0300
> > @@ -9049,22 +9049,17 @@
> >         throttle_bytes.put(costs);
> >           PExtentVector bluefs_gift_extents;
> > -      if (bluefs &&
> > -      after_flush - bluefs_last_balance >
> > -      cct->_conf->bluestore_bluefs_balance_interval) {
> > -    bluefs_last_balance = after_flush;
> > -    int r = _balance_bluefs_freespace(&bluefs_gift_extents);
> > -    assert(r >= 0);
> > -    if (r > 0) {
> > -      for (auto& p : bluefs_gift_extents) {
> > -        bluefs_extents.insert(p.offset, p.length);
> > -      }
> > -      bufferlist bl;
> > -      encode(bluefs_extents, bl);
> > -      dout(10) << __func__ << " bluefs_extents now 0x" << std::hex
> > -           << bluefs_extents << std::dec << dendl;
> > -      synct->set(PREFIX_SUPER, "bluefs_extents", bl);
> > +      int r = _balance_bluefs_freespace(&bluefs_gift_extents);
> > +      ceph_assert(r >= 0);
> > +      if (r > 0) {
> > +    for (auto& p : bluefs_gift_extents) {
> > +      bluefs_extents.insert(p.offset, p.length);
> >       }
> > +    bufferlist bl;
> > +    encode(bluefs_extents, bl);
> > +    dout(10) << __func__ << " bluefs_extents now 0x" << std::hex
> > +         << bluefs_extents << std::dec << dendl;
> > +    synct->set(PREFIX_SUPER, "bluefs_extents", bl);
> >         }
> >           // cleanup sync deferred keys
> >
> > > On 1.10.2018, at 18:39, Igor Fedotov <ifedotov@xxxxxxx> wrote:
> > >
> > > So you have just a single main device per OSD....
> > >
> > > Then bluestore-tool wouldn't help, it's unable to expand BlueFS 
> > > partition at main device, standalone devices are supported only.
> > >
> > > Given that you're able to rebuild the code I can suggest to make a 
> > > patch that triggers BlueFS rebalance (see code snippet below) on 
> > > repairing.
> > >      PExtentVector bluefs_gift_extents;
> > >      int r = _balance_bluefs_freespace(&bluefs_gift_extents);
> > >      ceph_assert(r >= 0);
> > >      if (r > 0) {
> > >        for (auto& p : bluefs_gift_extents) {
> > >          bluefs_extents.insert(p.offset, p.length);
> > >        }
> > >        bufferlist bl;
> > >        encode(bluefs_extents, bl);
> > >        dout(10) << __func__ << " bluefs_extents now 0x" << std::hex
> > >             << bluefs_extents << std::dec << dendl;
> > >        synct->set(PREFIX_SUPER, "bluefs_extents", bl);
> > >      }
> > >
> > > If it waits I can probably make a corresponding PR tomorrow.
> > >
> > > Thanks,
> > > Igor
> > > On 10/1/2018 6:16 PM, Sergey Malinin wrote:
> > > > I have rebuilt the tool, but none of my OSDs no matter dead or 
> > > > alive have any symlinks other than 'block' pointing to LVM.
> > > > I adjusted main device size but it looks like it needs even more 
> > > > space for db compaction. After executing bluefs-bdev-expand OSD 
> > > > fails to start, however 'fsck' and 'repair' commands finished 
> > > > successfully.
> > > >
> > > > 2018-10-01 18:02:39.755 7fc9226c6240  1 freelist init
> > > > 2018-10-01 18:02:39.763 7fc9226c6240  1 
> > > > bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc opening allocation 
> > > > metadata
> > > > 2018-10-01 18:02:40.907 7fc9226c6240  1 
> > > > bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc loaded 285 GiB in 
> > > > 2249899 extents
> > > > 2018-10-01 18:02:40.951 7fc9226c6240 -1 
> > > > bluestore(/var/lib/ceph/osd/ceph-1) _reconcile_bluefs_freespace 
> > > > bluefs extra 0x[6d6f000000~50c800000]
> > > > 2018-10-01 18:02:40.951 7fc9226c6240  1 stupidalloc 
> > > > 0x0x55d053fb9180 shutdown
> > > > 2018-10-01 18:02:40.963 7fc9226c6240  1 freelist shutdown
> > > > 2018-10-01 18:02:40.963 7fc9226c6240  4 rocksdb: 
> > > > [/build/ceph-13.2.2/src/rocksdb/db/db_impl.cc:252] Shutdown: 
> > > > canceling all background work
> > > > 2018-10-01 18:02:40.967 7fc9226c6240  4 rocksdb: 
> > > > [/build/ceph-13.2.2/src/rocksdb/db/db_impl.cc:397] Shutdown complete
> > > > 2018-10-01 18:02:40.971 7fc9226c6240  1 bluefs umount
> > > > 2018-10-01 18:02:40.975 7fc9226c6240  1 stupidalloc 
> > > > 0x0x55d053883800 shutdown
> > > > 2018-10-01 18:02:40.975 7fc9226c6240  1 bdev(0x55d053c32e00 
> > > > /var/lib/ceph/osd/ceph-1/block) close
> > > > 2018-10-01 18:02:41.267 7fc9226c6240  1 bdev(0x55d053c32a80 
> > > > /var/lib/ceph/osd/ceph-1/block) close
> > > > 2018-10-01 18:02:41.443 7fc9226c6240 -1 osd.1 0 OSD:init: unable to 
> > > > mount object store
> > > > 2018-10-01 18:02:41.443 7fc9226c6240 -1  ** ERROR: osd init failed: 
> > > > (5) Input/output error
> > > >
> > > >
> > > > > On 1.10.2018, at 18:09, Igor Fedotov <ifedotov@xxxxxxx> wrote:
> > > > >
> > > > > Well, actually you can avoid bluestore-tool rebuild.
> > > > >
> > > > > You'll need to edit the first chunk of blocks.db where labels are 
> > > > > stored. (Please make a backup first!!!)
> > > > >
> > > > > Size label is stored at offset 0x52 and is 8 bytes long - 
> > > > > little-endian 64bit integer encoding. (Please verify that old 
> > > > > value at this offset exactly corresponds to you original volume 
> > > > > size and/or 'size' label reported by ceph-bluestore-tool).
> > > > >
> > > > > So you have to put new DB volume size there. Or you can send the 
> > > > > first 4K chunk (e.g. extracted with dd) along with new DB volume 
> > > > > size (in bytes) to me and I'll do that for you.
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Igor
> > > > >
> > > > >
> > > > > On 10/1/2018 5:32 PM, Igor Fedotov wrote:
> > > > > > On 10/1/2018 5:03 PM, Sergey Malinin wrote:
> > > > > > > Before I received your response, I had already added 20GB to the 
> > > > > > > OSD (by epanding LV followed by bluefs-bdev-expand) and ran 
> > > > > > > "ceph-kvstore-tool bluestore-kv <path> compact", however it 
> > > > > > > still needs more space.
> > > > > > > Is that because I didn't update DB size with set-label-key?
> > > > > > In mimic you need to run both "bluefs-bdev-expand" and 
> > > > > > "set-label-key" command to commit bluefs volume expansion.
> > > > > > Unfortunately the last command doesn't handle "size" label 
> > > > > > properly. That's why you might need to backport and rebuild with 
> > > > > > the mentioned commits.
> > > > > >
> > > > > > > What exactly is the label-key that needs to be updated, as I 
> > > > > > > couldn't find which one is related to DB:
> > > > > > >
> > > > > > > # ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-1
> > > > > > > inferring bluefs devices from bluestore path
> > > > > > > {
> > > > > > >       "/var/lib/ceph/osd/ceph-1/block": {
> > > > > > >           "osd_uuid": "f8f122ee-70a6-4c54-8eb0-9b42205b1ecc",
> > > > > > >           "size": 471305551872,
> > > > > > >           "btime": "2018-07-31 03:06:43.751243",
> > > > > > >           "description": "main",
> > > > > > >           "bluefs": "1",
> > > > > > >           "ceph_fsid": "7d320499-5b3f-453e-831f-60d4db9a4533",
> > > > > > >           "kv_backend": "rocksdb",
> > > > > > >           "magic": "ceph osd volume v026",
> > > > > > >           "mkfs_done": "yes",
> > > > > > >           "osd_key": "XXX",
> > > > > > >           "ready": "ready",
> > > > > > >           "whoami": "1"
> > > > > > >       }
> > > > > > > }
> > > > > > 'size' label but your output is for block(aka slow) device.
> > > > > >
> > > > > > It should return labels for db/wal devices as well (block.db and 
> > > > > > block.wal symlinks respectively). It works for me in master, 
> > > > > > can't verify with mimic at the moment though..
> > > > > > Here is output for master:
> > > > > >
> > > > > > # bin/ceph-bluestore-tool show-label --path dev/osd0
> > > > > > inferring bluefs devices from bluestore path
> > > > > > {
> > > > > >      "dev/osd0/block": {
> > > > > >          "osd_uuid": "404dcbe9-3f8d-4ef5-ac59-2582454a9a75",
> > > > > >          "size": 21474836480,
> > > > > >          "btime": "2018-09-10 15:55:09.044039",
> > > > > >          "description": "main",
> > > > > >          "bluefs": "1",
> > > > > >          "ceph_fsid": "56eddc15-11b9-4e0b-9192-e391fbae551c",
> > > > > >          "kv_backend": "rocksdb",
> > > > > >          "magic": "ceph osd volume v026",
> > > > > >          "mkfs_done": "yes",
> > > > > >          "osd_key": "AQCsaZZbYTxXJBAAe3jJI4p6WbMjvA8CBBUJbA==",
> > > > > >          "ready": "ready",
> > > > > >          "whoami": "0"
> > > > > >      },
> > > > > >      "dev/osd0/block.wal": {
> > > > > >          "osd_uuid": "404dcbe9-3f8d-4ef5-ac59-2582454a9a75",
> > > > > >          "size": 1048576000,
> > > > > >          "btime": "2018-09-10 15:55:09.044985",
> > > > > >          "description": "bluefs wal"
> > > > > >      },
> > > > > >      "dev/osd0/block.db": {
> > > > > >          "osd_uuid": "404dcbe9-3f8d-4ef5-ac59-2582454a9a75",
> > > > > >          "size": 1048576000,
> > > > > >          "btime": "2018-09-10 15:55:09.044469",
> > > > > >          "description": "bluefs db"
> > > > > >      }
> > > > > > }
> > > > > >
> > > > > >
> > > > > > You can try --dev option instead of --path, e.g.
> > > > > > ceph-bluestore-tool show-label --dev <path-to-block.db>
> > > > > >
> > > > > >
> > > > > > > > On 1.10.2018, at 16:48, Igor Fedotov <ifedotov@xxxxxxx> wrote:
> > > > > > > >
> > > > > > > > This looks like a sort of deadlock when BlueFS needs some 
> > > > > > > > additional space to replay the log left after the crash. Which 
> > > > > > > > happens during BlueFS open.
> > > > > > > >
> > > > > > > > But such a space (at slow device as DB is full) is gifted in 
> > > > > > > > background during bluefs rebalance procedure which will occur 
> > > > > > > > after the open.
> > > > > > > >
> > > > > > > > Hence OSDs stuck in permanent crashing..
> > > > > > > >
> > > > > > > > The only way to recover I can suggest for now is to expand DB 
> > > > > > > > volumes. You can do that with lvm tools if you have any spare 
> > > > > > > > space for that.
> > > > > > > >
> > > > > > > > Once resized you'll need ceph-bluestore-tool to indicate volume 
> > > > > > > > expansion to BlueFS (bluefs-bdev-expand command ) and finally 
> > > > > > > > update DB volume size label with  set-label-key command.
> > > > > > > >
> > > > > > > > The latter is a bit tricky for mimic - you might need to 
> > > > > > > > backport 
> > > > > > > > https://github.com/ceph/ceph/pull/22085/commits/ffac450da5d6e09cf14b8363b35f21819b48f38b
> > > > > > > >
> > > > > > > > and rebuild ceph-bluestore-tool. Alternatively you can backport 
> > > > > > > > https://github.com/ceph/ceph/pull/22085/commits/71c3b58da4e7ced3422bce2b1da0e3fa9331530b
> > > > > > > >
> > > > > > > > then bluefs expansion and label updates will occur in a single 
> > > > > > > > step.
> > > > > > > >
> > > > > > > > I'll do these backports in upstream but this will take some 
> > > > > > > > time to pass all the procedures and get into official mimic 
> > > > > > > > release.
> > > > > > > >
> > > > > > > > Will fire a ticket to fix the original issue as well.
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Igor
> > > > > > > >
> > > > > > > >
> > > > > > > > On 10/1/2018 3:28 PM, Sergey Malinin wrote:
> > > > > > > > > These are LVM bluestore NVMe SSDs created with "ceph-volume 
> > > > > > > > > --lvm prepare --bluestore /dev/nvme0n1p3" i.e. without 
> > > > > > > > > specifying wal/db devices.
> > > > > > > > > OSDs were created with bluestore_min_alloc_size_ssd=4096, 
> > > > > > > > > another modified setting is bluestore_cache_kv_max=1073741824
> > > > > > > > >
> > > > > > > > > DB/block usage collected by prometheus module for 3 failed and 
> > > > > > > > > 1 survived OSDs:
> > > > > > > > >
> > > > > > > > > ceph_bluefs_db_total_bytes{ceph_daemon="osd.0"} 65493008384.0
> > > > > > > > > ceph_bluefs_db_total_bytes{ceph_daemon="osd.1"} 49013587968.0
> > > > > > > > > ceph_bluefs_db_total_bytes{ceph_daemon="osd.2"} 76834406400.0 
> > > > > > > > > --> this one has survived
> > > > > > > > > ceph_bluefs_db_total_bytes{ceph_daemon="osd.3"} 63726157824.0
> > > > > > > > >
> > > > > > > > > ceph_bluefs_db_used_bytes{ceph_daemon="osd.0"} 65217232896.0
> > > > > > > > > ceph_bluefs_db_used_bytes{ceph_daemon="osd.1"} 48944381952.0
> > > > > > > > > ceph_bluefs_db_used_bytes{ceph_daemon="osd.2"} 68093476864.0
> > > > > > > > > ceph_bluefs_db_used_bytes{ceph_daemon="osd.3"} 63632834560.0
> > > > > > > > >
> > > > > > > > > ceph_osd_stat_bytes{ceph_daemon="osd.0"} 471305551872.0
> > > > > > > > > ceph_osd_stat_bytes{ceph_daemon="osd.1"} 471305551872.0
> > > > > > > > > ceph_osd_stat_bytes{ceph_daemon="osd.2"} 471305551872.0
> > > > > > > > > ceph_osd_stat_bytes{ceph_daemon="osd.3"} 471305551872.0
> > > > > > > > >
> > > > > > > > > ceph_osd_stat_bytes_used{ceph_daemon="osd.0"} 222328213504.0
> > > > > > > > > ceph_osd_stat_bytes_used{ceph_daemon="osd.1"} 214472544256.0
> > > > > > > > > ceph_osd_stat_bytes_used{ceph_daemon="osd.2"} 163603996672.0
> > > > > > > > > ceph_osd_stat_bytes_used{ceph_daemon="osd.3"} 212806815744.0
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > First crashed OSD was doing DB compaction, others crashed 
> > > > > > > > > shortly after during backfilling. Workload was "ceph-data-scan 
> > > > > > > > > scan_inodes" filling metadata pool located on these OSDs at 
> > > > > > > > > the rate close to 10k objects/second.
> > > > > > > > > Here is the log excerpt of the first crash occurrence:
> > > > > > > > >
> > > > > > > > > 2018-10-01 03:27:12.762 7fbf16dd6700  0 
> > > > > > > > > bluestore(/var/lib/ceph/osd/ceph-1) _balance_bluefs_freespace 
> > > > > > > > > no allocate on 0x80000000 min_alloc_size 0x1000
> > > > > > > > > 2018-10-01 03:27:12.886 7fbf1e5e5700  4 rocksdb: 
> > > > > > > > > [/build/ceph-13.2.2/src/rocksdb/db/compaction_job.cc:1166] 
> > > > > > > > > [default] [JOB 24] Generated table #89741: 106356 keys, 
> > > > > > > > > 68110589 bytes
> > > > > > > > > 2018-10-01 03:27:12.886 7fbf1e5e5700  4 rocksdb: EVENT_LOG_v1 
> > > > > > > > > {"time_micros": 1538353632892744, "cf_name": "default", "job": 
> > > > > > > > > 24, "event": "table_file_creation", "file_number": 89741, 
> > > > > > > > > "file_size": 68110589, "table_properties": {"data_size": 
> > > > > > > > > 67112903, "index_size": 579319, "filter_size": 417316, 
> > > > > > > > > "raw_key_size": 6733561, "raw_average_key_size": 63, 
> > > > > > > > > "raw_value_size": 60994583, "raw_average_value_size": 573, 
> > > > > > > > > "num_data_blocks": 16336, "num_entries": 106356, 
> > > > > > > > > "filter_policy_name": "rocksdb.BuiltinBloomFilter", 
> > > > > > > > > "kDeletedKeys": "14444", "kMergeOperands": "0"}}
> > > > > > > > > 2018-10-01 03:27:12.934 7fbf1e5e5700  4 rocksdb: 
> > > > > > > > > [/build/ceph-13.2.2/src/rocksdb/db/compaction_job.cc:1166] 
> > > > > > > > > [default] [JOB 24] Generated table #89742: 23214 keys, 
> > > > > > > > > 16352315 bytes
> > > > > > > > > 2018-10-01 03:27:12.934 7fbf1e5e5700  4 rocksdb: EVENT_LOG_v1 
> > > > > > > > > {"time_micros": 1538353632938670, "cf_name": "default", "job": 
> > > > > > > > > 24, "event": "table_file_creation", "file_number": 89742, 
> > > > > > > > > "file_size": 16352315, "table_properties": {"data_size": 
> > > > > > > > > 16116986, "index_size": 139894, "filter_size": 94386, 
> > > > > > > > > "raw_key_size": 1470883, "raw_average_key_size": 63, 
> > > > > > > > > "raw_value_size": 14775006, "raw_average_value_size": 636, 
> > > > > > > > > "num_data_blocks": 3928, "num_entries": 23214, 
> > > > > > > > > "filter_policy_name": "rocksdb.BuiltinBloomFilter", 
> > > > > > > > > "kDeletedKeys": "90", "kMergeOperands": "0"}}
> > > > > > > > > 2018-10-01 03:27:13.042 7fbf1e5e5700  1 bluefs _allocate 
> > > > > > > > > failed to allocate 0x4100000 on bdev 1, free 0x1a00000; 
> > > > > > > > > fallback to bdev 2
> > > > > > > > > 2018-10-01 03:27:13.042 7fbf1e5e5700 -1 bluefs _allocate 
> > > > > > > > > failed to allocate 0x4100000 on bdev 2, dne
> > > > > > > > > 2018-10-01 03:27:13.042 7fbf1e5e5700 -1 bluefs _flush_range 
> > > > > > > > > allocated: 0x0 offset: 0x0 length: 0x40ea9f1
> > > > > > > > > 2018-10-01 03:27:13.046 7fbf1e5e5700 -1 
> > > > > > > > > /build/ceph-13.2.2/src/os/bluestore/BlueFS.cc: In function 
> > > > > > > > > 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, 
> > > > > > > > > uint64_t)' thread 7fbf1e5e5700 time 2018-10-01 03:27:13.048298
> > > > > > > > > /build/ceph-13.2.2/src/os/bluestore/BlueFS.cc: 1663: FAILED 
> > > > > > > > > assert(0 == "bluefs enospc")
> > > > > > > > >
> > > > > > > > >    ceph version 13.2.2 
> > > > > > > > > (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)
> > > > > > > > >    1: (ceph::__ceph_assert_fail(char const*, char const*, int, 
> > > > > > > > > char const*)+0x102) [0x7fbf2d4fe5c2]
> > > > > > > > >    2: (()+0x26c787) [0x7fbf2d4fe787]
> > > > > > > > >    3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned 
> > > > > > > > > long, unsigned long)+0x1ab4) [0x5619325114b4]
> > > > > > > > >    4: (BlueRocksWritableFile::Flush()+0x3d) [0x561932527c1d]
> > > > > > > > >    5: (rocksdb::WritableFileWriter::Flush()+0x1b9) 
> > > > > > > > > [0x56193271c399]
> > > > > > > > >    6: (rocksdb::WritableFileWriter::Sync(bool)+0x3b) 
> > > > > > > > > [0x56193271d42b]
> > > > > > > > >    7: 
> > > > > > > > > (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status 
> > > > > > > > > const&, rocksdb::CompactionJob::SubcompactionState*, 
> > > > > > > > > rocksdb::RangeDelAggregator*, CompactionIterationStats*, 
> > > > > > > > > rocksdb::Slice const*)+0x3db) [0x56193276098b]
> > > > > > > > >    8: 
> > > > > > > > > (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0x7d9) 
> > > > > > > > > [0x561932763da9]
> > > > > > > > >    9: (rocksdb::CompactionJob::Run()+0x314) [0x561932765504]
> > > > > > > > >    10: (rocksdb::DBImpl::BackgroundCompaction(bool*, 
> > > > > > > > > rocksdb::JobContext*, rocksdb::LogBuffer*, 
> > > > > > > > > rocksdb::DBImpl::PrepickedCompaction*)+0xc54) [0x5619325b5c44]
> > > > > > > > >    11: 
> > > > > > > > > (rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, 
> > > > > > > > > rocksdb::Env::Priority)+0x397) [0x5619325b8557]
> > > > > > > > >    12: (rocksdb::DBImpl::BGWorkCompaction(void*)+0x97) 
> > > > > > > > > [0x5619325b8cd7]
> > > > > > > > >    13: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned 
> > > > > > > > > long)+0x266) [0x5619327a5e36]
> > > > > > > > >    14: 
> > > > > > > > > (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x47) 
> > > > > > > > > [0x5619327a5fb7]
> > > > > > > > >    15: (()+0xbe733) [0x7fbf2b500733]
> > > > > > > > >    16: (()+0x76db) [0x7fbf2bbf86db]
> > > > > > > > >    17: (clone()+0x3f) [0x7fbf2abbc88f]
> > > > > > > > >    NOTE: a copy of the executable, or `objdump -rdS 
> > > > > > > > > <executable>` is needed to interpret this.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > On 1.10.2018, at 15:01, Igor Fedotov <ifedotov@xxxxxxx> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Sergey,
> > > > > > > > > >
> > > > > > > > > > could you please provide more details on your OSDs ?
> > > > > > > > > >
> > > > > > > > > > What are sizes for DB/block devices?
> > > > > > > > > >
> > > > > > > > > > Do you have any modifications in BlueStore config settings?
> > > > > > > > > >
> > > > > > > > > > Can you share stats you're referring to?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Igor
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 10/1/2018 12:29 PM, Sergey Malinin wrote:
> > > > > > > > > > > Hello,
> > > > > > > > > > > 3 of 4 NVME OSDs crashed at the same time on assert(0 == 
> > > > > > > > > > > "bluefs enospc") and no longer start.
> > > > > > > > > > > Stats collected just before crash show that 
> > > > > > > > > > > ceph_bluefs_db_used_bytes is 100% used. Although OSDs have 
> > > > > > > > > > > over 50% of free space, it is not reallocated for DB usage.
> > > > > > > > > > >
> > > > > > > > > > > 2018-10-01 12:18:06.744 7f1d6a04d240  1 bluefs _allocate 
> > > > > > > > > > > failed to allocate 0x100000 on bdev 1, free 0x0; fallback to 
> > > > > > > > > > > bdev 2
> > > > > > > > > > > 2018-10-01 12:18:06.744 7f1d6a04d240 -1 bluefs _allocate 
> > > > > > > > > > > failed to allocate 0x100000 on bdev 2, dne
> > > > > > > > > > > 2018-10-01 12:18:06.744 7f1d6a04d240 -1 bluefs _flush_range 
> > > > > > > > > > > allocated: 0x0 offset: 0x0 length: 0xa8700
> > > > > > > > > > > 2018-10-01 12:18:06.748 7f1d6a04d240 -1 
> > > > > > > > > > > /build/ceph-13.2.2/src/os/bluestore/BlueFS.cc: In function 
> > > > > > > > > > > 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, 
> > > > > > > > > > > uint64_t)' thread 7f1d6a04d240 time 2018-10-01 12:18:06.746800
> > > > > > > > > > > /build/ceph-13.2.2/src/os/bluestore/BlueFS.cc: 1663: FAILED 
> > > > > > > > > > > assert(0 == "bluefs enospc")
> > > > > > > > > > >
> > > > > > > > > > >    ceph version 13.2.2 
> > > > > > > > > > > (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)
> > > > > > > > > > >    1: (ceph::__ceph_assert_fail(char const*, char const*, 
> > > > > > > > > > > int, char const*)+0x102) [0x7f1d6146f5c2]
> > > > > > > > > > >    2: (()+0x26c787) [0x7f1d6146f787]
> > > > > > > > > > >    3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned 
> > > > > > > > > > > long, unsigned long)+0x1ab4) [0x5586b22684b4]
> > > > > > > > > > >    4: (BlueRocksWritableFile::Flush()+0x3d) [0x5586b227ec1d]
> > > > > > > > > > >    5: (rocksdb::WritableFileWriter::Flush()+0x1b9) 
> > > > > > > > > > > [0x5586b2473399]
> > > > > > > > > > >    6: (rocksdb::WritableFileWriter::Sync(bool)+0x3b) 
> > > > > > > > > > > [0x5586b247442b]
> > > > > > > > > > >    7: (rocksdb::BuildTable(std::__cxx11::basic_string<char, 
> > > > > > > > > > > std::char_traits<char>, std::allocator<char> > const&, 
> > > > > > > > > > > rocksdb::Env*, rocksdb::ImmutableCFOptions const&, 
> > > > > > > > > > > rocksdb::MutableCFOptions const&, rocksdb::EnvOptions 
> > > > > > > > > > > const&, rock
> > > > > > > > > > > sdb::TableCache*, rocksdb::InternalIterator*, 
> > > > > > > > > > > std::unique_ptr<rocksdb::InternalIterator, 
> > > > > > > > > > > std::default_delete<rocksdb::InternalIterator> >, 
> > > > > > > > > > > rocksdb::FileMetaData*, rocksdb::InternalKeyComparator 
> > > > > > > > > > > const&, std::vector<std::unique_ptr<
> > > > > > > > > > > rocksdb::IntTblPropCollectorFactory, 
> > > > > > > > > > > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, 
> > > > > > > > > > > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, 
> > > > > > > > > > > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > 
> > > > > > > > > > > > co
> > > > > > > > > > > nst*, unsigned int, std::__cxx11::basic_string<char, 
> > > > > > > > > > > std::char_traits<char>, std::allocator<char> > const&, 
> > > > > > > > > > > std::vector<unsigned long, std::allocator<unsigned long> >, 
> > > > > > > > > > > unsigned long, rocksdb::SnapshotChecker*, rocksdb::Compression
> > > > > > > > > > > Type, rocksdb::CompressionOptions const&, bool, 
> > > > > > > > > > > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, 
> > > > > > > > > > > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, 
> > > > > > > > > > > rocksdb::TableProperties*, int, unsigned long, unsigned 
> > > > > > > > > > > long, rocksdb
> > > > > > > > > > > ::Env::WriteLifeTimeHint)+0x1e24) [0x5586b249ef94]
> > > > > > > > > > >    8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, 
> > > > > > > > > > > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, 
> > > > > > > > > > > rocksdb::VersionEdit*)+0xcb7) [0x5586b2321457]
> > > > > > > > > > >    9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned 
> > > > > > > > > > > long, std::allocator<unsigned long> > const&, unsigned 
> > > > > > > > > > > long*, bool)+0x19de) [0x5586b232373e]
> > > > > > > > > > >    10: 
> > > > > > > > > > > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, 
> > > > > > > > > > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, 
> > > > > > > > > > > bool, bool, bool)+0x5d4) [0x5586b23242f4]
> > > > > > > > > > >    11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, 
> > > > > > > > > > > std::__cxx11::basic_string<char, std::char_traits<char>, 
> > > > > > > > > > > std::allocator<char> > const&, 
> > > > > > > > > > > std::vector<rocksdb::ColumnFamilyDescriptor, 
> > > > > > > > > > > std::allocator<rocksdb::ColumnFamilyDescri
> > > > > > > > > > > ptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, 
> > > > > > > > > > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, 
> > > > > > > > > > > rocksdb::DB**, bool)+0x68b) [0x5586b232559b]
> > > > > > > > > > >    12: (rocksdb::DB::Open(rocksdb::DBOptions const&, 
> > > > > > > > > > > std::__cxx11::basic_string<char, std::char_traits<char>, 
> > > > > > > > > > > std::allocator<char> > const&, 
> > > > > > > > > > > std::vector<rocksdb::ColumnFamilyDescriptor, 
> > > > > > > > > > > std::allocator<rocksdb::ColumnFamilyDescriptor
> > > > > > > > > > > > > const&, std::vector<rocksdb::ColumnFamilyHandle*, 
> > > > > > > > > > > > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, 
> > > > > > > > > > > > > rocksdb::DB**)+0x22) [0x5586b2326e72]
> > > > > > > > > > >    13: (RocksDBStore::do_open(std::ostream&, bool, 
> > > > > > > > > > > std::vector<KeyValueDB::ColumnFamily, 
> > > > > > > > > > > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x170c) 
> > > > > > > > > > > [0x5586b220219c]
> > > > > > > > > > >    14: (BlueStore::_open_db(bool, bool)+0xd8e) [0x5586b218ee1e]
> > > > > > > > > > >    15: (BlueStore::_mount(bool, bool)+0x4b7) [0x5586b21bf807]
> > > > > > > > > > >    16: (OSD::init()+0x295) [0x5586b1d673c5]
> > > > > > > > > > >    17: (main()+0x268d) [0x5586b1c554ed]
> > > > > > > > > > >    18: (__libc_start_main()+0xe7) [0x7f1d5ea2db97]
> > > > > > > > > > >    19: (_start()+0x2a) [0x5586b1d1d7fa]
> > > > > > > > > > >    NOTE: a copy of the executable, or `objdump -rdS 
> > > > > > > > > > > <executable>` is needed to interpret this.
> > > > > > > > > > >
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > ceph-users mailing list
> > > > > > > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > > _______________________________________________
> > > > > > ceph-users mailing list
> > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com