Yes, except that this happens on 8 different clusters with different hw but same ceph version and same kernel version. Greets, Stefan > Am 19.01.2020 um 11:53 schrieb Igor Fedotov <ifedotov@xxxxxxx>: > > So the intermediate summary is: > > Any OSD in the cluster can experience interim RocksDB checksum failure. Which isn't present after OSD restart. > > No HW issues observed, no persistent artifacts (except OSD log) afterwards. > > And looks like the issue is rather specific to the cluster as no similar reports from other users seem to be present. > > > Sorry, I'm out of ideas other then collect all the failure logs and try to find something common in them. May be this will shed some light.. > > BTW from my experience it might make sense to inspect OSD log prior to failure (any error messages and/or prior restarts, etc) sometimes this might provide some hints. > > > Thanks, > > Igor > > >> On 1/17/2020 2:30 PM, Stefan Priebe - Profihost AG wrote: >> HI Igor, >> >>> Am 17.01.20 um 12:10 schrieb Igor Fedotov: >>> hmmm.. >>> >>> Just in case - suggest to check H/W errors with dmesg. >> this happens on around 80 nodes - i don't expect all of those have not >> identified hw errors. Also all of them are monitored - no dmesg outpout >> contains any errors. >> >>> Also there are some (not very much though) chances this is another >>> incarnation of the following bug: >>> https://tracker.ceph.com/issues/22464 >>> https://github.com/ceph/ceph/pull/24649 >>> >>> The corresponding PR works around it for main device reads (user data >>> only!) but theoretically it might still happen >>> >>> either for DB device or DB data at main device. >>> >>> Can you observe any bluefs spillovers? Are there any correlation between >>> failing OSDs and spillover presence if any, e.g. failing OSDs always >>> have a spillover. While OSDs without spillovers never face the issue... >>> >>> To validate this hypothesis one can try to monitor/check (e.g. once a >>> day for a week or something) "bluestore_reads_with_retries" counter over >>> OSDs to learn if the issue is happening >>> >>> in the system. Non-zero values mean it's there for user data/main >>> device and hence is likely to happen for DB ones as well (which doesn't >>> have any workaround yet). >> OK i checked bluestore_reads_with_retries on 360 osds but all of them say 0. >> >> >>> Additionally you might want to monitor memory usage as the above >>> mentioned PR denotes high memory pressure as potential trigger for these >>> read errors. So if such pressure happens the hypothesis becomes more valid. >> we already do this heavily and have around 10GB of memory per OSD. Also >> no of those machines show any io pressure at all. >> >> All hosts show a constant rate of around 38GB to 45GB mem available in >> /proc/meminfo. >> >> Stefan >> >>> Thanks, >>> >>> Igor >>> >>> PS. Everything above is rather a speculation for now. Available >>> information is definitely not enough for extensive troubleshooting the >>> cases which happens that rarely. >>> >>> You might want to start collecting failure-related information >>> (including but not limited to failure logs, perf counter dumps, system >>> resource reports etc) for future analysis. >>> >>> >>> >>> On 1/16/2020 11:58 PM, Stefan Priebe - Profihost AG wrote: >>>> Hi Igor, >>>> >>>> answers inline. >>>> >>>> Am 16.01.20 um 21:34 schrieb Igor Fedotov: >>>>> you may want to run fsck against failing OSDs. Hopefully it will shed >>>>> some light. >>>> fsck just says everything fine: >>>> >>>> # ceph-bluestore-tool --command fsck --path /var/lib/ceph/osd/ceph-27/ >>>> fsck success >>>> >>>> >>>>> Also wondering if OSD is able to recover (startup and proceed working) >>>>> after facing the issue? >>>> no recover needed. It just runs forever after restarting. >>>> >>>>> If so do you have any one which failed multiple times? Do you have logs >>>>> for these occurrences? >>>> may be but there are most probably weeks or month between those failures >>>> - most probably logs are already deleted. >>>> >>>>> Also please note that patch you mentioned doesn't fix previous issues >>>>> (i.e. duplicate allocations), it prevents from new ones only. >>>>> >>>>> But fsck should show them if any... >>>> None showed. >>>> >>>> Stefan >>>> >>>>> Thanks, >>>>> >>>>> Igor >>>>> >>>>> >>>>> >>>>> On 1/16/2020 10:04 PM, Stefan Priebe - Profihost AG wrote: >>>>>> Hi Igor, >>>>>> >>>>>> ouch sorry. Here we go: >>>>>> >>>>>> -1> 2020-01-16 01:10:13.404090 7f3350a14700 -1 rocksdb: >>>>>> submit_transaction error: Corruption: block checksum mismatch code = 2 >>>>>> Rocksdb transaction: >>>>>> Put( Prefix = M key = >>>>>> 0x0000000000000402'.OBJ_0000000000000002.953BFD0A.bb85c.rbd%udata%e3e8eac6b8b4567%e0000000000001f2e..' >>>>>> >>>>>> >>>>>> Value size = 97) >>>>>> Put( Prefix = M key = >>>>>> 0x0000000000000402'.MAP_00000000000BB85C_0000000000000002.953BFD0A.bb85c.rbd%udata%e3e8eac6b8b4567%e0000000000001f2e..' >>>>>> >>>>>> >>>>>> Value size = 93) >>>>>> Put( Prefix = M key = >>>>>> 0x0000000000000916'.0000823257.00000000000073922044' Value size = 196) >>>>>> Put( Prefix = M key = >>>>>> 0x0000000000000916'.0000823257.00000000000073922045' Value size = 184) >>>>>> Put( Prefix = M key = 0x0000000000000916'._info' Value size = 899) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00000000'x' >>>>>> >>>>>> >>>>>> Value size = 418) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00030000'x' >>>>>> >>>>>> >>>>>> Value size = 474) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f0007c000'x' >>>>>> >>>>>> >>>>>> Value size = 392) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00090000'x' >>>>>> >>>>>> >>>>>> Value size = 317) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f000a0000'x' >>>>>> >>>>>> >>>>>> Value size = 521) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f000f4000'x' >>>>>> >>>>>> >>>>>> Value size = 558) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00130000'x' >>>>>> >>>>>> >>>>>> Value size = 649) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00194000'x' >>>>>> >>>>>> >>>>>> Value size = 449) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f001cc000'x' >>>>>> >>>>>> >>>>>> Value size = 580) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00200000'x' >>>>>> >>>>>> >>>>>> Value size = 435) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00240000'x' >>>>>> >>>>>> >>>>>> Value size = 569) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00290000'x' >>>>>> >>>>>> >>>>>> Value size = 465) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f002e0000'x' >>>>>> >>>>>> >>>>>> Value size = 710) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f00300000'x' >>>>>> >>>>>> >>>>>> Value size = 599) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f0036c000'x' >>>>>> >>>>>> >>>>>> Value size = 372) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f003a6000'x' >>>>>> >>>>>> >>>>>> Value size = 130) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f003b4000'x' >>>>>> >>>>>> >>>>>> Value size = 540) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff6f003fc000'x' >>>>>> >>>>>> >>>>>> Value size = 47) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0x00000000000bb85cffffffffffffffff'o' >>>>>> >>>>>> >>>>>> Value size = 1731) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0xfffffffffffffffeffffffffffffffff6f00040000'x' >>>>>> >>>>>> >>>>>> Value size = 675) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0xfffffffffffffffeffffffffffffffff6f00080000'x' >>>>>> >>>>>> >>>>>> Value size = 395) >>>>>> Put( Prefix = O key = >>>>>> 0x7f80000000000000029acdfb05217262'd_data.3e8eac6b8b4567.0000000000001f2e!='0xfffffffffffffffeffffffffffffffff'o' >>>>>> >>>>>> >>>>>> Value size = 1328) >>>>>> Put( Prefix = X key = 0x0000000018a38deb Value size = 14) >>>>>> Put( Prefix = X key = 0x0000000018a38dea Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a035b Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a035c Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a0355 Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a0356 Value size = 17) >>>>>> Put( Prefix = X key = 0x000000001a54f6e4 Value size = 14) >>>>>> Put( Prefix = X key = 0x000000001b1c061e Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a038f Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a0389 Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a0358 Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a035f Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a0357 Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a0387 Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a038a Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a0388 Value size = 14) >>>>>> Put( Prefix = X key = 0x00000000134c3fbe Value size = 14) >>>>>> Put( Prefix = X key = 0x00000000134c3fb5 Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a036e Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a036d Value size = 14) >>>>>> Put( Prefix = X key = 0x00000000134c3fb8 Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a0371 Value size = 14) >>>>>> Put( Prefix = X key = 0x000000000d7a036a Value size = 14) >>>>>> 0> 2020-01-16 01:10:13.413759 7f3350a14700 -1 >>>>>> /build/ceph/src/os/bluestore/BlueStore.cc: In function 'void >>>>>> BlueStore::_kv_sync_thread()' thread 7f3350a14700 time 2020-01-16 >>>>>> 01:10:13.404113 >>>>>> /build/ceph/src/os/bluestore/BlueStore.cc: 8808: FAILED assert(r == 0) >>>>>> >>>>>> ceph version 12.2.12-11-gd3eae83543 >>>>>> (d3eae83543bffc0fc6c43823feb637fa851b6213) luminous (stable) >>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>>>>> const*)+0x102) [0x55c9a712d232] >>>>>> 2: (BlueStore::_kv_sync_thread()+0x24c5) [0x55c9a6fb54b5] >>>>>> 3: (BlueStore::KVSyncThread::entry()+0xd) [0x55c9a6ff608d] >>>>>> 4: (()+0x7494) [0x7f33615f9494] >>>>>> 5: (clone()+0x3f) [0x7f3360680acf] >>>>>> >>>>>> I already picked those: >>>>>> https://github.com/ceph/ceph/pull/28644 >>>>>> >>>>>> Greets, >>>>>> Stefan >>>>>> Am 16.01.20 um 17:00 schrieb Igor Fedotov: >>>>>>> Hi Stefan, >>>>>>> >>>>>>> would you please share log snippet prior the assertions? Looks like >>>>>>> RocksDB is failing during transaction submission... >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Igor >>>>>>> >>>>>>> On 1/16/2020 11:56 AM, Stefan Priebe - Profihost AG wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> does anybody know a fix for this ASSERT / crash? >>>>>>>> >>>>>>>> 2020-01-16 02:02:31.316394 7f8c3f5ab700 -1 >>>>>>>> /build/ceph/src/os/bluestore/BlueStore.cc: In function 'void >>>>>>>> BlueStore::_kv_sync_thread()' thread 7f8c3f5ab700 time 2020-01-16 >>>>>>>> 02:02:31.304993 >>>>>>>> /build/ceph/src/os/bluestore/BlueStore.cc: 8808: FAILED assert(r >>>>>>>> == 0) >>>>>>>> >>>>>>>> ceph version 12.2.12-11-gd3eae83543 >>>>>>>> (d3eae83543bffc0fc6c43823feb637fa851b6213) luminous (stable) >>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >>>>>>>> const*)+0x102) [0x55e6df9d9232] >>>>>>>> 2: (BlueStore::_kv_sync_thread()+0x24c5) [0x55e6df8614b5] >>>>>>>> 3: (BlueStore::KVSyncThread::entry()+0xd) [0x55e6df8a208d] >>>>>>>> 4: (()+0x7494) [0x7f8c50190494] >>>>>>>> 5: (clone()+0x3f) [0x7f8c4f217acf] >>>>>>>> >>>>>>>> all bluestore OSDs are randomly crashing sometimes (once a week). >>>>>>>> >>>>>>>> Greets, >>>>>>>> Stefan >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list >>>>>>>> ceph-users@xxxxxxxxxxxxxx >>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |