Hello Eugen, thank you for you reply. Yes, restarting all OSDs, monitors, also increasing osd_map_cache_size to 5000 (this helped us in case of problem with not pruning OSD maps). none of this helped.. with best regards nik On Wed, Nov 03, 2021 at 11:41:28AM +0000, Eugen Block wrote: > Hi, > > I don't have an explanation but I remember having a similar issue a > year ago or so. IIRC a simple OSD restart fixed that, so I never got > to the bottom of it. Have you tried to restart OSD daemons? > > > Zitat von Nikola Ciprich <nikola.ciprich@xxxxxxxxxxx>: > > >Hello fellow ceph users, > > > >I'm trying to catch ghost here.. On one of our clusters, 6 nodes, > >14.2.15, EC pool 4+2, 6*32 SATA bluestore OSDs we got into very strange > >state. > > > >The cluster is clean (except for pgs not deep-scrubbed in time warning, > >since we've disabled scrubbing while investigating), there is absolutely > >no activity on EC pool, but according to atop, all OSDs are still reading > >furiously, without any apparent reason. even when increasing osd loglevel, > >I don't see anything interesting, except for occasional > >2021-11-03 12:04:52.664 7fb8652e3700 5 osd.0 9347 heartbeat > >osd_stat(store_statfs(0xb80056c0000/0x26b570000/0xe8d7fc00000, > >data 0x2f0ddd813e8/0x30b0ee60000, compress 0x0/0x0/0x0, omap > >0x98b706, meta 0x26abe48fa), peers [1,26,27,34,36,40,44,49,52,55,57,65,69,75,76,78,82,83,87,93,96,97,104,105,107,108,111,112,114,120,121,122,123,135,136,137,143,147,154,156,157,169,171,187,192,196,200,204,208,212,217,218,220,222,224,226,227] > >op hist []) > >and also compactions stats. > > > >trying to sequentially read data from the pool leads to very poor > >performance (ie 8MB/s) > > > >We've had very similar problem on different cluster (replicated, no EC), when > >osdmaps were not pruned correctly, but I checked and those seem to > >be OK, it's just > >OSD are still reading something and I'm unable to find out what. > > > >here's output of crush for one node, others are pretty similar: > > > > -1 2803.19824 - 2.7 PiB 609 TiB 607 TiB 1.9 GiB > >1.9 TiB 2.1 PiB 21.78 1.01 - root sata > > -2 467.19971 - 466 TiB 102 TiB 101 TiB 320 MiB > >328 GiB 364 TiB 21.83 1.01 - host spbstdv1a-sata > > 0 hdd 14.59999 1.00000 15 TiB 3.1 TiB 3.0 TiB 9.5 MiB > >9.7 GiB 12 TiB 20.98 0.97 51 up osd.0 > > 1 hdd 14.59999 1.00000 15 TiB 2.4 TiB 2.4 TiB 7.4 MiB > >7.7 GiB 12 TiB 16.34 0.76 50 up osd.1 > > 2 hdd 14.59999 1.00000 15 TiB 3.5 TiB 3.5 TiB 11 MiB > >11 GiB 11 TiB 24.33 1.13 51 up osd.2 > > 3 hdd 14.59999 1.00000 15 TiB 2.9 TiB 2.8 TiB 9.3 MiB > >9.1 GiB 12 TiB 19.58 0.91 48 up osd.3 > > 4 hdd 14.59999 1.00000 15 TiB 3.3 TiB 3.3 TiB 11 MiB > >11 GiB 11 TiB 22.94 1.06 51 up osd.4 > > 5 hdd 14.59999 1.00000 15 TiB 3.5 TiB 3.5 TiB 12 MiB > >12 GiB 11 TiB 23.94 1.11 50 up osd.5 > > 6 hdd 14.59999 1.00000 15 TiB 2.8 TiB 2.8 TiB 9.6 MiB > >9.6 GiB 12 TiB 19.11 0.89 49 up osd.6 > > 7 hdd 14.59999 1.00000 15 TiB 3.4 TiB 3.4 TiB 4.9 MiB > >11 GiB 11 TiB 23.68 1.10 50 up osd.7 > > 8 hdd 14.59998 1.00000 15 TiB 3.2 TiB 3.2 TiB 10 MiB > >10 GiB 11 TiB 22.18 1.03 51 up osd.8 > > 9 hdd 14.59999 1.00000 15 TiB 3.4 TiB 3.4 TiB 4.9 MiB > >11 GiB 11 TiB 23.52 1.09 50 up osd.9 > > 10 hdd 14.59999 1.00000 15 TiB 2.7 TiB 2.6 TiB 8.5 MiB > >8.5 GiB 12 TiB 18.25 0.85 50 up osd.10 > > 11 hdd 14.59999 1.00000 15 TiB 3.4 TiB 3.3 TiB 10 MiB > >11 GiB 11 TiB 23.02 1.07 51 up osd.11 > > 12 hdd 14.59999 1.00000 15 TiB 2.8 TiB 2.8 TiB 10 MiB > >9.7 GiB 12 TiB 19.53 0.91 49 up osd.12 > > 13 hdd 14.59999 1.00000 15 TiB 3.7 TiB 3.7 TiB 11 MiB > >12 GiB 11 TiB 25.62 1.19 49 up osd.13 > > 14 hdd 14.59999 1.00000 15 TiB 2.6 TiB 2.6 TiB 8.2 MiB > >8.3 GiB 12 TiB 17.65 0.82 53 up osd.14 > > 15 hdd 14.59999 1.00000 15 TiB 2.5 TiB 2.5 TiB 7.6 MiB > >7.8 GiB 12 TiB 17.42 0.81 50 up osd.15 > > 16 hdd 14.59999 1.00000 15 TiB 3.5 TiB 3.5 TiB 11 MiB > >11 GiB 11 TiB 24.37 1.13 50 up osd.16 > > 17 hdd 14.59999 1.00000 15 TiB 3.5 TiB 3.5 TiB 12 MiB > >12 GiB 11 TiB 24.09 1.12 52 up osd.17 > > 18 hdd 14.59999 1.00000 15 TiB 2.4 TiB 2.4 TiB 6.9 MiB > >7.5 GiB 12 TiB 16.79 0.78 49 up osd.18 > > 19 hdd 14.59999 1.00000 15 TiB 3.3 TiB 3.3 TiB 9.9 MiB > >10 GiB 11 TiB 22.91 1.06 50 up osd.19 > > 20 hdd 14.59999 1.00000 15 TiB 3.6 TiB 3.6 TiB 12 MiB > >12 GiB 11 TiB 25.02 1.16 49 up osd.20 > > 21 hdd 14.59999 1.00000 15 TiB 3.4 TiB 3.4 TiB 14 MiB > >12 GiB 11 TiB 23.45 1.09 51 up osd.21 > > 22 hdd 14.59999 1.00000 15 TiB 3.3 TiB 3.3 TiB 12 MiB > >11 GiB 11 TiB 22.64 1.05 51 up osd.22 > > 23 hdd 14.59999 1.00000 15 TiB 2.9 TiB 2.8 TiB 9.2 MiB > >9.3 GiB 12 TiB 19.59 0.91 51 up osd.23 > > 24 hdd 14.59999 1.00000 15 TiB 3.4 TiB 3.3 TiB 12 MiB > >11 GiB 11 TiB 23.04 1.07 50 up osd.24 > > 25 hdd 14.59999 1.00000 15 TiB 3.1 TiB 3.1 TiB 10 MiB > >9.9 GiB 11 TiB 21.61 1.00 50 up osd.25 > >162 hdd 14.59999 1.00000 15 TiB 3.2 TiB 3.2 TiB 10 MiB > >10 GiB 11 TiB 21.76 1.01 50 up osd.162 > >163 hdd 14.59999 1.00000 15 TiB 3.4 TiB 3.4 TiB 11 MiB > >11 GiB 11 TiB 23.60 1.09 50 up osd.163 > >164 hdd 14.59999 1.00000 15 TiB 3.5 TiB 3.5 TiB 12 MiB > >11 GiB 11 TiB 24.38 1.13 51 up osd.164 > >165 hdd 14.59999 1.00000 15 TiB 2.9 TiB 2.9 TiB 9.1 MiB > >9.5 GiB 12 TiB 20.18 0.94 50 up osd.165 > >166 hdd 14.59999 1.00000 15 TiB 3.3 TiB 3.3 TiB 11 MiB > >11 GiB 11 TiB 22.62 1.05 50 up osd.166 > >167 hdd 14.59999 1.00000 15 TiB 3.5 TiB 3.5 TiB 12 MiB > >12 GiB 11 TiB 24.36 1.13 52 up osd.167 > > > >most of OSD settings are defaults, cache autotune, memory_target 4GB etc. > > > >there is absolutely no activity on this (or any related) pool, > >just on one replicated, on different > >drives, there are about 30MB/s writes. al lboxes are almost idle, > >have enough RAM. unfortunately > >OSDs do not use any fast storage for WAL or any DB. > > > >anyone met similar problem? Or somebody has hint on how to debug > >what are OSDs reading all the time? > > > >I'd be very grateful > > > >with best regards > > > >nikola ciprich > > > > > >-- > >------------------------------------- > >Ing. Nikola CIPRICH > >LinuxBox.cz, s.r.o. > >28.rijna 168, 709 00 Ostrava > > > >tel.: +420 591 166 214 > >fax: +420 596 621 273 > >mobil: +420 777 093 799 > >www.linuxbox.cz > > > >mobil servis: +420 737 238 656 > >email servis: servis@xxxxxxxxxxx > >------------------------------------- > >_______________________________________________ > >ceph-users mailing list -- ceph-users@xxxxxxx > >To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis@xxxxxxxxxxx ------------------------------------- _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |