Ok, I did the compact on 1 osd.
The utilization is back to normal, so that's good... Thumbs up to you guys!
Though, one thing I want to get out of the way before adapting the other
OSDs:
When I now get the RocksDb stats, my L1, L2 and L3 are gone:
db_statistics {
"rocksdb_compaction_statistics": "",
"": "",
"": "** Compaction Stats [default] **",
"": "Level Files Size Score Read(GB) Rn(GB) Rnp1(GB)
Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
"":
"----------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
"": " L0 1/0 968.45 KB 0.2 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": " L4 1557/0 98.10 GB 0.4 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.00 0.00
0 0.000 0 0",
"": " Sum 1558/0 98.10 GB 0.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": " Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": "",
"": "** Compaction Stats [default] **",
"": "Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB)
Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec)
CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
"":
"-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
"": "User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 105.1 0.01 0.00
1 0.009 0 0",
"": "Uptime(secs): 0.3 total, 0.3 interval",
"": "Flush(GB): cumulative 0.001, interval 0.001",
"": "AddFile(GB): cumulative 0.000, interval 0.000",
"": "AddFile(Total Files): cumulative 0, interval 0",
"": "AddFile(L0 Files): cumulative 0, interval 0",
"": "AddFile(Keys): cumulative 0, interval 0",
"": "Cumulative compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB
read, 0.00 MB/s read, 0.0 seconds",
"": "Interval compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB read,
0.00 MB/s read, 0.0 seconds",
"": "Stalls(count): 0 level0_slowdown, 0
level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0
slowdown for pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count",
"": "",
"": "** File Read Latency Histogram By Level [default] **",
"": "** Level 0 read latency histogram (micros):",
"": "Count: 5 Average: 69.2000 StdDev: 85.92",
"": "Min: 0 Median: 1.5000 Max: 201",
"": "Percentiles: P50: 1.50 P75: 155.00 P99: 201.00 P99.9: 201.00
P99.99: 201.00",
"": "------------------------------------------------------",
"": "[ 0, 1 ] 2 40.000% 40.000% ########",
"": "( 1, 2 ] 1 20.000% 60.000% ####",
"": "( 110, 170 ] 1 20.000% 80.000% ####",
"": "( 170, 250 ] 1 20.000% 100.000% ####",
"": "",
"": "** Level 4 read latency histogram (micros):",
"": "Count: 4664 Average: 0.6895 StdDev: 0.82",
"": "Min: 0 Median: 0.5258 Max: 27",
"": "Percentiles: P50: 0.53 P75: 0.79 P99: 2.61 P99.9: 6.45 P99.99:
13.83",
"": "------------------------------------------------------",
"": "[ 0, 1 ] 4435 95.090% 95.090%
###################",
"": "( 1, 2 ] 149 3.195% 98.285% #",
"": "( 2, 3 ] 55 1.179% 99.464% ",
"": "( 3, 4 ] 12 0.257% 99.721% ",
"": "( 4, 6 ] 8 0.172% 99.893% ",
"": "( 6, 10 ] 3 0.064% 99.957% ",
"": "( 10, 15 ] 2 0.043% 100.000% ",
"": "( 22, 34 ] 1 0.021% 100.021% ",
"": "",
"": "",
"": "** DB Stats **",
"": "Uptime(secs): 0.3 total, 0.3 interval",
"": "Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes
per commit group, ingest: 0.00 GB, 0.00 MB/s",
"": "Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written:
0.00 GB, 0.00 MB/s",
"": "Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent",
"": "Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per
commit group, ingest: 0.00 MB, 0.00 MB/s",
"": "Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written:
0.00 MB, 0.00 MB/s",
"": "Interval stall: 00:00:0.000 H:M:S, 0.0 percent"
}
We use the NVMe's to store the RocksDb, but with the spillover towards the
spinning drives.
L4 is intended to be stored on the spinning drives...
Will the other levels be created automatically?
Op di 6 okt. 2020 om 13:18 schreef Stefan Kooman <stefan@xxxxxx>:
> On 2020-10-06 13:05, Igor Fedotov wrote:
> >
> > On 10/6/2020 1:04 PM, Kristof Coucke wrote:
> >> Another strange thing is going on:
> >>
> >> No client software is using the system any longer, so we would expect
> >> that all IOs are related to the recovery (fixing of the degraded PG).
> >> However, the disks that are reaching high IO are not a member of the
> >> PGs that are being fixed.
> >>
> >> So, something is heavily using the disk, but I can't find the process
> >> immediately. I've read something that there can be old client
> >> processes that keep on connecting to an OSD for retrieving data for a
> >> specific PG while that PG is no longer available on that disk.
> >>
> >>
> > I bet it's rather PG removal happening in background....
>
> ^^ This, and probably the accompanying RocksDB housekeeping that goes
> with it. As only removing PGs shouldn't be a too big a deal at all.
> Especially with very small files (and a lot of them) you probably have a
> lot of OMAP / META data, (ceph osd df will tell you).
>
> If that's indeed the case than there is a (way) quicker option to get
> out of this situation: offline compacting of the OSDs. This process
> happens orders of magnitude faster than when the OSDs are still online.
>
> To check if this hypothesis is true: are the OSD servers under CPU
> stress where the PGs were located previously (and not the new hosts)?
>
> Offline compaction per host:
>
> systemctl stop ceph-osd.target
>
> for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv
> /var/lib/ceph/osd/$osd compact &);done
>
> Gr. Stefan
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
