Ok, I did the compact on 1 osd.
The utilization is back to normal, so that's good... Thumbs up to
you guys!
Though, one thing I want to get out of the way before adapting
the other OSDs:
When I now get the RocksDb stats, my L1, L2 and L3 are gone:
db_statistics {
"rocksdb_compaction_statistics": "",
"": "",
"": "** Compaction Stats [default] **",
"": "Level Files Size Score Read(GB) Rn(GB)
Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s)
Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
"":
"----------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
"": " L0 1/0 968.45 KB 0.2 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 105.1 0.01
0.00 1 0.009 0 0",
"": " L4 1557/0 98.10 GB 0.4 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.00
0.00 0 0.000 0 0",
"": " Sum 1558/0 98.10 GB 0.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 105.1 0.01
0.00 1 0.009 0 0",
"": " Int 0/0 0.00 KB 0.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 105.1 0.01
0.00 1 0.009 0 0",
"": "",
"": "** Compaction Stats [default] **",
"": "Priority Files Size Score Read(GB) Rn(GB)
Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s)
Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
"":
"-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
"": "User 0/0 0.00 KB 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 105.1 0.01
0.00 1 0.009 0 0",
"": "Uptime(secs): 0.3 total, 0.3 interval",
"": "Flush(GB): cumulative 0.001, interval 0.001",
"": "AddFile(GB): cumulative 0.000, interval 0.000",
"": "AddFile(Total Files): cumulative 0, interval 0",
"": "AddFile(L0 Files): cumulative 0, interval 0",
"": "AddFile(Keys): cumulative 0, interval 0",
"": "Cumulative compaction: 0.00 GB write, 2.84 MB/s write,
0.00 GB read, 0.00 MB/s read, 0.0 seconds",
"": "Interval compaction: 0.00 GB write, 2.84 MB/s write,
0.00 GB read, 0.00 MB/s read, 0.0 seconds",
"": "Stalls(count): 0 level0_slowdown, 0
level0_slowdown_with_compaction, 0 level0_numfiles, 0
level0_numfiles_with_compaction, 0 stop for
pending_compaction_bytes, 0 slowdown for
pending_compaction_bytes, 0 memtable_compaction, 0
memtable_slowdown, interval 0 total count",
"": "",
"": "** File Read Latency Histogram By Level [default] **",
"": "** Level 0 read latency histogram (micros):",
"": "Count: 5 Average: 69.2000 StdDev: 85.92",
"": "Min: 0 Median: 1.5000 Max: 201",
"": "Percentiles: P50: 1.50 P75: 155.00 P99: 201.00 P99.9:
201.00 P99.99: 201.00",
"": "------------------------------------------------------",
"": "[ 0, 1 ] 2 40.000% 40.000% ########",
"": "( 1, 2 ] 1 20.000% 60.000% ####",
"": "( 110, 170 ] 1 20.000% 80.000% ####",
"": "( 170, 250 ] 1 20.000% 100.000% ####",
"": "",
"": "** Level 4 read latency histogram (micros):",
"": "Count: 4664 Average: 0.6895 StdDev: 0.82",
"": "Min: 0 Median: 0.5258 Max: 27",
"": "Percentiles: P50: 0.53 P75: 0.79 P99: 2.61 P99.9: 6.45
P99.99: 13.83",
"": "------------------------------------------------------",
"": "[ 0, 1 ] 4435 95.090% 95.090%
###################",
"": "( 1, 2 ] 149 3.195% 98.285% #",
"": "( 2, 3 ] 55 1.179% 99.464% ",
"": "( 3, 4 ] 12 0.257% 99.721% ",
"": "( 4, 6 ] 8 0.172% 99.893% ",
"": "( 6, 10 ] 3 0.064% 99.957% ",
"": "( 10, 15 ] 2 0.043% 100.000% ",
"": "( 22, 34 ] 1 0.021% 100.021% ",
"": "",
"": "",
"": "** DB Stats **",
"": "Uptime(secs): 0.3 total, 0.3 interval",
"": "Cumulative writes: 0 writes, 0 keys, 0 commit groups,
0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s",
"": "Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync,
written: 0.00 GB, 0.00 MB/s",
"": "Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent",
"": "Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0
writes per commit group, ingest: 0.00 MB, 0.00 MB/s",
"": "Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync,
written: 0.00 MB, 0.00 MB/s",
"": "Interval stall: 00:00:0.000 H:M:S, 0.0 percent"
}
We use the NVMe's to store the RocksDb, but with the spillover
towards the spinning drives.
L4 is intended to be stored on the spinning drives...
Will the other levels be created automatically?
Op di 6 okt. 2020 om 13:18 schreef Stefan Kooman <stefan@xxxxxx
<mailto:stefan@xxxxxx>>:
On 2020-10-06 13:05, Igor Fedotov wrote:
>
> On 10/6/2020 1:04 PM, Kristof Coucke wrote:
>> Another strange thing is going on:
>>
>> No client software is using the system any longer, so we
would expect
>> that all IOs are related to the recovery (fixing of the
degraded PG).
>> However, the disks that are reaching high IO are not a
member of the
>> PGs that are being fixed.
>>
>> So, something is heavily using the disk, but I can't find
the process
>> immediately. I've read something that there can be old client
>> processes that keep on connecting to an OSD for retrieving
data for a
>> specific PG while that PG is no longer available on that disk.
>>
>>
> I bet it's rather PG removal happening in background....
^^ This, and probably the accompanying RocksDB housekeeping
that goes
with it. As only removing PGs shouldn't be a too big a deal
at all.
Especially with very small files (and a lot of them) you
probably have a
lot of OMAP / META data, (ceph osd df will tell you).
If that's indeed the case than there is a (way) quicker
option to get
out of this situation: offline compacting of the OSDs. This
process
happens orders of magnitude faster than when the OSDs are
still online.
To check if this hypothesis is true: are the OSD servers
under CPU
stress where the PGs were located previously (and not the new
hosts)?
Offline compaction per host:
systemctl stop ceph-osd.target
for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool
bluestore-kv
/var/lib/ceph/osd/$osd compact &);done
Gr. Stefan
