It says: FAILED assert(0 == "bluefs enospc") Could it be that the OSD disks you use are very very small? Den mån 17 aug. 2020 kl 20:26 skrev Khodayar Doustar <khodayard@xxxxxxxxx>: > Hi, > > I have a 3 node cluster of mimic with 9 osds (3 osds on each node). > I use this cluster to test integration of an application with S3 api. > > The problem is that after a few days all OSD starts filling up with > bluestore logs and goes down and out one by one! > I cannot stop the logs and I cannot find the setting to fix this leakage, > this should be a leakage in logs because it's not logical to fill up all > OSD with bluefs logs. > > This is an example of logs which is being repeated in bluestore logs: > > [root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path > /var/lib/ceph/osd/ceph-5 > . > . > > [root@server1 ~]# ceph osd df tree > ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE > VAR PGS TYPE NAME > -1 0.16727 - 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 - root default > -3 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 - host server1 > 0 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.0 > 1 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.1 > 2 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.2 > -5 0.05576 - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 0 > 0 - host server2 > 3 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.3 > 4 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.4 > 5 hdd 0.01859 1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 7.11 > 1.04 99 osd.5 > -7 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 - host server3 > 6 hdd 0.01859 1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB 6.55 > 0.96 78 osd.6 > 7 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.7 > 8 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.8 > TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB 2.0 GiB 35 GiB 6.83 > > MIN/MAX VAR: 0/1.04 STDDEV: 5.58 > [root@server1 ~]# > > > I'm kind of newbie to ceph, so any help or hint would be appreciated. > Did I hit a bug or something is wrong with my configuration? > Make the disks larger, those sizes are far too small for any usable cluster, so I don't think that use case gets tested at all. The database preallocations, WAL and things OSDs create in order to be good for 100G -> 12-14-18TB drives makes them less useful for 0.018TB drives. I don't think the logs are the real problem, the OSD processes are crashing because you give them no room and then they log repeatedly that they can't restart because they are still out of space. -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |