Re: Tons of "cls_rgw.cc:3284: gc_iterate_entries end_key=" records in OSD logs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Still don't know why only particular OSDs write this information to log files.

Jakub

On Wed, Aug 8, 2018 at 12:02 PM Jakub Jaszewski <jaszewski.jakub@xxxxxxxxx> wrote:
Hi All, exactly the same story today, same 8 OSDs and a lot of garbage collection objects to process

Below is the number of "cls_rgw.cc:3284: gc_iterate_entries end_key=" entries per OSD log file
hostA:
  /var/log/ceph/ceph-osd.58.log
  1826467
hostB:
  /var/log/ceph/ceph-osd.88.log
  2924241
hostC:
  /var/log/ceph/ceph-osd.153.log
  581002
  /var/log/ceph/ceph-osd.164.log
  3278606
hostD:
  /var/log/ceph/ceph-osd.95.log
  1426963
hostE:
  /var/log/ceph/ceph-osd.4.log
  2716914
  /var/log/ceph/ceph-osd.53.log
  943749
hostF:
  /var/log/ceph/ceph-osd.172.log
  4085334


# radosgw-admin gc list --include-all|grep oid |wc -l
302357
#

Can anyone please explain what is going on ?

Thanks!
Jakub

On Tue, Aug 7, 2018 at 3:03 PM Jakub Jaszewski <jaszewski.jakub@xxxxxxxxx> wrote:
Hi,

8 out of 192 OSDs in our cluster (version 12.2.5) write plenty of records like "cls_rgw.cc:3284: gc_iterate_entries end_key=" to the corresponding log files, e.g.

2018-08-07 04:34:06.000585 7fdd8f012700  0 <cls> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries end_key=1_01533616446.000580407
2018-08-07 04:34:06.001888 7fdd8f012700  0 <cls> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries end_key=1_01533616446.001886318
2018-08-07 04:34:06.003395 7fdd8f012700  0 <cls> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries end_key=1_01533616446.003390299
2018-08-07 04:34:06.005205 7fdd8f012700  0 <cls> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries end_key=1_01533616446.005200341

# grep '2018-08-07 04:34:06' /var/log/ceph/ceph-osd.4.log |wc -l
712
#

At the same time there were like 500 000 expired garbage collection objects.

Log level of OSD subsystem is set to default 1/5 across all OSDs.

I wonder why only few OSDs record this information and is it something to be logged in log level = 1 or maybe higher? https://github.com/ceph/ceph/blob/v12.2.5/src/cls/rgw/cls_rgw.cc#L3284

Thanks
Jakub
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux