Hi Zakhar, > > I did try to play with various debug settings. The issue is that mons > produce logs of all commands issued by clients, not just mgr. For example, > an Openstack Cinder node asking for space it can use: > > Oct 9 07:59:01 ceph03 bash[4019]: debug 2023-10-09T07:59:01.303+0000 > 7f489da8f700 0 log_channel(audit) log [DBG] : from='client.? > 10.208.1.11:0/3286277243 <http://10.208.1.11:0/3286277243> ' > entity='client.cinder' cmd=[{"prefix":"osd pool get-quota", "pool": > "volumes-ssd", "format":"json"}]: dispatch I am on a older version of ceph still so I am not sure if I even have these. There is also an option ceph.conf to do client logging [client] #debug client = 5 > > It is unclear which particular mon debug option out of many controls this > particular type of debug. I tried searching for documentation of mon debug > options to no avail. > Maybe there is something equal to this for logging? ceph daemon mon.a perf schema|less ceph daemon osd.0 perf schema|less > > > Did you do something like this > > Getting keys with > ceph daemon mon.a config show | grep debug_ | grep mgr > > ceph tell mon.* injectargs --$monk=0/0 > > > > > Any input from anyone, please? > > > > This part of Ceph is very poorly documented. Perhaps there's a > better place > > to ask this question? Please let me know. > > > > /Z > > > > On Sat, 7 Oct 2023 at 22:00, Zakhar Kirpichenko <zakhar@xxxxxxxxx > <mailto:zakhar@xxxxxxxxx> > wrote: > > > > > Hi! > > > > > > I am still fighting excessive logging. I've reduced unnecessary > logging > > > from most components except for mon audit: > https://pastebin.com/jjWvUEcQ > > > > > > How can I stop logging this particular type of messages? > > > > > > I would appreciate your help and advice. > > > > > > /Z > > > > > > > > >> Thank you for your response, Igor. > > >> > > >> Currently debug_rocksdb is set to 4/5: > > >> > > >> # ceph config get osd debug_rocksdb > > >> 4/5 > > >> > > >> This setting seems to be default. Is my understanding correct > that > > you're > > >> suggesting setting it to 3/5 or even 0/5? Would setting it to > 0/5 have > > any > > >> negative effects on the cluster? > > >> > > >> /Z > > >> > > >> On Wed, 4 Oct 2023 at 21:23, Igor Fedotov <igor.fedotov@xxxxxxxx > <mailto:igor.fedotov@xxxxxxxx> > wrote: > > >> > > >>> Hi Zakhar, > > >>> > > >>> do reduce rocksdb logging verbosity you might want to set > debug_rocksdb > > >>> to 3 (or 0). > > >>> > > >>> I presume it produces a significant part of the logging > traffic. > > >>> > > >>> > > >>> Thanks, > > >>> > > >>> Igor > > >>> > > >>> On 04/10/2023 20:51, Zakhar Kirpichenko wrote: > > >>> > Any input from anyone, please? > > >>> > > > >>> > On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko > <zakhar@xxxxxxxxx <mailto:zakhar@xxxxxxxxx> > > > >>> wrote: > > >>> > > > >>> >> Hi, > > >>> >> > > >>> >> Our Ceph 16.2.x cluster managed by cephadm is logging a lot > of very > > >>> >> detailed messages, Ceph logs alone on hosts with monitors > and > > several > > >>> OSDs > > >>> >> has already eaten through 50% of the endurance of the flash > system > > >>> drives > > >>> >> over a couple of years. > > >>> >> > > >>> >> Cluster logging settings are default, and it seems that all > daemons > > >>> are > > >>> >> writing lots and lots of debug information to the logs, such > as for > > >>> >> example: https://pastebin.com/ebZq8KZk (it's just a snippet, > but > > >>> there's > > >>> >> lots and lots of various information). > > >>> >> > > >>> >> Is there a way to reduce the amount of logging and, for > example, > > >>> limit the > > >>> >> logging to warnings or important messages so that it doesn't > include > > >>> every > > >>> >> successful authentication attempt, compaction etc, etc, when > the > > >>> cluster is > > >>> >> healthy and operating normally? > > >>> >> > > >>> >> I would very much appreciate your advice on this. > > >>> >> > > >>> >> Best regards, > > >>> >> Zakhar > > >>> >> > > >>> >> > > >>> >> > > >>> > _______________________________________________ > > >>> > ceph-users mailing list -- ceph-users@xxxxxxx <mailto:ceph- > users@xxxxxxx> > > >>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > <mailto:ceph-users-leave@xxxxxxx> > > >>> > > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx <mailto:ceph- > users@xxxxxxx> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > <mailto:ceph-users-leave@xxxxxxx> > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |