Re: Upgrade to 16.2.6 and osd+mds crash after bluestore_fsck_quick_fix_on_mount true

[mgrzybowski <marek.grzybowski+ceph-users@xxxxxxxxx>]

Hi Igor

In ceph.conf i added:

[osd]
debug bluestore = 20


next: systemctl start ceph-osd@2


Log is large :

# ls -alh  /var/log/ceph/ceph-osd.2.log
-rw-r--r-- 1 ceph ceph 1,5G paź 22 21:14 /var/log/ceph/ceph-osd.2.log

Entire file: https://drive.google.com/file/d/1PqTQCVuFlgAYirtAYGcu3YN1j0BCuriE/view?usp=sharing

~# tail -n 700   /var/log/ceph/ceph-osd.2.log
  -578> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401af2d:::1000425e348.00000014:head# end GHMAX
  -577> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401b880:::1000425bf50.00000000:head# end GHMAX
  -576> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401bc06:::100005acc48.00000037:head# end GHMAX
  -575> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401c2e1:::10004f5879e.00000005:head# end GHMAX
  -574> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401c2f2:::1000022be2d.0000001b:head# end GHMAX
  -573> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401c84d:::1000352e0f4.0000000e:head# end GHMAX
  -572> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401cb11:::10000231f34.00000036:head# end GHMAX
  -571> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401d016:::100005d5971.00000029:head# end GHMAX
  -570> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401d855:::100042bed1e.00000000:head# end GHMAX
  -569> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401dbcc:::10004263aa2.00000000:head# end GHMAX
  -568> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401de32:::10003ff6114.0000003a:head# end GHMAX
  -567> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401deef:::10004681051.00000064:head# end GHMAX
  -566> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401e467:::10001907a42.00000036:head# end GHMAX
  -565> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401e8e4:::100046815bb.0000001d:head# end GHMAX
  -564> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6401f618:::1000001543b.00000224:head# end GHMAX
  -563> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6402218b:::100000fdd77.00000007:head# end GHMAX
  -562> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:64022348:::1000022da8b.0000000a:head# end GHMAX
  -561> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:64022520:::100000feae3.0000000d:head# end GHMAX
  -560> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:640239bf:::10005803551.0000000e:head# end GHMAX
  -559> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:64024882:::1000022d905.0000004c:head# end GHMAX
  -558> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:64024a5d:::100024e90a5.00000026:head# end GHMAX
  -557> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:6402533a:::100025b195c.0000001b:head# end GHMAX
  -556> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:64025c50:::1000111c11b.00000018:head# end GHMAX
  -555> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:640260e9:::1000426445a.0000022c:head# end GHMAX
  -554> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:64026115:::1000426686b.00000011:head# end GHMAX
  -553> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:64027c28:::1000001543d.000001ab:head# end GHMAX
  -552> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list reached max 64
  -551> 2021-10-22T21:14:52.418+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) collection_list 24.26s0_head start 0#24:64027c28:::1000001543d.000001ab:head# end GHMAX max 64 = 0, ls.size() = 64, next = 0#24:64027c28:::1000001543d.000001ab:head#
  -550> 2021-10-22T21:14:52.418+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) collection_list 24.as0_head start GHMIN end GHMAX max 64
  -549> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list range 0#-26:50000000::::0#0 to 0#-26:54000000::::0#0 and 0#24:50000000::::0#0 to 0#24:54000000::::0#0 start GHMIN
  -548> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list pend 0#-26:54000000::::0#0
  -547> 2021-10-22T21:14:52.418+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#22:a0000000::::head# >= 0#-26:54000000::::0#0
  -546> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50000000::::head# end GHMAX
  -545> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000002e:::1000022fa47.00000023:head# end GHMAX
  -544> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50000725:::10004279a70.00000067:head# end GHMAX
  -543> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50000828:::100042b7d79.00008b1f:head# end GHMAX
  -542> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50000b57:::10003ff5bbf.0000002c:head# end GHMAX
  -541> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50001c58:::100000fe001.00000062:head# end GHMAX
  -540> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50001c85:::10000015416.00000037:head# end GHMAX
  -539> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:500030bf:::100000154c3.00000431:head# end GHMAX
  -538> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000516b:::100007f4dde.00000073:head# end GHMAX
  -537> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50005904:::100000154c2.00000253:head# end GHMAX
  -536> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50005b85:::1000001529e.000002df:head# end GHMAX
  -535> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:500065af:::10000015490.0000039f:head# end GHMAX
  -534> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:500069ab:::10004258f26.00000002:head# end GHMAX
  -533> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:500073f9:::1000016dd1b.00000013:head# end GHMAX
  -532> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:500076f8:::10000015405.000002c1:head# end GHMAX
  -531> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50008c49:::100002301d0.00000021:head# end GHMAX
  -530> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000903f:::100000152ab.000000df:head# end GHMAX
  -529> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:500093b2:::10002e6d31e.00000030:head# end GHMAX
  -528> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50009d24:::10004ee2d66.00000021:head# end GHMAX
  -527> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000a35a:::10003c73247.00000016:head# end GHMAX
  -526> 2021-10-22T21:14:52.438+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000a83c:::10000015506.0000042e:head# end GHMAX
  -525> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000c303:::100049a5025.0000003a:head# end GHMAX
  -524> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000cd88:::10001530bbe.00000048:head# end GHMAX
  -523> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000d0e6:::100042c3ce6.00000000:head# end GHMAX
  -522> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000daa0:::10000015427.0000001e:head# end GHMAX
  -521> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000daa0:::10001a605a8.00000035:head# end GHMAX
  -520> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000db3c:::10002f1fc8e.00000004:head# end GHMAX
  -519> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000e1e8:::100001143bf.00000181:head# end GHMAX
  -518> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000e328:::10002bc2cb7.0000001a:head# end GHMAX
  -517> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5000ee71:::1000022ff03.00000039:head# end GHMAX
  -516> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50012333:::100019912ce.0000002e:head# end GHMAX
  -515> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50013cf7:::10000375b8d.00000025:head# end GHMAX
  -514> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50014b8f:::1000069f0f4.0000002a:head# end GHMAX
  -513> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:500159c5:::100000f8eb0.00000044:head# end GHMAX
  -512> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50016ad9:::1000172e3f2.00000020:head# end GHMAX
  -511> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50016f85:::100000fec3a.00000016:head# end GHMAX
  -510> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:500178cd:::1000425e34a.00000021:head# end GHMAX
  -509> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50017b25:::1000043176c.00000034:head# end GHMAX
  -508> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50017b32:::100000f96e8.00000002:head# end GHMAX
  -507> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5001863e:::1000022bd5f.00000006:head# end GHMAX
  -506> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5001890b:::10000bd2eb5.00000007:head# end GHMAX
  -505> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:500195eb:::100011d38fd.00000004:head# end GHMAX
  -504> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5001994b:::100000f8458.00000020:head# end GHMAX
  -503> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5001c006:::100000ee73b.00000000:head# end GHMAX
  -502> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5001e841:::10001e55dba.00000013:head# end GHMAX
  -501> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50020f90:::1000001542b.0000040d:head# end GHMAX
  -500> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:500214d2:::100035cefca.0000001f:head# end GHMAX
  -499> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50022899:::100000f8d6f.00000000:head# end GHMAX
  -498> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50022aa4:::100000f87d9.00000001:head# end GHMAX
  -497> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50022d00:::1000022c8f2.00000018:head# end GHMAX
  -496> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50022d36:::100026cf859.0000002a:head# end GHMAX
  -495> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50023eb5:::10000015466.00000123:head# end GHMAX
  -494> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50023f61:::1000307394a.00000009:head# end GHMAX
  -493> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50024809:::10000bc89a9.00000006:head# end GHMAX
  -492> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5002720a:::100047e53b8.00000031:head# end GHMAX
  -491> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50027681:::100000154ab.0000015c:head# end GHMAX
  -490> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:500278bd:::10000015427.000000a9:head# end GHMAX
  -489> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:50027f63:::10000a3f514.0000002c:head# end GHMAX
  -488> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5002810b:::100001913b7.00000015:head# end GHMAX
  -487> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5002828c:::10000015465.000002e7:head# end GHMAX
  -486> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:500285e9:::100002301d9.0000003b:head# end GHMAX
  -485> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5002a575:::10001c33f37.00000000:head# end GHMAX
  -484> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5002a871:::100042b7d79.00001e1b:head# end GHMAX
  -483> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5002b5d4:::100000154fc.00000075:head# end GHMAX
  -482> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list oid 0#24:5002bbf3:::100005acc50.00000004:head# end GHMAX
  -481> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _collection_list reached max 64
  -480> 2021-10-22T21:14:52.442+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) collection_list 24.as0_head start 0#24:5002bbf3:::100005acc50.00000004:head# end GHMAX max 64 = 0, ls.size() = 64, next = 0#24:5002bbf3:::100005acc50.00000004:head#
  -479> 2021-10-22T21:14:52.442+0200 7fb023c77f00  0 osd.2 23169 load_pgs
  -478> 2021-10-22T21:14:52.442+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) read meta #-1:46e858f0:::pg_num_history:head# 0x0~0
  -477> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(meta 0x55f8c87d2960) get_onode oid #-1:46e858f0:::pg_num_history:head# key 0x7F7FFFFFFFFFFFFFFF46E858F0'!pg_num_history!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F
  -476> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(meta 0x55f8c87d2960)  r 0 v.len 110
  -475> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore.blob(0x55f8cddb0a10) get_ref 0x0~2e Blob(0x55f8cddb0a10 blob([0x4090000~10000] csum+has_unused crc32c/0x1000 unused=0xfffe) use_tracker(0x0 0x0) SharedBlob(0x55f8cddb0a80 sbid 0x0))
  -474> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore.blob(0x55f8cddb0a10) get_ref init 0x10000, 10000
  -473> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore.OnodeSpace(0x55f8c87d2aa0 in 0x55f8c802c000) add #-1:46e858f0:::pg_num_history:head# 0x55f8d0387b80
  -472> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 _add 0x55f8c802c000 #-1:46e858f0:::pg_num_history:head# added, num=5
  -471> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read 0x0~2e size 0x2e (46)
  -470> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read defaulting to buffered read
  -469> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _read_cache  blob Blob(0x55f8cddb0a10 blob([0x4090000~10000] csum+has_unused crc32c/0x1000 unused=0xfffe) use_tracker(0x10000 0x2e) SharedBlob(0x55f8cddb0a80 sbid 0x0)) need 0x0~2e cache has 0x[]
  -468> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_read_ioc  blob Blob(0x55f8cddb0a10 blob([0x4090000~10000] csum+has_unused crc32c/0x1000 unused=0xfffe) use_tracker(0x10000 0x2e) SharedBlob(0x55f8cddb0a80 sbid 0x0)) need {<0x0, 0x1000> : [0x0:0~2e]}
  -467> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_read_ioc    region 0x0: 0x0 reading 0x0~1000
  -466> 2021-10-22T21:14:52.442+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read waiting for aio
  -465> 2021-10-22T21:14:52.446+0200 7fb017135700 20 bluestore.MempoolThread(0x55f8c7380b90) _resize_shards cache_size: 733711452 kv_alloc: 285212672 kv_used: 56349448 kv_onode_alloc: 42949672 kv_onode_used: -22 meta_alloc: 281018368 meta_used: 22752 data_alloc: 96468992 data_used: 20480
  -464> 2021-10-22T21:14:52.454+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _generate_read_result_bl  blob Blob(0x55f8cddb0a10 blob([0x4090000~10000] csum+has_unused crc32c/0x1000 unused=0xfffe) use_tracker(0x10000 0x2e) SharedBlob(0x55f8cddb0a80 sbid 0x0)) need 0x{<0x0, 0x1000> : [0x0:0~2e]}
  -463> 2021-10-22T21:14:52.454+0200 7fb023c77f00 20 bluestore.BufferSpace(0x55f8cddb0a98 in 0x55f8c7320f70) _discard 0x0~1000
  -462> 2021-10-22T21:14:52.454+0200 7fb023c77f00 20 _add level 1 near 0 on buffer(0x55f8cf1bc900 space 0x55f8cddb0a98 0x0~1000 clean) which has cache_private 0
  -461> 2021-10-22T21:14:52.454+0200 7fb023c77f00 20 _unpin0x55f8c802c000   #-1:46e858f0:::pg_num_history:head# unpinned
  -460> 2021-10-22T21:14:52.454+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) read meta #-1:46e858f0:::pg_num_history:head# 0x0~2e = 46
  -459> 2021-10-22T21:14:52.454+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.b_head oid #25:d0000000::::head#
  -458> 2021-10-22T21:14:52.454+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(25.b_head 0x55f8c87d2780) get_onode oid #25:d0000000::::head# key 0x7F8000000000000019D0000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F
  -457> 2021-10-22T21:14:52.454+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(25.b_head 0x55f8c87d2780)  r 0 v.len 31
  -456> 2021-10-22T21:14:52.454+0200 7fb023c77f00 20 bluestore.OnodeSpace(0x55f8c87d28c0 in 0x55f8c813f000) add #25:d0000000::::head# 0x55f8d7058000
  -455> 2021-10-22T21:14:52.454+0200 7fb023c77f00 20 _add 0x55f8c813f000 #25:d0000000::::head# added, num=1
  -454> 2021-10-22T21:14:52.454+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058000).flush flush done
  -453> 2021-10-22T21:14:52.494+0200 7fb017135700 20 bluestore.MempoolThread(0x55f8c7380b90) _resize_shards cache_size: 733711452 kv_alloc: 285212672 kv_used: 56349448 kv_onode_alloc: 42949672 kv_onode_used: -22 meta_alloc: 281018368 meta_used: 23818 data_alloc: 96468992 data_used: 24576
  -452> 2021-10-22T21:14:52.506+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.b_head oid #25:d0000000::::head# = 0
  -451> 2021-10-22T21:14:52.506+0200 7fb023c77f00 20 _unpin0x55f8c813f000   #25:d0000000::::head# unpinned
  -450> 2021-10-22T21:14:52.506+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.b_head oid #25:d0000000::::head#
  -449> 2021-10-22T21:14:52.506+0200 7fb023c77f00 20 _pin0x55f8c813f000   #25:d0000000::::head# pinned
  -448> 2021-10-22T21:14:52.506+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058000).flush flush done
  -447> 2021-10-22T21:14:52.538+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.b_head oid #25:d0000000::::head# = 0
  -446> 2021-10-22T21:14:52.538+0200 7fb023c77f00 20 _unpin0x55f8c813f000   #25:d0000000::::head# unpinned
  -445> 2021-10-22T21:14:52.538+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) read meta #-1:9bbddd72:::osdmap.23167:0# 0x0~0
  -444> 2021-10-22T21:14:52.538+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(meta 0x55f8c87d2960) get_onode oid #-1:9bbddd72:::osdmap.23167:0# key 0x7F7FFFFFFFFFFFFFFF9BBDDD'r!osdmap.23167!='0x0000000000000000FFFFFFFFFFFFFFFF6F
  -443> 2021-10-22T21:14:52.546+0200 7fb017135700 20 bluestore.MempoolThread(0x55f8c7380b90) _resize_shards cache_size: 733711452 kv_alloc: 285212672 kv_used: 61396560 kv_onode_alloc: 42949672 kv_onode_used: -22 meta_alloc: 281018368 meta_used: 23818 data_alloc: 96468992 data_used: 24576
  -442> 2021-10-22T21:14:52.558+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(meta 0x55f8c87d2960)  r 0 v.len 112
  -441> 2021-10-22T21:14:52.562+0200 7fb023c77f00 20 bluestore.blob(0x55f8cddb0bd0) get_ref 0x0~3fd7 Blob(0x55f8cddb0bd0 blob([0x545c0000~10000] csum+has_unused crc32c/0x1000 unused=0xfff0) use_tracker(0x0 0x0) SharedBlob(0x55f8cddb0c40 sbid 0x0))
  -440> 2021-10-22T21:14:52.562+0200 7fb023c77f00 20 bluestore.blob(0x55f8cddb0bd0) get_ref init 0x10000, 10000
  -439> 2021-10-22T21:14:52.562+0200 7fb023c77f00 20 bluestore.OnodeSpace(0x55f8c87d2aa0 in 0x55f8c802c000) add #-1:9bbddd72:::osdmap.23167:0# 0x55f8d7058280
  -438> 2021-10-22T21:14:52.562+0200 7fb023c77f00 20 _add 0x55f8c802c000 #-1:9bbddd72:::osdmap.23167:0# added, num=6
  -437> 2021-10-22T21:14:52.562+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read 0x0~3fd7 size 0x3fd7 (16343)
  -436> 2021-10-22T21:14:52.562+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read will do buffered read
  -435> 2021-10-22T21:14:52.562+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _read_cache  blob Blob(0x55f8cddb0bd0 blob([0x545c0000~10000] csum+has_unused crc32c/0x1000 unused=0xfff0) use_tracker(0x10000 0x3fd7) SharedBlob(0x55f8cddb0c40 sbid 0x0)) need 0x0~3fd7 cache has 0x[]
  -434> 2021-10-22T21:14:52.562+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_read_ioc  blob Blob(0x55f8cddb0bd0 blob([0x545c0000~10000] csum+has_unused crc32c/0x1000 unused=0xfff0) use_tracker(0x10000 0x3fd7) SharedBlob(0x55f8cddb0c40 sbid 0x0)) need {<0x0, 0x4000> : [0x0:0~3fd7]}
  -433> 2021-10-22T21:14:52.562+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_read_ioc    region 0x0: 0x0 reading 0x0~4000
  -432> 2021-10-22T21:14:52.562+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read waiting for aio
  -431> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _generate_read_result_bl  blob Blob(0x55f8cddb0bd0 blob([0x545c0000~10000] csum+has_unused crc32c/0x1000 unused=0xfff0) use_tracker(0x10000 0x3fd7) SharedBlob(0x55f8cddb0c40 sbid 0x0)) need 0x{<0x0, 0x4000> : [0x0:0~3fd7]}
  -430> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.BufferSpace(0x55f8cddb0c58 in 0x55f8c7320f70) _discard 0x0~4000
  -429> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _add level 1 near 0 on buffer(0x55f8d99f9d40 space 0x55f8cddb0c58 0x0~4000 clean) which has cache_private 0
  -428> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _unpin0x55f8c802c000   #-1:9bbddd72:::osdmap.23167:0# unpinned
  -427> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) read meta #-1:9bbddd72:::osdmap.23167:0# 0x0~3fd7 = 16343
  -426> 2021-10-22T21:14:52.578+0200 7fb023c77f00  5 osd.2 pg_epoch: 23167 pg[25.b(unlocked)] enter Initial
  -425> 2021-10-22T21:14:52.578+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.b_head oid #25:d0000000::::head#
  -424> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _pin0x55f8c813f000   #25:d0000000::::head# pinned
  -423> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058000).flush flush done
  -422> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.b_head oid #25:d0000000::::head# = 0
  -421> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _unpin0x55f8c813f000   #25:d0000000::::head# unpinned
  -420> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) stat 25.b_head #25:d0000000::::head#
  -419> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _pin0x55f8c813f000   #25:d0000000::::head# pinned
  -418> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _unpin0x55f8c813f000   #25:d0000000::::head# unpinned
  -417> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) get_omap_iterator 25.b_head #25:d0000000::::head#
  -416> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _pin0x55f8c813f000   #25:d0000000::::head# pinned
  -415> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058000).flush flush done
  -414> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) get_omap_iterator has_omap = 1
  -413> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1180) valid is at 0x0000000004F830E1'._biginfo'
  -412> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1180) valid is at 0x0000000004F830E1'._epoch'
  -411> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1180) valid is at 0x0000000004F830E1'._info'
  -410> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1180) valid is at 0x0000000004F830E1'._infover'
  -409> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1180) valid is at 0x0000000004F830E1'.may_include_deletes_in_missing'
  -408> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1180) valid is at 0x0000000004F830E17E
  -407> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _unpin0x55f8c813f000   #25:d0000000::::head# unpinned
  -406> 2021-10-22T21:14:52.578+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) set_collection_opts 25.b_head options  pg_num_min 1
  -405> 2021-10-22T21:14:52.578+0200 7fb023c77f00  5 osd.2 pg_epoch: 23167 pg[25.b( empty local-lis/les=12563/12564 n=0 ec=10991/10977 lis/c=12563/12563 les/c/f=12564/12564/0 sis=12563) [2,12,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Initial 0.000387 0 0.000000
  -404> 2021-10-22T21:14:52.578+0200 7fb023c77f00  5 osd.2 pg_epoch: 23167 pg[25.b( empty local-lis/les=12563/12564 n=0 ec=10991/10977 lis/c=12563/12563 les/c/f=12564/12564/0 sis=12563) [2,12,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Reset
  -403> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) queue_transactions ch 0x55f8c87d2780 25.b_head
  -402> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_create osr 0x55f8c87d4580 = 0x55f8c88dbc00 seq 1
  -401> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _pin0x55f8c813f000   #25:d0000000::::head# pinned
  -400> 2021-10-22T21:14:52.578+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) _touch 25.b_head #25:d0000000::::head#
  -399> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _touch 25.b_head #25:d0000000::::head# = 0
  -398> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_calc_cost 0x55f8c88dbc00 cost 670195 (1 ios * 670000 + 195 bytes)
  -397> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_write_nodes txc 0x55f8c88dbc00 onodes 0x55f8d7058000 shared_blobs
  -396> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.extentmap(0x55f8d7058150) update #25:d0000000::::head#
  -395> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _record_onode onode #25:d0000000::::head# is 31 (23 bytes onode + 2 bytes spanning blobs + 6 bytes inline extents)
  -394> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finalize_kv txc 0x55f8c88dbc00 allocated 0x[] released 0x[]
  -393> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c88dbc00 prepare
  -392> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish_io 0x55f8c88dbc00
  -391> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c88dbc00 io_done
  -390> 2021-10-22T21:14:52.578+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.a_head oid #25:50000000::::head#
  -389> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(25.a_head 0x55f8c87d25a0) get_onode oid #25:50000000::::head# key 0x7F800000000000001950000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F
  -388> 2021-10-22T21:14:52.578+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread wake
  -387> 2021-10-22T21:14:52.578+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread committing 1 submitting 1 deferred done 0 stable 0
  -386> 2021-10-22T21:14:52.578+0200 7fb01292c700 10 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread new_nid_max 83561921
  -385> 2021-10-22T21:14:52.578+0200 7fb01292c700 10 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread new_blobid_max 2294655
  -384> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(25.a_head 0x55f8c87d25a0)  r 0 v.len 31
  -383> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OnodeSpace(0x55f8c87d26e0 in 0x55f8c813c800) add #25:50000000::::head# 0x55f8d7058500
  -382> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _add 0x55f8c813c800 #25:50000000::::head# added, num=1
  -381> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058500).flush flush done
  -380> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.a_head oid #25:50000000::::head# = 0
  -379> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _unpin0x55f8c813c800   #25:50000000::::head# unpinned
  -378> 2021-10-22T21:14:52.578+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.a_head oid #25:50000000::::head#
  -377> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _pin0x55f8c813c800   #25:50000000::::head# pinned
  -376> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058500).flush flush done
  -375> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.a_head oid #25:50000000::::head# = 0
  -374> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _unpin0x55f8c813c800   #25:50000000::::head# unpinned
  -373> 2021-10-22T21:14:52.578+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_apply_kv onode 0x55f8d7058000 had 1
  -372> 2021-10-22T21:14:52.578+0200 7fb01292c700 10 HybridAllocator allocate want 0x10000 unit 0x10000 max_alloc_size 0x10000 hint 0x0
  -371> 2021-10-22T21:14:52.578+0200 7fb023c77f00  5 osd.2 pg_epoch: 23167 pg[25.a(unlocked)] enter Initial
  -370> 2021-10-22T21:14:52.578+0200 7fb01292c700 20 AvlAllocator _allocate first fit=12582912 size=65536
  -369> 2021-10-22T21:14:52.578+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.a_head oid #25:50000000::::head#
  -368> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _pin0x55f8c813c800   #25:50000000::::head# pinned
  -367> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058500).flush flush done
  -366> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.a_head oid #25:50000000::::head# = 0
  -365> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _unpin0x55f8c813c800   #25:50000000::::head# unpinned
  -364> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) stat 25.a_head #25:50000000::::head#
  -363> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _pin0x55f8c813c800   #25:50000000::::head# pinned
  -362> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _unpin0x55f8c813c800   #25:50000000::::head# unpinned
  -361> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) get_omap_iterator 25.a_head #25:50000000::::head#
  -360> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _pin0x55f8c813c800   #25:50000000::::head# pinned
  -359> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058500).flush flush done
  -358> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) get_omap_iterator has_omap = 1
  -357> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1490) valid is at 0x0000000004F830DB'._biginfo'
  -356> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1490) valid is at 0x0000000004F830DB'._epoch'
  -355> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1490) valid is at 0x0000000004F830DB'._info'
  -354> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1490) valid is at 0x0000000004F830DB'._infover'
  -353> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1490) valid is at 0x0000000004F830DB'.may_include_deletes_in_missing'
  -352> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1490) valid is at 0x0000000004F830DB7E
  -351> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _unpin0x55f8c813c800   #25:50000000::::head# unpinned
  -350> 2021-10-22T21:14:52.578+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) set_collection_opts 25.a_head options  pg_num_min 1
  -349> 2021-10-22T21:14:52.578+0200 7fb023c77f00  5 osd.2 pg_epoch: 23167 pg[25.a( empty local-lis/les=11033/11034 n=0 ec=10991/10977 lis/c=11033/11033 les/c/f=11034/11034/0 sis=11033) [8,2,4] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Initial 0.000327 0 0.000000
  -348> 2021-10-22T21:14:52.578+0200 7fb023c77f00  5 osd.2 pg_epoch: 23167 pg[25.a( empty local-lis/les=11033/11034 n=0 ec=10991/10977 lis/c=11033/11033 les/c/f=11034/11034/0 sis=11033) [8,2,4] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Reset
  -347> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) queue_transactions ch 0x55f8c87d25a0 25.a_head
  -346> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_create osr 0x55f8c87d4420 = 0x55f8c737d180 seq 1
  -345> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _pin0x55f8c813c800   #25:50000000::::head# pinned
  -344> 2021-10-22T21:14:52.578+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) _touch 25.a_head #25:50000000::::head#
  -343> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _touch 25.a_head #25:50000000::::head# = 0
  -342> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_calc_cost 0x55f8c737d180 cost 670195 (1 ios * 670000 + 195 bytes)
  -341> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_write_nodes txc 0x55f8c737d180 onodes 0x55f8d7058500 shared_blobs
  -340> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.extentmap(0x55f8d7058650) update #25:50000000::::head#
  -339> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _record_onode onode #25:50000000::::head# is 31 (23 bytes onode + 2 bytes spanning blobs + 6 bytes inline extents)
  -338> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finalize_kv txc 0x55f8c737d180 allocated 0x[] released 0x[]
  -337> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c737d180 prepare
  -336> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish_io 0x55f8c737d180
  -335> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c737d180 io_done
  -334> 2021-10-22T21:14:52.578+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.9_head oid #25:90000000::::head#
  -333> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(25.9_head 0x55f8c87d23c0) get_onode oid #25:90000000::::head# key 0x7F800000000000001990000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F
  -332> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(25.9_head 0x55f8c87d23c0)  r 0 v.len 31
  -331> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.OnodeSpace(0x55f8c87d2500 in 0x55f8c813a000) add #25:90000000::::head# 0x55f8d7058780
  -330> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _add 0x55f8c813a000 #25:90000000::::head# added, num=1
  -329> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058780).flush flush done
  -328> 2021-10-22T21:14:52.578+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.9_head oid #25:90000000::::head# = 0
  -327> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _unpin0x55f8c813a000   #25:90000000::::head# unpinned
  -326> 2021-10-22T21:14:52.578+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.9_head oid #25:90000000::::head#
  -325> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 _pin0x55f8c813a000   #25:90000000::::head# pinned
  -324> 2021-10-22T21:14:52.578+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058780).flush flush done
  -323> 2021-10-22T21:14:52.594+0200 7fb017135700 20 bluestore.MempoolThread(0x55f8c7380b90) _resize_shards cache_size: 733711452 kv_alloc: 285212672 kv_used: 61400752 kv_onode_alloc: 42949672 kv_onode_used: -22 meta_alloc: 281018368 meta_used: 26980 data_alloc: 96468992 data_used: 40960
  -322> 2021-10-22T21:14:52.598+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.9_head oid #25:90000000::::head# = 0
  -321> 2021-10-22T21:14:52.598+0200 7fb023c77f00 20 _unpin0x55f8c813a000   #25:90000000::::head# unpinned
  -320> 2021-10-22T21:14:52.598+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) read meta #-1:9325dd72:::osdmap.23155:0# 0x0~0
  -319> 2021-10-22T21:14:52.598+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(meta 0x55f8c87d2960) get_onode oid #-1:9325dd72:::osdmap.23155:0# key 0x7F7FFFFFFFFFFFFFFF9325DD'r!osdmap.23155!='0x0000000000000000FFFFFFFFFFFFFFFF6F
  -318> 2021-10-22T21:14:52.618+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(meta 0x55f8c87d2960)  r 0 v.len 112
  -317> 2021-10-22T21:14:52.618+0200 7fb023c77f00 20 bluestore.blob(0x55f8cddb17a0) get_ref 0x0~3fd7 Blob(0x55f8cddb17a0 blob([0x51ba0000~10000] csum+has_unused crc32c/0x1000 unused=0xfff0) use_tracker(0x0 0x0) SharedBlob(0x55f8cddb1810 sbid 0x0))
  -316> 2021-10-22T21:14:52.618+0200 7fb023c77f00 20 bluestore.blob(0x55f8cddb17a0) get_ref init 0x10000, 10000
  -315> 2021-10-22T21:14:52.618+0200 7fb023c77f00 20 bluestore.OnodeSpace(0x55f8c87d2aa0 in 0x55f8c802c000) add #-1:9325dd72:::osdmap.23155:0# 0x55f8d7058a00
  -314> 2021-10-22T21:14:52.618+0200 7fb023c77f00 20 _add 0x55f8c802c000 #-1:9325dd72:::osdmap.23155:0# added, num=7
  -313> 2021-10-22T21:14:52.618+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read 0x0~3fd7 size 0x3fd7 (16343)
  -312> 2021-10-22T21:14:52.618+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read will do buffered read
  -311> 2021-10-22T21:14:52.618+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _read_cache  blob Blob(0x55f8cddb17a0 blob([0x51ba0000~10000] csum+has_unused crc32c/0x1000 unused=0xfff0) use_tracker(0x10000 0x3fd7) SharedBlob(0x55f8cddb1810 sbid 0x0)) need 0x0~3fd7 cache has 0x[]
  -310> 2021-10-22T21:14:52.618+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_read_ioc  blob Blob(0x55f8cddb17a0 blob([0x51ba0000~10000] csum+has_unused crc32c/0x1000 unused=0xfff0) use_tracker(0x10000 0x3fd7) SharedBlob(0x55f8cddb1810 sbid 0x0)) need {<0x0, 0x4000> : [0x0:0~3fd7]}
  -309> 2021-10-22T21:14:52.618+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_read_ioc    region 0x0: 0x0 reading 0x0~4000
  -308> 2021-10-22T21:14:52.618+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read waiting for aio
  -307> 2021-10-22T21:14:52.622+0200 7fb01292c700 10 AvlAllocator _release offset 0x4e0000 length 0x10000
  -306> 2021-10-22T21:14:52.622+0200 7fb01292c700 10 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread nid_max now 83561921
  -305> 2021-10-22T21:14:52.622+0200 7fb01292c700 10 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread blobid_max now 2294655
  -304> 2021-10-22T21:14:52.622+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread committed 1 cleaned 0 in 0.045136102s (0.000000199s flush + 0.045135900s kv commit)
  -303> 2021-10-22T21:14:52.622+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread wake
  -302> 2021-10-22T21:14:52.622+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread kv_committed <0x55f8c88dbc00>
  -301> 2021-10-22T21:14:52.622+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread deferred_stable <>
  -300> 2021-10-22T21:14:52.622+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread committing 1 submitting 1 deferred done 0 stable 0
  -299> 2021-10-22T21:14:52.622+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c88dbc00 kv_submitted
  -298> 2021-10-22T21:14:52.622+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_committed_kv txc 0x55f8c88dbc00
  -297> 2021-10-22T21:14:52.622+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c88dbc00 finishing
  -296> 2021-10-22T21:14:52.622+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish 0x55f8c88dbc00 onodes 0x55f8d7058000
  -295> 2021-10-22T21:14:52.622+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish  txc 0x55f8c88dbc00 done
  -294> 2021-10-22T21:14:52.622+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish osr 0x55f8c87d4580 q now empty
  -293> 2021-10-22T21:14:52.622+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_release_alloc(sync) 0x55f8c88dbc00 []
  -292> 2021-10-22T21:14:52.622+0200 7fb017936700 20 _unpin0x55f8c813f000   #25:d0000000::::head# unpinned
  -291> 2021-10-22T21:14:52.622+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread sleep
  -290> 2021-10-22T21:14:52.622+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_apply_kv onode 0x55f8d7058500 had 1
  -289> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _generate_read_result_bl  blob Blob(0x55f8cddb17a0 blob([0x51ba0000~10000] csum+has_unused crc32c/0x1000 unused=0xfff0) use_tracker(0x10000 0x3fd7) SharedBlob(0x55f8cddb1810 sbid 0x0)) need 0x{<0x0, 0x4000> : [0x0:0~3fd7]}
  -288> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.BufferSpace(0x55f8cddb1828 in 0x55f8c7320f70) _discard 0x0~4000
  -287> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _add level 1 near 0 on buffer(0x55f8cda0c840 space 0x55f8cddb1828 0x0~4000 clean) which has cache_private 0
  -286> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _unpin0x55f8c802c000   #-1:9325dd72:::osdmap.23155:0# unpinned
  -285> 2021-10-22T21:14:52.642+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) read meta #-1:9325dd72:::osdmap.23155:0# 0x0~3fd7 = 16343
  -284> 2021-10-22T21:14:52.642+0200 7fb023c77f00  5 osd.2 pg_epoch: 23155 pg[25.9(unlocked)] enter Initial
  -283> 2021-10-22T21:14:52.642+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.9_head oid #25:90000000::::head#
  -282> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _pin0x55f8c813a000   #25:90000000::::head# pinned
  -281> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058780).flush flush done
  -280> 2021-10-22T21:14:52.642+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.9_head oid #25:90000000::::head# = 0
  -279> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _unpin0x55f8c813a000   #25:90000000::::head# unpinned
  -278> 2021-10-22T21:14:52.642+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) stat 25.9_head #25:90000000::::head#
  -277> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _pin0x55f8c813a000   #25:90000000::::head# pinned
  -276> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _unpin0x55f8c813a000   #25:90000000::::head# unpinned
  -275> 2021-10-22T21:14:52.642+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) get_omap_iterator 25.9_head #25:90000000::::head#
  -274> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _pin0x55f8c813a000   #25:90000000::::head# pinned
  -273> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058780).flush flush done
  -272> 2021-10-22T21:14:52.642+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) get_omap_iterator has_omap = 1
  -271> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1d50) valid is at 0x0000000004F830E6'._biginfo'
  -270> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1d50) valid is at 0x0000000004F830E6'._epoch'
  -269> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1d50) valid is at 0x0000000004F830E6'._info'
  -268> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1d50) valid is at 0x0000000004F830E6'._infover'
  -267> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1d50) valid is at 0x0000000004F830E6'.may_include_deletes_in_missing'
  -266> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8cddb1d50) valid is at 0x0000000004F830E67E
  -265> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _unpin0x55f8c813a000   #25:90000000::::head# unpinned
  -264> 2021-10-22T21:14:52.642+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) set_collection_opts 25.9_head options  pg_num_min 1
  -263> 2021-10-22T21:14:52.642+0200 7fb023c77f00  5 osd.2 pg_epoch: 23155 pg[25.9( empty local-lis/les=12563/12564 n=0 ec=10991/10977 lis/c=12563/12563 les/c/f=12564/12564/0 sis=12563) [2,9,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Initial 0.000326 0 0.000000
  -262> 2021-10-22T21:14:52.642+0200 7fb023c77f00  5 osd.2 pg_epoch: 23155 pg[25.9( empty local-lis/les=12563/12564 n=0 ec=10991/10977 lis/c=12563/12563 les/c/f=12564/12564/0 sis=12563) [2,9,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Reset
  -261> 2021-10-22T21:14:52.642+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) queue_transactions ch 0x55f8c87d23c0 25.9_head
  -260> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_create osr 0x55f8c87d42c0 = 0x55f8c737ca80 seq 1
  -259> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _pin0x55f8c813a000   #25:90000000::::head# pinned
  -258> 2021-10-22T21:14:52.642+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) _touch 25.9_head #25:90000000::::head#
  -257> 2021-10-22T21:14:52.642+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _touch 25.9_head #25:90000000::::head# = 0
  -256> 2021-10-22T21:14:52.642+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_calc_cost 0x55f8c737ca80 cost 670195 (1 ios * 670000 + 195 bytes)
  -255> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_write_nodes txc 0x55f8c737ca80 onodes 0x55f8d7058780 shared_blobs
  -254> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.extentmap(0x55f8d70588d0) update #25:90000000::::head#
  -253> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _record_onode onode #25:90000000::::head# is 31 (23 bytes onode + 2 bytes spanning blobs + 6 bytes inline extents)
  -252> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finalize_kv txc 0x55f8c737ca80 allocated 0x[] released 0x[]
  -251> 2021-10-22T21:14:52.642+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c737ca80 prepare
  -250> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish_io 0x55f8c737ca80
  -249> 2021-10-22T21:14:52.642+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c737ca80 io_done
  -248> 2021-10-22T21:14:52.642+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.18_head oid #25:18000000::::head#
  -247> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(25.18_head 0x55f8c87d2000) get_onode oid #25:18000000::::head# key 0x7F800000000000001918000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F
  -246> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(25.18_head 0x55f8c87d2000)  r 0 v.len 31
  -245> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.OnodeSpace(0x55f8c87d2140 in 0x55f8c8162000) add #25:18000000::::head# 0x55f8d7058c80
  -244> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _add 0x55f8c8162000 #25:18000000::::head# added, num=1
  -243> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058c80).flush flush done
  -242> 2021-10-22T21:14:52.642+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.18_head oid #25:18000000::::head# = 0
  -241> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _unpin0x55f8c8162000   #25:18000000::::head# unpinned
  -240> 2021-10-22T21:14:52.642+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.18_head oid #25:18000000::::head#
  -239> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _pin0x55f8c8162000   #25:18000000::::head# pinned
  -238> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058c80).flush flush done
  -237> 2021-10-22T21:14:52.642+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.18_head oid #25:18000000::::head# = 0
  -236> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 _unpin0x55f8c8162000   #25:18000000::::head# unpinned
  -235> 2021-10-22T21:14:52.642+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) read meta #-1:987ddd72:::osdmap.23163:0# 0x0~0
  -234> 2021-10-22T21:14:52.642+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(meta 0x55f8c87d2960) get_onode oid #-1:987ddd72:::osdmap.23163:0# key 0x7F7FFFFFFFFFFFFFFF987DDD'r!osdmap.23163!='0x0000000000000000FFFFFFFFFFFFFFFF6F
  -233> 2021-10-22T21:14:52.646+0200 7fb017135700 20 bluestore.MempoolThread(0x55f8c7380b90) _resize_shards cache_size: 733711452 kv_alloc: 285212672 kv_used: 61409136 kv_onode_alloc: 42949672 kv_onode_used: -22 meta_alloc: 281018368 meta_used: 29220 data_alloc: 96468992 data_used: 57344
  -232> 2021-10-22T21:14:52.650+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(meta 0x55f8c87d2960)  r 0 v.len 112
  -231> 2021-10-22T21:14:52.650+0200 7fb023c77f00 20 bluestore.blob(0x55f8d6cfe070) get_ref 0x0~3fd7 Blob(0x55f8d6cfe070 blob([0x53bc0000~10000] csum+has_unused crc32c/0x1000 unused=0xfff0) use_tracker(0x0 0x0) SharedBlob(0x55f8d6cfe0e0 sbid 0x0))
  -230> 2021-10-22T21:14:52.650+0200 7fb023c77f00 20 bluestore.blob(0x55f8d6cfe070) get_ref init 0x10000, 10000
  -229> 2021-10-22T21:14:52.650+0200 7fb023c77f00 20 bluestore.OnodeSpace(0x55f8c87d2aa0 in 0x55f8c802c000) add #-1:987ddd72:::osdmap.23163:0# 0x55f8d7058f00
  -228> 2021-10-22T21:14:52.650+0200 7fb023c77f00 20 _add 0x55f8c802c000 #-1:987ddd72:::osdmap.23163:0# added, num=8
  -227> 2021-10-22T21:14:52.650+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read 0x0~3fd7 size 0x3fd7 (16343)
  -226> 2021-10-22T21:14:52.650+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read will do buffered read
  -225> 2021-10-22T21:14:52.650+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _read_cache  blob Blob(0x55f8d6cfe070 blob([0x53bc0000~10000] csum+has_unused crc32c/0x1000 unused=0xfff0) use_tracker(0x10000 0x3fd7) SharedBlob(0x55f8d6cfe0e0 sbid 0x0)) need 0x0~3fd7 cache has 0x[]
  -224> 2021-10-22T21:14:52.650+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_read_ioc  blob Blob(0x55f8d6cfe070 blob([0x53bc0000~10000] csum+has_unused crc32c/0x1000 unused=0xfff0) use_tracker(0x10000 0x3fd7) SharedBlob(0x55f8d6cfe0e0 sbid 0x0)) need {<0x0, 0x4000> : [0x0:0~3fd7]}
  -223> 2021-10-22T21:14:52.650+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_read_ioc    region 0x0: 0x0 reading 0x0~4000
  -222> 2021-10-22T21:14:52.650+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read waiting for aio
  -221> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _generate_read_result_bl  blob Blob(0x55f8d6cfe070 blob([0x53bc0000~10000] csum+has_unused crc32c/0x1000 unused=0xfff0) use_tracker(0x10000 0x3fd7) SharedBlob(0x55f8d6cfe0e0 sbid 0x0)) need 0x{<0x0, 0x4000> : [0x0:0~3fd7]}
  -220> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.BufferSpace(0x55f8d6cfe0f8 in 0x55f8c7320f70) _discard 0x0~4000
  -219> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _add level 1 near 0 on buffer(0x55f8cbdaf740 space 0x55f8d6cfe0f8 0x0~4000 clean) which has cache_private 0
  -218> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _unpin0x55f8c802c000   #-1:987ddd72:::osdmap.23163:0# unpinned
  -217> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) read meta #-1:987ddd72:::osdmap.23163:0# 0x0~3fd7 = 16343
  -216> 2021-10-22T21:14:52.654+0200 7fb023c77f00  5 osd.2 pg_epoch: 23163 pg[25.18(unlocked)] enter Initial
  -215> 2021-10-22T21:14:52.654+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.18_head oid #25:18000000::::head#
  -214> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _pin0x55f8c8162000   #25:18000000::::head# pinned
  -213> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058c80).flush flush done
  -212> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.18_head oid #25:18000000::::head# = 0
  -211> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _unpin0x55f8c8162000   #25:18000000::::head# unpinned
  -210> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) stat 25.18_head #25:18000000::::head#
  -209> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _pin0x55f8c8162000   #25:18000000::::head# pinned
  -208> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _unpin0x55f8c8162000   #25:18000000::::head# unpinned
  -207> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) get_omap_iterator 25.18_head #25:18000000::::head#
  -206> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _pin0x55f8c8162000   #25:18000000::::head# pinned
  -205> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7058c80).flush flush done
  -204> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) get_omap_iterator has_omap = 1
  -203> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cfe620) valid is at 0x0000000004F830E4'._biginfo'
  -202> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cfe620) valid is at 0x0000000004F830E4'._epoch'
  -201> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cfe620) valid is at 0x0000000004F830E4'._info'
  -200> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cfe620) valid is at 0x0000000004F830E4'._infover'
  -199> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cfe620) valid is at 0x0000000004F830E4'.may_include_deletes_in_missing'
  -198> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cfe620) valid is at 0x0000000004F830E47E
  -197> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _unpin0x55f8c8162000   #25:18000000::::head# unpinned
  -196> 2021-10-22T21:14:52.654+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) set_collection_opts 25.18_head options  pg_num_min 1
  -195> 2021-10-22T21:14:52.654+0200 7fb023c77f00  5 osd.2 pg_epoch: 23163 pg[25.18( empty local-lis/les=12563/12564 n=0 ec=10991/10977 lis/c=12563/12563 les/c/f=12564/12564/0 sis=12563) [0,2,12] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Initial 0.000562 0 0.000000
  -194> 2021-10-22T21:14:52.654+0200 7fb023c77f00  5 osd.2 pg_epoch: 23163 pg[25.18( empty local-lis/les=12563/12564 n=0 ec=10991/10977 lis/c=12563/12563 les/c/f=12564/12564/0 sis=12563) [0,2,12] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Reset
  -193> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) queue_transactions ch 0x55f8c87d2000 25.18_head
  -192> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_create osr 0x55f8c87d4000 = 0x55f8c8034380 seq 1
  -191> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _pin0x55f8c8162000   #25:18000000::::head# pinned
  -190> 2021-10-22T21:14:52.654+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) _touch 25.18_head #25:18000000::::head#
  -189> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _touch 25.18_head #25:18000000::::head# = 0
  -188> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_calc_cost 0x55f8c8034380 cost 670195 (1 ios * 670000 + 195 bytes)
  -187> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_write_nodes txc 0x55f8c8034380 onodes 0x55f8d7058c80 shared_blobs
  -186> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.extentmap(0x55f8d7058dd0) update #25:18000000::::head#
  -185> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _record_onode onode #25:18000000::::head# is 31 (23 bytes onode + 2 bytes spanning blobs + 6 bytes inline extents)
  -184> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finalize_kv txc 0x55f8c8034380 allocated 0x[] released 0x[]
  -183> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c8034380 prepare
  -182> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish_io 0x55f8c8034380
  -181> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c8034380 io_done
  -180> 2021-10-22T21:14:52.654+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.15_head oid #25:a8000000::::head#
  -179> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(25.15_head 0x55f8d3e33e00) get_onode oid #25:a8000000::::head# key 0x7F8000000000000019A8000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F
  -178> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(25.15_head 0x55f8d3e33e00)  r 0 v.len 31
  -177> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OnodeSpace(0x55f8d3e33f40 in 0x55f8c815a000) add #25:a8000000::::head# 0x55f8d7059180
  -176> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _add 0x55f8c815a000 #25:a8000000::::head# added, num=1
  -175> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7059180).flush flush done
  -174> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.15_head oid #25:a8000000::::head# = 0
  -173> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _unpin0x55f8c815a000   #25:a8000000::::head# unpinned
  -172> 2021-10-22T21:14:52.654+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.15_head oid #25:a8000000::::head#
  -171> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _pin0x55f8c815a000   #25:a8000000::::head# pinned
  -170> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7059180).flush flush done
  -169> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.15_head oid #25:a8000000::::head# = 0
  -168> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _unpin0x55f8c815a000   #25:a8000000::::head# unpinned
  -167> 2021-10-22T21:14:52.654+0200 7fb023c77f00  5 osd.2 pg_epoch: 23163 pg[25.15(unlocked)] enter Initial
  -166> 2021-10-22T21:14:52.654+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.15_head oid #25:a8000000::::head#
  -165> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _pin0x55f8c815a000   #25:a8000000::::head# pinned
  -164> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7059180).flush flush done
  -163> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.15_head oid #25:a8000000::::head# = 0
  -162> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _unpin0x55f8c815a000   #25:a8000000::::head# unpinned
  -161> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) stat 25.15_head #25:a8000000::::head#
  -160> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _pin0x55f8c815a000   #25:a8000000::::head# pinned
  -159> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _unpin0x55f8c815a000   #25:a8000000::::head# unpinned
  -158> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) get_omap_iterator 25.15_head #25:a8000000::::head#
  -157> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _pin0x55f8c815a000   #25:a8000000::::head# pinned
  -156> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7059180).flush flush done
  -155> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) get_omap_iterator has_omap = 1
  -154> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cfe8c0) valid is at 0x0000000004F830DF'._biginfo'
  -153> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cfe8c0) valid is at 0x0000000004F830DF'._epoch'
  -152> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cfe8c0) valid is at 0x0000000004F830DF'._info'
  -151> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cfe8c0) valid is at 0x0000000004F830DF'._infover'
  -150> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cfe8c0) valid is at 0x0000000004F830DF'.may_include_deletes_in_missing'
  -149> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cfe8c0) valid is at 0x0000000004F830DF7E
  -148> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _unpin0x55f8c815a000   #25:a8000000::::head# unpinned
  -147> 2021-10-22T21:14:52.654+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) set_collection_opts 25.15_head options  pg_num_min 1
  -146> 2021-10-22T21:14:52.654+0200 7fb023c77f00  5 osd.2 pg_epoch: 23163 pg[25.15( empty local-lis/les=12563/12564 n=0 ec=10991/10977 lis/c=12563/12563 les/c/f=12564/12564/0 sis=12563) [0,5,2] r=2 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Initial 0.000517 0 0.000000
  -145> 2021-10-22T21:14:52.654+0200 7fb023c77f00  5 osd.2 pg_epoch: 23163 pg[25.15( empty local-lis/les=12563/12564 n=0 ec=10991/10977 lis/c=12563/12563 les/c/f=12564/12564/0 sis=12563) [0,5,2] r=2 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Reset
  -144> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) queue_transactions ch 0x55f8d3e33e00 25.15_head
  -143> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_create osr 0x55f8d3e31e40 = 0x55f8c88db500 seq 1
  -142> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _pin0x55f8c815a000   #25:a8000000::::head# pinned
  -141> 2021-10-22T21:14:52.654+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) _touch 25.15_head #25:a8000000::::head#
  -140> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _touch 25.15_head #25:a8000000::::head# = 0
  -139> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_calc_cost 0x55f8c88db500 cost 670195 (1 ios * 670000 + 195 bytes)
  -138> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_write_nodes txc 0x55f8c88db500 onodes 0x55f8d7059180 shared_blobs
  -137> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.extentmap(0x55f8d70592d0) update #25:a8000000::::head#
  -136> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _record_onode onode #25:a8000000::::head# is 31 (23 bytes onode + 2 bytes spanning blobs + 6 bytes inline extents)
  -135> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finalize_kv txc 0x55f8c88db500 allocated 0x[] released 0x[]
  -134> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c88db500 prepare
  -133> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish_io 0x55f8c88db500
  -132> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c88db500 io_done
  -131> 2021-10-22T21:14:52.654+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.0_head oid #25:00000000::::head#
  -130> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(25.0_head 0x55f8d3e33c20) get_onode oid #25:00000000::::head# key 0x7F800000000000001900000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F
  -129> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(25.0_head 0x55f8d3e33c20)  r 0 v.len 31
  -128> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.OnodeSpace(0x55f8d3e33d60 in 0x55f8c802c000) add #25:00000000::::head# 0x55f8d7059400
  -127> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _add 0x55f8c802c000 #25:00000000::::head# added, num=9
  -126> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7059400).flush flush done
  -125> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.0_head oid #25:00000000::::head# = 0
  -124> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _unpin0x55f8c802c000   #25:00000000::::head# unpinned
  -123> 2021-10-22T21:14:52.654+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.0_head oid #25:00000000::::head#
  -122> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _pin0x55f8c802c000   #25:00000000::::head# pinned
  -121> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7059400).flush flush done
  -120> 2021-10-22T21:14:52.654+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.0_head oid #25:00000000::::head# = 0
  -119> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 _unpin0x55f8c802c000   #25:00000000::::head# unpinned
  -118> 2021-10-22T21:14:52.654+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) read meta #-1:9829dd72:::osdmap.23134:0# 0x0~0
  -117> 2021-10-22T21:14:52.654+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(meta 0x55f8c87d2960) get_onode oid #-1:9829dd72:::osdmap.23134:0# key 0x7F7FFFFFFFFFFFFFFF9829DD'r!osdmap.23134!='0x0000000000000000FFFFFFFFFFFFFFFF6F
  -116> 2021-10-22T21:14:52.674+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread committed 1 cleaned 0 in 0.050118767s (0.000000073s flush + 0.050118696s kv commit)
  -115> 2021-10-22T21:14:52.674+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread committing 3 submitting 3 deferred done 0 stable 0
  -114> 2021-10-22T21:14:52.674+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread wake
  -113> 2021-10-22T21:14:52.674+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread kv_committed <0x55f8c737d180>
  -112> 2021-10-22T21:14:52.674+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread deferred_stable <>
  -111> 2021-10-22T21:14:52.674+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_apply_kv onode 0x55f8d7058780 had 1
  -110> 2021-10-22T21:14:52.674+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c737d180 kv_submitted
  -109> 2021-10-22T21:14:52.674+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_committed_kv txc 0x55f8c737d180
  -108> 2021-10-22T21:14:52.674+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c737d180 finishing
  -107> 2021-10-22T21:14:52.674+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish 0x55f8c737d180 onodes 0x55f8d7058500
  -106> 2021-10-22T21:14:52.674+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish  txc 0x55f8c737d180 done
  -105> 2021-10-22T21:14:52.674+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish osr 0x55f8c87d4420 q now empty
  -104> 2021-10-22T21:14:52.674+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_release_alloc(sync) 0x55f8c737d180 []
  -103> 2021-10-22T21:14:52.674+0200 7fb017936700 20 _unpin0x55f8c813c800   #25:50000000::::head# unpinned
  -102> 2021-10-22T21:14:52.674+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread sleep
  -101> 2021-10-22T21:14:52.674+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_apply_kv onode 0x55f8d7058c80 had 1
  -100> 2021-10-22T21:14:52.674+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_apply_kv onode 0x55f8d7059180 had 1
   -99> 2021-10-22T21:14:52.674+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(meta 0x55f8c87d2960)  r 0 v.len 113
   -98> 2021-10-22T21:14:52.674+0200 7fb023c77f00 20 bluestore.blob(0x55f8d6cfeb60) get_ref 0x0~4083 Blob(0x55f8d6cfeb60 blob([0x49800000~10000] csum+has_unused crc32c/0x1000 unused=0xffe0) use_tracker(0x0 0x0) SharedBlob(0x55f8d6cfebd0 sbid 0x0))
   -97> 2021-10-22T21:14:52.674+0200 7fb023c77f00 20 bluestore.blob(0x55f8d6cfeb60) get_ref init 0x10000, 10000
   -96> 2021-10-22T21:14:52.674+0200 7fb023c77f00 20 bluestore.OnodeSpace(0x55f8c87d2aa0 in 0x55f8c802c000) add #-1:9829dd72:::osdmap.23134:0# 0x55f8d7059680
   -95> 2021-10-22T21:14:52.674+0200 7fb023c77f00 20 _add 0x55f8c802c000 #-1:9829dd72:::osdmap.23134:0# added, num=10
   -94> 2021-10-22T21:14:52.674+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read 0x0~4083 size 0x4083 (16515)
   -93> 2021-10-22T21:14:52.674+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read will do buffered read
   -92> 2021-10-22T21:14:52.674+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _read_cache  blob Blob(0x55f8d6cfeb60 blob([0x49800000~10000] csum+has_unused crc32c/0x1000 unused=0xffe0) use_tracker(0x10000 0x4083) SharedBlob(0x55f8d6cfebd0 sbid 0x0)) need 0x0~4083 cache has 0x[]
   -91> 2021-10-22T21:14:52.674+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_read_ioc  blob Blob(0x55f8d6cfeb60 blob([0x49800000~10000] csum+has_unused crc32c/0x1000 unused=0xffe0) use_tracker(0x10000 0x4083) SharedBlob(0x55f8d6cfebd0 sbid 0x0)) need {<0x0, 0x5000> : [0x0:0~4083]}
   -90> 2021-10-22T21:14:52.674+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _prepare_read_ioc    region 0x0: 0x0 reading 0x0~5000
   -89> 2021-10-22T21:14:52.674+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _do_read waiting for aio
   -88> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _generate_read_result_bl  blob Blob(0x55f8d6cfeb60 blob([0x49800000~10000] csum+has_unused crc32c/0x1000 unused=0xffe0) use_tracker(0x10000 0x4083) SharedBlob(0x55f8d6cfebd0 sbid 0x0)) need 0x{<0x0, 0x5000> : [0x0:0~4083]}
   -87> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore.BufferSpace(0x55f8d6cfebe8 in 0x55f8c7320f70) _discard 0x0~5000
   -86> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 _add level 1 near 0 on buffer(0x55f8ceb0e0c0 space 0x55f8d6cfebe8 0x0~5000 clean) which has cache_private 0
   -85> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 _unpin0x55f8c802c000   #-1:9829dd72:::osdmap.23134:0# unpinned
   -84> 2021-10-22T21:14:52.690+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) read meta #-1:9829dd72:::osdmap.23134:0# 0x0~4083 = 16515
   -83> 2021-10-22T21:14:52.690+0200 7fb023c77f00  5 osd.2 pg_epoch: 23134 pg[25.0(unlocked)] enter Initial
   -82> 2021-10-22T21:14:52.690+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.0_head oid #25:00000000::::head#
   -81> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 _pin0x55f8c802c000   #25:00000000::::head# pinned
   -80> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7059400).flush flush done
   -79> 2021-10-22T21:14:52.690+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 25.0_head oid #25:00000000::::head# = 0
   -78> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 _unpin0x55f8c802c000   #25:00000000::::head# unpinned
   -77> 2021-10-22T21:14:52.690+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) stat 25.0_head #25:00000000::::head#
   -76> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 _pin0x55f8c802c000   #25:00000000::::head# pinned
   -75> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 _unpin0x55f8c802c000   #25:00000000::::head# unpinned
   -74> 2021-10-22T21:14:52.690+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) get_omap_iterator 25.0_head #25:00000000::::head#
   -73> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 _pin0x55f8c802c000   #25:00000000::::head# pinned
   -72> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7059400).flush flush done
   -71> 2021-10-22T21:14:52.690+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) get_omap_iterator has_omap = 1
   -70> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cff180) valid is at 0x0000000004F830EC'._biginfo'
   -69> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cff180) valid is at 0x0000000004F830EC'._epoch'
   -68> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cff180) valid is at 0x0000000004F830EC'._info'
   -67> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cff180) valid is at 0x0000000004F830EC'._infover'
   -66> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cff180) valid is at 0x0000000004F830EC'.may_include_deletes_in_missing'
   -65> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore.OmapIteratorImpl(0x55f8d6cff180) valid is at 0x0000000004F830EC7E
   -64> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 _unpin0x55f8c802c000   #25:00000000::::head# unpinned
   -63> 2021-10-22T21:14:52.690+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) set_collection_opts 25.0_head options  pg_num_min 1
   -62> 2021-10-22T21:14:52.690+0200 7fb023c77f00  5 osd.2 pg_epoch: 23134 pg[25.0( empty local-lis/les=11017/11018 n=0 ec=10977/10977 lis/c=11017/11017 les/c/f=11018/11018/0 sis=11017) [6,2,12] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] exit Initial 0.000704 0 0.000000
   -61> 2021-10-22T21:14:52.690+0200 7fb023c77f00  5 osd.2 pg_epoch: 23134 pg[25.0( empty local-lis/les=11017/11018 n=0 ec=10977/10977 lis/c=11017/11017 les/c/f=11018/11018/0 sis=11017) [6,2,12] r=1 lpr=0 crt=0'0 mlcod 0'0 unknown mbc={}] enter Reset
   -60> 2021-10-22T21:14:52.690+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) queue_transactions ch 0x55f8d3e33c20 25.0_head
   -59> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_create osr 0x55f8d3e31ce0 = 0x55f8c88db880 seq 1
   -58> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 _pin0x55f8c802c000   #25:00000000::::head# pinned
   -57> 2021-10-22T21:14:52.690+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) _touch 25.0_head #25:00000000::::head#
   -56> 2021-10-22T21:14:52.690+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _touch 25.0_head #25:00000000::::head# = 0
   -55> 2021-10-22T21:14:52.690+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_calc_cost 0x55f8c88db880 cost 670195 (1 ios * 670000 + 195 bytes)
   -54> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_write_nodes txc 0x55f8c88db880 onodes 0x55f8d7059400 shared_blobs
   -53> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore.extentmap(0x55f8d7059550) update #25:00000000::::head#
   -52> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _record_onode onode #25:00000000::::head# is 31 (23 bytes onode + 2 bytes spanning blobs + 6 bytes inline extents)
   -51> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finalize_kv txc 0x55f8c88db880 allocated 0x[] released 0x[]
   -50> 2021-10-22T21:14:52.690+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c88db880 prepare
   -49> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish_io 0x55f8c88db880
   -48> 2021-10-22T21:14:52.690+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c88db880 io_done
   -47> 2021-10-22T21:14:52.690+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 23.4s3_head oid 3#23:20000000::::head#
   -46> 2021-10-22T21:14:52.690+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(23.4s3_head 0x55f8d19f1c20) get_onode oid 3#23:20000000::::head# key 0x83800000000000001720000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F
   -45> 2021-10-22T21:14:52.694+0200 7fb017135700 20 bluestore.MempoolThread(0x55f8c7380b90) _resize_shards cache_size: 733711452 kv_alloc: 285212672 kv_used: 61417520 kv_onode_alloc: 42949672 kv_onode_used: -22 meta_alloc: 281018368 meta_used: 33700 data_alloc: 96468992 data_used: 94208
   -44> 2021-10-22T21:14:52.718+0200 7fb023c77f00 20 bluestore(/var/lib/ceph/osd/ceph-2).collection(23.4s3_head 0x55f8d19f1c20)  r 0 v.len 31
   -43> 2021-10-22T21:14:52.718+0200 7fb023c77f00 20 bluestore.OnodeSpace(0x55f8d19f1d60 in 0x55f8c812c800) add 3#23:20000000::::head# 0x55f8d7059900
   -42> 2021-10-22T21:14:52.718+0200 7fb023c77f00 20 _add 0x55f8c812c800 3#23:20000000::::head# added, num=1
   -41> 2021-10-22T21:14:52.718+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7059900).flush flush done
   -40> 2021-10-22T21:14:52.734+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread committed 3 cleaned 0 in 0.060137559s (0.000000088s flush + 0.060137473s kv commit)
   -39> 2021-10-22T21:14:52.734+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread committing 1 submitting 1 deferred done 0 stable 0
   -38> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread wake
   -37> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread kv_committed <0x55f8c737ca80,0x55f8c8034380,0x55f8c88db500>
   -36> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread deferred_stable <>
   -35> 2021-10-22T21:14:52.734+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c737ca80 kv_submitted
   -34> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_committed_kv txc 0x55f8c737ca80
   -33> 2021-10-22T21:14:52.734+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c737ca80 finishing
   -32> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish 0x55f8c737ca80 onodes 0x55f8d7058780
   -31> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish  txc 0x55f8c737ca80 done
   -30> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish osr 0x55f8c87d42c0 q now empty
   -29> 2021-10-22T21:14:52.734+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_release_alloc(sync) 0x55f8c737ca80 []
   -28> 2021-10-22T21:14:52.734+0200 7fb017936700 20 _unpin0x55f8c813a000   #25:90000000::::head# unpinned
   -27> 2021-10-22T21:14:52.734+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c8034380 kv_submitted
   -26> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_committed_kv txc 0x55f8c8034380
   -25> 2021-10-22T21:14:52.734+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c8034380 finishing
   -24> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish 0x55f8c8034380 onodes 0x55f8d7058c80
   -23> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish  txc 0x55f8c8034380 done
   -22> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish osr 0x55f8c87d4000 q now empty
   -21> 2021-10-22T21:14:52.734+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_release_alloc(sync) 0x55f8c8034380 []
   -20> 2021-10-22T21:14:52.734+0200 7fb017936700 20 _unpin0x55f8c8162000   #25:18000000::::head# unpinned
   -19> 2021-10-22T21:14:52.734+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c88db500 kv_submitted
   -18> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_committed_kv txc 0x55f8c88db500
   -17> 2021-10-22T21:14:52.734+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_apply_kv onode 0x55f8d7059400 had 1
   -16> 2021-10-22T21:14:52.734+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c88db500 finishing
   -15> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish 0x55f8c88db500 onodes 0x55f8d7059180
   -14> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish  txc 0x55f8c88db500 done
   -13> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish osr 0x55f8d3e31e40 q now empty
   -12> 2021-10-22T21:14:52.734+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_release_alloc(sync) 0x55f8c88db500 []
   -11> 2021-10-22T21:14:52.734+0200 7fb017936700 20 _unpin0x55f8c815a000   #25:a8000000::::head# unpinned
   -10> 2021-10-22T21:14:52.734+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread sleep
    -9> 2021-10-22T21:14:52.746+0200 7fb017135700 20 bluestore.MempoolThread(0x55f8c7380b90) _resize_shards cache_size: 733711452 kv_alloc: 285212672 kv_used: 61565016 kv_onode_alloc: 42949672 kv_onode_used: -22 meta_alloc: 281018368 meta_used: 34622 data_alloc: 96468992 data_used: 94208
    -8> 2021-10-22T21:14:52.754+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 23.4s3_head oid 3#23:20000000::::head# = 0
    -7> 2021-10-22T21:14:52.754+0200 7fb023c77f00 20 _unpin0x55f8c812c800   3#23:20000000::::head# unpinned
    -6> 2021-10-22T21:14:52.754+0200 7fb023c77f00 15 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 23.4s3_head oid 3#23:20000000::::head#
    -5> 2021-10-22T21:14:52.754+0200 7fb023c77f00 20 _pin0x55f8c812c800   3#23:20000000::::head# pinned
    -4> 2021-10-22T21:14:52.754+0200 7fb023c77f00 20 bluestore.onode(0x55f8d7059900).flush flush done
    -3> 2021-10-22T21:14:52.754+0200 7fb023c77f00 10 bluestore(/var/lib/ceph/osd/ceph-2) omap_get_values 23.4s3_head oid 3#23:20000000::::head# = 0
    -2> 2021-10-22T21:14:52.754+0200 7fb023c77f00 20 _unpin0x55f8c812c800   3#23:20000000::::head# unpinned
    -1> 2021-10-22T21:14:52.758+0200 7fb023c77f00 -1 /build/ceph-16.2.6/src/osd/PG.cc: In function 'static int PG::peek_map_epoch(ObjectStore*, spg_t, epoch_t*)' thread 7fb023c77f00 time 2021-10-22T21:14:52.757014+0200
/build/ceph-16.2.6/src/osd/PG.cc: 1009: FAILED ceph_assert(values.size() == 2)

 ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55f8bc092e61]
 2: /usr/bin/ceph-osd(+0xac6069) [0x55f8bc093069]
 3: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0xa17) [0x55f8bc21d057]
 4: (OSD::load_pgs()+0x6b4) [0x55f8bc18dec4]
 5: (OSD::init()+0x2b4e) [0x55f8bc19aa6e]
 6: main()
 7: __libc_start_main()
 8: _start()

     0> 2021-10-22T21:14:52.762+0200 7fb023c77f00 -1 *** Caught signal (Aborted) **
 in thread 7fb023c77f00 thread_name:ceph-osd

 ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fb0243213c0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ad) [0x55f8bc092ebc]
 5: /usr/bin/ceph-osd(+0xac6069) [0x55f8bc093069]
 6: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0xa17) [0x55f8bc21d057]
 7: (OSD::load_pgs()+0x6b4) [0x55f8bc18dec4]
 8: (OSD::init()+0x2b4e) [0x55f8bc19aa6e]
 9: main()
 10: __libc_start_main()
 11: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 rbd_pwl
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 immutable_obj_cache
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 0 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 compressor
  20/20 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 fuse
   2/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
   1/ 5 prioritycache
   0/ 5 test
   0/ 5 cephfs_mirror
   0/ 5 cephsqlite
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
  140394177402624 / rocksdb:dump_st
  140394202580736 / bstore_kv_sync
  140394278115072 / bstore_mempool
  140394286507776 / bstore_kv_final
  140394491248384 / ceph-osd
  max_recent     10000
  max_new        10000
  log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2021-10-22T21:14:52.786+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread committed 1 cleaned 0 in 0.050166298s (0.000000118s flush + 0.050166178s kv commit)
2021-10-22T21:14:52.786+0200 7fb01292c700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_sync_thread sleep
2021-10-22T21:14:52.786+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread wake
2021-10-22T21:14:52.786+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread kv_committed <0x55f8c88db880>
2021-10-22T21:14:52.786+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread deferred_stable <>
2021-10-22T21:14:52.786+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c88db880 kv_submitted
2021-10-22T21:14:52.786+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_committed_kv txc 0x55f8c88db880
2021-10-22T21:14:52.786+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_state_proc txc 0x55f8c88db880 finishing
2021-10-22T21:14:52.786+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish 0x55f8c88db880 onodes 0x55f8d7059400
2021-10-22T21:14:52.786+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish  txc 0x55f8c88db880 done
2021-10-22T21:14:52.786+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _txc_finish osr 0x55f8d3e31ce0 q now empty
2021-10-22T21:14:52.786+0200 7fb017936700 10 bluestore(/var/lib/ceph/osd/ceph-2) _txc_release_alloc(sync) 0x55f8c88db880 []
2021-10-22T21:14:52.786+0200 7fb017936700 20 _unpin0x55f8c802c000   #25:00000000::::head# unpinned
2021-10-22T21:14:52.786+0200 7fb017936700 20 bluestore(/var/lib/ceph/osd/ceph-2) _kv_finalize_thread sleep
2021-10-22T21:14:52.794+0200 7fb017135700 20 bluestore.MempoolThread(0x55f8c7380b90) _resize_shards cache_size: 733711452 kv_alloc: 285212672 kv_used: 62318720 kv_onode_alloc: 42949672 kv_onode_used: -22 meta_alloc: 281018368 meta_used: 34622 data_alloc: 96468992 data_used: 94208
2021-10-22T21:14:52.846+0200 7fb017135700 20 bluestore.MempoolThread(0x55f8c7380b90) _resize_shards cache_size: 733714101 kv_alloc: 285212672 kv_used: 62318720 kv_onode_alloc: 42949672 kv_onode_used: -22 meta_alloc: 281018368 meta_used: 34622 data_alloc: 96468992 data_used: 94208


On 21.10.2021 00:19, Igor Fedotov wrote:
> Hey mgrzybowski!
>
> Never seen that before but perhaps some omaps have been improperly converted to new format and aren't read any more...
>
> I'll take a more detailed look at what's happening during that load_pgs call and what exact information is missing.
>
> Meanwhile could you please set debug-bluestore to 20 and collect OSD startup log?
>
>
> Thanks,
>
> Igor
>
> On 10/21/2021 12:56 AM, mgrzybowski wrote:
> > Hi
> >   Recently I did perform upgrades on single node cephfs server i have.
> >
> > # ceph fs ls
> > name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ecpoolk3m1osd ecpoolk5m1osd ecpoolk4m2osd
> > ~# ceph osd pool ls detail
> > pool 20 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 10674 lfor 0/0/5088 flags hashpspool stripe_width 0 application cephfs
> > pool 21 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 10674 lfor 0/0/5179 flags hashpspool stripe_width 0 application cephfs
> > pool 22 'ecpoolk3m1osd' erasure profile myprofilek3m1osd size 4 min_size 3 crush_rule 3 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn last_change 10674 lfor 0/0/1442 flags hashpspool,ec_overwrites stripe_width 12288 compression_algorithm zstd compression_mode aggressive application cephfs
> > pool 23 'ecpoolk5m1osd' erasure profile myprofilek5m1osd size 6 min_size 5 crush_rule 5 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 12517 lfor 0/0/7892 flags hashpspool,ec_overwrites stripe_width 20480 compression_algorithm zstd compression_mode aggressive application cephfs
> > pool 24 'ecpoolk4m2osd' erasure profile myprofilek4m2osd size 6 min_size 5 crush_rule 6 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 10674 flags hashpspool,ec_overwrites stripe_width 16384 compression_algorithm zstd compression_mode aggressive application cephfs
> > pool 25 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 11033 lfor 0/0/10991 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth
> >
> >
> > I started this upgrade from ubuntu 16.04 and luminous ( there were upgrades in the past and some osd's could be started in Kraken ) ):
> > - first i upgraded ceph to Nautilus,  all seems to went well and accoording to the docs, no warning in status
> > - then i did "do-release-upgrade" to ubuntu to 18.04 ( ceph packaged were not touch by that upgrade )
> > - then i did "do-release-upgrade" to ubuntu to 20.04 ( this upgrade bumped ceph
> >   packages to 15.2.1-0ubuntu1, before each do-release-upgrade i removed /etc/ceph/ceph.conf,
> >   so at least mon deamon was down. osd should not start ( siple volumes are encrypted )
> > - next i upgraded ceph packages to  16.2.6-1focal m started deamons .
> >
> > All seems to work well, only what left was warning:
> >
> > 10 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats
> >
> > I found on the list that it is recommend to set:
> >
> > ceph config set osd bluestore_fsck_quick_fix_on_mount true
> >
> > and rolling restart OSDs. After first restart+fsck i got crash on OSD ( and on MDS to) :
> >
> >     -1> 2021-10-14T22:02:45.877+0200 7f7f080a4f00 -1 /build/ceph-16.2.6/src/osd/PG.cc: In function 'static int PG::peek_map_epoch(ObjectStore*, spg_t, epoch_t*)' thread 7f7f080a4f00 time 2021-10-14T22:02:45.878154+0200
> > /build/ceph-16.2.6/src/osd/PG.cc: 1009: FAILED ceph_assert(values.size() == 2)
> >  ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
> >  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55e29cd0ce61]
> >  2: /usr/bin/ceph-osd(+0xac6069) [0x55e29cd0d069]
> >  3: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0xa17) [0x55e29ce97057]
> >  4: (OSD::load_pgs()+0x6b4) [0x55e29ce07ec4]
> >  5: (OSD::init()+0x2b4e) [0x55e29ce14a6e]
> >  6: main()
> >  7: __libc_start_main()
> >  8: _start()
> >
> >
> > The same went on next restart+fsck  osd:
> >
> >     -1> 2021-10-17T22:47:49.291+0200 7f98877bff00 -1 /build/ceph-16.2.6/src/osd/PG.cc: In function 'static int PG::peek_map_epoch(ObjectStore*, spg_t, epoch_t*)' thread 7f98877bff00 time 2021-10-17T22:47:49.292912+0200
> > /build/ceph-16.2.6/src/osd/PG.cc: 1009: FAILED ceph_assert(values.size() == 2)
> >
> >  ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
> >  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x560e09af7e61]
> >  2: /usr/bin/ceph-osd(+0xac6069) [0x560e09af8069]
> >  3: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0xa17) [0x560e09c82057]
> >  4: (OSD::load_pgs()+0x6b4) [0x560e09bf2ec4]
> >  5: (OSD::init()+0x2b4e) [0x560e09bffa6e]
> >  6: main()
> >  7: __libc_start_main()
> >  8: _start()
> >
> >
> > Once crashed OSDs could not be bring back online, they will crash again if i try start them.
> > Deep fsck did not found anything:
> >
> > ~# ceph-bluestore-tool --command fsck  --deep yes --path /var/lib/ceph/osd/ceph-2
> > fsck success
> >
> >
> > Any ideas what could cause this crashes and is it possible to bring online crashed osd this way  ?

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx