oh. OSD 87 (one of the replica partners) crashed. Here some lines from
the log
-10> 2022-02-10T14:28:46.840+0100 7fd1b306d700 5 osd.87 pg_epoch: 2016357 pg[7.3ff( v 2016317'1 (0'0,2016317'1] local-lis/les=2016308/2016309 n=1 ec=2016308/2016308 lis/c=2016308/2016308 les/c/f=2016309/2016309/0 sis=2016356) [123,92,85] r=-1 lpr=2016356 pi=[2016308,2016356)/1 luod=0'0 crt=2016317'1 lcod 0'0 mlcod 0'0 active mbc={}] exit Started/ReplicaActive/RepNotRecovering 0.073328 2 0.000058
-9> 2022-02-10T14:28:46.841+0100 7fd1b306d700 5 osd.87 pg_epoch: 2016357 pg[7.3ff( v 2016317'1 (0'0,2016317'1] local-lis/les=2016308/2016309 n=1 ec=2016308/2016308 lis/c=2016308/2016308 les/c/f=2016309/2016309/0 sis=2016356) [123,92,85] r=-1 lpr=2016356 pi=[2016308,2016356)/1 luod=0'0 crt=2016317'1 lcod 0'0 mlcod 0'0 active mbc={}] exit Started/ReplicaActive 0.073376 0 0.000000
-8> 2022-02-10T14:28:46.841+0100 7fd1b306d700 5 osd.87 pg_epoch: 2016357 pg[7.3ff( v 2016317'1 (0'0,2016317'1] local-lis/les=2016308/2016309 n=1 ec=2016308/2016308 lis/c=2016308/2016308 les/c/f=2016309/2016309/0 sis=2016356) [123,92,85] r=-1 lpr=2016356 pi=[2016308,2016356)/1 luod=0'0 crt=2016317'1 lcod 0'0 mlcod 0'0 active mbc={}] enter Started/ToDelete
-7> 2022-02-10T14:28:46.841+0100 7fd1b306d700 5 osd.87 pg_epoch: 2016357 pg[7.3ff( v 2016317'1 (0'0,2016317'1] local-lis/les=2016308/2016309 n=1 ec=2016308/2016308 lis/c=2016308/2016308 les/c/f=2016309/2016309/0 sis=2016356) [123,92,85] r=-1 lpr=2016356 pi=[2016308,2016356)/1 luod=0'0 crt=2016317'1 lcod 0'0 mlcod 0'0 active mbc={}] enter Started/ToDelete/WaitDeleteReseved
-6> 2022-02-10T14:28:46.841+0100 7fd1b306d700 5 osd.87 pg_epoch: 2016357 pg[7.3ff( v 2016317'1 (0'0,2016317'1] local-lis/les=2016308/2016309 n=1 ec=2016308/2016308 lis/c=2016308/2016308 les/c/f=2016309/2016309/0 sis=2016356) [123,92,85] r=-1 lpr=2016356 pi=[2016308,2016356)/1 luod=0'0 crt=2016317'1 lcod 0'0 mlcod 0'0 active mbc={}] exit Started/ToDelete/WaitDeleteReseved 0.000044 1 0.000092
-5> 2022-02-10T14:28:46.841+0100 7fd1b306d700 5 osd.87 pg_epoch: 2016357 pg[7.3ff( v 2016317'1 (0'0,2016317'1] local-lis/les=2016308/2016309 n=1 ec=2016308/2016308 lis/c=2016308/2016308 les/c/f=2016309/2016309/0 sis=2016356) [123,92,85] r=-1 lpr=2016356 pi=[2016308,2016356)/1 luod=0'0 crt=2016317'1 lcod 0'0 mlcod 0'0 active mbc={}] enter Started/ToDelete/Deleting
-4> 2022-02-10T14:28:46.843+0100 7fd1b306d700 1 bluestore(/var/lib/ceph/osd/ceph-87) operator() #7:ffffffff:::c76c7ac2014adb9f0f0837ac1e85fd1e241af225908b6a0c3d3a44d6b866e732_00400000:head# 0x55fe306aac80 exists in onode_map
-3> 2022-02-10T14:28:46.843+0100 7fd1b306d700 -1 bluestore(/var/lib/ceph/osd/ceph-87) _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
-2> 2022-02-10T14:28:46.843+0100 7fd1b306d700 0 _dump_transaction transaction dump:
{
"ops": [
{
"op_num": 0,
"op_name": "remove",
"collection": "7.3ff_head",
"oid": "#7:ffc00000::::head#"
},
{
"op_num": 1,
"op_name": "rmcoll",
"collection": "7.3ff_head"
}
]
}
-1> 2022-02-10T14:28:46.848+0100 7fd1b306d700 -1 /root/rpmbuild/BUILD/ceph-16.2.6-4-g5651163a235/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7fd1b306d700 time 2022-02-10T14:28:46.844725+0100
/root/rpmbuild/BUILD/ceph-16.2.6-4-g5651163a235/src/os/bluestore/BlueStore.cc: 12922: ceph_abort_msg("unexpected error")
On Thu, 10 Feb 2022 14:15:37 +0100
Manuel Lausch <manuel.lausch@xxxxxxxx> wrote:
> yes the pool on the testcluster contains a lot of objects
>
> I created a new pool, put the object (this time only 100K, just to test
> it) and run a deep-scrub -> error
>
> # dd if=/dev/urandom of=test_obj bs=1K count=100
>
> # rados -p nameplosion put c76c7ac2014adb9f0f0837ac1e85fd1e241af225908b6a0c3d3a44d6b866e732_00400000 test_obj
>
> # ceph osd map nameplosion c76c7ac2014adb9f0f0837ac1e85fd1e241af225908b6a0c3d3a44d6b866e732_00400000
> osdmap e2016317 pool 'nameplosion' (7) object 'c76c7ac2014adb9f0f0837ac1e85fd1e241af225908b6a0c3d3a44d6b866e732_00400000' -> pg 7.ffffffff (7.3ff) -> up ([123,87,85], p123) acting ([123,87,85], p123)
>
> # ceph pg deep-scrub 7.3ff
>
>
> and here the ceph-osd.123.log snipped
>
> 2022-02-10T14:12:13.287+0100 7f9f792ad700 -1 log_channel(cluster) log [ERR] : 7.3ff deep-scrub : stat mismatch, got 0/1 objects, 0/0 clones, 0/1 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/102400 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.
> 2022-02-10T14:12:13.287+0100 7f9f792ad700 -1 log_channel(cluster) log [ERR] : 7.3ff deep-scrub 1 errors
>
>
> Manuel
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
