Hi I have single node cephfs on top of EC pool on top of bluestore ( my dream setup ;) ) . I hit bug that is crushing my osd ( first only one , then second before data backfilled ): http://tracker.ceph.com/issues/20997 After some investigation i found that there is object ("3#22:addebba8:::10000014f8e.00000145:head#32ce3" ( please note abnormal "#32ce3" at the end ) ) that is probably causing this. - ceph-bluestore-tool deep fsck : ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-1/ --deep yes --command fsck "finish with 0 errors" - ceph-objectstore-tool fsck : ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1/ --op fsck "fsck found no errors" - export of the affected pg (22.5s3) is failing: ceph-objectstore-tool --force --data-path /var/lib/ceph/osd/ceph-1/ --pgid 22.5s3 --op export --file /tank1/ceph-recovery/recover.22.5s3v2 2>&1 | tee recover.22.5s3v2.log .... Read 3#22:addea9c4:::1000000d42b.00000073:head# Read 3#22:addeb3f7:::10000009d3b.00000440:head# Read 3#22:addeb487:::1000000a84f.00000e50:head# Read 3#22:addebba8:::10000014f8e.00000145:head#32ce3 getattr failure object_info -61 export_files error -61 - tools like ceph-objectstore-tool shows warning when they for example list objects for pg : Error getting attr on : 22.5s3_head,3#22:addebba8:::10000014f8e.00000145:head#/32ce3, (61) No data available - I can clearly see directory for this strange object, when i fuse-mount bluestore. /mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3 /mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/bitwise_hash /mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/data /mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/omap /mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/attr /mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/omap_header I read all documentation, and a loot of therads on this list ( this list is really great for info about tools like ceph-objectstore-tool ), but i can not find any way to remove/modify this broken object from bluestore. My data are anacessable, i do not know when and if this bug will be fixed in ceph code. Should i focus on removing this strange/broken object (like hack ceph-objectstore-tool or ceph-bluestore-tool), or is there something else i can/should do to get my osd.1 up and running again ? -- Kind Regards Marek Grzybowski _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |