Is it possible to extract just this 4MB object from every disk?
Is there a command that just forces the one from osd.0 to be replicated
to the other 2 osds?
This is for osd.0 (more below)
bluestore(/var/lib/ceph/osd/ceph-0) _verify_csum bad crc32c/0x1000
checksum at blob offset 0x0, got 0x1a128a93, expected 0x90407f75, device
location [0x5826c10000~1000], logical extent 0x0~1000
bluestore(/var/lib/ceph/osd/ceph-12) _verify_csum bad crc32c/0x1000
checksum at blob offset 0x0, got 0x100ac314, expected 0x90407f75, device
location [0x15a0170000~1000], logical extent 0x0~1000,
bluestore(/var/lib/ceph/osd/ceph-9) _verify_csum bad crc32c/0x1000
checksum at blob offset 0x0, got 0xb40b26a7, expected 0x90407f75, device
location [0x2daea0000~1000], logical extent 0x0~1000,
I could not get this to work.
ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-12 --log-file out
--debug-bluestore 30 --no-log-to-stderr too many positional options have
been specified on the command line
ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-12 --out-dir /tmp
> /tmp/fsck.out 2>&1
2017-08-09 11:01:34.208832 7fafd0171b80 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore
2017-08-09 11:01:34.208927 7fafd0171b80 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore action fsck
2017-08-09 11:01:34.238734 7fafd0171b80 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore
2017-08-09 11:01:34.238937 7fafd0171b80 1
bluestore(/var/lib/ceph/osd/ceph-12) fsck (shallow) start
2017-08-09 11:01:34.239001 7fafd0171b80 1 bdev create path
/var/lib/ceph/osd/ceph-12/block type kernel
2017-08-09 11:01:34.239013 7fafd0171b80 1 bdev(0x7fafd2ac9200
/var/lib/ceph/osd/ceph-12/block) open path
/var/lib/ceph/osd/ceph-12/block
2017-08-09 11:01:34.239263 7fafd0171b80 1 bdev(0x7fafd2ac9200
/var/lib/ceph/osd/ceph-12/block) open size 4000681103360 (0x3a37b2d1000,
3725 GB) block_size 4096 (4096 B) rotational
2017-08-09 11:01:34.239549 7fafd0171b80 1
bluestore(/var/lib/ceph/osd/ceph-12) _set_cache_sizes max 0.5 < ratio
0.99
2017-08-09 11:01:34.239577 7fafd0171b80 1
bluestore(/var/lib/ceph/osd/ceph-12) _set_cache_sizes cache_size
1073741824 meta 0.5 kv 0.5 data 0
2017-08-09 11:01:34.239659 7fafd0171b80 1 bdev create path
/var/lib/ceph/osd/ceph-12/block type kernel
2017-08-09 11:01:34.239664 7fafd0171b80 1 bdev(0x7fafd2ac8e00
/var/lib/ceph/osd/ceph-12/block) open path
/var/lib/ceph/osd/ceph-12/block
2017-08-09 11:01:34.239815 7fafd0171b80 1 bdev(0x7fafd2ac8e00
/var/lib/ceph/osd/ceph-12/block) open size 4000681103360 (0x3a37b2d1000,
3725 GB) block_size 4096 (4096 B) rotational
2017-08-09 11:01:34.239827 7fafd0171b80 1 bluefs add_block_device bdev
1 path /var/lib/ceph/osd/ceph-12/block size 3725 GB
2017-08-09 11:01:34.239857 7fafd0171b80 1 bluefs mount
2017-08-09 11:01:35.239635 7fafd0171b80 0 set rocksdb option
compaction_readahead_size = 2097152
2017-08-09 11:01:35.239654 7fafd0171b80 0 set rocksdb option
compression = kNoCompression
2017-08-09 11:01:35.239677 7fafd0171b80 0 set rocksdb option
max_write_buffer_number = 4
2017-08-09 11:01:35.239683 7fafd0171b80 0 set rocksdb option
min_write_buffer_number_to_merge = 1
2017-08-09 11:01:35.239686 7fafd0171b80 0 set rocksdb option
recycle_log_file_num = 4
2017-08-09 11:01:35.239690 7fafd0171b80 0 set rocksdb option
writable_file_max_buffer_size = 0
2017-08-09 11:01:35.239694 7fafd0171b80 0 set rocksdb option
write_buffer_size = 268435456
2017-08-09 11:01:35.239725 7fafd0171b80 0 set rocksdb option
compaction_readahead_size = 2097152
2017-08-09 11:01:35.239730 7fafd0171b80 0 set rocksdb option
compression = kNoCompression
2017-08-09 11:01:35.239735 7fafd0171b80 0 set rocksdb option
max_write_buffer_number = 4
2017-08-09 11:01:35.239739 7fafd0171b80 0 set rocksdb option
min_write_buffer_number_to_merge = 1
2017-08-09 11:01:35.239742 7fafd0171b80 0 set rocksdb option
recycle_log_file_num = 4
2017-08-09 11:01:35.239746 7fafd0171b80 0 set rocksdb option
writable_file_max_buffer_size = 0
2017-08-09 11:01:35.239749 7fafd0171b80 0 set rocksdb option
write_buffer_size = 268435456
2017-08-09 11:01:35.239948 7fafd0171b80 4 rocksdb: RocksDB version:
5.4.0
2017-08-09 11:01:35.239959 7fafd0171b80 4 rocksdb: Git sha
rocksdb_build_git_sha:@0@
2017-08-09 11:01:35.239961 7fafd0171b80 4 rocksdb: Compile date Jul 17
2017
2017-08-09 11:01:35.239964 7fafd0171b80 4 rocksdb: DB SUMMARY
2017-08-09 11:01:35.240004 7fafd0171b80 4 rocksdb: CURRENT file:
CURRENT
2017-08-09 11:01:35.240006 7fafd0171b80 4 rocksdb: IDENTITY file:
IDENTITY
2017-08-09 11:01:35.240010 7fafd0171b80 4 rocksdb: MANIFEST file:
MANIFEST-001083 size: 1868 Bytes
2017-08-09 11:01:35.240013 7fafd0171b80 4 rocksdb: SST files in db dir,
Total Num: 15, files: 000529.sst 000733.sst 000904.sst 000905.sst
000906.sst 000907.sst 000908.sst 000909.sst 000910.sst
2017-08-09 11:01:35.240015 7fafd0171b80 4 rocksdb: Write Ahead Log file
in db: 001084.log size: 160853870 ;
2017-08-09 11:01:35.240018 7fafd0171b80 4 rocksdb:
Options.error_if_exists: 0
2017-08-09 11:01:35.240019 7fafd0171b80 4 rocksdb:
Options.create_if_missing: 0
2017-08-09 11:01:35.240019 7fafd0171b80 4 rocksdb:
Options.paranoid_checks: 1
2017-08-09 11:01:35.240029 7fafd0171b80 4 rocksdb:
Options.env: 0x7fafd2c6bd40
2017-08-09 11:01:35.240030 7fafd0171b80 4 rocksdb:
Options.info_log: 0x7fafd2c6bf00
2017-08-09 11:01:35.240031 7fafd0171b80 4 rocksdb:
Options.max_open_files: -1
2017-08-09 11:01:35.240032 7fafd0171b80 4 rocksdb:
Options.max_file_opening_threads: 16
2017-08-09 11:01:35.240033 7fafd0171b80 4 rocksdb:
Options.use_fsync: 0
2017-08-09 11:01:35.240034 7fafd0171b80 4 rocksdb:
Options.max_log_file_size: 0
2017-08-09 11:01:35.240035 7fafd0171b80 4 rocksdb:
Options.max_manifest_file_size: 18446744073709551615
2017-08-09 11:01:35.240037 7fafd0171b80 4 rocksdb:
Options.log_file_time_to_roll: 0
2017-08-09 11:01:35.240037 7fafd0171b80 4 rocksdb:
Options.keep_log_file_num: 1000
2017-08-09 11:01:35.240038 7fafd0171b80 4 rocksdb:
Options.recycle_log_file_num: 4
2017-08-09 11:01:35.240039 7fafd0171b80 4 rocksdb:
Options.allow_fallocate: 1
2017-08-09 11:01:35.240040 7fafd0171b80 4 rocksdb:
Options.allow_mmap_reads: 0
2017-08-09 11:01:35.240041 7fafd0171b80 4 rocksdb:
Options.allow_mmap_writes: 0
2017-08-09 11:01:35.240051 7fafd0171b80 4 rocksdb:
Options.use_direct_reads: 0
2017-08-09 11:01:35.240053 7fafd0171b80 4 rocksdb:
Options.use_direct_io_for_flush_and_compaction: 0
2017-08-09 11:01:35.240054 7fafd0171b80 4 rocksdb:
Options.create_missing_column_families: 0
2017-08-09 11:01:35.240055 7fafd0171b80 4 rocksdb:
Options.db_log_dir:
2017-08-09 11:01:35.240056 7fafd0171b80 4 rocksdb:
Options.wal_dir: db
2017-08-09 11:01:35.240062 7fafd0171b80 4 rocksdb:
Options.table_cache_numshardbits: 6
2017-08-09 11:01:35.240063 7fafd0171b80 4 rocksdb:
Options.max_subcompactions: 1
2017-08-09 11:01:35.240064 7fafd0171b80 4 rocksdb:
Options.max_background_flushes: 1
2017-08-09 11:01:35.240065 7fafd0171b80 4 rocksdb:
Options.WAL_ttl_seconds: 0
2017-08-09 11:01:35.240067 7fafd0171b80 4 rocksdb:
Options.WAL_size_limit_MB: 0
2017-08-09 11:01:35.240068 7fafd0171b80 4 rocksdb:
Options.manifest_preallocation_size: 4194304
2017-08-09 11:01:35.240069 7fafd0171b80 4 rocksdb:
Options.is_fd_close_on_exec: 1
2017-08-09 11:01:35.240070 7fafd0171b80 4 rocksdb:
Options.advise_random_on_open: 1
2017-08-09 11:01:35.240072 7fafd0171b80 4 rocksdb:
Options.db_write_buffer_size: 0
2017-08-09 11:01:35.240073 7fafd0171b80 4 rocksdb:
Options.access_hint_on_compaction_start: 1
2017-08-09 11:01:35.240074 7fafd0171b80 4 rocksdb:
Options.new_table_reader_for_compaction_inputs: 1
2017-08-09 11:01:35.240075 7fafd0171b80 4 rocksdb:
Options.compaction_readahead_size: 2097152
2017-08-09 11:01:35.240076 7fafd0171b80 4 rocksdb:
Options.random_access_max_buffer_size: 1048576
2017-08-09 11:01:35.240076 7fafd0171b80 4 rocksdb:
Options.writable_file_max_buffer_size: 0
2017-08-09 11:01:35.240077 7fafd0171b80 4 rocksdb:
Options.use_adaptive_mutex: 0
2017-08-09 11:01:35.240078 7fafd0171b80 4 rocksdb:
Options.rate_limiter: (nil)
2017-08-09 11:01:35.240080 7fafd0171b80 4 rocksdb:
Options.sst_file_manager.rate_bytes_per_sec: 0
2017-08-09 11:01:35.240081 7fafd0171b80 4 rocksdb:
Options.bytes_per_sync: 0
2017-08-09 11:01:35.240081 7fafd0171b80 4 rocksdb:
Options.wal_bytes_per_sync: 0
2017-08-09 11:01:35.240082 7fafd0171b80 4 rocksdb:
Options.wal_recovery_mode: 0
2017-08-09 11:01:35.240083 7fafd0171b80 4 rocksdb:
Options.enable_thread_tracking: 0
2017-08-09 11:01:35.240084 7fafd0171b80 4 rocksdb:
Options.allow_concurrent_memtable_write: 1
2017-08-09 11:01:35.240085 7fafd0171b80 4 rocksdb:
Options.enable_write_thread_adaptive_yield: 1
2017-08-09 11:01:35.240086 7fafd0171b80 4 rocksdb:
Options.write_thread_max_yield_usec: 100
2017-08-09 11:01:35.240087 7fafd0171b80 4 rocksdb:
Options.write_thread_slow_yield_usec: 3
2017-08-09 11:01:35.240088 7fafd0171b80 4 rocksdb:
Options.row_cache: None
2017-08-09 11:01:35.240089 7fafd0171b80 4 rocksdb:
Options.wal_filter: None
2017-08-09 11:01:35.240090 7fafd0171b80 4 rocksdb:
Options.avoid_flush_during_recovery: 0
2017-08-09 11:01:35.240091 7fafd0171b80 4 rocksdb:
Options.base_background_compactions: 1
2017-08-09 11:01:35.240092 7fafd0171b80 4 rocksdb:
Options.max_background_compactions: 1
2017-08-09 11:01:35.240093 7fafd0171b80 4 rocksdb:
Options.avoid_flush_during_shutdown: 0
2017-08-09 11:01:35.240094 7fafd0171b80 4 rocksdb:
Options.delayed_write_rate : 16777216
2017-08-09 11:01:35.240095 7fafd0171b80 4 rocksdb:
Options.max_total_wal_size: 0
2017-08-09 11:01:35.240096 7fafd0171b80 4 rocksdb:
Options.delete_obsolete_files_period_micros: 21600000000
2017-08-09 11:01:35.240097 7fafd0171b80 4 rocksdb:
Options.stats_dump_period_sec: 600
2017-08-09 11:01:35.240099 7fafd0171b80 4 rocksdb: Compression
algorithms supported:
2017-08-09 11:01:35.240101 7fafd0171b80 4 rocksdb: Snappy
supported: 0
2017-08-09 11:01:35.240102 7fafd0171b80 4 rocksdb: Zlib supported:
0
2017-08-09 11:01:35.240103 7fafd0171b80 4 rocksdb: Bzip supported:
0
2017-08-09 11:01:35.240104 7fafd0171b80 4 rocksdb: LZ4 supported:
0
2017-08-09 11:01:35.240105 7fafd0171b80 4 rocksdb: ZSTD supported:
0
2017-08-09 11:01:35.240108 7fafd0171b80 4 rocksdb: Fast CRC32
supported: 1
2017-08-09 11:01:35.246606 7fafd0171b80 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR
CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/
12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/version_set.cc:2609]
Recovering from manifest file: MANIFEST-001083
2017-08-09 11:01:35.246707 7fafd0171b80 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR
CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/
12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/column_family.cc:407]
--------------- Options for column family [default]:
2017-08-09 11:01:35.246712 7fafd0171b80 4 rocksdb:
Options.comparator: leveldb.BytewiseComparator
2017-08-09 11:01:35.246721 7fafd0171b80 4 rocksdb:
Options.merge_operator: .T:int64_array.b:bitwise_xor
2017-08-09 11:01:35.246724 7fafd0171b80 4 rocksdb:
Options.compaction_filter: None
2017-08-09 11:01:35.246726 7fafd0171b80 4 rocksdb:
Options.compaction_filter_factory: None
2017-08-09 11:01:35.246727 7fafd0171b80 4 rocksdb:
Options.memtable_factory: SkipListFactory
2017-08-09 11:01:35.246728 7fafd0171b80 4 rocksdb:
Options.table_factory: BlockBasedTable
2017-08-09 11:01:35.246752 7fafd0171b80 4 rocksdb:
table_factory options: flush_block_policy_factory:
FlushBlockBySizePolicyFactory (0x7fafd2aaa1f0)
cache_index_and_filter_blocks: 0
cache_index_and_filter_blocks_with_high_priority: 0
pin_l0_filter_and_index_blocks_in_cache: 0
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x7fafd2b03048
block_cache_name: LRUCache
block_cache_options:
capacity : 536870912
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: nullptr
whole_key_filtering: 1
format_version: 2
2017-08-09 11:01:35.246758 7fafd0171b80 4 rocksdb:
Options.write_buffer_size: 268435456
2017-08-09 11:01:35.246759 7fafd0171b80 4 rocksdb:
Options.max_write_buffer_number: 4
2017-08-09 11:01:35.246761 7fafd0171b80 4 rocksdb:
Options.compression: NoCompression
2017-08-09 11:01:35.246762 7fafd0171b80 4 rocksdb:
Options.bottommost_compression: Disabled
2017-08-09 11:01:35.246763 7fafd0171b80 4 rocksdb:
Options.prefix_extractor: nullptr
2017-08-09 11:01:35.246764 7fafd0171b80 4 rocksdb:
Options.memtable_insert_with_hint_prefix_extractor: nullptr
2017-08-09 11:01:35.246765 7fafd0171b80 4 rocksdb:
Options.num_levels: 7
2017-08-09 11:01:35.246766 7fafd0171b80 4 rocksdb:
Options.min_write_buffer_number_to_merge: 1
2017-08-09 11:01:35.246767 7fafd0171b80 4 rocksdb:
Options.max_write_buffer_number_to_maintain: 0
2017-08-09 11:01:35.246768 7fafd0171b80 4 rocksdb:
Options.compression_opts.window_bits: -14
2017-08-09 11:01:35.246769 7fafd0171b80 4 rocksdb:
Options.compression_opts.level: -1
2017-08-09 11:01:35.246770 7fafd0171b80 4 rocksdb:
Options.compression_opts.strategy: 0
2017-08-09 11:01:35.246771 7fafd0171b80 4 rocksdb:
Options.compression_opts.max_dict_bytes: 0
2017-08-09 11:01:35.246772 7fafd0171b80 4 rocksdb:
Options.level0_file_num_compaction_trigger: 4
2017-08-09 11:01:35.246775 7fafd0171b80 4 rocksdb:
Options.level0_slowdown_writes_trigger: 20
2017-08-09 11:01:35.246777 7fafd0171b80 4 rocksdb:
Options.level0_stop_writes_trigger: 36
2017-08-09 11:01:35.246778 7fafd0171b80 4 rocksdb:
Options.target_file_size_base: 67108864
2017-08-09 11:01:35.246779 7fafd0171b80 4 rocksdb:
Options.target_file_size_multiplier: 1
2017-08-09 11:01:35.246790 7fafd0171b80 4 rocksdb:
Options.max_bytes_for_level_base: 268435456
2017-08-09 11:01:35.246793 7fafd0171b80 4 rocksdb:
Options.level_compaction_dynamic_level_bytes: 0
2017-08-09 11:01:35.246794 7fafd0171b80 4 rocksdb:
Options.max_bytes_for_level_multiplier: 10.000000
2017-08-09 11:01:35.246799 7fafd0171b80 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[0]: 1
2017-08-09 11:01:35.246801 7fafd0171b80 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[1]: 1
2017-08-09 11:01:35.246802 7fafd0171b80 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[2]: 1
2017-08-09 11:01:35.246803 7fafd0171b80 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[3]: 1
2017-08-09 11:01:35.246804 7fafd0171b80 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[4]: 1
2017-08-09 11:01:35.246805 7fafd0171b80 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[5]: 1
2017-08-09 11:01:35.246806 7fafd0171b80 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[6]: 1
2017-08-09 11:01:35.246807 7fafd0171b80 4 rocksdb:
Options.max_sequential_skip_in_iterations: 8
2017-08-09 11:01:35.246808 7fafd0171b80 4 rocksdb:
Options.max_compaction_bytes: 1677721600
2017-08-09 11:01:35.246809 7fafd0171b80 4 rocksdb:
Options.arena_block_size: 33554432
2017-08-09 11:01:35.246810 7fafd0171b80 4 rocksdb:
Options.soft_pending_compaction_bytes_limit: 68719476736
2017-08-09 11:01:35.246811 7fafd0171b80 4 rocksdb:
Options.hard_pending_compaction_bytes_limit: 274877906944
2017-08-09 11:01:35.246812 7fafd0171b80 4 rocksdb:
Options.rate_limit_delay_max_milliseconds: 100
2017-08-09 11:01:35.246813 7fafd0171b80 4 rocksdb:
Options.disable_auto_compactions: 0
2017-08-09 11:01:35.246814 7fafd0171b80 4 rocksdb:
Options.compaction_style: kCompactionStyleLevel
2017-08-09 11:01:35.246824 7fafd0171b80 4 rocksdb:
Options.compaction_pri: kByCompensatedSize
2017-08-09 11:01:35.246826 7fafd0171b80 4 rocksdb:
Options.compaction_options_universal.size_ratio: 1
2017-08-09 11:01:35.246827 7fafd0171b80 4 rocksdb:
Options.compaction_options_universal.min_merge_width: 2
2017-08-09 11:01:35.246828 7fafd0171b80 4 rocksdb:
Options.compaction_options_universal.max_merge_width: 4294967295
2017-08-09 11:01:35.246829 7fafd0171b80 4 rocksdb:
Options.compaction_options_universal.max_size_amplification_percent: 200
2017-08-09 11:01:35.246831 7fafd0171b80 4 rocksdb:
Options.compaction_options_universal.compression_size_percent: -1
2017-08-09 11:01:35.246832 7fafd0171b80 4 rocksdb:
Options.compaction_options_fifo.max_table_files_size: 1073741824
2017-08-09 11:01:35.246833 7fafd0171b80 4 rocksdb:
Options.table_properties_collectors:
2017-08-09 11:01:35.246834 7fafd0171b80 4 rocksdb:
Options.inplace_update_support: 0
2017-08-09 11:01:35.246835 7fafd0171b80 4 rocksdb:
Options.inplace_update_num_locks: 10000
2017-08-09 11:01:35.246836 7fafd0171b80 4 rocksdb:
Options.memtable_prefix_bloom_size_ratio: 0.000000
2017-08-09 11:01:35.246837 7fafd0171b80 4 rocksdb:
Options.memtable_huge_page_size: 0
2017-08-09 11:01:35.246839 7fafd0171b80 4 rocksdb:
Options.bloom_locality: 0
2017-08-09 11:01:35.246840 7fafd0171b80 4 rocksdb:
Options.max_successive_merges: 0
2017-08-09 11:01:35.246841 7fafd0171b80 4 rocksdb:
Options.optimize_filters_for_hits: 0
2017-08-09 11:01:35.246842 7fafd0171b80 4 rocksdb:
Options.paranoid_file_checks: 0
2017-08-09 11:01:35.246843 7fafd0171b80 4 rocksdb:
Options.force_consistency_checks: 0
2017-08-09 11:01:35.246844 7fafd0171b80 4 rocksdb:
Options.report_bg_io_stats: 0
2017-08-09 11:01:35.473935 7fafd0171b80 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR
CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/
12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/version_set.cc:2859]
Recovered from manifest file:db/MANIFEST-001083
succeeded,manifest_file_number is 1083, next_file_number is 1085,
last_sequence is 53818783, log_number is 0,prev_log_number is
0,max_column_family is 0
2017-08-09 11:01:35.473953 7fafd0171b80 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR
CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/
12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/version_set.cc:2867]
Column family [default] (ID 0), log number is 1082
2017-08-09 11:01:35.474064 7fafd0171b80 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1502269295474050, "job": 1, "event": "recovery_started",
"log_files": [1084]}
2017-08-09 11:01:35.474071 7fafd0171b80 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR
CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/
12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl_open.cc:482]
Recovering log #1084 mode 0
2017-08-09 11:01:36.905449 7fafd0171b80 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1502269296905427, "cf_name": "default", "job": 1,
"event": "table_file_creation", "file_number": 1085, "file_size":
1123741, "table_properties": {"data_size": 1103795, "index_size": 19072,
"filter_size": 0, "raw_key_size": 247455, "raw_average_key_size": 52,
"raw_value_size": 1007202, "raw_average_value_size": 215,
"num_data_blocks": 277, "num_entries": 4674, "filter_policy_name": "",
"kDeletedKeys": "0", "kMergeOperands": "225"}}
2017-08-09 11:01:36.907614 7fafd0171b80 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR
CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/
12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/version_set.cc:2395]
Creating manifest 1086
2017-08-09 11:01:36.949137 7fafd0171b80 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1502269296949132, "job": 1, "event":
"recovery_finished"}
2017-08-09 11:01:36.983107 7fafd0171b80 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR
CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/
12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl_open.cc:1063] DB
pointer 0x7fafd2da2000
2017-08-09 11:01:36.983155 7fafd0171b80 1
bluestore(/var/lib/ceph/osd/ceph-12) _open_db opened rocksdb path db
options
compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_nu
mber_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writa
ble_file_max_buffer_size=0,compaction_readahead_size=2097152
2017-08-09 11:01:37.003047 7fafd0171b80 1 freelist init
2017-08-09 11:01:37.027518 7fafd0171b80 1
bluestore(/var/lib/ceph/osd/ceph-12) _open_alloc opening allocation
metadata
2017-08-09 11:01:41.875967 7fafd0171b80 1
bluestore(/var/lib/ceph/osd/ceph-12) _open_alloc loaded 3718 G in 3275
extents
2017-08-09 11:01:42.524478 7fafd0171b80 1 bluefs fsck
2017-08-09 11:01:42.524502 7fafd0171b80 1
bluestore(/var/lib/ceph/osd/ceph-12) fsck walking object keyspace
2017-08-09 11:02:01.008048 7fafd0171b80 1
bluestore(/var/lib/ceph/osd/ceph-12) fsck checking shared_blobs
2017-08-09 11:02:01.008594 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12) fsck error: found stray shared blob
data for sbid 0xc959
2017-08-09 11:02:01.008811 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12) fsck error: shared blob 0xcadc
ref_map ref_map(0x15a0170000~80000=2) != expected
ref_map(0x15a0170000~80000=1)
2017-08-09 11:02:01.008827 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12)
#17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4# extent
0x15a0170000~80000 or a subset is already allocated
2017-08-09 11:02:01.009713 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12) fsck error: found stray shared blob
data for sbid 0xcdb3
2017-08-09 11:02:01.009735 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12) fsck error: shared blob 0xcdd4
ref_map ref_map(0x7a14b00000~10000=2,0x81357c0000~70000=2) != expected
ref_map(0x7a14b00000~10000=1,0x81357c0000~70000=1)
2017-08-09 11:02:01.009741 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12)
#17:6d8fbe37:::rbd_data.1f114174b0dc51.000000000000006b:4# extent
0x7a14b00000~10000 or a subset is already allocated
2017-08-09 11:02:01.009748 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12)
#17:6d8fbe37:::rbd_data.1f114174b0dc51.000000000000006b:4# extent
0x81357c0000~70000 or a subset is already allocated
2017-08-09 11:02:01.010672 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12) fsck error: found stray shared blob
data for sbid 0xd08b
2017-08-09 11:02:01.016220 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12) fsck error: actual
store_statfs(0x3a12ef00000/0x3a37b2d1000, stored
0x1db028d2b/0x1dd4c0000, compress 0x0/0x0/0x0) != expected
store_statfs(0x3a12ef00000/0x3a37b2d1000, stored
0x1db028d2b/0x1dd5c0000, compress 0x0/0x0/0x0)
2017-08-09 11:02:01.016249 7fafd0171b80 1
bluestore(/var/lib/ceph/osd/ceph-12) fsck checking for stray omap data
2017-08-09 11:02:01.334870 7fafd0171b80 1
bluestore(/var/lib/ceph/osd/ceph-12) fsck checking deferred events
2017-08-09 11:02:01.334934 7fafd0171b80 1
bluestore(/var/lib/ceph/osd/ceph-12) fsck checking freelist vs allocated
2017-08-09 11:02:03.091036 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12) fsck error: free extent
0x159f970000~810000 intersects allocated blocks
2017-08-09 11:02:03.091055 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12) fsck error: free extent
0x15a01b0000~20000 intersects allocated blocks
2017-08-09 11:02:03.091058 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12) fsck error: free extent
0x15a01e0000~10000 intersects allocated blocks
2017-08-09 11:02:25.466606 7fafd0171b80 -1
bluestore(/var/lib/ceph/osd/ceph-12) fsck error: leaked some space;65536
bytes leaked
2017-08-09 11:02:25.513900 7fafd0171b80 1 freelist shutdown
2017-08-09 11:02:25.514045 7fafd0171b80 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR
CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/
12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:217]
Shutdown: canceling all background work
2017-08-09 11:02:25.516412 7fafd0171b80 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR
CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/
12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:343] Shutdown
complete
2017-08-09 11:02:25.851561 7fafd0171b80 1 bluefs umount
2017-08-09 11:02:25.872039 7fafd0171b80 1 bdev(0x7fafd2ac8e00
/var/lib/ceph/osd/ceph-12/block) close
2017-08-09 11:02:26.107504 7fafd0171b80 1 bdev(0x7fafd2ac9200
/var/lib/ceph/osd/ceph-12/block) close
2017-08-09 11:02:26.259819 7fafd0171b80 1
bluestore(/var/lib/ceph/osd/ceph-12) fsck finish with 13 errors in
52.020867 seconds
========================================================================
=========================
objectstore.out
========================================================================
=========================
data section offset=1048576 len=1048576
data section offset=2097152 len=1048576
data section offset=3145728 len=1048576
attrs size 2
omap map size 0
Read #17:6ca10b29:::rbd_data.1fff61238e1f29.0000000000009923:head#
size=4194304
object_info:
17:6ca10b29:::rbd_data.1fff61238e1f29.0000000000009923:head(5387'35157
client.2096993.0:78941 dirty|data_digest|omap_digest s 4194304 uv 35356
dd f53dff2e od ffffffff alloc_hint [4194304 4194304 0]) data section
offset=0 len=1048576 data section offset=1048576 len=1048576 data
section offset=2097152 len=1048576 data section offset=3145728
len=1048576 attrs size 2 omap map size 0 Read
#17:6ca11ab9:::rbd_data.1fa8ef2ae8944a.00000000000011b4:head#
size=4194304
object_info:
17:6ca11ab9:::rbd_data.1fa8ef2ae8944a.00000000000011b4:head(5163'7136
client.2074638.1:483264 dirty|data_digest|omap_digest s 4194304 uv 7418
dd 43d61c5d od ffffffff alloc_hint [4194304 4194304 0]) data section
offset=0 len=1048576 data section offset=1048576 len=1048576 data
section offset=2097152 len=1048576 data section offset=3145728
len=1048576 attrs size 2 omap map size 0 Read
#17:6ca13bed:::rbd_data.1f114174b0dc51.00000000000002c6:head#
size=4194304
object_info:
17:6ca13bed:::rbd_data.1f114174b0dc51.00000000000002c6:head(5236'7640
client.2074638.1:704364 dirty|data_digest|omap_digest s 4194304 uv 7922
dd 3bcff64d od ffffffff alloc_hint [4194304 4194304 0]) data section
offset=0 len=1048576 data section offset=1048576 len=1048576 data
section offset=2097152 len=1048576 data section offset=3145728
len=1048576 attrs size 2 omap map size 0 Read
#17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head#
size=4194304
object_info:
17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head(5387'35553
client.2096993.0:123721 dirty|data_digest|omap_digest s 4194304 uv 35752
dd f9bc0fbd od ffffffff alloc_hint [4194304 4194304 0]) data section
offset=0 len=1048576 data section offset=1048576 len=1048576 data
section offset=2097152 len=1048576 data section offset=3145728
len=1048576 attrs size 2 omap map size 0 Read
#17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4#
size=4194304
object_info:
17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4(5390'56613
client.2096907.1:3222443 dirty|omap_digest s 4194304 uv 55477 od
ffffffff alloc_hint [0 0 0]) export_files error -5
2017-08-09 11:41:25.273003 7f26db8ae100 -1
bluestore(/var/lib/ceph/osd/ceph-0) _verify_csum bad crc32c/0x1000
checksum at blob offset 0x0, got 0x1a128a93, expected 0x90407f75, device
location [0x5826c10000~1000], logical extent 0x0~1000, object
#17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4#
2017-08-09 11:41:25.286087 7f26db8ae100 1
bluestore(/var/lib/ceph/osd/ceph-0) umount
2017-08-09 11:41:25.455280 7f26db8ae100 1 freelist shutdown
2017-08-09 11:41:25.455386 7f26db8ae100 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR
CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/
12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:217]
Shutdown: canceling all background work
2017-08-09 11:41:25.471514 7f26db8ae100 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR
CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/
12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:343] Shutdown
complete
2017-08-09 11:41:25.686088 7f26db8ae100 1 bluefs umount
2017-08-09 11:41:25.705389 7f26db8ae100 1 bdev(0x7f26de472e00
/var/lib/ceph/osd/ceph-0/block) close
2017-08-09 11:41:25.944548 7f26db8ae100 1 bdev(0x7f26de2b3a00
/var/lib/ceph/osd/ceph-0/block) close
-----Original Message-----
From: Sage Weil [mailto:sage@xxxxxxxxxxxx]
Sent: woensdag 9 augustus 2017 4:44
To: Brad Hubbard
Cc: Marc Roos; ceph-users
Subject: Re: Pg inconsistent / export_files error -5
On Wed, 9 Aug 2017, Brad Hubbard wrote:
> Wee
>
> On Wed, Aug 9, 2017 at 12:41 AM, Marc Roos <M.Roos@xxxxxxxxxxxxxxxxx>
wrote:
> >
> >
> >
> > The --debug indeed comes up with something
> > bluestore(/var/lib/ceph/osd/ceph-12) _verify_csum bad crc32c/0x1000
> > checksum at blob offset 0x0, got 0x100ac314, expected 0x90407f75,
> > device location [0x15a0170000~1000], logical extent 0x0~1000,
> > bluestore(/var/lib/ceph/osd/ceph-9) _verify_csum bad crc32c/0x1000
> > checksum at blob offset 0x0, got 0xb40b26a7, expected 0x90407f75,
> > device location [0x2daea0000~1000], logical extent 0x0~1000,
What about the 3rd OSD?
It would be interesting to capture the fsck output for one of these.
Stop the OSD, and then run
ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-12 --log-file
out \
--debug-bluestore 30 --no-log-to-stderr
That'll generate a pretty huge log, but should include dumps of onode
metadata and will hopefully include something else with the checksum of
0x100ac314 so we can get some clue as to where the bad data came from.
Thanks!
sage
> >
> > I dont know how to interpret this, but am I correct to understand
> > that data has been written across the cluster to these 3 osd's and
> > all 3 have somehow received something different?
>
> Did you run this command on OSD 0? What was the output in that case?
>
> Possibly, all we currently know for sure is that the crc32c checksum
> for the object on OSDs 12 and 9 do not match the expected checksum
> according to the code when we attempt to read the object
> #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4#. There
> seems to be some history behind this based on your previous emails
> regarding these OSDs (12,9,0, and possibly 13) could you give us as
> much detail as possible about how this issue came about and what you
> have done in the interim to try to resolve it?
>
> When was the first indication there was a problem with pg 17.36? Did
> this correspond with any significant event?
>
> Are these OSDs all on separate hosts?
>
> It's possible ceph-bluestore-tool may help here but I would hold off
> on that option until we understand the issue better.
>
>
> >
> >
> > size=4194304 object_info:
> > 17:6ca10b29:::rbd_data.1fff61238e1f29.0000000000009923:head(5387'351
> > 57
> > client.2096993.0:78941 dirty|data_digest|omap_digest s 4194304 uv
> > 35356 dd f53dff2e od ffffffff alloc_hint [4194304 4194304 0]) data
> > section offset=0
> > len=1048576 data section offset=1048576 len=1048576 data section
> > offset=2097152 len=1048576 data section offset=3145728 len=1048576
> > attrs size
> > 2 omap map size 0 Read
> > #17:6ca11ab9:::rbd_data.1fa8ef2ae8944a.00000000000011b4:head#
> > size=4194304
> > object_info:
> > 17:6ca11ab9:::rbd_data.1fa8ef2ae8944a.00000000000011b4:head(5163'713
> > 6
> > client.2074638.1:483264 dirty|data_digest|omap_digest s 4194304 uv
> > 7418 dd 43d61c5d od ffffffff alloc_hint [4194304 4194304 0]) data
> > section offset=0
> > len=1048576 data section offset=1048576 len=1048576 data section
> > offset=2097152 len=1048576 data section offset=3145728 len=1048576
> > attrs size
> > 2 omap map size 0 Read
> > #17:6ca13bed:::rbd_data.1f114174b0dc51.00000000000002c6:head#
> > size=4194304
> > object_info:
> > 17:6ca13bed:::rbd_data.1f114174b0dc51.00000000000002c6:head(5236'764
> > 0
> > client.2074638.1:704364 dirty|data_digest|omap_digest s 4194304 uv
> > 7922 dd 3bcff64d od ffffffff alloc_hint [4194304 4194304 0]) data
> > section offset=0
> > len=1048576 data section offset=1048576 len=1048576 data section
> > offset=2097152 len=1048576 data section offset=3145728 len=1048576
> > attrs size
> > 2 omap map size 0 Read
> > #17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head#
> > size=4194304
> > object_info:
> > 17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head(5387'355
> > 53
> > client.2096993.0:123721 dirty|data_digest|omap_digest s 4194304 uv
> > 35752 dd f9bc0fbd od ffffffff alloc_hint [4194304 4194304 0]) data
> > section offset=0
> > len=1048576 data section offset=1048576 len=1048576 data section
> > offset=2097152 len=1048576 data section offset=3145728 len=1048576
> > attrs size
> > 2 omap map size 0 Read
> > #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4#
> > size=4194304
> > object_info:
> > 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4(5390'56613
> > client.2096907.1:3222443 dirty|omap_digest s 4194304 uv 55477 od
> > ffffffff alloc_hint [0 0 0]) 2017-08-08 15:57:45.078348 7fad08fa4100
> > -1
> > bluestore(/var/lib/ceph/osd/ceph-12) _verify_csum bad crc32c/0x1000
> > checksum at blob offset 0x0, got 0x100ac314, expected 0x90407f75,
> > device location [0x15a0170000~1000], logical extent 0x0~1000, object
> > #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4#
> > export_files error
> > -5 2017-08-08 15:57:45.081279 7fad08fa4100 1
> > bluestore(/var/lib/ceph/osd/ceph-12) umount 2017-08-08
> > 15:57:45.150210 7fad08fa4100 1 freelist shutdown 2017-08-08
> > 15:57:45.150307 7fad08fa4100 4
> > rocksdb:
> > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABL
> > E_AR
> > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rele
> > ase/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:217]
> > Shutdown:
> > canceling all background work 2017-08-08 15:57:45.152099
> > 7fad08fa4100 4
> > rocksdb:
> > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABL
> > E_AR
> > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rele
> > ase/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:343]
> > Shutdown complete 2017-08-08 15:57:45.184742 7fad08fa4100 1 bluefs
> > umount 2017-08-08
> > 15:57:45.203674 7fad08fa4100 1 bdev(0x7fad0b260e00
> > /var/lib/ceph/osd/ceph-12/block) close 2017-08-08 15:57:45.442499
> > 7fad08fa4100
> > 1 bdev(0x7fad0b0a5a00 /var/lib/ceph/osd/ceph-12/block) close
> >
> > grep -i export_files strace.out -C 10
> >
> > 814 16:08:19.261144 futex(0x7fffea9378c0, FUTEX_WAKE_PRIVATE, 1) =
> > 0 <0.000010> 6814 16:08:19.261242 futex(0x7f4832bb60bc,
> > FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f4832bb60b8, {FUTEX_OP_SET, 0,
> > FUTEX_OP_CMP_GT, 1}) = 1 <0.000012>
> > 6814 16:08:19.261281 madvise(0x7f4843bf0000, 524288, MADV_DONTNEED
> > <unfinished ...> 6815 16:08:19.261382 <... futex resumed> ) = 0
> > <14.990766>
> > 6814 16:08:19.261412 <... madvise resumed> ) = 0 <0.000123> 6814
> > 16:08:19.261446 madvise(0x7f4843b70000, 1048576, MADV_DONTNEED
> > <unfinished ...> 6815 16:08:19.261474 futex(0x7f4832bb6038,
> > FUTEX_WAKE_PRIVATE, 1 <unfinished ...> 6814 16:08:19.261535 <...
> > madvise resumed> ) = 0 <0.000067>
> > 6815 16:08:19.261557 <... futex resumed> ) = 0 <0.000069> 6815
> > 16:08:19.261647 futex(0x7f4832bb60bc, FUTEX_WAIT_PRIVATE, 45, NULL
> > <unfinished ...> 6814 16:08:19.261700 write(2</dev/pts/0>,
> > "export_files error ", 19) =
> > 19 <0.000024> 6814 16:08:19.261774 write(2</dev/pts/0>, "-5", 2) =
> > 2 <0.000018> 6814 16:08:19.261841 write(2</dev/pts/0>, "\n", 1) = 1
> > <0.000016>
> > 6814 16:08:19.262191 madvise(0x7f4839106000, 16384, MADV_DONTNEED)
> > = 0 <0.000015> 6814 16:08:19.262229 madvise(0x7f483914e000, 16384,
> > MADV_DONTNEED) = 0 <0.000012> 6814 16:08:19.262295
> > madvise(0x7f48389e6000, 49152,
> > MADV_DONTNEED) = 0 <0.000013> 6814 16:08:19.262498
> > madvise(0x7f48390ea000, 16384, MADV_DONTNEED) = 0 <0.000013> 6814
> > 16:08:19.262538 madvise(0x7f48390ce000, 16384, MADV_DONTNEED) = 0
> > <0.000012> 6814 16:08:19.262580 madvise(0x7f483c228000, 24576,
> > MADV_DONTNEED) = 0 <0.000012>
> > 6814 16:08:19.263047 madvise(0x7f48393d8000, 16384, MADV_DONTNEED)
> > = 0 <0.000013> 6814 16:08:19.263081 madvise(0x7f48393d8000, 32768,
> > MADV_DONTNEED) = 0 <0.000016>
> >
> >
> > I was curious how this would compare to the osd.9
> >
> > object_info:
> > 17:6ca13bed:::rbd_data.1f114174b0dc51.00000000000002c6:head(5236'764
> > 0
> > client.2074638.1:704364 dirty|data_digest|omap_digest s 4194304 uv
> > 7922 dd 3bcff64d od ffffffff alloc_hint [4194304 4194304 0]) data
> > section offset=0
> > len=1048576 data section offset=1048576 len=1048576 data section
> > offset=2097152 len=1048576 data section offset=3145728 len=1048576
> > attrs size
> > 2 omap map size 0 Read
> > #17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head#
> > size=4194304
> > object_info:
> > 17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head(5387'355
> > 53
> > client.2096993.0:123721 dirty|data_digest|omap_digest s 4194304 uv
> > 35752 dd f9bc0fbd od ffffffff alloc_hint [4194304 4194304 0]) data
> > section offset=0
> > len=1048576 data section offset=1048576 len=1048576 data section
> > offset=2097152 len=1048576 data section offset=3145728 len=1048576
> > attrs size
> > 2 omap map size 0 Read
> > #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4#
> > size=4194304
> > object_info:
> > 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4(5390'56613
> > client.2096907.1:3222443 dirty|omap_digest s 4194304 uv 55477 od
> > ffffffff alloc_hint [0 0 0]) 2017-08-08 16:22:00.893216 7f94e10f5100
> > -1
> > bluestore(/var/lib/ceph/osd/ceph-9) _verify_csum bad crc32c/0x1000
> > checksum at blob offset 0x0, got 0xb40b26a7, expected 0x90407f75,
> > device location [0x2daea0000~1000], logical extent 0x0~1000, object
> > #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4#
> > export_files error
> > -5 2017-08-08 16:22:00.895439 7f94e10f5100 1
> > bluestore(/var/lib/ceph/osd/ceph-9) umount 2017-08-08
> > 16:22:00.963774 7f94e10f5100 1 freelist shutdown 2017-08-08
> > 16:22:00.963861 7f94e10f5100 4
> > rocksdb:
> > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABL
> > E_AR
> > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rele
> > ase/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:217]
> > Shutdown:
> > canceling all background work 2017-08-08 16:22:00.968438
> > 7f94e10f5100 4
> > rocksdb:
> > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABL
> > E_AR
> > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rele
> > ase/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:343]
> > Shutdown complete 2017-08-08 16:22:00.984583 7f94e10f5100 1 bluefs
> > umount 2017-08-08
> > 16:22:01.026784 7f94e10f5100 1 bdev(0x7f94e3670e00
> > /var/lib/ceph/osd/ceph-9/block) close 2017-08-08 16:22:01.243361
> > 7f94e10f5100
> > 1 bdev(0x7f94e34b5a00 /var/lib/ceph/osd/ceph-9/block) close
> >
> >
> > 23555 16:26:31.336061 io_getevents(139955679129600, 1, 16,
> > <unfinished ...>
> > 23552 16:26:31.336081 futex(0x7ffe7e4c9210, FUTEX_WAKE_PRIVATE, 1) =
> > 0 <0.000155> 23552 16:26:31.336452 futex(0x7f49fb4d20bc,
> > FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f49fb4d20b8, {FUTEX_OP_SET, 0,
> > FUTEX_OP_CMP_GT, 1}) = 1 <0.000129>
> > 23553 16:26:31.336637 <... futex resumed> ) = 0 <16.434259> 23553
> > 16:26:31.336758 futex(0x7f49fb4d2038, FUTEX_WAKE_PRIVATE, 1
> > <unfinished ...>
> > 23552 16:26:31.336801 madvise(0x7f4a0cafa000, 2555904, MADV_DONTNEED
> > <unfinished ...> 23553 16:26:31.336915 <... futex resumed> ) = 0
> > <0.000113>
> > 23552 16:26:31.336959 <... madvise resumed> ) = 0 <0.000148> 23553
> > 16:26:31.337040 futex(0x7f49fb4d20bc, FUTEX_WAIT_PRIVATE, 55, NULL
> > <unfinished ...> 23552 16:26:31.337070 madvise(0x7f4a0ca7a000,
> > 3080192, MADV_DONTNEED) = 0 <0.000180> 23552 16:26:31.337424
> > write(2</dev/pts/1>, "export_files error ",
> > 19) = 19 <0.000104> 23552 16:26:31.337615 write(2</dev/pts/1>, "-5",
> > 2) = 2 <0.000017> 23552 16:26:31.337674 write(2</dev/pts/1>, "\n",
> > 1) = 1 <0.000037>
> > 23552 16:26:31.338270 madvise(0x7f4a01ae4000, 16384, MADV_DONTNEED)
> > = 0 <0.000020> 23552 16:26:31.338320 madvise(0x7f4a018cc000, 49152,
> > MADV_DONTNEED) = 0 <0.000014> 23552 16:26:31.338561
> > madvise(0x7f4a0770a000, 24576,
> > MADV_DONTNEED) = 0 <0.000015> 23552 16:26:31.339161
> > madvise(0x7f4a02102000, 16384, MADV_DONTNEED) = 0 <0.000015> 23552
> > 16:26:31.339201 madvise(0x7f4a02132000, 16384, MADV_DONTNEED) = 0
> > <0.000013> 23552
> > 16:26:31.339235 madvise(0x7f4a02102000, 32768, MADV_DONTNEED) = 0
> > <0.000014>
> > 23552 16:26:31.339331 madvise(0x7f4a01df8000, 16384, MADV_DONTNEED)
> > = 0 <0.000019> 23552 16:26:31.339372 madvise(0x7f4a01df8000, 32768,
> > MADV_DONTNEED) = 0 <0.000013>
> >
> >
> > -----Original Message----- From: Brad Hubbard
> > [mailto:bhubbard@xxxxxxxxxx]
> > Sent: 07 August 2017 02:34 To: Marc Roos Cc: ceph-users Subject: Re:
> > Pg inconsistent / export_files error -5
> >
> >
> >
> > On Sat, Aug 5, 2017 at 1:21 AM, Marc Roos <M.Roos@xxxxxxxxxxxxxxxxx>
wrote:
> >>
> >> I have got a placement group inconsistency, and saw some manual
> >> where you can export and import this on another osd. But I am
> >> getting an export error on every osd.
> >>
> >> What does this export_files error -5 actually mean? I thought 3
> >> copies
> >
> > #define EIO 5 /* I/O error */
> >
> >> should be enough to secure your data.
> >>
> >>
> >>> PG_DAMAGED Possible data damage: 1 pg inconsistent pg 17.36 is
> >>> active+clean+inconsistent, acting [9,0,12]
> >>
> >>
> >>> 2017-08-04 05:39:51.534489 7f2f623d6700 -1 log_channel(cluster)
> >>> log
> >> [ERR] : 17.36 soid
17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4:
> >> failed to pick suitable object info
> >>> 2017-08-04 05:41:12.715393 7f2f623d6700 -1 log_channel(cluster)
> >>> log
> >> [ERR] : 17.36 deep-scrub 3 errors
> >>> 2017-08-04 15:21:12.445799 7f2f623d6700 -1 log_channel(cluster)
> >>> log
> >> [ERR] : 17.36 soid
17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4:
> >> failed to pick suitable object info
> >>> 2017-08-04 15:22:35.646635 7f2f623d6700 -1 log_channel(cluster)
> >>> log
> >> [ERR] : 17.36 repair 3 errors, 0 fixed
> >>
> >> ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 --pgid
> >> 17.36 --op export --file /tmp/recover.17.36
> >
> > Can you run this command under strace like so?
> >
> > # strace -fvttyyTo /tmp/strace.out -s 1024 ceph-objectstore-tool
> > --data-path
> > /var/lib/ceph/osd/ceph-12 --pgid 17.36 --op export --file
> > /tmp/recover.17.36
> >
> > Then see if you can find which syscall is returning EIO.
> >
> > # grep "= \-5" /tmp/strace.out
> >
> >>
> >> ... Read
> >> #17:6c9f811c:::rbd_data.1b42f52ae8944a.0000000000001a32:head# Read
> >> #17:6ca035fc:::rbd_data.1fff61238e1f29.000000000000b31a:head# Read
> >> #17:6ca0b4f8:::rbd_data.1fff61238e1f29.0000000000006fcc:head# Read
> >> #17:6ca0ffbc:::rbd_data.1fff61238e1f29.000000000000a214:head# Read
> >> #17:6ca10b29:::rbd_data.1fff61238e1f29.0000000000009923:head# Read
> >> #17:6ca11ab9:::rbd_data.1fa8ef2ae8944a.00000000000011b4:head# Read
> >> #17:6ca13bed:::rbd_data.1f114174b0dc51.00000000000002c6:head# Read
> >> #17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head# Read
> >> #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4#
> >> export_files error
> >> -5
> >
> > Running the command with "--debug" appended will give more output
> > which may shed more light as well.
> >
> >> _______________________________________________ ceph-users mailing
> >> list ceph-users@xxxxxxxxxxxxxx
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
> > -- Cheers, Brad
> >
> >
>
>
>
> --
> Cheers,
> Brad
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
