-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
I don't think this does what you think it does.....
This will almost certainly starve the client of IO. This is the number
of seconds between backfills, not the number of objects being scanned
during a backfill. Setting these to higher values will make recovery
take longer, but hardly affect the client. Setting these to low values
will increase the rate of recovery so it takes less time, but will
impact the performance of the clients.
Also, I haven't had much luck changing these on the fly for
recovery/backfill already in progress or queued.
- ----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Wed, Nov 25, 2015 at 2:42 PM, Tomasz Kuzemko wrote:
> To ease on clients you can change osd_backfill_scan_min and
> osd_backfill_scan_max to 1. It's possible to change this online:
> ceph tell osd.\* injectargs '--osd_backfill_scan_min 1'
> ceph tell osd.\* injectargs '--osd_backfill_scan_max 1'
>
> 2015-11-24 16:52 GMT+01:00 Joe Ryner :
>>
>> Hi,
>>
>> Last night I upgraded my cluster from Centos 6.5 -> Centos 7.1 and in the
>> process upgraded from Emperor -> Firefly -> Hammer
>>
>> When I finished I changed the crush tunables from
>> ceph osd crush tunables legacy -> ceph osd crush tunables optimal
>>
>> I knew this would cause data movement. But the IO for my clients is
>> unacceptable. Can any please tell what the best settings are for my
>> configuration. I have 2 Dell R720 Servers and 2 Dell R730 servers. I have
>> 36 1TB SATA SSD Drives in my cluster. The servers have 128 GB of RAM.
>>
>> Below is some detail the might help. According to my calculations the
>> rebalance will take over a day.
>>
>> I would greatly appreciate some help on this.
>>
>> Thank you,
>>
>> Joe
>>
>> -----
>> BEGIN --------------------------------------------------------------
>> NODE: gold.sys.cu.cait.org
>> CMD : free -m
>> total used free shared buff/cache
>> available
>> Mem: 128726 71316 2077 20 55332
>> 56767
>> Swap: 0 0 0
>> END --------------------------------------------------------------
>> BEGIN --------------------------------------------------------------
>> NODE: gallo.sys.cu.cait.org
>> CMD : free -m
>> total used free shared buff/cache
>> available
>> Mem: 128726 79489 462 36 48774
>> 48547
>> Swap: 8191 0 8191
>> END --------------------------------------------------------------
>> BEGIN --------------------------------------------------------------
>> NODE: hamms.sys.cu.cait.org
>> CMD : free -m
>> total used free shared buff/cache
>> available
>> Mem: 128536 69412 659 19 58464
>> 58342
>> Swap: 16383 0 16383
>> END --------------------------------------------------------------
>> BEGIN --------------------------------------------------------------
>> NODE: helm.sys.cu.cait.org
>> CMD : free -m
>> total used free shared buff/cache
>> available
>> Mem: 128536 66216 8799 26 53520
>> 61603
>> Swap: 16383 1739 14644
>> END --------------------------------------------------------------
>>
>> ceph osd tree
>> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
>> -1 36.00000 root default
>> -3 36.00000 rack curack
>> -10 9.00000 host helm
>> 0 1.00000 osd.0 up 1.00000 1.00000
>> 1 1.00000 osd.1 up 1.00000 1.00000
>> 2 1.00000 osd.2 up 1.00000 1.00000
>> 3 1.00000 osd.3 up 1.00000 1.00000
>> 4 1.00000 osd.4 up 1.00000 1.00000
>> 5 1.00000 osd.5 up 1.00000 1.00000
>> 6 1.00000 osd.6 up 1.00000 1.00000
>> 7 1.00000 osd.7 up 1.00000 1.00000
>> 8 1.00000 osd.8 up 1.00000 1.00000
>> -7 9.00000 host gold
>> 16 1.00000 osd.16 up 1.00000 1.00000
>> 17 1.00000 osd.17 up 1.00000 1.00000
>> 18 1.00000 osd.18 up 1.00000 1.00000
>> 19 1.00000 osd.19 up 1.00000 1.00000
>> 20 1.00000 osd.20 up 1.00000 1.00000
>> 21 1.00000 osd.21 up 1.00000 1.00000
>> 9 1.00000 osd.9 up 1.00000 1.00000
>> 10 1.00000 osd.10 up 1.00000 1.00000
>> 34 1.00000 osd.34 up 1.00000 1.00000
>> -8 9.00000 host gallo
>> 22 1.00000 osd.22 up 1.00000 1.00000
>> 23 1.00000 osd.23 up 1.00000 1.00000
>> 24 1.00000 osd.24 up 1.00000 1.00000
>> 25 1.00000 osd.25 up 1.00000 1.00000
>> 26 1.00000 osd.26 up 1.00000 1.00000
>> 27 1.00000 osd.27 up 1.00000 1.00000
>> 11 1.00000 osd.11 up 1.00000 1.00000
>> 12 1.00000 osd.12 up 1.00000 1.00000
>> 35 1.00000 osd.35 up 1.00000 1.00000
>> -9 9.00000 host hamms
>> 13 1.00000 osd.13 up 1.00000 1.00000
>> 14 1.00000 osd.14 up 1.00000 1.00000
>> 15 1.00000 osd.15 up 1.00000 1.00000
>> 28 1.00000 osd.28 up 1.00000 1.00000
>> 29 1.00000 osd.29 up 1.00000 1.00000
>> 30 1.00000 osd.30 up 1.00000 1.00000
>> 31 1.00000 osd.31 up 1.00000 1.00000
>> 32 1.00000 osd.32 up 1.00000 1.00000
>> 33 1.00000 osd.33 up 1.00000 1.00000
>>
>> ceph -s
>> cluster 0c4fe157-88d3-4e83-8603-4dae31f63af6
>> health HEALTH_WARN
>> 2767 pgs backfill
>> 7 pgs backfilling
>> 2774 pgs stuck unclean
>> recovery 328/4281405 objects degraded (0.008%)
>> recovery 2476616/4281405 objects misplaced (57.846%)
>> monmap e34: 3 mons at
>> {gold=10.66.4.7:6789/0,hamms=10.66.4.9:6789/0,helm=10.66.4.4:6789/0}
>> election epoch 824, quorum 0,1,2 helm,gold,hamms
>> mdsmap e2870: 0/0/1 up
>> osdmap e116330: 36 osds: 36 up, 36 in; 2774 remapped pgs
>> pgmap v73418676: 9166 pgs, 7 pools, 3865 GB data, 990 kobjects
>> 11682 GB used, 19765 GB / 31448 GB avail
>> 328/4281405 objects degraded (0.008%)
>> 2476616/4281405 objects misplaced (57.846%)
>> 6392 active+clean
>> 2767 active+remapped+wait_backfill
>> 7 active+remapped+backfilling
>> recovery io 77061 kB/s, 19 objects/s
>> client io 118 kB/s rd, 956 kB/s wr, 265 op/s
>>
>>
>> ------
>> ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show
>> {
>> "name": "osd.0",
>> "cluster": "ceph",
>> "debug_none": "0\/5",
>> "debug_lockdep": "0\/1",
>> "debug_context": "0\/1",
>> "debug_crush": "1\/1",
>> "debug_mds": "1\/5",
>> "debug_mds_balancer": "1\/5",
>> "debug_mds_locker": "1\/5",
>> "debug_mds_log": "1\/5",
>> "debug_mds_log_expire": "1\/5",
>> "debug_mds_migrator": "1\/5",
>> "debug_buffer": "0\/1",
>> "debug_timer": "0\/1",
>> "debug_filer": "0\/1",
>> "debug_striper": "0\/1",
>> "debug_objecter": "0\/1",
>> "debug_rados": "0\/5",
>> "debug_rbd": "0\/5",
>> "debug_rbd_replay": "0\/5",
>> "debug_journaler": "0\/5",
>> "debug_objectcacher": "0\/5",
>> "debug_client": "0\/5",
>> "debug_osd": "0\/5",
>> "debug_optracker": "0\/5",
>> "debug_objclass": "0\/5",
>> "debug_filestore": "1\/3",
>> "debug_keyvaluestore": "1\/3",
>> "debug_journal": "1\/3",
>> "debug_ms": "0\/5",
>> "debug_mon": "1\/5",
>> "debug_monc": "0\/10",
>> "debug_paxos": "1\/5",
>> "debug_tp": "0\/5",
>> "debug_auth": "1\/5",
>> "debug_crypto": "1\/5",
>> "debug_finisher": "1\/1",
>> "debug_heartbeatmap": "1\/5",
>> "debug_perfcounter": "1\/5",
>> "debug_rgw": "1\/5",
>> "debug_civetweb": "1\/10",
>> "debug_javaclient": "1\/5",
>> "debug_asok": "1\/5",
>> "debug_throttle": "1\/1",
>> "debug_refs": "0\/0",
>> "debug_xio": "1\/5",
>> "host": "helm",
>> "fsid": "00000000-0000-0000-0000-000000000000",
>> "public_addr": "10.66.4.4:0\/0",
>> "cluster_addr": "10.66.4.4:0\/0",
>> "public_network": "10.66.4.0\/24",
>> "cluster_network": "10.66.4.0\/24",
>> "num_client": "1",
>> "monmap": "",
>> "mon_host": "",
>> "lockdep": "false",
>> "lockdep_force_backtrace": "false",
>> "run_dir": "\/var\/run\/ceph",
>> "admin_socket": "\/var\/run\/ceph\/ceph-osd.0.asok",
>> "daemonize": "false",
>> "pid_file": "\/var\/run\/ceph\/osd.0.pid",
>> "chdir": "\/",
>> "max_open_files": "0",
>> "restapi_log_level": "",
>> "restapi_base_url": "",
>> "fatal_signal_handlers": "true",
>> "log_file": "\/var\/log\/ceph\/ceph-osd.0.log",
>> "log_max_new": "1000",
>> "log_max_recent": "10000",
>> "log_to_stderr": "false",
>> "err_to_stderr": "true",
>> "log_to_syslog": "false",
>> "err_to_syslog": "false",
>> "log_flush_on_exit": "true",
>> "log_stop_at_utilization": "0.97",
>> "clog_to_monitors": "default=true",
>> "clog_to_syslog": "false",
>> "clog_to_syslog_level": "info",
>> "clog_to_syslog_facility": "default=daemon audit=local0",
>> "mon_cluster_log_to_syslog": "default=false",
>> "mon_cluster_log_to_syslog_level": "info",
>> "mon_cluster_log_to_syslog_facility": "daemon",
>> "mon_cluster_log_file": "default=\/var\/log\/ceph\/ceph.$channel.log
>> cluster=\/var\/log\/ceph\/ceph.log",
>> "mon_cluster_log_file_level": "info",
>> "enable_experimental_unrecoverable_data_corrupting_features": "",
>> "xio_trace_mempool": "false",
>> "xio_trace_msgcnt": "false",
>> "xio_trace_xcon": "false",
>> "xio_queue_depth": "512",
>> "xio_mp_min": "128",
>> "xio_mp_max_64": "65536",
>> "xio_mp_max_256": "8192",
>> "xio_mp_max_1k": "8192",
>> "xio_mp_max_page": "4096",
>> "xio_mp_max_hint": "4096",
>> "xio_portal_threads": "2",
>> "key": "",
>> "keyfile": "",
>> "keyring": "\/var\/lib\/ceph\/osd\/ceph-0\/keyring",
>> "heartbeat_interval": "5",
>> "heartbeat_file": "",
>> "heartbeat_inject_failure": "0",
>> "perf": "true",
>> "ms_type": "simple",
>> "ms_tcp_nodelay": "true",
>> "ms_tcp_rcvbuf": "0",
>> "ms_tcp_prefetch_max_size": "4096",
>> "ms_initial_backoff": "0.2",
>> "ms_max_backoff": "15",
>> "ms_crc_data": "true",
>> "ms_crc_header": "true",
>> "ms_die_on_bad_msg": "false",
>> "ms_die_on_unhandled_msg": "false",
>> "ms_die_on_old_message": "false",
>> "ms_die_on_skipped_message": "false",
>> "ms_dispatch_throttle_bytes": "104857600",
>> "ms_bind_ipv6": "false",
>> "ms_bind_port_min": "6800",
>> "ms_bind_port_max": "7300",
>> "ms_bind_retry_count": "3",
>> "ms_bind_retry_delay": "5",
>> "ms_rwthread_stack_bytes": "1048576",
>> "ms_tcp_read_timeout": "900",
>> "ms_pq_max_tokens_per_priority": "16777216",
>> "ms_pq_min_cost": "65536",
>> "ms_inject_socket_failures": "0",
>> "ms_inject_delay_type": "",
>> "ms_inject_delay_msg_type": "",
>> "ms_inject_delay_max": "1",
>> "ms_inject_delay_probability": "0",
>> "ms_inject_internal_delays": "0",
>> "ms_dump_on_send": "false",
>> "ms_dump_corrupt_message_level": "1",
>> "ms_async_op_threads": "2",
>> "ms_async_set_affinity": "true",
>> "ms_async_affinity_cores": "",
>> "inject_early_sigterm": "false",
>> "mon_data": "\/var\/lib\/ceph\/mon\/ceph-0",
>> "mon_initial_members": "",
>> "mon_sync_fs_threshold": "5",
>> "mon_compact_on_start": "false",
>> "mon_compact_on_bootstrap": "false",
>> "mon_compact_on_trim": "true",
>> "mon_osd_cache_size": "10",
>> "mon_tick_interval": "5",
>> "mon_subscribe_interval": "300",
>> "mon_delta_reset_interval": "10",
>> "mon_osd_laggy_halflife": "3600",
>> "mon_osd_laggy_weight": "0.3",
>> "mon_osd_adjust_heartbeat_grace": "true",
>> "mon_osd_adjust_down_out_interval": "true",
>> "mon_osd_auto_mark_in": "false",
>> "mon_osd_auto_mark_auto_out_in": "true",
>> "mon_osd_auto_mark_new_in": "true",
>> "mon_osd_down_out_interval": "300",
>> "mon_osd_down_out_subtree_limit": "rack",
>> "mon_osd_min_up_ratio": "0.3",
>> "mon_osd_min_in_ratio": "0.3",
>> "mon_osd_max_op_age": "32",
>> "mon_osd_max_split_count": "32",
>> "mon_osd_allow_primary_temp": "false",
>> "mon_osd_allow_primary_affinity": "false",
>> "mon_stat_smooth_intervals": "2",
>> "mon_lease": "5",
>> "mon_lease_renew_interval": "3",
>> "mon_lease_ack_timeout": "10",
>> "mon_clock_drift_allowed": "0.05",
>> "mon_clock_drift_warn_backoff": "5",
>> "mon_timecheck_interval": "300",
>> "mon_accept_timeout": "10",
>> "mon_pg_create_interval": "30",
>> "mon_pg_stuck_threshold": "300",
>> "mon_pg_warn_min_per_osd": "30",
>> "mon_pg_warn_max_per_osd": "300",
>> "mon_pg_warn_max_object_skew": "10",
>> "mon_pg_warn_min_objects": "10000",
>> "mon_pg_warn_min_pool_objects": "1000",
>> "mon_cache_target_full_warn_ratio": "0.66",
>> "mon_osd_full_ratio": "0.98",
>> "mon_osd_nearfull_ratio": "0.94",
>> "mon_allow_pool_delete": "true",
>> "mon_globalid_prealloc": "10000",
>> "mon_osd_report_timeout": "900",
>> "mon_force_standby_active": "true",
>> "mon_warn_on_old_mons": "true",
>> "mon_warn_on_legacy_crush_tunables": "true",
>> "mon_warn_on_osd_down_out_interval_zero": "true",
>> "mon_warn_on_cache_pools_without_hit_sets": "true",
>> "mon_min_osdmap_epochs": "500",
>> "mon_max_pgmap_epochs": "500",
>> "mon_max_log_epochs": "500",
>> "mon_max_mdsmap_epochs": "500",
>> "mon_max_osd": "10000",
>> "mon_probe_timeout": "2",
>> "mon_slurp_timeout": "10",
>> "mon_slurp_bytes": "262144",
>> "mon_client_bytes": "104857600",
>> "mon_daemon_bytes": "419430400",
>> "mon_max_log_entries_per_event": "4096",
>> "mon_reweight_min_pgs_per_osd": "10",
>> "mon_reweight_min_bytes_per_osd": "104857600",
>> "mon_health_data_update_interval": "60",
>> "mon_health_to_clog": "true",
>> "mon_health_to_clog_interval": "3600",
>> "mon_health_to_clog_tick_interval": "60",
>> "mon_data_avail_crit": "5",
>> "mon_data_avail_warn": "30",
>> "mon_data_size_warn": "16106127360",
>> "mon_config_key_max_entry_size": "4096",
>> "mon_sync_timeout": "60",
>> "mon_sync_max_payload_size": "1048576",
>> "mon_sync_debug": "false",
>> "mon_sync_debug_leader": "-1",
>> "mon_sync_debug_provider": "-1",
>> "mon_sync_debug_provider_fallback": "-1",
>> "mon_inject_sync_get_chunk_delay": "0",
>> "mon_osd_min_down_reporters": "1",
>> "mon_osd_min_down_reports": "3",
>> "mon_osd_force_trim_to": "0",
>> "mon_mds_force_trim_to": "0",
>> "crushtool": "crushtool",
>> "mon_debug_dump_transactions": "false",
>> "mon_debug_dump_location": "\/var\/log\/ceph\/ceph-osd.0.tdump",
>> "mon_inject_transaction_delay_max": "10",
>> "mon_inject_transaction_delay_probability": "0",
>> "mon_sync_provider_kill_at": "0",
>> "mon_sync_requester_kill_at": "0",
>> "mon_force_quorum_join": "false",
>> "mon_keyvaluedb": "leveldb",
>> "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false",
>> "paxos_stash_full_interval": "25",
>> "paxos_max_join_drift": "10",
>> "paxos_propose_interval": "1",
>> "paxos_min_wait": "0.05",
>> "paxos_min": "500",
>> "paxos_trim_min": "250",
>> "paxos_trim_max": "500",
>> "paxos_service_trim_min": "250",
>> "paxos_service_trim_max": "500",
>> "paxos_kill_at": "0",
>> "clock_offset": "0",
>> "auth_cluster_required": "cephx",
>> "auth_service_required": "cephx",
>> "auth_client_required": "cephx",
>> "auth_supported": "",
>> "cephx_require_signatures": "false",
>> "cephx_cluster_require_signatures": "false",
>> "cephx_service_require_signatures": "false",
>> "cephx_sign_messages": "true",
>> "auth_mon_ticket_ttl": "43200",
>> "auth_service_ticket_ttl": "3600",
>> "auth_debug": "false",
>> "mon_client_hunt_interval": "3",
>> "mon_client_ping_interval": "10",
>> "mon_client_ping_timeout": "30",
>> "mon_client_hunt_interval_backoff": "2",
>> "mon_client_hunt_interval_max_multiple": "10",
>> "mon_client_max_log_entries_per_message": "1000",
>> "mon_max_pool_pg_num": "65536",
>> "mon_pool_quota_warn_threshold": "0",
>> "mon_pool_quota_crit_threshold": "0",
>> "client_cache_size": "16384",
>> "client_cache_mid": "0.75",
>> "client_use_random_mds": "false",
>> "client_mount_timeout": "300",
>> "client_tick_interval": "1",
>> "client_trace": "",
>> "client_readahead_min": "131072",
>> "client_readahead_max_bytes": "0",
>> "client_readahead_max_periods": "4",
>> "client_snapdir": ".snap",
>> "client_mountpoint": "\/",
>> "client_notify_timeout": "10",
>> "osd_client_watch_timeout": "30",
>> "client_caps_release_delay": "5",
>> "client_quota": "false",
>> "client_oc": "true",
>> "client_oc_size": "209715200",
>> "client_oc_max_dirty": "104857600",
>> "client_oc_target_dirty": "8388608",
>> "client_oc_max_dirty_age": "5",
>> "client_oc_max_objects": "1000",
>> "client_debug_force_sync_read": "false",
>> "client_debug_inject_tick_delay": "0",
>> "client_max_inline_size": "4096",
>> "client_inject_release_failure": "false",
>> "fuse_use_invalidate_cb": "false",
>> "fuse_allow_other": "true",
>> "fuse_default_permissions": "true",
>> "fuse_big_writes": "true",
>> "fuse_atomic_o_trunc": "true",
>> "fuse_debug": "false",
>> "fuse_multithreaded": "true",
>> "client_try_dentry_invalidate": "true",
>> "client_die_on_failed_remount": "true",
>> "client_check_pool_perm": "true",
>> "crush_location": "",
>> "objecter_tick_interval": "5",
>> "objecter_timeout": "10",
>> "objecter_inflight_op_bytes": "104857600",
>> "objecter_inflight_ops": "1024",
>> "objecter_completion_locks_per_session": "32",
>> "objecter_inject_no_watch_ping": "false",
>> "journaler_allow_split_entries": "true",
>> "journaler_write_head_interval": "15",
>> "journaler_prefetch_periods": "10",
>> "journaler_prezero_periods": "5",
>> "journaler_batch_interval": "0.001",
>> "journaler_batch_max": "0",
>> "mds_data": "\/var\/lib\/ceph\/mds\/ceph-0",
>> "mds_max_file_size": "1099511627776",
>> "mds_cache_size": "100000",
>> "mds_cache_mid": "0.7",
>> "mds_max_file_recover": "32",
>> "mds_mem_max": "1048576",
>> "mds_dir_max_commit_size": "10",
>> "mds_decay_halflife": "5",
>> "mds_beacon_interval": "4",
>> "mds_beacon_grace": "15",
>> "mds_enforce_unique_name": "true",
>> "mds_blacklist_interval": "1440",
>> "mds_session_timeout": "60",
>> "mds_revoke_cap_timeout": "60",
>> "mds_recall_state_timeout": "60",
>> "mds_freeze_tree_timeout": "30",
>> "mds_session_autoclose": "300",
>> "mds_health_summarize_threshold": "10",
>> "mds_reconnect_timeout": "45",
>> "mds_tick_interval": "5",
>> "mds_dirstat_min_interval": "1",
>> "mds_scatter_nudge_interval": "5",
>> "mds_client_prealloc_inos": "1000",
>> "mds_early_reply": "true",
>> "mds_default_dir_hash": "2",
>> "mds_log": "true",
>> "mds_log_skip_corrupt_events": "false",
>> "mds_log_max_events": "-1",
>> "mds_log_events_per_segment": "1024",
>> "mds_log_segment_size": "0",
>> "mds_log_max_segments": "30",
>> "mds_log_max_expiring": "20",
>> "mds_bal_sample_interval": "3",
>> "mds_bal_replicate_threshold": "8000",
>> "mds_bal_unreplicate_threshold": "0",
>> "mds_bal_frag": "false",
>> "mds_bal_split_size": "10000",
>> "mds_bal_split_rd": "25000",
>> "mds_bal_split_wr": "10000",
>> "mds_bal_split_bits": "3",
>> "mds_bal_merge_size": "50",
>> "mds_bal_merge_rd": "1000",
>> "mds_bal_merge_wr": "1000",
>> "mds_bal_interval": "10",
>> "mds_bal_fragment_interval": "5",
>> "mds_bal_idle_threshold": "0",
>> "mds_bal_max": "-1",
>> "mds_bal_max_until": "-1",
>> "mds_bal_mode": "0",
>> "mds_bal_min_rebalance": "0.1",
>> "mds_bal_min_start": "0.2",
>> "mds_bal_need_min": "0.8",
>> "mds_bal_need_max": "1.2",
>> "mds_bal_midchunk": "0.3",
>> "mds_bal_minchunk": "0.001",
>> "mds_bal_target_removal_min": "5",
>> "mds_bal_target_removal_max": "10",
>> "mds_replay_interval": "1",
>> "mds_shutdown_check": "0",
>> "mds_thrash_exports": "0",
>> "mds_thrash_fragments": "0",
>> "mds_dump_cache_on_map": "false",
>> "mds_dump_cache_after_rejoin": "false",
>> "mds_verify_scatter": "false",
>> "mds_debug_scatterstat": "false",
>> "mds_debug_frag": "false",
>> "mds_debug_auth_pins": "false",
>> "mds_debug_subtrees": "false",
>> "mds_kill_mdstable_at": "0",
>> "mds_kill_export_at": "0",
>> "mds_kill_import_at": "0",
>> "mds_kill_link_at": "0",
>> "mds_kill_rename_at": "0",
>> "mds_kill_openc_at": "0",
>> "mds_kill_journal_at": "0",
>> "mds_kill_journal_expire_at": "0",
>> "mds_kill_journal_replay_at": "0",
>> "mds_journal_format": "1",
>> "mds_kill_create_at": "0",
>> "mds_inject_traceless_reply_probability": "0",
>> "mds_wipe_sessions": "false",
>> "mds_wipe_ino_prealloc": "false",
>> "mds_skip_ino": "0",
>> "max_mds": "1",
>> "mds_standby_for_name": "",
>> "mds_standby_for_rank": "-1",
>> "mds_standby_replay": "false",
>> "mds_enable_op_tracker": "true",
>> "mds_op_history_size": "20",
>> "mds_op_history_duration": "600",
>> "mds_op_complaint_time": "30",
>> "mds_op_log_threshold": "5",
>> "mds_snap_min_uid": "0",
>> "mds_snap_max_uid": "65536",
>> "mds_verify_backtrace": "1",
>> "mds_action_on_write_error": "1",
>> "osd_compact_leveldb_on_mount": "false",
>> "osd_max_backfills": "1",
>> "osd_min_recovery_priority": "0",
>> "osd_backfill_full_ratio": "0.93",
>> "osd_backfill_retry_interval": "10",
>> "osd_agent_max_ops": "4",
>> "osd_agent_min_evict_effort": "0.1",
>> "osd_agent_quantize_effort": "0.1",
>> "osd_agent_delay_time": "5",
>> "osd_find_best_info_ignore_history_les": "false",
>> "osd_agent_hist_halflife": "1000",
>> "osd_agent_slop": "0.02",
>> "osd_uuid": "00000000-0000-0000-0000-000000000000",
>> "osd_data": "\/var\/lib\/ceph\/osd\/ceph-0",
>> "osd_journal": "\/dev\/vg-osd-0\/lv-journal",
>> "osd_journal_size": "5000",
>> "osd_max_write_size": "90",
>> "osd_max_pgls": "1024",
>> "osd_client_message_size_cap": "524288000",
>> "osd_client_message_cap": "100",
>> "osd_pg_bits": "6",
>> "osd_pgp_bits": "6",
>> "osd_crush_chooseleaf_type": "1",
>> "osd_pool_default_crush_rule": "-1",
>> "osd_pool_default_crush_replicated_ruleset": "0",
>> "osd_pool_erasure_code_stripe_width": "4096",
>> "osd_pool_default_size": "3",
>> "osd_pool_default_min_size": "0",
>> "osd_pool_default_pg_num": "8",
>> "osd_pool_default_pgp_num": "8",
>> "osd_pool_default_erasure_code_directory":
>> "\/usr\/lib64\/ceph\/erasure-code",
>> "osd_pool_default_erasure_code_profile": "plugin=jerasure
>> technique=reed_sol_van k=2 m=1 ",
>> "osd_erasure_code_plugins": "jerasure lrc isa",
>> "osd_allow_recovery_below_min_size": "true",
>> "osd_pool_default_flags": "0",
>> "osd_pool_default_flag_hashpspool": "true",
>> "osd_pool_default_flag_nodelete": "false",
>> "osd_pool_default_flag_nopgchange": "false",
>> "osd_pool_default_flag_nosizechange": "false",
>> "osd_pool_default_hit_set_bloom_fpp": "0.05",
>> "osd_pool_default_cache_target_dirty_ratio": "0.4",
>> "osd_pool_default_cache_target_full_ratio": "0.8",
>> "osd_pool_default_cache_min_flush_age": "0",
>> "osd_pool_default_cache_min_evict_age": "0",
>> "osd_hit_set_min_size": "1000",
>> "osd_hit_set_max_size": "100000",
>> "osd_hit_set_namespace": ".ceph-internal",
>> "osd_tier_default_cache_mode": "writeback",
>> "osd_tier_default_cache_hit_set_count": "4",
>> "osd_tier_default_cache_hit_set_period": "1200",
>> "osd_tier_default_cache_hit_set_type": "bloom",
>> "osd_tier_default_cache_min_read_recency_for_promote": "1",
>> "osd_map_dedup": "true",
>> "osd_map_max_advance": "200",
>> "osd_map_cache_size": "500",
>> "osd_map_message_max": "100",
>> "osd_map_share_max_epochs": "100",
>> "osd_inject_bad_map_crc_probability": "0",
>> "osd_inject_failure_on_pg_removal": "false",
>> "osd_op_threads": "2",
>> "osd_peering_wq_batch_size": "20",
>> "osd_op_pq_max_tokens_per_priority": "4194304",
>> "osd_op_pq_min_cost": "65536",
>> "osd_disk_threads": "1",
>> "osd_disk_thread_ioprio_class": "",
>> "osd_disk_thread_ioprio_priority": "-1",
>> "osd_recovery_threads": "1",
>> "osd_recover_clone_overlap": "true",
>> "osd_op_num_threads_per_shard": "2",
>> "osd_op_num_shards": "5",
>> "osd_read_eio_on_bad_digest": "true",
>> "osd_recover_clone_overlap_limit": "10",
>> "osd_backfill_scan_min": "64",
>> "osd_backfill_scan_max": "512",
>> "osd_op_thread_timeout": "15",
>> "osd_op_thread_suicide_timeout": "150",
>> "osd_recovery_thread_timeout": "30",
>> "osd_recovery_thread_suicide_timeout": "300",
>> "osd_snap_trim_thread_timeout": "3600",
>> "osd_snap_trim_thread_suicide_timeout": "36000",
>> "osd_snap_trim_sleep": "0",
>> "osd_scrub_thread_timeout": "60",
>> "osd_scrub_thread_suicide_timeout": "60",
>> "osd_scrub_finalize_thread_timeout": "600",
>> "osd_scrub_invalid_stats": "true",
>> "osd_remove_thread_timeout": "3600",
>> "osd_remove_thread_suicide_timeout": "36000",
>> "osd_command_thread_timeout": "600",
>> "osd_age": "0.8",
>> "osd_age_time": "0",
>> "osd_command_thread_suicide_timeout": "900",
>> "osd_heartbeat_addr": ":\/0",
>> "osd_heartbeat_interval": "6",
>> "osd_heartbeat_grace": "20",
>> "osd_heartbeat_min_peers": "10",
>> "osd_heartbeat_use_min_delay_socket": "false",
>> "osd_pg_max_concurrent_snap_trims": "2",
>> "osd_heartbeat_min_healthy_ratio": "0.33",
>> "osd_mon_heartbeat_interval": "30",
>> "osd_mon_report_interval_max": "120",
>> "osd_mon_report_interval_min": "5",
>> "osd_pg_stat_report_interval_max": "500",
>> "osd_mon_ack_timeout": "30",
>> "osd_default_data_pool_replay_window": "45",
>> "osd_preserve_trimmed_log": "false",
>> "osd_auto_mark_unfound_lost": "false",
>> "osd_recovery_delay_start": "0",
>> "osd_recovery_max_active": "1",
>> "osd_recovery_max_single_start": "5",
>> "osd_recovery_max_chunk": "8388608",
>> "osd_copyfrom_max_chunk": "8388608",
>> "osd_push_per_object_cost": "1000",
>> "osd_max_push_cost": "8388608",
>> "osd_max_push_objects": "10",
>> "osd_recovery_forget_lost_objects": "false",
>> "osd_max_scrubs": "1",
>> "osd_scrub_begin_hour": "0",
>> "osd_scrub_end_hour": "24",
>> "osd_scrub_load_threshold": "0.5",
>> "osd_scrub_min_interval": "86400",
>> "osd_scrub_max_interval": "604800",
>> "osd_scrub_chunk_min": "5",
>> "osd_scrub_chunk_max": "25",
>> "osd_scrub_sleep": "0",
>> "osd_deep_scrub_interval": "604800",
>> "osd_deep_scrub_stride": "524288",
>> "osd_deep_scrub_update_digest_min_age": "7200",
>> "osd_scan_list_ping_tp_interval": "100",
>> "osd_auto_weight": "false",
>> "osd_class_dir": "\/usr\/lib64\/rados-classes",
>> "osd_open_classes_on_start": "true",
>> "osd_check_for_log_corruption": "false",
>> "osd_use_stale_snap": "false",
>> "osd_rollback_to_cluster_snap": "",
>> "osd_default_notify_timeout": "30",
>> "osd_kill_backfill_at": "0",
>> "osd_pg_epoch_persisted_max_stale": "200",
>> "osd_min_pg_log_entries": "3000",
>> "osd_max_pg_log_entries": "10000",
>> "osd_pg_log_trim_min": "100",
>> "osd_op_complaint_time": "30",
>> "osd_command_max_records": "256",
>> "osd_max_pg_blocked_by": "16",
>> "osd_op_log_threshold": "5",
>> "osd_verify_sparse_read_holes": "false",
>> "osd_debug_drop_ping_probability": "0",
>> "osd_debug_drop_ping_duration": "0",
>> "osd_debug_drop_pg_create_probability": "0",
>> "osd_debug_drop_pg_create_duration": "1",
>> "osd_debug_drop_op_probability": "0",
>> "osd_debug_op_order": "false",
>> "osd_debug_scrub_chance_rewrite_digest": "0",
>> "osd_debug_verify_snaps_on_info": "false",
>> "osd_debug_verify_stray_on_activate": "false",
>> "osd_debug_skip_full_check_in_backfill_reservation": "false",
>> "osd_debug_reject_backfill_probability": "0",
>> "osd_debug_inject_copyfrom_error": "false",
>> "osd_enable_op_tracker": "true",
>> "osd_num_op_tracker_shard": "32",
>> "osd_op_history_size": "20",
>> "osd_op_history_duration": "600",
>> "osd_target_transaction_size": "30",
>> "osd_failsafe_full_ratio": "0.97",
>> "osd_failsafe_nearfull_ratio": "0.9",
>> "osd_pg_object_context_cache_count": "64",
>> "osd_debug_pg_log_writeout": "false",
>> "threadpool_default_timeout": "60",
>> "threadpool_empty_queue_max_wait": "2",
>> "leveldb_write_buffer_size": "8388608",
>> "leveldb_cache_size": "134217728",
>> "leveldb_block_size": "0",
>> "leveldb_bloom_size": "0",
>> "leveldb_max_open_files": "0",
>> "leveldb_compression": "true",
>> "leveldb_paranoid": "false",
>> "leveldb_log": "",
>> "leveldb_compact_on_mount": "false",
>> "kinetic_host": "",
>> "kinetic_port": "8123",
>> "kinetic_user_id": "1",
>> "kinetic_hmac_key": "asdfasdf",
>> "kinetic_use_ssl": "false",
>> "rocksdb_compact_on_mount": "false",
>> "rocksdb_write_buffer_size": "0",
>> "rocksdb_target_file_size_base": "0",
>> "rocksdb_cache_size": "0",
>> "rocksdb_block_size": "0",
>> "rocksdb_bloom_size": "0",
>> "rocksdb_write_buffer_num": "0",
>> "rocksdb_background_compactions": "0",
>> "rocksdb_background_flushes": "0",
>> "rocksdb_max_open_files": "0",
>> "rocksdb_compression": "",
>> "rocksdb_paranoid": "false",
>> "rocksdb_log": "\/dev\/null",
>> "rocksdb_level0_file_num_compaction_trigger": "0",
>> "rocksdb_level0_slowdown_writes_trigger": "0",
>> "rocksdb_level0_stop_writes_trigger": "0",
>> "rocksdb_disableDataSync": "true",
>> "rocksdb_disableWAL": "false",
>> "rocksdb_num_levels": "0",
>> "rocksdb_wal_dir": "",
>> "rocksdb_info_log_level": "info",
>> "osd_client_op_priority": "63",
>> "osd_recovery_op_priority": "1",
>> "osd_recovery_op_warn_multiple": "16",
>> "osd_mon_shutdown_timeout": "5",
>> "osd_max_object_size": "107374182400",
>> "osd_max_object_name_len": "2048",
>> "osd_max_attr_name_len": "100",
>> "osd_max_attr_size": "0",
>> "osd_objectstore": "filestore",
>> "osd_debug_override_acting_compat": "false",
>> "osd_bench_small_size_max_iops": "100",
>> "osd_bench_large_size_max_throughput": "104857600",
>> "osd_bench_max_block_size": "67108864",
>> "osd_bench_duration": "30",
>> "memstore_device_bytes": "1073741824",
>> "filestore_omap_backend": "leveldb",
>> "filestore_debug_disable_sharded_check": "false",
>> "filestore_wbthrottle_enable": "true",
>> "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040",
>> "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400",
>> "filestore_wbthrottle_btrfs_ios_start_flusher": "500",
>> "filestore_wbthrottle_btrfs_ios_hard_limit": "5000",
>> "filestore_wbthrottle_btrfs_inodes_start_flusher": "500",
>> "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040",
>> "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400",
>> "filestore_wbthrottle_xfs_ios_start_flusher": "500",
>> "filestore_wbthrottle_xfs_ios_hard_limit": "5000",
>> "filestore_wbthrottle_xfs_inodes_start_flusher": "500",
>> "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000",
>> "filestore_wbthrottle_xfs_inodes_hard_limit": "5000",
>> "filestore_index_retry_probability": "0",
>> "filestore_debug_inject_read_err": "false",
>> "filestore_debug_omap_check": "false",
>> "filestore_omap_header_cache_size": "1024",
>> "filestore_max_inline_xattr_size": "0",
>> "filestore_max_inline_xattr_size_xfs": "65536",
>> "filestore_max_inline_xattr_size_btrfs": "2048",
>> "filestore_max_inline_xattr_size_other": "512",
>> "filestore_max_inline_xattrs": "0",
>> "filestore_max_inline_xattrs_xfs": "10",
>> "filestore_max_inline_xattrs_btrfs": "10",
>> "filestore_max_inline_xattrs_other": "2",
>> "filestore_sloppy_crc": "false",
>> "filestore_sloppy_crc_block_size": "65536",
>> "filestore_max_alloc_hint_size": "1048576",
>> "filestore_max_sync_interval": "5",
>> "filestore_min_sync_interval": "0.01",
>> "filestore_btrfs_snap": "true",
>> "filestore_btrfs_clone_range": "true",
>> "filestore_zfs_snap": "false",
>> "filestore_fsync_flushes_journal_data": "false",
>> "filestore_fiemap": "false",
>> "filestore_fadvise": "true",
>> "filestore_xfs_extsize": "true",
>> "filestore_journal_parallel": "false",
>> "filestore_journal_writeahead": "false",
>> "filestore_journal_trailing": "false",
>> "filestore_queue_max_ops": "50",
>> "filestore_queue_max_bytes": "104857600",
>> "filestore_queue_committing_max_ops": "500",
>> "filestore_queue_committing_max_bytes": "104857600",
>> "filestore_op_threads": "2",
>> "filestore_op_thread_timeout": "60",
>> "filestore_op_thread_suicide_timeout": "180",
>> "filestore_commit_timeout": "600",
>> "filestore_fiemap_threshold": "4096",
>> "filestore_merge_threshold": "10",
>> "filestore_split_multiple": "2",
>> "filestore_update_to": "1000",
>> "filestore_blackhole": "false",
>> "filestore_fd_cache_size": "128",
>> "filestore_fd_cache_shards": "16",
>> "filestore_dump_file": "",
>> "filestore_kill_at": "0",
>> "filestore_inject_stall": "0",
>> "filestore_fail_eio": "true",
>> "filestore_debug_verify_split": "false",
>> "journal_dio": "true",
>> "journal_aio": "true",
>> "journal_force_aio": "false",
>> "keyvaluestore_queue_max_ops": "50",
>> "keyvaluestore_queue_max_bytes": "104857600",
>> "keyvaluestore_debug_check_backend": "false",
>> "keyvaluestore_op_threads": "2",
>> "keyvaluestore_op_thread_timeout": "60",
>> "keyvaluestore_op_thread_suicide_timeout": "180",
>> "keyvaluestore_default_strip_size": "4096",
>> "keyvaluestore_max_expected_write_size": "16777216",
>> "keyvaluestore_header_cache_size": "4096",
>> "keyvaluestore_backend": "leveldb",
>> "journal_max_corrupt_search": "10485760",
>> "journal_block_align": "true",
>> "journal_write_header_frequency": "0",
>> "journal_max_write_bytes": "10485760",
>> "journal_max_write_entries": "100",
>> "journal_queue_max_ops": "300",
>> "journal_queue_max_bytes": "33554432",
>> "journal_align_min_size": "65536",
>> "journal_replay_from": "0",
>> "journal_zero_on_create": "false",
>> "journal_ignore_corruption": "false",
>> "journal_discard": "false",
>> "rados_mon_op_timeout": "0",
>> "rados_osd_op_timeout": "0",
>> "rbd_op_threads": "1",
>> "rbd_op_thread_timeout": "60",
>> "rbd_non_blocking_aio": "true",
>> "rbd_cache": "true",
>> "rbd_cache_writethrough_until_flush": "true",
>> "rbd_cache_size": "33554432",
>> "rbd_cache_max_dirty": "25165824",
>> "rbd_cache_target_dirty": "16777216",
>> "rbd_cache_max_dirty_age": "1",
>> "rbd_cache_max_dirty_object": "0",
>> "rbd_cache_block_writes_upfront": "false",
>> "rbd_concurrent_management_ops": "10",
>> "rbd_balance_snap_reads": "false",
>> "rbd_localize_snap_reads": "false",
>> "rbd_balance_parent_reads": "false",
>> "rbd_localize_parent_reads": "true",
>> "rbd_readahead_trigger_requests": "10",
>> "rbd_readahead_max_bytes": "524288",
>> "rbd_readahead_disable_after_bytes": "52428800",
>> "rbd_clone_copy_on_read": "false",
>> "rbd_blacklist_on_break_lock": "true",
>> "rbd_blacklist_expire_seconds": "0",
>> "rbd_request_timed_out_seconds": "30",
>> "rbd_default_format": "1",
>> "rbd_default_order": "22",
>> "rbd_default_stripe_count": "0",
>> "rbd_default_stripe_unit": "0",
>> "rbd_default_features": "3",
>> "nss_db_path": "",
>> "rgw_max_chunk_size": "524288",
>> "rgw_override_bucket_index_max_shards": "0",
>> "rgw_bucket_index_max_aio": "8",
>> "rgw_enable_quota_threads": "true",
>> "rgw_enable_gc_threads": "true",
>> "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-0",
>> "rgw_enable_apis": "s3, swift, swift_auth, admin",
>> "rgw_cache_enabled": "true",
>> "rgw_cache_lru_size": "10000",
>> "rgw_socket_path": "",
>> "rgw_host": "",
>> "rgw_port": "",
>> "rgw_dns_name": "",
>> "rgw_content_length_compat": "false",
>> "rgw_script_uri": "",
>> "rgw_request_uri": "",
>> "rgw_swift_url": "",
>> "rgw_swift_url_prefix": "swift",
>> "rgw_swift_auth_url": "",
>> "rgw_swift_auth_entry": "auth",
>> "rgw_swift_tenant_name": "",
>> "rgw_swift_enforce_content_length": "false",
>> "rgw_keystone_url": "",
>> "rgw_keystone_admin_token": "",
>> "rgw_keystone_admin_user": "",
>> "rgw_keystone_admin_password": "",
>> "rgw_keystone_admin_tenant": "",
>> "rgw_keystone_accepted_roles": "Member, admin",
>> "rgw_keystone_token_cache_size": "10000",
>> "rgw_keystone_revocation_interval": "900",
>> "rgw_s3_auth_use_rados": "true",
>> "rgw_s3_auth_use_keystone": "false",
>> "rgw_admin_entry": "admin",
>> "rgw_enforce_swift_acls": "true",
>> "rgw_swift_token_expiration": "86400",
>> "rgw_print_continue": "true",
>> "rgw_remote_addr_param": "REMOTE_ADDR",
>> "rgw_op_thread_timeout": "600",
>> "rgw_op_thread_suicide_timeout": "0",
>> "rgw_thread_pool_size": "100",
>> "rgw_num_control_oids": "8",
>> "rgw_num_rados_handles": "1",
>> "rgw_zone": "",
>> "rgw_zone_root_pool": ".rgw.root",
>> "rgw_region": "",
>> "rgw_region_root_pool": ".rgw.root",
>> "rgw_default_region_info_oid": "default.region",
>> "rgw_log_nonexistent_bucket": "false",
>> "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
>> "rgw_log_object_name_utc": "false",
>> "rgw_usage_max_shards": "32",
>> "rgw_usage_max_user_shards": "1",
>> "rgw_enable_ops_log": "false",
>> "rgw_enable_usage_log": "false",
>> "rgw_ops_log_rados": "true",
>> "rgw_ops_log_socket_path": "",
>> "rgw_ops_log_data_backlog": "5242880",
>> "rgw_usage_log_flush_threshold": "1024",
>> "rgw_usage_log_tick_interval": "30",
>> "rgw_intent_log_object_name": "%Y-%m-%d-%i-%n",
>> "rgw_intent_log_object_name_utc": "false",
>> "rgw_init_timeout": "300",
>> "rgw_mime_types_file": "\/etc\/mime.types",
>> "rgw_gc_max_objs": "32",
>> "rgw_gc_obj_min_wait": "7200",
>> "rgw_gc_processor_max_time": "3600",
>> "rgw_gc_processor_period": "3600",
>> "rgw_s3_success_create_obj_status": "0",
>> "rgw_resolve_cname": "false",
>> "rgw_obj_stripe_size": "4194304",
>> "rgw_extended_http_attrs": "",
>> "rgw_exit_timeout_secs": "120",
>> "rgw_get_obj_window_size": "16777216",
>> "rgw_get_obj_max_req_size": "4194304",
>> "rgw_relaxed_s3_bucket_names": "false",
>> "rgw_defer_to_bucket_acls": "",
>> "rgw_list_buckets_max_chunk": "1000",
>> "rgw_md_log_max_shards": "64",
>> "rgw_num_zone_opstate_shards": "128",
>> "rgw_opstate_ratelimit_sec": "30",
>> "rgw_curl_wait_timeout_ms": "1000",
>> "rgw_copy_obj_progress": "true",
>> "rgw_copy_obj_progress_every_bytes": "1048576",
>> "rgw_data_log_window": "30",
>> "rgw_data_log_changes_size": "1000",
>> "rgw_data_log_num_shards": "128",
>> "rgw_data_log_obj_prefix": "data_log",
>> "rgw_replica_log_obj_prefix": "replica_log",
>> "rgw_bucket_quota_ttl": "600",
>> "rgw_bucket_quota_soft_threshold": "0.95",
>> "rgw_bucket_quota_cache_size": "10000",
>> "rgw_expose_bucket": "false",
>> "rgw_frontends": "fastcgi, civetweb port=7480",
>> "rgw_user_quota_bucket_sync_interval": "180",
>> "rgw_user_quota_sync_interval": "86400",
>> "rgw_user_quota_sync_idle_users": "false",
>> "rgw_user_quota_sync_wait_time": "86400",
>> "rgw_multipart_min_part_size": "5242880",
>> "rgw_olh_pending_timeout_sec": "3600",
>> "rgw_user_max_buckets": "1000",
>> "mutex_perf_counter": "false",
>> "throttler_perf_counter": "true",
>> "internal_safe_to_start_threads": "true"
>> }
>>
>> ceph.conf
>>
>> cat /etc/ceph/ceph.conf
>> [client]
>> rbd cache = true
>> # 256 MB of cache
>> rbd cache size = 268435456
>> # 64 MB Max Dirty
>> rbd cache max dirty = 67108864
>> # 32 MB Target Dirty
>> rbd cache target dirty = 33554432
>> # 2 Seconds worth can be dirty
>> rbd cache max dirty age = 2.0
>> [global]
>> public network = 10.66.4.0/24
>> cluster network = 10.66.4.0/24
>>
>> # For version 0.55 and beyond, you must explicitly enable
>> # or disable authentication with "auth" entries in [global].
>>
>> auth cluster required = cephx
>> auth service required = cephx
>> auth client required = cephx
>>
>> mon osd full ratio = .98
>> mon osd nearfull ratio = .94
>>
>>
>> mon_pg_warn_max_per_osd = 0
>>
>> [osd]
>> osd journal size = 5000
>> journal aio = true
>>
>> # See
>> https://ceph.com/docs/master/rados/configuration/osd-config-ref/
>> osd client op priority = 90
>> osd recovery op priority = 10
>> osd recovery max active = 1
>> osd max backfills = 1
>> osd backfill full ratio = 0.93
>>
>> #The following assumes ext4 filesystem.
>> # filestore xattr use omap = true
>>
>>
>> # For Bobtail (v 0.56) and subsequent versions, you may
>> # add settings for mkcephfs so that it will create and mount
>> # the file system on a particular OSD for you. Remove the comment
>> `#`
>> # character for the following settings and replace the values
>> # in braces with appropriate values, or leave the following
>> settings
>> # commented out to accept the default values. You must specify the
>> # --mkfs option with mkcephfs in order for the deployment script
>> to
>> # utilize the following settings, and you must define the 'devs'
>> # option for each osd instance; see below.
>>
>> osd mkfs type = btrfs
>> osd mkfs options btrfs = -L $name
>> # xfs options by Dell RAID CARD specifies 64k stripe size(su) and
>> each osd is a single drive(1) for (sw)
>> # see :
>> http://www.mysqlperformanceblog.com/2011/12/16/setting-up-xfs-the-simple-edition/
>> # also see : https://raid.wiki.kernel.org/index.php/RAID_setup#XFS
>> # mkfs -t xfs -f -L ceph-xx -d su=64k,sw=1 /dev/sdX1
>> osd mkfs options xfs = -L -d su=64k,sw=1 $name
>> osd mount options btrfs = rw,noatime
>> osd mount options xfs = rw,noatime,nodiratime
>> # mount -o rw,noatime,nodiratime /dev/disk/by-label/ceph-X
>> /var/lib/ceph/osd/ceph-X
>>
>> # For example, for ext4, the mount option might look like this:
>>
>> #osd mkfs options ext4 = user_xattr,rw,noatime
>>
>> # Execute $ hostname to retrieve the name of your host,
>> # and replace {hostname} with the name of your host.
>> # For the monitor, replace {ip-address} with the IP
>> # address of your host.
>>
>>
>> # CAIT -- Manual commands to make and mount file system
>>
>> # -- Make xfs file system
>> # mkfs -t xfs -f -L ceph-X -d su=64k,sw=1 /dev/sdX1
>>
>> # -- Rescan Parition Label
>> # partprobe /dev/sdX1
>>
>> # -- Mount ceph file system
>> # mount -o rw,noatime,nodiratime /dev/disk/by-label/ceph-X
>> /var/lib/ceph/osd/ceph-X
>>
>>
>>
>>
>>
>> [mon]
>> mon debug dump transactions = false
>> # mon warn on legacy crush tunables = false
>>
>> # [mon.splinter]
>> # host = splinter
>> # mon addr = 10.66.4.2:6789
>>
>> [mon.helm]
>> host = helm
>> mon addr = 10.66.4.4:6789
>>
>> [mon.hamms]
>> host = hamms
>> mon addr = 10.66.4.9:6789
>>
>> [mon.gold]
>> host = gold
>> mon addr = 10.66.4.7:6789
>>
>> [osd.0]
>> host = helm
>> devs = /dev/vg-osd-0/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-0/lv-journal
>>
>> [osd.1]
>> host = helm
>> devs = /dev/vg-osd-1/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-1/lv-journal
>>
>> [osd.2]
>> host = helm
>> devs = /dev/vg-osd-2/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-2/lv-journal
>>
>> [osd.3]
>> host = helm
>> devs = /dev/vg-osd-3/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-3/lv-journal
>>
>> [osd.4]
>> host = helm
>> devs = /dev/vg-osd-4/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-4/lv-journal
>>
>> [osd.5]
>> host = helm
>> devs = /dev/vg-osd-5/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-5/lv-journal
>>
>> [osd.6]
>> host = helm
>> devs = /dev/vg-osd-6/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-6/lv-journal
>>
>> [osd.7]
>> host = helm
>> devs = /dev/vg-osd-7/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-7/lv-journal
>>
>> [osd.8]
>> host = helm
>> devs = /dev/vg-osd-8/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-8/lv-journal
>>
>> [osd.16]
>> host = gold
>> devs = /dev/disk/by-label/ceph-16
>> osd mkfs type = xfs
>>
>> [osd.17]
>> host = gold
>> devs = /dev/disk/by-label/ceph-17
>> osd mkfs type = xfs
>>
>> [osd.18]
>> host = gold
>> devs = /dev/disk/by-label/ceph-18
>> osd mkfs type = xfs
>>
>> [osd.19]
>> host = gold
>> devs = /dev/disk/by-label/ceph-19
>> osd mkfs type = xfs
>>
>> [osd.20]
>> host = gold
>> devs = /dev/disk/by-label/ceph-20
>> osd mkfs type = xfs
>>
>> [osd.21]
>> host = gold
>> devs = /dev/disk/by-label/ceph-21
>> osd mkfs type = xfs
>>
>> [osd.9]
>> host = gold
>> devs = /dev/vg-osd-9/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-9/lv-journal
>>
>> [osd.10]
>> host = gold
>> devs = /dev/vg-osd-10/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-10/lv-journal
>>
>> [osd.34]
>> host = gold
>> devs = /dev/vg-osd-34/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-34/lv-journal
>>
>> [osd.22]
>> host = gallo
>> devs = /dev/disk/by-label/ceph-22
>> osd mkfs type = xfs
>>
>> [osd.23]
>> host = gallo
>> devs = /dev/disk/by-label/ceph-23
>> osd mkfs type = xfs
>>
>> [osd.24]
>> host = gallo
>> devs = /dev/disk/by-label/ceph-24
>> osd mkfs type = xfs
>>
>> [osd.25]
>> host = gallo
>> devs = /dev/disk/by-label/ceph-25
>> osd mkfs type = xfs
>>
>> [osd.26]
>> host = gallo
>> devs = /dev/disk/by-label/ceph-26
>> osd mkfs type = xfs
>>
>> [osd.27]
>> host = gallo
>> devs = /dev/disk/by-label/ceph-27
>> osd mkfs type = xfs
>>
>> [osd.11]
>> host = gallo
>> devs = /dev/vg-osd-11/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-11/lv-journal
>>
>> [osd.12]
>> host = gallo
>> devs = /dev/vg-osd-12/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-12/lv-journal
>>
>> [osd.35]
>> host = gallo
>> devs = /dev/vg-osd-35/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-35/lv-journal
>>
>> [osd.13]
>> host = hamms
>> devs = /dev/vg-osd-13/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-13/lv-journal
>>
>> [osd.14]
>> host = hamms
>> devs = /dev/vg-osd-14/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-14/lv-journal
>>
>> [osd.15]
>> host = hamms
>> devs = /dev/vg-osd-15/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-15/lv-journal
>>
>> [osd.28]
>> host = hamms
>> devs = /dev/vg-osd-28/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-28/lv-journal
>>
>> [osd.29]
>> host = hamms
>> devs = /dev/vg-osd-29/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-29/lv-journal
>>
>> [osd.30]
>> host = hamms
>> devs = /dev/vg-osd-30/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-30/lv-journal
>>
>> [osd.31]
>> host = hamms
>> devs = /dev/vg-osd-31/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-31/lv-journal
>>
>> [osd.32]
>> host = hamms
>> devs = /dev/vg-osd-32/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-32/lv-journal
>>
>> [osd.33]
>> host = hamms
>> devs = /dev/vg-osd-33/lv-osd
>> osd mkfs type = xfs
>> osd journal = /dev/vg-osd-33/lv-journal
>>
>>
>> --
>> Joe Ryner
>> Center for the Application of Information Technologies (CAIT)
>> Production Coordinator
>> P: (309) 298-1804
>> F: (309) 298-2806
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
> --
> Pozdrawiam,
> Tomasz Kuzemko
> tomasz@xxxxxxxxxxx
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.2.3
Comment: https://www.mailvelope.com
wsFcBAEBCAAQBQJWVkM0CRDmVDuy+mK58QAAo94P/iv1YroW77EebBGpTUjU
9m9DVryCmz7NJuoif5yVe5L4jFl1wqQ0yMJbWHWKxZRtazAYIOnGPo7vm805
+Ph035l/b/SV9EixgsbGnCW/TFP+RV9gapHn8y57ei5ICdFf/ZmmyWqAmGbl
s27/gzQhKzGcuyAi9Q8R5CJcmgyGxY3CObX62TE0Xi70tMsNV8TtQC4UEbtV
kKcYMxs+HsLjKS8Wfw6XerXtwWjbxdqPd0S+Hf4rmIeqfDs72yoy6dzObT/3
YH6EXBwc8/YrrWdNwB6HDY87smK5pRD0NvRUf/C8jUJPSIBLQvPK6pmgd+pe
w8xNkCrn26Npx72I+vf1+lhEr6SvwahImbiaAkMVJD2HxHq5tjyKJkPxAvrM
soOaFNdvZhUJ7yKv0DDShxS8NrwrqWgE/t2vIGM8RO3Q3hAIAM+eC2WIevC/
2jIt/co9QPyXZIFRe0qlLKbDs6Nco8VQoyU+ZZFwdRNxYT8btooxFegiKttH
ELQSr0BV5dZjBa4bhM2smXrTOXMH7tZ9hqDwMlrmoGkLWv7zJYwjfD9ODZyV
pJH9Ugr28h49BuoH3CWvoOKmkrh7GGU5Y/RdMO1tEMe80mn2Lblt573KRTlo
bPv4vsvd8yY1ETZtNwhe1xkx8sgW16iJIMv5BI69zAYaqo+o0+kbZ4aaBU8z
RpEX
=q7Xr
-----END PGP SIGNATURE-----
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
