Re: RDMA/RoCE enablement failed with (113) No route to host

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

I don't know. 
Ceph documentation on Mimic doesn't appear to go into too much details on RDMA in general, but still it's mentioned in the Ceph docs here and there.  Some examples:

I want to believe that the official docs wouldn't mention something that's completely broken?

There are multiple posts in this very mailing list from people trying to make it work. 
--
Michael Green
Customer Support & Integration
Tel. +1 (518) 9862385
green@xxxxxxxxxxxxx

E8 Storage has a new look, find out more 










On Dec 18, 2018, at 6:55 AM, Виталий Филиппов <vitalif@xxxxxxxxxx> wrote:

Is RDMA officially supported? I'm asking because I recently tried to use DPDK and it seems it's broken... i.e the code is there, but does not compile until I fix cmake scripts, and after fixing the build OSDs just get segfaults and die after processing something like 40-50 incoming packets.

Maybe RDMA is in the same state?

13 декабря 2018 г. 2:42:23 GMT+03:00, Michael Green <green@xxxxxxxxxxxxx> пишет:
Sorry for bumping the thread. I refuse to believe there are no people on this list who have successfully enabled and run RDMA with Mimic. :)

Mike

Hello collective wisdom,

ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable) here.

I have a working cluster here consisting of 3 monitor hosts,  64 OSD processes across 4 osd hosts, plus 2 MDSs, plus 2 MGRs. All of that is consumed by 10 client nodes.

Every host in the cluster, including clients is 
RHEL 7.5
Mellanox OFED 4.4-2.0.7.0
RoCE NICs are either MCX416A-CCAT or MCX414A-CCAT @ 50Gbit/sec
The NICs are all mlx5_0 port 1

ring and ib_send_bw work fine both ways on any two nodes in the cluster.

Full configuration of the cluster is pasted below, but RDMA related parameters are configured as following:


ms_public_type = async+rdma
ms_cluster = async+rdma
# Exclude clients for now 
ms_type = async+posix

ms_async_rdma_device_name = mlx5_0
ms_async_rdma_polling_us = 0
ms_async_rdma_port_num=1

When I try to start MON, it immediately fails as below. Anybody has seen this or could give any pointers what to/where to look next?


------ceph-mon.rio.log--begin------
2018-12-12 22:35:30.011 7f515dc39140  0 set uid:gid to 167:167 (ceph:ceph)
2018-12-12 22:35:30.011 7f515dc39140  0 ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable), process ceph-mon, pid 2129843
2018-12-12 22:35:30.011 7f515dc39140  0 pidfile_write: ignore empty --pid-file
2018-12-12 22:35:30.036 7f515dc39140  0 load: jerasure load: lrc load: isa
2018-12-12 22:35:30.036 7f515dc39140  0  set rocksdb option compression = kNoCompression
2018-12-12 22:35:30.036 7f515dc39140  0  set rocksdb option level_compaction_dynamic_level_bytes = true
2018-12-12 22:35:30.036 7f515dc39140  0  set rocksdb option write_buffer_size = 33554432
2018-12-12 22:35:30.036 7f515dc39140  0  set rocksdb option compression = kNoCompression
2018-12-12 22:35:30.036 7f515dc39140  0  set rocksdb option level_compaction_dynamic_level_bytes = true
2018-12-12 22:35:30.036 7f515dc39140  0  set rocksdb option write_buffer_size = 33554432
2018-12-12 22:35:30.147 7f51442ed700  2 Event(0x55d927e95700 nevent=5000 time_id=1).set_owner idx=1 owner=139987012998912
2018-12-12 22:35:30.147 7f51442ed700 10 stack operator() starting
2018-12-12 22:35:30.147 7f5143aec700  2 Event(0x55d927e95200 nevent=5000 time_id=1).set_owner idx=0 owner=139987004606208
2018-12-12 22:35:30.147 7f5144aee700  2 Event(0x55d927e95c00 nevent=5000 time_id=1).set_owner idx=2 owner=139987021391616
2018-12-12 22:35:30.147 7f5143aec700 10 stack operator() starting
2018-12-12 22:35:30.147 7f5144aee700 10 stack operator() starting
2018-12-12 22:35:30.147 7f515dc39140  0 starting mon.rio rank 0 at public addr 192.168.1.58:6789/0 at bind addr 192.168.1.58:6789/0 mon_data /var/lib/ceph/mon/ceph-rio fsid 376540c8-a362-41cc-9a58-9c8ceca0e4ee
2018-12-12 22:35:30.147 7f515dc39140 10 -- - bind bind 192.168.1.58:6789/0
2018-12-12 22:35:30.147 7f515dc39140 10 -- - bind Network Stack is not ready for bind yet - postponed
2018-12-12 22:35:30.147 7f515dc39140  0 starting mon.rio rank 0 at 192.168.1.58:6789/0 mon_data /var/lib/ceph/mon/ceph-rio fsid 376540c8-a362-41cc-9a58-9c8ceca0e4ee
2018-12-12 22:35:30.148 7f515dc39140  0 mon.rio@-1(probing).mds e84 new map
2018-12-12 22:35:30.148 7f515dc39140  0 mon.rio@-1(probing).mds e84 print_map
e84
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: -1

No filesystems configured
Standby daemons:

5906437:        192.168.1.152:6800/1077205146 'prince' mds.-1.0 up:standby seq 2
6284118:        192.168.1.59:6800/1266235911 'salvador' mds.-1.0 up:standby seq 2

2018-12-12 22:35:30.148 7f515dc39140  0 mon.rio@-1(probing).osd e25894 crush map has features 288514051259236352, adjusting msgr requires
2018-12-12 22:35:30.148 7f515dc39140  0 mon.rio@-1(probing).osd e25894 crush map has features 288514051259236352, adjusting msgr requires
2018-12-12 22:35:30.148 7f515dc39140  0 mon.rio@-1(probing).osd e25894 crush map has features 1009089991638532096, adjusting msgr requires
2018-12-12 22:35:30.148 7f515dc39140  0 mon.rio@-1(probing).osd e25894 crush map has features 288514051259236352, adjusting msgr requires
2018-12-12 22:35:30.149 7f515dc39140 10 -- - create_connect 192.168.1.88:6800/1638, creating connection and registering
2018-12-12 22:35:30.149 7f515dc39140 10 -- - >> 192.168.1.88:6800/1638 conn(0x55d9281fbe00 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._connect csq=0
2018-12-12 22:35:30.149 7f515dc39140 10 -- - get_connection mgr.5894115 192.168.1.88:6800/1638 new 0x55d9281fbe00
2018-12-12 22:35:30.150 7f515dc39140  1 -- - --> 192.168.1.88:6800/1638 -- mgropen(unknown.rio) v3 -- 0x55d92844e000 con 0
2018-12-12 22:35:30.151 7f515dc39140  1 -- - start start
2018-12-12 22:35:30.151 7f515dc39140  1 -- - start start
2018-12-12 22:35:30.151 7f515dc39140 10 -- - ready -
2018-12-12 22:35:30.151 7f515dc39140 10 -- - bind bind 192.168.1.58:6789/0
2018-12-12 22:35:30.151 7f515dc39140 10  Processor -- bind
2018-12-12 22:35:30.154 7f5144aee700  1 Infiniband binding_port found active port 1
2018-12-12 22:35:30.154 7f5144aee700  1 Infiniband init receive queue length is 4096 receive buffers
2018-12-12 22:35:30.154 7f5144aee700  1 Infiniband init assigning: 1024 send buffers
2018-12-12 22:35:30.154 7f5144aee700  1 Infiniband init device allow 4194303 completion entries
2018-12-12 22:35:30.509 7f515dc39140 10  Processor -- bind bound to 192.168.1.58:6789/0
2018-12-12 22:35:30.509 7f515dc39140  1 -- 192.168.1.58:6789/0 learned_addr learned my addr 192.168.1.58:6789/0
2018-12-12 22:35:30.509 7f515dc39140  1 -- 192.168.1.58:6789/0 _finish_bind bind my_inst.addr is 192.168.1.58:6789/0
2018-12-12 22:35:30.510 7f515dc39140 10 -- - ready -
2018-12-12 22:35:30.510 7f515dc39140  1  Processor -- start
2018-12-12 22:35:30.510 7f515dc39140  0 mon.rio@-1(probing) e5  my rank is now 0 (was -1)
2018-12-12 22:35:30.510 7f515dc39140  1 -- 192.168.1.58:6789/0 shutdown_connections
2018-12-12 22:35:30.510 7f515dc39140  1 -- 192.168.1.58:6789/0 _send_message--> mon.1 192.168.1.59:6789/0 -- mon_probe(probe 376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- ?+0 0x55d928525680
2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 create_connect 192.168.1.59:6789/0, creating connection and registering
2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >> 192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._connect csq=0
2018-12-12 22:35:30.510 7f515dc39140  1 -- 192.168.1.58:6789/0 --> 192.168.1.59:6789/0 -- mon_probe(probe 376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- 0x55d928525680 con 0
2018-12-12 22:35:30.510 7f515dc39140  1 -- 192.168.1.58:6789/0 _send_message--> mon.2 192.168.1.65:6789/0 -- mon_probe(probe 376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- ?+0 0x55d928525900
2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 create_connect 192.168.1.65:6789/0, creating connection and registering
2018-12-12 22:35:30.510 7f515dc39140 10 -- 192.168.1.58:6789/0 >> 192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._connect csq=0
2018-12-12 22:35:30.510 7f515dc39140  1 -- 192.168.1.58:6789/0 --> 192.168.1.65:6789/0 -- mon_probe(probe 376540c8-a362-41cc-9a58-9c8ceca0e4ee name rio) v6 -- 0x55d928525900 con 0
2018-12-12 22:35:30.513 7f5143aec700 10 NetHandler generic_connect connect: (111) Connection refused
2018-12-12 22:35:30.513 7f51442ed700 10 NetHandler generic_connect connect: (111) Connection refused
2018-12-12 22:35:30.513 7f5144aee700 10 Infiniband send_msg sending: 0, 12894, 0, 0, fe800000000000007efe90fffe1e2524
2018-12-12 22:35:30.513 7f5143aec700  1 RDMAStack connect try connecting failed.
2018-12-12 22:35:30.513 7f51442ed700  1 RDMAStack connect try connecting failed.
2018-12-12 22:35:30.513 7f5144aee700 10 -- - >> 192.168.1.88:6800/1638 conn(0x55d9281fbe00 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection nonblock connect inprogress
2018-12-12 22:35:30.513 7f5143aec700 10 -- 192.168.1.58:6789/0 >> 192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=0).fault waiting 0.200000
2018-12-12 22:35:30.513 7f51442ed700 10 -- 192.168.1.58:6789/0 >> 192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=0).fault waiting 0.200000
2018-12-12 22:35:30.513 7f5144aee700 10 -- - >> 192.168.1.88:6800/1638 conn(0x55d9281fbe00 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0).handle_write
2018-12-12 22:35:30.513 7f5143aec700 10 -- 192.168.1.58:6789/0 >> 192.168.1.59:6789/0 conn(0x55d9281fc400 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=0).handle_write
2018-12-12 22:35:30.513 7f51442ed700 10 -- 192.168.1.58:6789/0 >> 192.168.1.65:6789/0 conn(0x55d9281fca00 :-1 s=STATE_CONNECTING pgs=0 cs=0 l=0).handle_write
2018-12-12 22:35:30.513 7f5144aee700  5 Infiniband recv_msg recevd: 105, 0, 262144, 0,
2018-12-12 22:35:30.513 7f5144aee700 -1  RDMAConnectedSocketImpl activate failed to transition to RTR state: (113) No route to host
2018-12-12 22:35:30.515 7f5144aee700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.2/rpm/el7/BUILD/ceph-13.2.2/src/msg/async/rdma/RDMAConnectedSocketImpl.cc: In function 'void RDMAConnectedSocketImpl::handle_connection()' thread 7f5144aee700 time 2018-12-12 22:35:30.514762
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.2/rpm/el7/BUILD/ceph-13.2.2/src/msg/async/rdma/RDMAConnectedSocketImpl.cc: 224: FAILED assert(!r)

 ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f515506e6bf]
 2: (()+0x285887) [0x7f515506e887]
 3: (RDMAConnectedSocketImpl::handle_connection()+0x6f5) [0x7f51551b9655]
 4: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x695) [0x7f51551a76f5]
 5: (()+0x3c15cc) [0x7f51551aa5cc]
 6: (()+0x6afaef) [0x7f5155498aef]
 7: (()+0x7e25) [0x7f5154396e25]
 8: (clone()+0x6d) [0x7f5150cb7bad]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
------ceph-mon.rio.log--end------

-----ceph.conf---begin-----
[client]
rbd_cache = False
rbd_cache_writethrough_until_flush = False
#admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
#log file = /var/log/ceph/

[global]
ms_type = async+posix
# set RDMA messaging just for the public or cluster network
ms_public_type = async+rdma
ms_cluster = async+rdma
#
# set a device name according to IB or ROCE device used, e.g.
ms_async_rdma_device_name = mlx5_0
#
# for better performance if using LUMINOUS 12.2.x release
ms_async_rdma_polling_us = 0
ms_async_rdma_port_num=1

rgw_override_bucket_index_max_shards=1
auth client required = none
auth cluster required = none
auth service required = none
auth supported = none
cephx require signatures = False
cephx sign messages = False
cluster network = 192.168.1.0/24
debug asok = 0/0
debug auth = 0/0
debug bluefs = 0/0
debug bluestore = 0/0
debug buffer = 0/0
debug client = 0/0
debug context = 0/0
debug crush = 0/0
debug filer = 0/0
debug filestore = 0/0
debug finisher = 0/0
#debug hadoop = 0/0 -- doesn't work in E8 setup
debug heartbeatmap = 0/0
debug journal = 0/0
debug journaler = 0/0
debug lockdep = 0/0
debug log = 0
debug mds = 0/0
debug mds_balancer = 0/0
debug mds_locker = 0/0
debug mds_log = 0/0
debug mds_log_expire = 0/0
debug mds_migrator = 0/0
debug mon = 0/0
debug monc = 0/0
debug ms = 10/10
#debug ms = 0/0
debug objclass = 0/0
debug objectcacher = 0/0
debug objecter = 0/0
debug optracker = 0/0
debug osd = 0/0
debug paxos = 0/0
debug perfcounter = 0/0
debug rados = 0/0
debug rbd = 0/0
debug rgw = 0/0
debug rocksdb = 0/0
debug throttle = 0/0
debug timer = 0/0
debug tp = 0/0
#debug zs = 0/0 -- doesn't work in E8 setup
fsid = 376540c8-a362-41cc-9a58-9c8ceca0e4ee
mon_host = 192.168.1.58,192.168.1.65,192.168.1.59
mon pg warn max per osd = 800
mon_allow_pool_delete = True
mon_max_pg_per_osd = 800
ms type = async
ms_crc_data = False
#ms_crc_header = False -- broken!
osd objectstore = bluestore
osd_pool_default_size = 2
perf = True
public network = 192.168.1.0/24
rocksdb_perf = True
# Parameter not present in Micron's config, but introduced by ceph-deploy
mon_initial_members = rio
# The following param claims to reduce CPU usage; found it at
ms nocrc = true
#rbd_op_threads=4
[mon]
mon_max_pool_pg_num = 166496
mon_osd_max_split_count = 10000
[osd]
osd_min_pg_log_entries = 10
osd_max_pg_log_entries = 10
osd_pg_log_dups_tracked = 10
osd_pg_log_trim_min = 10
bluestore_cache_kv_max = 96G
bluestore_cache_kv_ratio = 0.2
bluestore_cache_meta_ratio = 0.8
bluestore_cache_size_ssd = 7G
bluestore_csum_type = none
bluestore_extent_map_shard_max_size = 200
bluestore_extent_map_shard_min_size = 50
bluestore_extent_map_shard_target_size = 100

bluestore_rocksdb_options = compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB

[client.rgw.sm26]
rgw_frontends = "civetweb port=7480"

#[osd.0]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.1]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.2]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.3]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.4]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.5]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.6]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.7]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.8]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.9]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.10]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.11]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.12]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.13]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.14]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
#[osd.15]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01ae
#
## BONJOVI1
#
#[osd.16]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.17]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.18]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.19]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.20]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.21]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.22]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.23]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.24]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.25]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.26]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.27]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.28]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.29]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.30]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
#[osd.31]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:01af
#
## PRINCE
#
#[osd.32]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.33]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#[osd.34]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.35]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.36]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.37]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.38]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.39]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.40]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.41]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.42]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.43]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.44]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.45]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.46]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#[osd.47]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0198
#
#
## RINGO
#
#[osd.48]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.49]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.50]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.51]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.52]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.53]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.54]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.55]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.56]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.57]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.58]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.59]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.60]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.61]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.62]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[osd.63]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:018e
#
#[mon.rio]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:013a
#
#[mon.salvador]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:013b
#
#[mon.medellin]
#ms_async_rdma_local_gid=0000:0000:0000:0000:0000:ffff:c0a8:0141

-----ceph.conf---end-----


--
Michael Green











--
With best regards,
Vitaliy Filippov

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux