Hi Missed that last one, no theres nothing special there really. Here's the conf, very simple. [global] fsid = a2974742-3805-4cd3-bc79-765f2bddaefe mon_initial_members = ceph1, ceph2, ceph3 mon_host = 10.20.0.11,10.20.0.12,10.20.0.13 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true osd_pool_default_pg_num = 128 osd_pool_default_pgp_num = 128 public_network = 10.20.0.0/16 Br, Tuomas -----Original Message----- From: Sage Weil [mailto:sweil@xxxxxxxxxx] Sent: 28. huhtikuuta 2015 21:57 To: Tuomas Juntunen Cc: ceph-users@xxxxxxxxxxxxxx Subject: RE: Upgrade from Giant to Hammer and after some basic operations most of the OSD's went down On Tue, 28 Apr 2015, Tuomas Juntunen wrote: > Hi > > Here's the file. Hrm. Can you also attach ceph osd dump -f json-pretty 17846 > The cluster was working totally fine for months before this, no > problems at all. > > 'ceph osd tree' still shows 11 osd's up, even though all of them are down. > If I add a new one, that will get added to the list and it goes up, > but when I stop it, the status will still show it as up. > > Here's a log for that osd http://beta.xaasbox.com/ceph/ceph-osd.37.log Does your ceph.conf have any changes to osd_heartbeat_* configurables? It is very strange that the OSDs aren't getting marked down. sage > > Br, > Tuomas > > > -----Original Message----- > From: Sage Weil [mailto:sweil@xxxxxxxxxx] > Sent: 28. huhtikuuta 2015 20:02 > To: Tuomas Juntunen > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: RE: Upgrade from Giant to Hammer and after some > basic operations most of the OSD's went down > > On Tue, 28 Apr 2015, Tuomas Juntunen wrote: > > Just to add some more interesting behavior to my problem, is that > > monitors are not updating the status of OSD's. > > Yeah, this is very strange. I see > > 2015-04-27 22:25:26.142245 7f78d793a900 10 osd.15 17882 _get_pool 4 > cached_removed_snaps [1~1,4~a,f~2,12~10,23~4,28~2,2e~2,32~c,3f~4] > > but pool 4 (images) has no snaps and a zeroed snap_seq in the osdmap > dump you provided. Can you attach > > ceph osd dump -f json-pretty 17882 > > ? > > The fact that your mon status isn't changing makes me wonder if these > are two different ceph clusters or something similarly silly... if > the mds daemons are down the status should show stale. If all osds > stop you might not see the osds go down for ~10 minutes (since they > aren't monitoring each other). > > sage > > > > > Even when I stop all the remaining OSD's, ceph osd tree shows them as up. > > Also there the status of mons and mds doesn't seem to update > > correctly in my opinion. > > > > Below is a copy of status when one mon and mds are stopped and all > > of the osd's are also stopped. > > > > monmap e7: 3 mons at > > {ceph1=10.20.0.11:6789/0,ceph2=10.20.0.12:6789/0,ceph3=10.20.0.13:6789/0} > > election epoch 48, quorum 0,1 ceph1,ceph2 > > mdsmap e1750: 1/1/1 up {0=ceph2=up:replay}, 1 up:standby > > osdmap e18132: 37 osds: 11 up, 11 in > > > > Br, > > Tuomas > > > > > > -----Original Message----- > > From: Sage Weil [mailto:sweil@xxxxxxxxxx] > > Sent: 27. huhtikuuta 2015 22:22 > > To: Tuomas Juntunen > > Cc: ceph-users@xxxxxxxxxxxxxx > > Subject: RE: Upgrade from Giant to Hammer and after > > some basic operations most of the OSD's went down > > > > On Mon, 27 Apr 2015, Tuomas Juntunen wrote: > > > Hey > > > > > > Got the log, you can get it from > > > http://beta.xaasbox.com/ceph/ceph-osd.15.log > > > > Can you repeat this with 'debug osd = 20'? Thanks! > > > > sage > > > > > > > > Br, > > > Tuomas > > > > > > > > > -----Original Message----- > > > From: Sage Weil [mailto:sweil@xxxxxxxxxx] > > > Sent: 27. huhtikuuta 2015 20:45 > > > To: Tuomas Juntunen > > > Cc: ceph-users@xxxxxxxxxxxxxx > > > Subject: Re: Upgrade from Giant to Hammer and after > > > some basic operations most of the OSD's went down > > > > > > Yeah, no snaps: > > > > > > images: > > > "snap_mode": "selfmanaged", > > > "snap_seq": 0, > > > "snap_epoch": 17882, > > > "pool_snaps": [], > > > "removed_snaps": "[]", > > > > > > img: > > > "snap_mode": "selfmanaged", > > > "snap_seq": 0, > > > "snap_epoch": 0, > > > "pool_snaps": [], > > > "removed_snaps": "[]", > > > > > > ...and actually the log shows this happens on pool 2 (rbd), which > > > has > > > > > > "snap_mode": "selfmanaged", > > > "snap_seq": 0, > > > "snap_epoch": 0, > > > "pool_snaps": [], > > > "removed_snaps": "[]", > > > > > > I'm guessin gthe offending code is > > > > > > pi->build_removed_snaps(newly_removed_snaps); > > > newly_removed_snaps.subtract(cached_removed_snaps); > > > > > > so newly_removed_snaps should be empty, and apparently > > > cached_removed_snaps is not? Maybe one of your older osdmaps has > > > snap info for rbd? It doesn't make sense. :/ Maybe > > > > > > ceph osd dump 18127 -f json-pretty > > > > > > just to be certain? I've pushed a branch 'wip-hammer-snaps' that > > > will appear at gitbuilder.ceph.com in 20-30 minutes that will > > > output some additional debug info. It will be at > > > > > > > > > http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/ref/wip-ha > > > mm > > > er > > > -sanps > > > > > > or similar, depending on your distro. Can you install it one on > > > node and start and osd with logging to reproduce the crash? > > > > > > Thanks! > > > sage > > > > > > > > > On Mon, 27 Apr 2015, Tuomas Juntunen wrote: > > > > > > > Hi > > > > > > > > Here you go > > > > > > > > Br, > > > > Tuomas > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Sage Weil [mailto:sweil@xxxxxxxxxx] > > > > Sent: 27. huhtikuuta 2015 19:23 > > > > To: Tuomas Juntunen > > > > Cc: 'Samuel Just'; ceph-users@xxxxxxxxxxxxxx > > > > Subject: Re: Upgrade from Giant to Hammer and after > > > > some basic operations most of the OSD's went down > > > > > > > > On Mon, 27 Apr 2015, Tuomas Juntunen wrote: > > > > > Thanks for the info. > > > > > > > > > > For my knowledge there was no snapshots on that pool, but > > > > > cannot verify that. > > > > > > > > Can you attach a 'ceph osd dump -f json-pretty'? That will shed > > > > a bit more light on what happened (and the simplest way to fix it). > > > > > > > > sage > > > > > > > > > > > > > Any way to make this work again? Removing the tier and other > > > > > settings didn't fix it, I tried it the second this happened. > > > > > > > > > > Br, > > > > > Tuomas > > > > > > > > > > -----Original Message----- > > > > > From: Samuel Just [mailto:sjust@xxxxxxxxxx] > > > > > Sent: 27. huhtikuuta 2015 15:50 > > > > > To: tuomas juntunen > > > > > Cc: ceph-users@xxxxxxxxxxxxxx > > > > > Subject: Re: Upgrade from Giant to Hammer and > > > > > after some basic operations most of the OSD's went down > > > > > > > > > > So, the base tier is what determines the snapshots for the > > > > > cache/base pool > > > > amalgam. You added a populated pool complete with snapshots on > > > > top of a base tier without snapshots. Apparently, it caused an > > > > existential crisis for the snapshot code. That's one of the > > > > reasons why there is a --force-nonempty flag for that operation, > > > > I think. I think the immediate answer is probably to disallow > > > > pools with snapshots as a cache tier altogether until we think > > > > of a good way to > > make it work. > > > > > -Sam > > > > > > > > > > ----- Original Message ----- > > > > > From: "tuomas juntunen" <tuomas.juntunen@xxxxxxxxxxxxxxx> > > > > > To: "Samuel Just" <sjust@xxxxxxxxxx> > > > > > Cc: ceph-users@xxxxxxxxxxxxxx > > > > > Sent: Monday, April 27, 2015 4:56:58 AM > > > > > Subject: Re: Upgrade from Giant to Hammer and > > > > > after some basic operations most of the OSD's went down > > > > > > > > > > > > > > > > > > > > The following: > > > > > > > > > > ceph osd tier add img images --force-nonempty ceph osd tier > > > > > cache-mode images forward ceph osd tier set-overlay img images > > > > > > > > > > Idea was to make images as a tier to img, move data to img > > > > > then change > > > > clients to use the new img pool. > > > > > > > > > > Br, > > > > > Tuomas > > > > > > > > > > > Can you explain exactly what you mean by: > > > > > > > > > > > > "Also I created one pool for tier to be able to move data > > > > > > without > > > > outage." > > > > > > > > > > > > -Sam > > > > > > ----- Original Message ----- > > > > > > From: "tuomas juntunen" <tuomas.juntunen@xxxxxxxxxxxxxxx> > > > > > > To: "Ian Colle" <icolle@xxxxxxxxxx> > > > > > > Cc: ceph-users@xxxxxxxxxxxxxx > > > > > > Sent: Monday, April 27, 2015 4:23:44 AM > > > > > > Subject: Re: Upgrade from Giant to Hammer and > > > > > > after some basic operations most of the OSD's went down > > > > > > > > > > > > Hi > > > > > > > > > > > > Any solution for this yet? > > > > > > > > > > > > Br, > > > > > > Tuomas > > > > > > > > > > > >> It looks like you may have hit > > > > > >> http://tracker.ceph.com/issues/7915 > > > > > >> > > > > > >> Ian R. Colle > > > > > >> Global Director > > > > > >> of Software Engineering > > > > > >> Red Hat (Inktank is now part of Red Hat!) > > > > > >> http://www.linkedin.com/in/ircolle > > > > > >> http://www.twitter.com/ircolle > > > > > >> Cell: +1.303.601.7713 > > > > > >> Email: icolle@xxxxxxxxxx > > > > > >> > > > > > >> ----- Original Message ----- > > > > > >> From: "tuomas juntunen" <tuomas.juntunen@xxxxxxxxxxxxxxx> > > > > > >> To: ceph-users@xxxxxxxxxxxxxx > > > > > >> Sent: Monday, April 27, 2015 1:56:29 PM > > > > > >> Subject: Upgrade from Giant to Hammer and > > > > > >> after some basic operations most of the OSD's went down > > > > > >> > > > > > >> > > > > > >> > > > > > >> I upgraded Ceph from 0.87 Giant to 0.94.1 Hammer > > > > > >> > > > > > >> Then created new pools and deleted some old ones. Also I > > > > > >> created one pool for tier to be able to move data without outage. > > > > > >> > > > > > >> After these operations all but 10 OSD's are down and > > > > > >> creating this kind of messages to logs, I get more than > > > > > >> 100gb of these in a > > > night: > > > > > >> > > > > > >> -19> 2015-04-27 10:17:08.808584 7fd8e748d700 5 osd.23 pg_epoch: > > > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0 > > > > > >> ec=1 les/c > > > > > >> 16609/16659 > > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 > > > > > >> pi=15659-16589/42 > > > > > >> crt=8480'7 lcod > > > > > >> 0'0 inactive NOTIFY] enter Started > > > > > >> -18> 2015-04-27 10:17:08.808596 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0 > > > > > >> ec=1 les/c > > > > > >> 16609/16659 > > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 > > > > > >> pi=15659-16589/42 > > > > > >> crt=8480'7 lcod > > > > > >> 0'0 inactive NOTIFY] enter Start > > > > > >> -17> 2015-04-27 10:17:08.808608 7fd8e748d700 1 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0 > > > > > >> ec=1 les/c > > > > > >> 16609/16659 > > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 > > > > > >> pi=15659-16589/42 > > > > > >> crt=8480'7 lcod > > > > > >> 0'0 inactive NOTIFY] state<Start>: transitioning to Stray > > > > > >> -16> 2015-04-27 10:17:08.808621 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0 > > > > > >> ec=1 les/c > > > > > >> 16609/16659 > > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 > > > > > >> pi=15659-16589/42 > > > > > >> crt=8480'7 lcod > > > > > >> 0'0 inactive NOTIFY] exit Start 0.000025 0 0.000000 > > > > > >> -15> 2015-04-27 10:17:08.808637 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0 > > > > > >> ec=1 les/c > > > > > >> 16609/16659 > > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 > > > > > >> pi=15659-16589/42 > > > > > >> crt=8480'7 lcod > > > > > >> 0'0 inactive NOTIFY] enter Started/Stray > > > > > >> -14> 2015-04-27 10:17:08.808796 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c > > > > > >> 17879/17879 > > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive > > > > > >> NOTIFY] exit Reset 0.119467 4 0.000037 > > > > > >> -13> 2015-04-27 10:17:08.808817 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c > > > > > >> 17879/17879 > > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive > > > > > >> NOTIFY] enter Started > > > > > >> -12> 2015-04-27 10:17:08.808828 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c > > > > > >> 17879/17879 > > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive > > > > > >> NOTIFY] enter Start > > > > > >> -11> 2015-04-27 10:17:08.808838 7fd8e748d700 1 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c > > > > > >> 17879/17879 > > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive > > > > > >> NOTIFY] > > > > > >> state<Start>: transitioning to Stray > > > > > >> -10> 2015-04-27 10:17:08.808849 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c > > > > > >> 17879/17879 > > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive > > > > > >> NOTIFY] exit Start 0.000020 0 0.000000 > > > > > >> -9> 2015-04-27 10:17:08.808861 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c > > > > > >> 17879/17879 > > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive > > > > > >> NOTIFY] enter Started/Stray > > > > > >> -8> 2015-04-27 10:17:08.809427 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c > > > > > >> 16127/16344 > > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 > > > > > >> inactive] exit Reset 7.511623 45 0.000165 > > > > > >> -7> 2015-04-27 10:17:08.809445 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c > > > > > >> 16127/16344 > > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 > > > > > >> inactive] enter Started > > > > > >> -6> 2015-04-27 10:17:08.809456 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c > > > > > >> 16127/16344 > > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 > > > > > >> inactive] enter Start > > > > > >> -5> 2015-04-27 10:17:08.809468 7fd8e748d700 1 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c > > > > > >> 16127/16344 > > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 > > > > > >> inactive] > > > > > >> state<Start>: transitioning to Primary > > > > > >> -4> 2015-04-27 10:17:08.809479 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c > > > > > >> 16127/16344 > > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 > > > > > >> inactive] exit Start 0.000023 0 0.000000 > > > > > >> -3> 2015-04-27 10:17:08.809492 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c > > > > > >> 16127/16344 > > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 > > > > > >> inactive] enter Started/Primary > > > > > >> -2> 2015-04-27 10:17:08.809502 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c > > > > > >> 16127/16344 > > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 > > > > > >> inactive] enter Started/Primary/Peering > > > > > >> -1> 2015-04-27 10:17:08.809513 7fd8e748d700 5 osd.23 > pg_epoch: > > > > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c > > > > > >> 16127/16344 > > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 > > > > > >> peering] enter Started/Primary/Peering/GetInfo > > > > > >> 0> 2015-04-27 10:17:08.813837 7fd8e748d700 -1 > > > > ./include/interval_set.h: > > > > > >> In > > > > > >> function 'void interval_set<T>::erase(T, T) [with T = snapid_t]' > > > > > >> thread > > > > > >> 7fd8e748d700 time 2015-04-27 10:17:08.809899 > > > > > >> ./include/interval_set.h: 385: FAILED assert(_size >= 0) > > > > > >> > > > > > >> ceph version 0.94.1 > > > > > >> (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) > > > > > >> 1: (ceph::__ceph_assert_fail(char const*, char const*, > > > > > >> int, char > > > > > >> const*)+0x8b) > > > > > >> [0xbc271b] > > > > > >> 2: > > > > > >> (interval_set<snapid_t>::subtract(interval_set<snapid_t> > > > > > >> const&)+0xb0) [0x82cd50] > > > > > >> 3: (PGPool::update(std::tr1::shared_ptr<OSDMap > > > > > >> const>)+0x52e) [0x80113e] > > > > > >> 4: (PG::handle_advance_map(std::tr1::shared_ptr<OSDMap > > > > > >> const>, std::tr1::shared_ptr<OSDMap const>, > > > > > >> const>std::vector<int, > > > > > >> std::allocator<int> >&, int, std::vector<int, > > > > > >> std::allocator<int> > > > > > >> >&, int, PG::RecoveryCtx*)+0x282) [0x801652] > > > > > >> 5: (OSD::advance_pg(unsigned int, PG*, > > > > > >> ThreadPool::TPHandle&, PG::RecoveryCtx*, > > > > > >> std::set<boost::intrusive_ptr<PG>, > > > > > >> std::less<boost::intrusive_ptr<PG> >, > > > > > >> std::allocator<boost::intrusive_ptr<PG> > >*)+0x2c3) > > > > > >> [0x6b0e43] > > > > > >> 6: (OSD::process_peering_events(std::list<PG*, > > > > > >> std::allocator<PG*> > > > > > >> > const&, > > > > > >> ThreadPool::TPHandle&)+0x21c) [0x6b191c] > > > > > >> 7: (OSD::PeeringWQ::_process(std::list<PG*, > > > > > >> std::allocator<PG*> > > > > > >> > const&, > > > > > >> ThreadPool::TPHandle&)+0x18) [0x709278] > > > > > >> 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e) > > > > > >> [0xbb38ae] > > > > > >> 9: (ThreadPool::WorkThread::entry()+0x10) [0xbb4950] > > > > > >> 10: (()+0x8182) [0x7fd906946182] > > > > > >> 11: (clone()+0x6d) [0x7fd904eb147d] > > > > > >> > > > > > >> Also by monitoring (ceph -w) I get the following messages, > > > > > >> also lots of > > > > them. > > > > > >> > > > > > >> 2015-04-27 10:39:52.935812 mon.0 [INF] from='client.? > > > > 10.20.0.13:0/1174409' > > > > > >> entity='osd.30' cmd=[{"prefix": "osd crush create-or-move", > "args": > > > > > >> ["host=ceph3", "root=default"], "id": 30, "weight": 1.82}]: > > > > > >> dispatch > > > > > >> 2015-04-27 10:39:53.297376 mon.0 [INF] from='client.? > > > > 10.20.0.13:0/1174483' > > > > > >> entity='osd.26' cmd=[{"prefix": "osd crush create-or-move", > "args": > > > > > >> ["host=ceph3", "root=default"], "id": 26, "weight": 1.82}]: > > > > > >> dispatch > > > > > >> > > > > > >> > > > > > >> This is a cluster of 3 nodes with 36 OSD's, nodes are also > > > > > >> mons and mds's to save servers. All run Ubuntu 14.04.2. > > > > > >> > > > > > >> I have pretty much tried everything I could think of. > > > > > >> > > > > > >> Restarting daemons doesn't help. > > > > > >> > > > > > >> Any help would be appreciated. I can also provide more logs > > > > > >> if necessary. They just seem to get pretty large in few moments. > > > > > >> > > > > > >> Thank you > > > > > >> Tuomas > > > > > >> > > > > > >> > > > > > >> _______________________________________________ > > > > > >> ceph-users mailing list > > > > > >> ceph-users@xxxxxxxxxxxxxx > > > > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list > > > > > > ceph-users@xxxxxxxxxxxxxx > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > ceph-users mailing list > > > > > ceph-users@xxxxxxxxxxxxxx > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > ceph-users mailing list > > > > > ceph-users@xxxxxxxxxxxxxx > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |