Yes, the unfound objects are due to a bug in the repair command. I suggest you don't repair anything, actually. I don't think any of the pgs are actually inconsistent. The log I have here is definitely a case of two objects showing up as inconsistent which are actually fine. -Sam On Mon, Mar 7, 2016 at 4:38 PM, Jeffrey McDonald <jmcdonal@xxxxxxx> wrote: > I have three more pgs now showing up as inconsistent. Should I turn on > debug for all OSDs to capture the transition from active+clean -> > inconsistent? Do I understand correction that the repair cause the > 'unfound' objects because of a bug in the repair command? > Regards, > Jeff > > > > On Mon, Mar 7, 2016 at 6:36 PM, Samuel Just <sjust@xxxxxxxxxx> wrote: >> >> Hmm, so much for that theory, still looking. If you can produce >> another set of logs (as before) from scrubbing that pg, it might help. >> -Sam >> >> On Mon, Mar 7, 2016 at 4:34 PM, Jeffrey McDonald <jmcdonal@xxxxxxx> wrote: >> > they're all the same.....see attached. >> > >> > On Mon, Mar 7, 2016 at 6:31 PM, Samuel Just <sjust@xxxxxxxxxx> wrote: >> >> >> >> Have you confirmed the versions? >> >> -Sam >> >> >> >> On Mon, Mar 7, 2016 at 4:29 PM, Jeffrey McDonald <jmcdonal@xxxxxxx> >> >> wrote: >> >> > I have one other very strange event happening, I've opened a ticket >> >> > on >> >> > it: >> >> > http://tracker.ceph.com/issues/14766 >> >> > >> >> > During this migration, OSD failed probably over 400 times while >> >> > moving >> >> > data >> >> > around. We move the empty directories and restarted the OSDs. I >> >> > can't >> >> > say if this is related--I have no reason to suspect it is. >> >> > >> >> > Jeff >> >> > >> >> > On Mon, Mar 7, 2016 at 5:31 PM, Shinobu Kinjo <shinobu.kj@xxxxxxxxx> >> >> > wrote: >> >> >> >> >> >> What could cause this kind of unexpected behaviour? >> >> >> Any assumption?? >> >> >> Sorry for interrupting you. >> >> >> >> >> >> Cheers, >> >> >> S >> >> >> >> >> >> On Tue, Mar 8, 2016 at 8:19 AM, Samuel Just <sjust@xxxxxxxxxx> >> >> >> wrote: >> >> >> > Hmm, at the end of the log, the pg is still inconsistent. Can you >> >> >> > attach a ceph pg query on that pg? >> >> >> > -Sam >> >> >> > >> >> >> > On Mon, Mar 7, 2016 at 3:05 PM, Samuel Just <sjust@xxxxxxxxxx> >> >> >> > wrote: >> >> >> >> If so, that strongly suggests that the pg was actually never >> >> >> >> inconsistent in the first place and that the bug is in scrub >> >> >> >> itself >> >> >> >> presumably getting confused about an object during a write. The >> >> >> >> next >> >> >> >> step would be to get logs like the above from a pg as it scrubs >> >> >> >> transitioning from clean to inconsistent. If it's really a race >> >> >> >> between scrub and a write, it's probably just non-deterministic, >> >> >> >> you >> >> >> >> could set logging on a set of osds and continuously scrub any pgs >> >> >> >> which only map to those osds until you reproduce the problem. >> >> >> >> -Sam >> >> >> >> >> >> >> >> On Mon, Mar 7, 2016 at 2:44 PM, Samuel Just <sjust@xxxxxxxxxx> >> >> >> >> wrote: >> >> >> >>> So after the scrub, it came up clean? The inconsistent/missing >> >> >> >>> objects reappeared? >> >> >> >>> -Sam >> >> >> >>> >> >> >> >>> On Mon, Mar 7, 2016 at 2:33 PM, Jeffrey McDonald >> >> >> >>> <jmcdonal@xxxxxxx> >> >> >> >>> wrote: >> >> >> >>>> Hi Sam, >> >> >> >>>> >> >> >> >>>> I've done as you requested: >> >> >> >>>> >> >> >> >>>> pg 70.459 is active+clean+inconsistent, acting >> >> >> >>>> [307,210,273,191,132,450] >> >> >> >>>> >> >> >> >>>> # for i in 307 210 273 191 132 450 ; do >> >> >> >>>>> ceph tell osd.$i injectargs '--debug-osd 20 --debug-filestore >> >> >> >>>>> 20 >> >> >> >>>>> --debug-ms 1' >> >> >> >>>>> done >> >> >> >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1 >> >> >> >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1 >> >> >> >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1 >> >> >> >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1 >> >> >> >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1 >> >> >> >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1 >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> # date >> >> >> >>>> Mon Mar 7 16:03:38 CST 2016 >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> # ceph pg deep-scrub 70.459 >> >> >> >>>> instructing pg 70.459 on osd.307 to deep-scrub >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> Scrub finished around >> >> >> >>>> >> >> >> >>>> # date >> >> >> >>>> Mon Mar 7 16:13:03 CST 2016 >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> I've tar'd+gziped the files which can be downloaded from here. >> >> >> >>>> The >> >> >> >>>> logs >> >> >> >>>> start a minute or two after today at 16:00. >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> https://drive.google.com/folderview?id=0Bzz8TrxFvfema2NQUmotd1BOTnM&usp=sharing >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> Oddly(to me anyways), this pg is now active+clean: >> >> >> >>>> >> >> >> >>>> # ceph pg dump | grep 70.459 >> >> >> >>>> dumped all in format plain >> >> >> >>>> 70.459 21377 0 0 0 0 64515446306 3088 3088 active+clean >> >> >> >>>> 2016-03-07 >> >> >> >>>> 16:26:57.796537 279563'212832 279602:628151 >> >> >> >>>> [307,210,273,191,132,450] >> >> >> >>>> 307 >> >> >> >>>> [307,210,273,191,132,450] 307 279563'212832 2016-03-07 >> >> >> >>>> 16:12:30.741984 >> >> >> >>>> 279563'212832 2016-03-07 16:12:30.741984 >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> Regards, >> >> >> >>>> Jeff >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> On Mon, Mar 7, 2016 at 4:11 PM, Samuel Just <sjust@xxxxxxxxxx> >> >> >> >>>> wrote: >> >> >> >>>>> >> >> >> >>>>> I think the unfound object on repair is fixed by >> >> >> >>>>> d51806f5b330d5f112281fbb95ea6addf994324e (not in hammer yet). >> >> >> >>>>> I >> >> >> >>>>> opened http://tracker.ceph.com/issues/15002 for the backport >> >> >> >>>>> and >> >> >> >>>>> to >> >> >> >>>>> make sure it's covered in ceph-qa-suite. No idea at this time >> >> >> >>>>> why >> >> >> >>>>> the >> >> >> >>>>> objects are disappearing though. >> >> >> >>>>> -Sam >> >> >> >>>>> >> >> >> >>>>> On Mon, Mar 7, 2016 at 1:57 PM, Samuel Just <sjust@xxxxxxxxxx> >> >> >> >>>>> wrote: >> >> >> >>>>> > The one just scrubbed and now inconsistent. >> >> >> >>>>> > -Sam >> >> >> >>>>> > >> >> >> >>>>> > On Mon, Mar 7, 2016 at 1:57 PM, Jeffrey McDonald >> >> >> >>>>> > <jmcdonal@xxxxxxx> >> >> >> >>>>> > wrote: >> >> >> >>>>> >> Do you want me to enable this for the pg already with >> >> >> >>>>> >> unfound >> >> >> >>>>> >> objects >> >> >> >>>>> >> or the >> >> >> >>>>> >> placement group just scrubbed and now inconsistent? >> >> >> >>>>> >> Jeff >> >> >> >>>>> >> >> >> >> >>>>> >> On Mon, Mar 7, 2016 at 3:54 PM, Samuel Just >> >> >> >>>>> >> <sjust@xxxxxxxxxx> >> >> >> >>>>> >> wrote: >> >> >> >>>>> >>> >> >> >> >>>>> >>> Can you enable >> >> >> >>>>> >>> >> >> >> >>>>> >>> debug osd = 20 >> >> >> >>>>> >>> debug filestore = 20 >> >> >> >>>>> >>> debug ms = 1 >> >> >> >>>>> >>> >> >> >> >>>>> >>> on all osds in that PG, rescrub, and convey to us the >> >> >> >>>>> >>> resulting >> >> >> >>>>> >>> logs? >> >> >> >>>>> >>> -Sam >> >> >> >>>>> >>> >> >> >> >>>>> >>> On Mon, Mar 7, 2016 at 1:36 PM, Jeffrey McDonald >> >> >> >>>>> >>> <jmcdonal@xxxxxxx> >> >> >> >>>>> >>> wrote: >> >> >> >>>>> >>> > Here is a PG which just went inconsistent: >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > pg 70.459 is active+clean+inconsistent, acting >> >> >> >>>>> >>> > [307,210,273,191,132,450] >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > Attached is the result of a pg query on this. I will >> >> >> >>>>> >>> > wait >> >> >> >>>>> >>> > for your >> >> >> >>>>> >>> > feedback before issuing a repair. >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > From what I read, the inconsistencies are more likely >> >> >> >>>>> >>> > the >> >> >> >>>>> >>> > result of >> >> >> >>>>> >>> > ntp, >> >> >> >>>>> >>> > but >> >> >> >>>>> >>> > all nodes have the local ntp master and all are showing >> >> >> >>>>> >>> > sync. >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > Regards, >> >> >> >>>>> >>> > Jeff >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > On Mon, Mar 7, 2016 at 3:15 PM, Gregory Farnum >> >> >> >>>>> >>> > <gfarnum@xxxxxxxxxx> >> >> >> >>>>> >>> > wrote: >> >> >> >>>>> >>> >> >> >> >> >>>>> >>> >> [ Keeping this on the users list. ] >> >> >> >>>>> >>> >> >> >> >> >>>>> >>> >> Okay, so next time this happens you probably want to do >> >> >> >>>>> >>> >> a >> >> >> >>>>> >>> >> pg >> >> >> >>>>> >>> >> query >> >> >> >>>>> >>> >> on >> >> >> >>>>> >>> >> the PG which has been reported as dirty. I can't help >> >> >> >>>>> >>> >> much >> >> >> >>>>> >>> >> beyond >> >> >> >>>>> >>> >> that, but hopefully Kefu or David will chime in once >> >> >> >>>>> >>> >> there's >> >> >> >>>>> >>> >> a >> >> >> >>>>> >>> >> little >> >> >> >>>>> >>> >> more for them to look at. >> >> >> >>>>> >>> >> -Greg >> >> >> >>>>> >>> >> >> >> >> >>>>> >>> >> On Mon, Mar 7, 2016 at 1:00 PM, Jeffrey McDonald >> >> >> >>>>> >>> >> <jmcdonal@xxxxxxx> >> >> >> >>>>> >>> >> wrote: >> >> >> >>>>> >>> >> > Hi Greg, >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > I'm running the ceph version hammer, >> >> >> >>>>> >>> >> > ceph version 0.94.5 >> >> >> >>>>> >>> >> > (9764da52395923e0b32908d83a9f7304401fee43) >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > The hardware migration was performed by just setting >> >> >> >>>>> >>> >> > the >> >> >> >>>>> >>> >> > crush >> >> >> >>>>> >>> >> > map to >> >> >> >>>>> >>> >> > zero >> >> >> >>>>> >>> >> > for the OSD we wanted to retire. The system was >> >> >> >>>>> >>> >> > performing >> >> >> >>>>> >>> >> > poorly >> >> >> >>>>> >>> >> > with >> >> >> >>>>> >>> >> > these older OSDs and we had a difficult time >> >> >> >>>>> >>> >> > maintaining >> >> >> >>>>> >>> >> > stability of >> >> >> >>>>> >>> >> > the >> >> >> >>>>> >>> >> > system. The old OSDs are still there but all of >> >> >> >>>>> >>> >> > the >> >> >> >>>>> >>> >> > data >> >> >> >>>>> >>> >> > is >> >> >> >>>>> >>> >> > now >> >> >> >>>>> >>> >> > migrated >> >> >> >>>>> >>> >> > to new and/or existing hardware. >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > Thanks, >> >> >> >>>>> >>> >> > Jeff >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > On Mon, Mar 7, 2016 at 2:56 PM, Gregory Farnum >> >> >> >>>>> >>> >> > <gfarnum@xxxxxxxxxx> >> >> >> >>>>> >>> >> > wrote: >> >> >> >>>>> >>> >> >> >> >> >> >>>>> >>> >> >> On Mon, Mar 7, 2016 at 12:07 PM, Jeffrey McDonald >> >> >> >>>>> >>> >> >> <jmcdonal@xxxxxxx> >> >> >> >>>>> >>> >> >> wrote: >> >> >> >>>>> >>> >> >> > Hi, >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > For a while, we've been seeing inconsistent >> >> >> >>>>> >>> >> >> > placement >> >> >> >>>>> >>> >> >> > groups >> >> >> >>>>> >>> >> >> > on >> >> >> >>>>> >>> >> >> > our >> >> >> >>>>> >>> >> >> > erasure >> >> >> >>>>> >>> >> >> > coded system. The placement groups go from a >> >> >> >>>>> >>> >> >> > state >> >> >> >>>>> >>> >> >> > of >> >> >> >>>>> >>> >> >> > active+clean >> >> >> >>>>> >>> >> >> > to >> >> >> >>>>> >>> >> >> > active+clean+inconsistent after a deep scrub: >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > 2016-03-07 13:45:42.044131 7f385d118700 -1 >> >> >> >>>>> >>> >> >> > log_channel(cluster) >> >> >> >>>>> >>> >> >> > log >> >> >> >>>>> >>> >> >> > [ERR] : >> >> >> >>>>> >>> >> >> > 70.320s0 deep-scrub stat mismatch, got 21446/21428 >> >> >> >>>>> >>> >> >> > objects, >> >> >> >>>>> >>> >> >> > 0/0 >> >> >> >>>>> >>> >> >> > clones, >> >> >> >>>>> >>> >> >> > 21446/21428 dirty, 0/0 omap, 0/0 hit_set_archive, >> >> >> >>>>> >>> >> >> > 0/0 >> >> >> >>>>> >>> >> >> > whiteouts, >> >> >> >>>>> >>> >> >> > 64682334170/64624353083 bytes,0/0 hit_set_archive >> >> >> >>>>> >>> >> >> > bytes. >> >> >> >>>>> >>> >> >> > 2016-03-07 13:45:42.044416 7f385d118700 -1 >> >> >> >>>>> >>> >> >> > log_channel(cluster) >> >> >> >>>>> >>> >> >> > log >> >> >> >>>>> >>> >> >> > [ERR] : >> >> >> >>>>> >>> >> >> > 70.320s0 deep-scrub 18 missing, 0 inconsistent >> >> >> >>>>> >>> >> >> > objects >> >> >> >>>>> >>> >> >> > 2016-03-07 13:45:42.044464 7f385d118700 -1 >> >> >> >>>>> >>> >> >> > log_channel(cluster) >> >> >> >>>>> >>> >> >> > log >> >> >> >>>>> >>> >> >> > [ERR] : >> >> >> >>>>> >>> >> >> > 70.320 deep-scrub 73 errors >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > So I tell the placement group to perform a repair: >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > 2016-03-07 13:49:26.047177 7f385d118700 0 >> >> >> >>>>> >>> >> >> > log_channel(cluster) >> >> >> >>>>> >>> >> >> > log >> >> >> >>>>> >>> >> >> > [INF] : >> >> >> >>>>> >>> >> >> > 70.320 repair starts >> >> >> >>>>> >>> >> >> > 2016-03-07 13:49:57.087291 7f3858b0a700 0 -- >> >> >> >>>>> >>> >> >> > 10.31.0.2:6874/13937 >> >> >> >>>>> >>> >> >> > >> >> >> >> >>>>> >>> >> >> > 10.31.0.6:6824/8127 pipe(0x2e578000 sd=697 :6874 >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > The repair finds missing shards and repairs them, >> >> >> >>>>> >>> >> >> > but >> >> >> >>>>> >>> >> >> > then I >> >> >> >>>>> >>> >> >> > have >> >> >> >>>>> >>> >> >> > 18 >> >> >> >>>>> >>> >> >> > 'unfound objects' : >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > 2016-03-07 13:51:28.467590 7f385d118700 -1 >> >> >> >>>>> >>> >> >> > log_channel(cluster) >> >> >> >>>>> >>> >> >> > log >> >> >> >>>>> >>> >> >> > [ERR] : >> >> >> >>>>> >>> >> >> > 70.320s0 repair stat mismatch, got 21446/21428 >> >> >> >>>>> >>> >> >> > objects, >> >> >> >>>>> >>> >> >> > 0/0 >> >> >> >>>>> >>> >> >> > clones, >> >> >> >>>>> >>> >> >> > 21446/21428 dirty, 0/0 omap, 0/0 hit_set_archive, >> >> >> >>>>> >>> >> >> > 0/0 >> >> >> >>>>> >>> >> >> > whiteouts, >> >> >> >>>>> >>> >> >> > 64682334170/64624353083 bytes,0/0 hit_set_archive >> >> >> >>>>> >>> >> >> > bytes. >> >> >> >>>>> >>> >> >> > 2016-03-07 13:51:28.468358 7f385d118700 -1 >> >> >> >>>>> >>> >> >> > log_channel(cluster) >> >> >> >>>>> >>> >> >> > log >> >> >> >>>>> >>> >> >> > [ERR] : >> >> >> >>>>> >>> >> >> > 70.320s0 repair 18 missing, 0 inconsistent objects >> >> >> >>>>> >>> >> >> > 2016-03-07 13:51:28.469431 7f385d118700 -1 >> >> >> >>>>> >>> >> >> > log_channel(cluster) >> >> >> >>>>> >>> >> >> > log >> >> >> >>>>> >>> >> >> > [ERR] : >> >> >> >>>>> >>> >> >> > 70.320 repair 73 errors, 73 fixed >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > I've traced one of the unfound objects all the way >> >> >> >>>>> >>> >> >> > through the >> >> >> >>>>> >>> >> >> > system >> >> >> >>>>> >>> >> >> > and >> >> >> >>>>> >>> >> >> > I've found that they are not really lost. I can >> >> >> >>>>> >>> >> >> > fail >> >> >> >>>>> >>> >> >> > over >> >> >> >>>>> >>> >> >> > the >> >> >> >>>>> >>> >> >> > osd >> >> >> >>>>> >>> >> >> > and >> >> >> >>>>> >>> >> >> > recover the files. This is happening quite >> >> >> >>>>> >>> >> >> > regularly >> >> >> >>>>> >>> >> >> > now >> >> >> >>>>> >>> >> >> > after a >> >> >> >>>>> >>> >> >> > large >> >> >> >>>>> >>> >> >> > migration of data from old hardware to >> >> >> >>>>> >>> >> >> > new(migration >> >> >> >>>>> >>> >> >> > is >> >> >> >>>>> >>> >> >> > now >> >> >> >>>>> >>> >> >> > complete). >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > The system sets the PG into 'recovery', but we've >> >> >> >>>>> >>> >> >> > seen >> >> >> >>>>> >>> >> >> > the >> >> >> >>>>> >>> >> >> > system >> >> >> >>>>> >>> >> >> > in >> >> >> >>>>> >>> >> >> > a >> >> >> >>>>> >>> >> >> > recovering state for many days. Should we just >> >> >> >>>>> >>> >> >> > be >> >> >> >>>>> >>> >> >> > patient >> >> >> >>>>> >>> >> >> > or do >> >> >> >>>>> >>> >> >> > we >> >> >> >>>>> >>> >> >> > need >> >> >> >>>>> >>> >> >> > to dig further into the issue? >> >> >> >>>>> >>> >> >> >> >> >> >>>>> >>> >> >> You may need to dig into this more, although I'm not >> >> >> >>>>> >>> >> >> sure >> >> >> >>>>> >>> >> >> what >> >> >> >>>>> >>> >> >> the >> >> >> >>>>> >>> >> >> issue is likely to be. What version of Ceph are you >> >> >> >>>>> >>> >> >> running? How >> >> >> >>>>> >>> >> >> did >> >> >> >>>>> >>> >> >> you do this hardware migration? >> >> >> >>>>> >>> >> >> -Greg >> >> >> >>>>> >>> >> >> >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > pg 70.320 is stuck unclean for 704.803040, current >> >> >> >>>>> >>> >> >> > state >> >> >> >>>>> >>> >> >> > active+recovering, >> >> >> >>>>> >>> >> >> > last acting [277,101,218,49,304,412] >> >> >> >>>>> >>> >> >> > pg 70.320 is active+recovering, acting >> >> >> >>>>> >>> >> >> > [277,101,218,49,304,412], >> >> >> >>>>> >>> >> >> > 18 >> >> >> >>>>> >>> >> >> > unfound >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > There is no indication of any problems with down >> >> >> >>>>> >>> >> >> > OSDs >> >> >> >>>>> >>> >> >> > or >> >> >> >>>>> >>> >> >> > network >> >> >> >>>>> >>> >> >> > issues >> >> >> >>>>> >>> >> >> > with >> >> >> >>>>> >>> >> >> > OSDs. >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > Thanks, >> >> >> >>>>> >>> >> >> > Jeff >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > -- >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > Jeffrey McDonald, PhD >> >> >> >>>>> >>> >> >> > Assistant Director for HPC Operations >> >> >> >>>>> >>> >> >> > Minnesota Supercomputing Institute >> >> >> >>>>> >>> >> >> > University of Minnesota Twin Cities >> >> >> >>>>> >>> >> >> > 599 Walter Library email: >> >> >> >>>>> >>> >> >> > jeffrey.mcdonald@xxxxxxxxxxx >> >> >> >>>>> >>> >> >> > 117 Pleasant St SE phone: +1 612 >> >> >> >>>>> >>> >> >> > 625-6905 >> >> >> >>>>> >>> >> >> > Minneapolis, MN 55455 fax: +1 612 >> >> >> >>>>> >>> >> >> > 624-8861 >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > _______________________________________________ >> >> >> >>>>> >>> >> >> > ceph-users mailing list >> >> >> >>>>> >>> >> >> > ceph-users@xxxxxxxxxxxxxx >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >>>>> >>> >> >> > >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > -- >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > Jeffrey McDonald, PhD >> >> >> >>>>> >>> >> > Assistant Director for HPC Operations >> >> >> >>>>> >>> >> > Minnesota Supercomputing Institute >> >> >> >>>>> >>> >> > University of Minnesota Twin Cities >> >> >> >>>>> >>> >> > 599 Walter Library email: >> >> >> >>>>> >>> >> > jeffrey.mcdonald@xxxxxxxxxxx >> >> >> >>>>> >>> >> > 117 Pleasant St SE phone: +1 612 625-6905 >> >> >> >>>>> >>> >> > Minneapolis, MN 55455 fax: +1 612 624-8861 >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> >> > >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > -- >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > Jeffrey McDonald, PhD >> >> >> >>>>> >>> > Assistant Director for HPC Operations >> >> >> >>>>> >>> > Minnesota Supercomputing Institute >> >> >> >>>>> >>> > University of Minnesota Twin Cities >> >> >> >>>>> >>> > 599 Walter Library email: >> >> >> >>>>> >>> > jeffrey.mcdonald@xxxxxxxxxxx >> >> >> >>>>> >>> > 117 Pleasant St SE phone: +1 612 625-6905 >> >> >> >>>>> >>> > Minneapolis, MN 55455 fax: +1 612 624-8861 >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > >> >> >> >>>>> >>> > _______________________________________________ >> >> >> >>>>> >>> > ceph-users mailing list >> >> >> >>>>> >>> > ceph-users@xxxxxxxxxxxxxx >> >> >> >>>>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >>>>> >>> > >> >> >> >>>>> >> >> >> >> >>>>> >> >> >> >> >>>>> >> >> >> >> >>>>> >> >> >> >> >>>>> >> -- >> >> >> >>>>> >> >> >> >> >>>>> >> Jeffrey McDonald, PhD >> >> >> >>>>> >> Assistant Director for HPC Operations >> >> >> >>>>> >> Minnesota Supercomputing Institute >> >> >> >>>>> >> University of Minnesota Twin Cities >> >> >> >>>>> >> 599 Walter Library email: >> >> >> >>>>> >> jeffrey.mcdonald@xxxxxxxxxxx >> >> >> >>>>> >> 117 Pleasant St SE phone: +1 612 625-6905 >> >> >> >>>>> >> Minneapolis, MN 55455 fax: +1 612 624-8861 >> >> >> >>>>> >> >> >> >> >>>>> >> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> -- >> >> >> >>>> >> >> >> >>>> Jeffrey McDonald, PhD >> >> >> >>>> Assistant Director for HPC Operations >> >> >> >>>> Minnesota Supercomputing Institute >> >> >> >>>> University of Minnesota Twin Cities >> >> >> >>>> 599 Walter Library email: >> >> >> >>>> jeffrey.mcdonald@xxxxxxxxxxx >> >> >> >>>> 117 Pleasant St SE phone: +1 612 625-6905 >> >> >> >>>> Minneapolis, MN 55455 fax: +1 612 624-8861 >> >> >> >>>> >> >> >> >>>> >> >> >> > _______________________________________________ >> >> >> > ceph-users mailing list >> >> >> > ceph-users@xxxxxxxxxxxxxx >> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Email: >> >> >> shinobu@xxxxxxxxx >> >> >> GitHub: >> >> >> shinobu-x >> >> >> Blog: >> >> >> Life with Distributed Computational System based on OpenSource >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > >> >> > Jeffrey McDonald, PhD >> >> > Assistant Director for HPC Operations >> >> > Minnesota Supercomputing Institute >> >> > University of Minnesota Twin Cities >> >> > 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx >> >> > 117 Pleasant St SE phone: +1 612 625-6905 >> >> > Minneapolis, MN 55455 fax: +1 612 624-8861 >> >> > >> >> > >> > >> > >> > >> > >> > -- >> > >> > Jeffrey McDonald, PhD >> > Assistant Director for HPC Operations >> > Minnesota Supercomputing Institute >> > University of Minnesota Twin Cities >> > 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx >> > 117 Pleasant St SE phone: +1 612 625-6905 >> > Minneapolis, MN 55455 fax: +1 612 624-8861 >> > >> > > > > > > -- > > Jeffrey McDonald, PhD > Assistant Director for HPC Operations > Minnesota Supercomputing Institute > University of Minnesota Twin Cities > 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx > 117 Pleasant St SE phone: +1 612 625-6905 > Minneapolis, MN 55455 fax: +1 612 624-8861 > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |