> On 08 Jun 2015, at 10:40, Christian Balzer <chibi@xxxxxxx> wrote: > > On Mon, 8 Jun 2015 10:12:02 +0200 Jan Schermer wrote: > >> >>> On 08 Jun 2015, at 10:07, Christian Balzer <chibi@xxxxxxx> wrote: >>> >>> On Mon, 8 Jun 2015 09:44:54 +0200 Jan Schermer wrote: >>> >>>> I recently did some testing of a few SSDs and found some surprising, >>>> and some not so surprising things: >>>> >>>> 1) performance varies wildly with firmware, especially with cheaper >>>> drives >>>> 2) performance varies with time - even with S3700 - slows down >>>> after ~40-80GB and then creeps back up >>>> 3) cheaper drives have almost no gain from bigger queues (at least >>>> with fio iodepth=XX) >>>> 4) all drives I tested reach higher IOPS for direct and >>>> synchronous writes with iodepth=1 when write cache is *DISABLED* (even >>>> S3700) >>>> - I suspect compression and write coalescing are disabled >>>> - Intel S3700 can reach almost the same IOPS with higher queue depths, >>>> but that’s sadly not a real scenario >>>> - in any case, disabling write cache doesn’t help real workloads on my >>>> cluster >>> >>>> 5) write amplification is worst for synchronous direct writes, >>>> so it’s much better to collocate journal and data on the same SSD if >>>> you worry about DWPD endurance rating >>>> >>> Or in his case _not_ have them on the low endurance EVOs. ^_- >> >> Not necesarrily - drives have to write the data sooner or later, writing >> the same block twice doesn’t always mean draining DWPD rating at twice >> the rate, in fact if it writes with 32kb blocks it might almost the same >> because SSDc have much larger blocks anyway… >> >> if there is an async IO queued (data) and another synchronous IO comes >> for the journal, then it might be a single atomic operation on the SSD >> because it has to flush everything it has. >> > While true, there are a number of might/maybes in there and with those > SSDs in particular I'd probably opt for anything that can reduce writes. > Never mind the less than stellar expected performance with a co-located > journal. > > All that of course if he can afford DC S3700s for OS/journal. > True. We might know how it should work, but there’a s lot of magic in the firmware these days, without real benchmarks it is a guesswork and with benchmarks it’s indicative only for the one SSD/firmware/revision anyway... For exmaple from what I can see I get sometimes 50% compression on my workloads, but still about 20% write amplification (on the SSD alone) Some interesting SMART metrics: This should indicate the real wear: 100 Gigabytes_Erased 0x0032 000 000 000 Old_age Always - 62936 Bytes written after compression: 233 SandForce_Internal 0x0000 000 000 000 Old_age Offline - 40464 Written bytes from OS perspective: 241 Lifetime_Writes_GiB 0x0032 000 000 000 Old_age Always - 53826 I wonder if it’s “write-mostly” for everyone… :) 242 Lifetime_Reads_GiB 0x0032 000 000 000 Old_age Always - 13085 >> >>> >>>> and bottom line >>>> 6) CEPH doesn’t utilize the SSD capabilities at all on my Dumpling - I >>>> hope it will be better on Giant :) 7) ext4 has much higher throughput >>>> for my benchmarks when not using a raw device 8) I lose ~50% IOPS on >>>> XFS compared to block device - ext4 loses ~10% >>>> >>> I can confirm that from my old (emperor days) comparison between EXT4 >>> and XFS. >> >> Do you have the benchmark somewhere? This wasn’t really what I was >> testing, some number would be very helpful… Did you test the various >> ext4 options? >> > Unfortunately, no. At least not in any meaningful detail. > But what I have left and remember matches that ratio you're seeing. > > I think the most significant change was to give EXT4 a maximum sized > journal. I tested it with journal_async_commit (which implies journal_checksum) - that’s something I am really excited about and I think this made the most difference. If any devs are reading - when we have 3 replica durability, each replica is running in its’ own datacentre - why should we bother with fsync() and stuff at all? If we had an option to just turn it off and buffer everything it would make everything run much happier. I of course assume operators know what they are doing and not just blindly “speeding” everything up… (innodb_flush_log_ at_trx_commit=0 is so common… and so sad…) > > Christian > >> Jan >> >>> >>> Christian >>> >>>> btw the real average request size on my SSDs is only about 32KiB >>>> (journal+data on the same device) >>>> >>>> Jan >>>> >>>>> On 08 Jun 2015, at 09:33, Christian Balzer <chibi@xxxxxxx> wrote: >>>>> >>>>> >>>>> Hello, >>>>> >>>>> On Mon, 8 Jun 2015 18:01:28 +1200 Cameron.Scrace@xxxxxxxxxxxx wrote: >>>>> >>>>>> Just used the method in the link you sent me to test one of the EVO >>>>>> 850s, with one job it reached a speed of around 2.5MB/s but it >>>>>> didn't max out until around 32 jobs at 24MB/s: >>>>>> >>>>> I'm not the author of that page, nor did I verify that they did use a >>>>> uniform methodology/environment, nor do I think that this test is a >>>>> particular close approach to what I see Ceph doing in reality (it >>>>> seems to write much larger chunks that 4KB). >>>>> I'd suggest keeping numjobs to 1 and ramping up the block size to >>>>> 4MB, see where you max out with that. >>>>> I can reach the theoretical max speed of my SSDs (350MB/s) at 4MB >>>>> blocks, but it's already at 90% with 1MB. >>>>> >>>>> That test does however produce interesting numbers that seem to be >>>>> consistent by themselves and match what people have reporting here. >>>>> >>>>> I can get 22MB/s with just one fio job with the above setting (alas >>>>> on filesystem, no spare raw partition right now) on a DC S3700 200GB >>>>> SSD, directly conntected to an onboard Intel SATA-3 port. >>>>> >>>>> Now think what that means, a fio numjob is equivalent to an OSD >>>>> daemon, so in this worst case 4KB scenario my journal and thus OSD >>>>> would be 10 times faster than yours. >>>>> Food for thought. >>>>> >>>>>> sudo fio --filename=/dev/sdh --direct=1 --sync=1 --rw=write --bs=4k >>>>>> --numjobs=32 --iodepth=1 --runtime=60 --time_based >>>>>> --group_reporting --name=journal-test >>>>>> write: io=1507.4MB, bw=25723KB/s, iops=6430, runt= 60007msec >>>>>> >>>>>> Also tested a Micron 550 we had sitting around and it maxed out at >>>>>> 2.5mb/s, both results conflict with the chart >>>>>> >>>>> Note that they disabled on SSD and controller caches, the former is >>>>> of course messing things up where this isn't needed. >>>>> >>>>> I'd suggest you go and do a test install of Ceph with your HW and >>>>> test that. >>>>> Paying close attention to your SSD utilization with atop or iostat, >>>>> etc. >>>>> >>>>> Christian >>>>> >>>>>> Regards, >>>>>> >>>>>> Cameron Scrace >>>>>> Infrastructure Engineer >>>>>> >>>>>> Mobile +64 22 610 4629 >>>>>> Phone +64 4 462 5085 >>>>>> Email cameron.scrace@xxxxxxxxxxxx >>>>>> Solnet Solutions Limited >>>>>> Level 12, Solnet House >>>>>> 70 The Terrace, Wellington 6011 >>>>>> PO Box 397, Wellington 6140 >>>>>> >>>>>> www.solnet.co.nz >>>>>> >>>>>> >>>>>> >>>>>> From: Christian Balzer <chibi@xxxxxxx> >>>>>> To: "ceph-users@xxxxxxxx" <ceph-users@xxxxxxxx> >>>>>> Cc: Cameron.Scrace@xxxxxxxxxxxx >>>>>> Date: 08/06/2015 02:40 p.m. >>>>>> Subject: Re: Multiple journals and an OSD on one >>>>>> SSD doable? >>>>>> >>>>>> >>>>>> >>>>>> On Mon, 8 Jun 2015 14:30:17 +1200 Cameron.Scrace@xxxxxxxxxxxx wrote: >>>>>> >>>>>>> Thanks for all the feedback. >>>>>>> >>>>>>> What makes the EVOs unusable? They should have plenty of speed but >>>>>>> your link has them at 1.9MB/s, is it just the way they handle >>>>>>> O_DIRECT and D_SYNC? >>>>>>> >>>>>> Precisely. >>>>>> Read that ML thread for details. >>>>>> >>>>>> And once more, they also are not very endurable. >>>>>> So depending on your usage pattern and Ceph (Ceph itself and the >>>>>> underlying FS) write amplification their TBW/$ will be horrible, >>>>>> costing you more in the end than more expensive, but an order of >>>>>> magnitude more endurable DC SSDs. >>>>>> >>>>>>> Not sure if we will be able to spend anymore, we may just have to >>>>>>> take the performance hit until we can get more money for the >>>>>>> project. >>>>>>> >>>>>> You could cheap out with 200GB DC S3700s (half the price), but they >>>>>> will definitely become the bottleneck at a combined max speed of >>>>>> about 700MB/s, as opposed to the 400GB ones at 900MB/s combined. >>>>>> >>>>>> Christian >>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Cameron Scrace >>>>>>> Infrastructure Engineer >>>>>>> >>>>>>> Mobile +64 22 610 4629 >>>>>>> Phone +64 4 462 5085 >>>>>>> Email cameron.scrace@xxxxxxxxxxxx >>>>>>> Solnet Solutions Limited >>>>>>> Level 12, Solnet House >>>>>>> 70 The Terrace, Wellington 6011 >>>>>>> PO Box 397, Wellington 6140 >>>>>>> >>>>>>> www.solnet.co.nz >>>>>>> >>>>>>> >>>>>>> >>>>>>> From: Christian Balzer <chibi@xxxxxxx> >>>>>>> To: "ceph-users@xxxxxxxx" <ceph-users@xxxxxxxx> >>>>>>> Cc: Cameron.Scrace@xxxxxxxxxxxx >>>>>>> Date: 08/06/2015 02:00 p.m. >>>>>>> Subject: Re: Multiple journals and an OSD on >>>>>>> one SSD >>>>>> >>>>>>> doable? >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Cameron, >>>>>>> >>>>>>> To offer at least some constructive advice here instead of just all >>>>>>> doom and gloom, here's what I'd do: >>>>>>> >>>>>>> Replace the OS SSDs with 2 400GB Intel DC S3700s (or S3710s). >>>>>>> They have enough BW to nearly saturate your network. >>>>>>> >>>>>>> Put all your journals on them (3 SSD OSD and 3 HDD OSD per). >>>>>>> While that's a bad move from a failure domain perspective, your >>>>>>> budget probably won't allow for anything better and those are VERY >>>>>>> reliable and just as important durable SSDs. >>>>>>> >>>>>>> This will give you the speed your current setup is capable of, >>>>>>> probably limited by the CPU when it comes to SSD pool operations. >>>>>>> >>>>>>> Christian >>>>>>> >>>>>>> On Mon, 8 Jun 2015 10:44:06 +0900 Christian Balzer wrote: >>>>>>> >>>>>>>> >>>>>>>> Hello Cameron, >>>>>>>> >>>>>>>> On Mon, 8 Jun 2015 13:13:33 +1200 Cameron.Scrace@xxxxxxxxxxxx >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Christian, >>>>>>>>> >>>>>>>>> Yes we have purchased all our hardware, was very hard to >>>>>>>>> convince management/finance to approve it, so some of the stuff >>>>>>>>> we have is a bit cheap. >>>>>>>>> >>>>>>>> Unfortunate. Both the done deal and the cheapness. >>>>>>>> >>>>>>>>> We have four storage nodes each with 6 x 6TB Western Digital Red >>>>>>>>> SATA Drives (WD60EFRX-68M) and 6 x 1TB Samsung EVO 850s SSDs and >>>>>>>>> 2x250GB Samsung EVO 850s (for OS raid). >>>>>>>>> CPUs are Intel Atom C2750 @ 2.40GHz (8 Cores) with 32 GB of >>>>>>>>> RAM. We have a 10Gig Network. >>>>>>>>> >>>>>>>> I wish there was a nice way to say this, but it unfortunately >>>>>>>> boils down to a "You're fooked". >>>>>>>> >>>>>>>> There have been many discussions about which SSDs are usable with >>>>>> Ceph, >>>>>>>> very recently as well. >>>>>>>> Samsung EVOs (the non DC type for sure) are basically unusable for >>>>>>>> journals. See the recent thread: >>>>>>>> Possible improvements for a slow write speed (excluding >>>>>>>> independent SSD journals) and: >>>>>>>> >>>>>>> >>>>>> http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ >>>>>> >>>>>>> >>>>>>>> for reference. >>>>>>>> >>>>>>>> I presume your intention for the 1TB SSDs is for a SSD backed >>>>>>>> pool? Note that the EVOs have a pretty low (guaranteed) >>>>>>>> endurance, so aside from needing journal SSDs that actually can >>>>>>>> do the job, you're looking >>>>>> >>>>>>> at >>>>>>>> wearing them out rather quickly (depending on your use case of >>>>>> course). >>>>>>>> >>>>>>>> Now with SSD based OSDs or even HDD based OSDs with SSD journals >>>>>>>> that >>>>>>> CPU >>>>>>>> looks a bit anemic. >>>>>>>> >>>>>>>> More below: >>>>>>>>> The two options we are considering are: >>>>>>>>> >>>>>>>>> 1) Use two of the 1TB SSDs for the spinning disk journals (3 >>>>>>>>> each) and >>>>>>> >>>>>>>>> then use the remaining 900+GB of each drive as an OSD to be part >>>>>>>>> of the cache pool. >>>>>>>>> >>>>>>>>> 2) Put the spinning disk journals on the OS SSDs and use the 2 >>>>>>>>> 1TB SSDs for the cache pool. >>>>>>>>> >>>>>>>> Cache pools aren't all that speedy currently (research the ML >>>>>>>> archives), even less so with the SSDs you have. >>>>>>>> >>>>>>>> Christian >>>>>>>> >>>>>>>>> In both cases the other 4 1TB SSDs will be part of their own >>>>>>>>> tier. >>>>>>>>> >>>>>>>>> Thanks a lot! >>>>>>>>> >>>>>>>>> Cameron Scrace >>>>>>>>> Infrastructure Engineer >>>>>>>>> >>>>>>>>> Mobile +64 22 610 4629 >>>>>>>>> Phone +64 4 462 5085 >>>>>>>>> Email cameron.scrace@xxxxxxxxxxxx >>>>>>>>> Solnet Solutions Limited >>>>>>>>> Level 12, Solnet House >>>>>>>>> 70 The Terrace, Wellington 6011 >>>>>>>>> PO Box 397, Wellington 6140 >>>>>>>>> >>>>>>>>> www.solnet.co.nz >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> From: Christian Balzer <chibi@xxxxxxx> >>>>>>>>> To: "ceph-users@xxxxxxxx" <ceph-users@xxxxxxxx> >>>>>>>>> Cc: Cameron.Scrace@xxxxxxxxxxxx >>>>>>>>> Date: 08/06/2015 12:18 p.m. >>>>>>>>> Subject: Re: Multiple journals and an OSD on >>>>>>>>> one SSD doable? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, 8 Jun 2015 09:55:56 +1200 Cameron.Scrace@xxxxxxxxxxxx >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> The other option we were considering was putting the journals on >>>>>>>>>> the OS SSDs, they are only 250GB and the rest would be for the >>>>>>>>>> OS. Is that a decent option? >>>>>>>>>> >>>>>>>>> You'll be getting a LOT better advice if you're telling us more >>>>>>>>> details. >>>>>>>>> >>>>>>>>> For starters, have you bought the hardware yet? >>>>>>>>> Tell us about your design, how many initial storage nodes, how >>>>>>>>> many HDDs/SSDs per node, what CPUs/RAM/network? >>>>>>>>> >>>>>>>>> What SSDs are we talking about, exact models please. >>>>>>>>> (Both the sizes you mentioned do not ring a bell for DC level >>>>>>>>> SSDs I'm aware of) >>>>>>>>> >>>>>>>>> That said, I'm using Intel DC S3700s for mixed OS and journal use >>>>>>>>> with >>>>>>> >>>>>>>>> good >>>>>>>>> results. >>>>>>>>> In your average Ceph storage node, normal OS (logging mostly) >>>>>>>>> activity is a >>>>>>>>> minute drop in the bucket for any decent SSD, so nearly all of >>>>>>>>> it's resources are available to journals. >>>>>>>>> >>>>>>>>> You want to match the number of journals per SSD according to the >>>>>>>>> capabilities of your SSD, HDDs and network. >>>>>>>>> >>>>>>>>> For example 8 HDD OSDs with 2 200GB DC S3700 and a 10Gb/s network >>>>>>>>> is a decent match. >>>>>>>>> The two SSDs at 900MB/s would appear to be the bottleneck, but in >>>>>>>>> reality I'd expect the HDDs to be it. >>>>>>>>> Never mind that you'd be more likely to be IOPS than bandwidth >>>>>> bound. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Christian >>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> >>>>>>>>>> Cameron Scrace >>>>>>>>>> Infrastructure Engineer >>>>>>>>>> >>>>>>>>>> Mobile +64 22 610 4629 >>>>>>>>>> Phone +64 4 462 5085 >>>>>>>>>> Email cameron.scrace@xxxxxxxxxxxx >>>>>>>>>> Solnet Solutions Limited >>>>>>>>>> Level 12, Solnet House >>>>>>>>>> 70 The Terrace, Wellington 6011 >>>>>>>>>> PO Box 397, Wellington 6140 >>>>>>>>>> >>>>>>>>>> www.solnet.co.nz >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> From: Somnath Roy <Somnath.Roy@xxxxxxxxxxx> >>>>>>>>>> To: "Cameron.Scrace@xxxxxxxxxxxx" >>>>>>>>>> <Cameron.Scrace@xxxxxxxxxxxx>, >>>>>>> >>>>>>>>>> "ceph-users@xxxxxxxx" <ceph-users@xxxxxxxx> >>>>>>>>>> Date: 08/06/2015 09:34 a.m. >>>>>>>>>> Subject: RE: Multiple journals and an OSD on >>>>>>>>>> one SSD >>>>>>>>> >>>>>>>>>> doable? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Cameron, >>>>>>>>>> Generally, it’s not a good idea. >>>>>>>>>> You want to protect your SSDs used as journal.If any problem on >>>>>>>>>> that disk, you will be losing all of your dependent OSDs. >>>>>>>>>> I don’t think a bigger journal will gain you much performance , >>>>>>>>>> so, default 5 GB journal size should be good enough. If you >>>>>>>>>> want >>>>>> to >>>>>>>>>> reduce the fault domain and want to put 3 journals on a SSD , go >>>>>>>>>> for minimum size and high endurance SSDs for that. >>>>>>>>>> Now, if you want to use your rest of space of 1 TB ssd, >>>>>>>>>> creating >>>>>>> just >>>>>>>>>> OSDs will not gain you much (rather may get some burst >>>>>>>>>> performance). You may want to consider the following. >>>>>>>>>> >>>>>>>>>> 1. If your spindle OSD size is much bigger than 900 GB , you >>>>>>>>>> don’t want to make all OSDs of similar sizes, cache pool could >>>>>>>>>> be one of your option. But, remember, cache pool can wear out >>>>>>>>>> your SSDs faster as presently I guess it is not optimizing the >>>>>>>>>> extra writes. Sorry, I don’t have exact data as I am yet to test >>>>>>>>>> that out. >>>>>>>>>> >>>>>>>>>> 2. If you want to make all the OSDs of similar sizes and you >>>>>>>>>> will be able to create a substantial number of OSDs with your >>>>>>>>>> unused SSDs (depends on how big the cluster is), you may want >>>>>>>>>> to put all of your primary OSDs to SSD and gain significant >>>>>>>>>> performance boost for read. Also, in this case, I don’t think >>>>>>>>>> you will be getting any burst performance. >>>>>>>>>> Thanks & Regards >>>>>>>>>> Somnath >>>>>>>>>> >>>>>>>>>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On >>>>>>> Behalf >>>>>>>>>> Of >>>>>>>>> >>>>>>>>>> Cameron.Scrace@xxxxxxxxxxxx >>>>>>>>>> Sent: Sunday, June 07, 2015 1:49 PM >>>>>>>>>> To: ceph-users@xxxxxxxx >>>>>>>>>> Subject: Multiple journals and an OSD on one SSD >>>>>>> doable? >>>>>>>>>> >>>>>>>>>> Setting up a Ceph cluster and we want the journals for our >>>>>> spinning >>>>>>>>>> disks to be on SSDs but all of our SSDs are 1TB. We were >>>>>>>>>> planning on putting 3 journals on each SSD, but that leaves >>>>>>>>>> 900+GB unused on the drive, is it possible to use the leftover >>>>>>>>>> space as another OSD or will it affect performance too much? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Cameron Scrace >>>>>>>>>> Infrastructure Engineer >>>>>>>>>> >>>>>>>>>> Mobile +64 22 610 4629 >>>>>>>>>> Phone +64 4 462 5085 >>>>>>>>>> Email cameron.scrace@xxxxxxxxxxxx >>>>>>>>>> Solnet Solutions Limited >>>>>>>>>> Level 12, Solnet House >>>>>>>>>> 70 The Terrace, Wellington 6011 >>>>>>>>>> PO Box 397, Wellington 6140 >>>>>>>>>> >>>>>>>>>> www.solnet.co.nzAttention: This email may contain information >>>>>>>>>> intended for the sole use of the original recipient. Please >>>>>> respect >>>>>>>>>> this when sharing or disclosing this email's contents with any >>>>>>>>>> third party. If you believe you have received this email in >>>>>>>>>> error, please delete it and notify the sender or >>>>>>>>>> postmaster@xxxxxxxxxxxxxxxxxxxxx as soon as possible. The >>>>>>>>>> content of this email does not necessarily reflect the views of >>>>>>>>>> Solnet Solutions Ltd. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> PLEASE NOTE: The information contained in this electronic mail >>>>>>>>>> message is intended only for the use of the designated >>>>>> recipient(s) >>>>>>>>>> named above. If the reader of this message is not the intended >>>>>>>>>> recipient, you are hereby notified that you have received this >>>>>>>>>> message in error and that any review, dissemination, >>>>>>>>>> distribution, or copying of this message is strictly prohibited. >>>>>>>>>> If you have received this communication in error, please notify >>>>>>>>>> the sender by telephone or e-mail (as shown above) immediately >>>>>>>>>> and destroy any and all copies of this message in your >>>>>>>>>> possession (whether hard copies or electronically stored >>>>>>>>>> copies). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Attention: >>>>>>>>>> This email may contain information intended for the sole use of >>>>>>>>>> the original recipient. Please respect this when sharing or >>>>>>>>>> disclosing this email's contents with any third party. If you >>>>>>>>>> believe you have received this email in error, please delete it >>>>>>>>>> and notify the sender or postmaster@xxxxxxxxxxxxxxxxxxxxx as >>>>>>>>>> soon as possible. The content of this email does not necessarily >>>>>>>>>> reflect the views of Solnet Solutions Ltd. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Christian Balzer Network/Systems Engineer >>>>> chibi@xxxxxxx Global OnLine Japan/Fusion Communications >>>>> http://www.gol.com/ >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@xxxxxxxxxxxxxx >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>>> >>> >>> >>> -- >>> Christian Balzer Network/Systems Engineer >>> chibi@xxxxxxx Global OnLine Japan/Fusion Communications >>> http://www.gol.com/ >> >> > > > -- > Christian Balzer Network/Systems Engineer > chibi@xxxxxxx Global OnLine Japan/Fusion Communications > http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |