Hi,
response in line
On 20 Apr 2016 7:45 a.m., "Christian Balzer" <chibi@xxxxxxx> wrote:
>
>
> Hello,
>
> On Wed, 20 Apr 2016 03:42:00 +0000 Stephen Lord wrote:
>
> >
> > OK, you asked ;-)
> >
>
> I certainly did. ^o^
>
> > This is all via RBD, I am running a single filesystem on top of 8 RBD
> > devices in an effort to get data striping across more OSDs, I had been
> > using that setup before adding the cache tier.
> >
> Nods.
> Depending on your use case (sequential writes) actual RADOS striping might
> be more advantageous than this (with 4MB writes still going to the same
> PG/OSD all the time).
>
>
> > 3 nodes with 11 6 Tbyte SATA drives each for a base RBD pool, this is
> > setup with replication size 3. No SSDs involved in those OSDs, since
> > ceph-disk does not let you break a bluestore configuration into more
> > than one device at the moment.
> >
> That's a pity, but supposedly just a limitation of ceph-disk.
> I'd venture you can work around that with symlinks to a raw SSD
> partition, same as with current filestore journals.
>
> As Sage recently wrote:
> ---
> BlueStore can use as many as three devices: one for the WAL (journal,
> though it can be much smaller than FileStores, e.g., 128MB), one for
> metadata (e.g., an SSD partition), and one for data.
> ---
I believe he also mentioned the use of bcache and friends for the osd, maybe a way forward in this case?
Regards
Josef
>
> > The 600 Mbytes/sec is an approx sustained number for the data rate I can
> > get going into this pool via RBD, that turns into 3 times that for raw
> > data rate, so at 33 drives that is mid 50s Mbytes/sec per drive. I have
> > pushed it harder than that from time to time, but the OSD really wants
> > to use fdatasync a lot and that tends to suck up a lot of the potential
> > of a device, these disks will do 160 Mbytes/sec if you stream data to
> > them.
> >
> > I just checked with rados bench to this set of 33 OSDs with a 3 replica
> > pool, and 600 Mbytes/sec is what it will do from the same client host.
> >
> This matches a cluster of mine with 32 OSDs (filestore of course) and SSD
> journals on 4 nodes with a replica of 3.
>
> So BlueStore is indeed faster than than filestore.
>
> > All the networking is 40 GB ethernet, single port per host, generally I
> > can push 2.2 Gbytes/sec in one direction between two hosts over a single
> > tcp link, the max I have seen is about 2.7 Gbytes/sec coming into a
> > node. Short of going to RDMA that appears to be about the limit for
> > these processors.
> >
> Yeah, didn't expect your network to be involved here bottleneck wise, but
> a good data point to have nevertheless.
>
> > There are a grand total of 2 400 GB P3700s which are running a pool with
> > a replication factor of 1, these are in 2 other nodes. Once I add in
> > replication perf goes downhill. If I had more hardware I would be
> > running more of these and using replication, but I am out of network
> > cards right now.
> >
> Alright, so at 900MB/s you're pretty close to what one would expect from 2
> of these: 1080MB/s*2/2(journal).
>
> How much downhill is that?
>
> I have a production cache tier with 2 nodes (replica 2 of course) and 4
> 800GB DC S3610s each, IPoIB QDR (40Gbs) interconnect and the performance
> is pretty much what I would expect.
>
> > So 5 nodes running OSDs, and a 6th node running the RBD client using the
> > kernel implementation.
> >
> I assume there's are reason for use the kernel RBD client (which kernel?),
> given that it tends to be behind the curve in terms of features and speed?
>
> > Complete set of commands for creating the cache tier, I pulled this from
> > history, so the line in the middle was a failed command actually so
> > sorry for the red herring.
> >
> > 982 ceph osd pool create nvme 512 512 replicated_nvme
> > 983 ceph osd pool set nvme size 1
> > 984 ceph osd tier add rbd nvme
> > 985 ceph osd tier cache-mode nvme writeback
> > 986 ceph osd tier set-overlay rbd nvme
> > 987 ceph osd pool set nvme hit_set_type bloom
> > 988 ceph osd pool set target_max_bytes 500000000000 <<—— typo here,
> > so never mind 989 ceph osd pool set nvme target_max_bytes 500000000000
> > 990 ceph osd pool set nvme target_max_objects 500000
> > 991 ceph osd pool set nvme cache_target_dirty_ratio 0.5
> > 992 ceph osd pool set nvme cache_target_full_ratio 0.8
> >
> > I wish the cache tier would cause a health warning if it does not have
> > a max size set, it lets you do that, flushes nothing and fills the OSDs.
> >
> Oh yes, people have been bitten by this over and over again.
> At least it's documented now.
>
> > As for what the actual test is, this is 4K uncompressed DPX video frames,
> > so 50 Mbyte files written at least 24 a second on a good day, ideally
> > more. This needs to sustain around 1.3 Gbytes/sec in either direction
> > from a single application and needs to do it consistently. There is a
> > certain amount of buffering to deal with fluctuations in perf. I am
> > pushing 4096 of these files sequentially with a queue depth of 32 so
> > there is rather a lot of data in flight at any one time. I know I do not
> > have enough hardware to achieve this rate on writes.
> >
> So this is your test AND actual intended use case I presume, right?
>
> > The are being written with direct I/O into a pool of 8 RBD LUNs. The 8
> > LUN setup will not really help here with the small number of OSDs in the
> > cache pool, it does help when the RBD LUNs are going directly to a large
> > pool of disk based OSDs as it gets all the OSDs moving in parallel.
> >
> > My basic point here is that there is a lot more potential bandwidth to
> > be had in the backing pool, but I cannot get the cache tier to use more
> > than a small fraction of the available bandwidth when flushing content.
> > Since the front end of the cache can sustain around 900 Mbytes/sec over
> > RBD, I am somewhat out of balance here:
> >
> > cache input rate 900 Mbytes/sec
> > backing pool input rate 600 Mbytes/sec
> >
> > But not by a significant amount.
> >
> > The question is really about is there anything I can do to get cache
> > flushing to take advantage of more of the bandwidth. If I do this
> > without the cache tier then the latency of the disk based OSDs is too
> > variable and you cannot sustain a consistent data rate.
>
> This should hopefully be reduced by having WAL and metadata on SSDs with
> bluestore.
> But HDD based storage will be more jittery, that's a given.
>
> >The NVMe devices
> > are better about consistent device latency, but the cache tier
> > implementation seems to have a problem driving the backing pool at
> > anything close to its capabilities. It really only needs to move 40 or
> > 50 objects in parallel to achieve that.
> >
> And this is clearly where the Ceph (cache-tier) code could probably use
> some attention.
> Mind you, most people have the exact OPPOSITE requirement of yours, they
> want a steady, slow, low-impact stream of flushes to the HDDs, not an
> avalanche. ^o^
>
> We know that this is all based on per PG ratios, so if one PG goes over
> the dirty ratio the tier-agent will start flushing objects from it.
> The singular spikes you're seeing.
>
> My bet is that this process is sequential and not parallel (or at least
> not massively so), meaning that until one PG has finished flushing,
> the next dirty'ish one won't start flushing.
> Would be nice to have this confirmed by somebody familiar with the code,
> though.
>
> Of course having fast OSDs in the backing pool will alleviate this
> somewhat.
>
> What's your cache_target_dirty_high_ratio set to and have you tried
> setting it below/identical to cache_target_dirty_ratio?
>
> Christian
>
> > I am not attempting to provision a cache tier large enough for whole
> > workload, but as more of a debounce zone to avoid jitter making it back
> > to the application. I am trying to categorize what can and cannot be
> > achieved with ceph here for this type of workload, not build a complete
> > production setup. My test represents 170 seconds of content and
> > generates 209 Gbytes of data, so this is a small scale test ;-)
> > fortunately this stuff is not always used realtime.
> >
> > All of those extra config options look to be around how fast promotion
> > into the cache can go, not how fast you can get things out of it :-(
> >
> > I have been using readforward and that is working OK, there is
> > sufficient read bandwidth that it does not matter if data is coming from
> > the cache pool or the disk backing pool.
> >
> > Steve
> >
> >
> > > On Apr 19, 2016, at 7:47 PM, Christian Balzer <chibi@xxxxxxx> wrote:
> > >
> > >
> > > Hello,
> > >
> > > On Tue, 19 Apr 2016 20:21:39 +0000 Stephen Lord wrote:
> > >
> > >>
> > >>
> > >> I Have a setup using some Intel P3700 devices as a cache tier, and 33
> > >> sata drives hosting the pool behind them.
> > >
> > > A bit more details about the setup would be nice, as in how many nodes,
> > > interconnect, replication size of the cache tier and the backing HDD
> > > pool, etc.
> > > And "some" isn't a number, how many P3700s (which size?) in how many
> > > nodes? One assumes there are no further SSDs involved with those SATA
> > > HDDs?
> >
> > >
> > >> I setup the cache tier with
> > >> writeback, gave it a size and max object count etc:
> > >>
> > >> ceph osd pool set target_max_bytes 500000000000
> > > ^^^
> > > This should have given you an error, it needs the pool name, as in your
> > > next line.
> > >
> > >> ceph osd pool set nvme target_max_bytes 500000000000
> > >> ceph osd pool set nvme target_max_objects 500000
> > >> ceph osd pool set nvme cache_target_dirty_ratio 0.5
> > >> ceph osd pool set nvme cache_target_full_ratio 0.8
> > >>
> > >> This is all running Jewel using bluestore OSDs (I know experimental).
> > > Make sure to report all pyrotechnics, trap doors and sharp edges. ^_-
> > >
> > >> The cache tier will write at about 900 Mbytes/sec and read at 2.2
> > >> Gbytes/sec, the sata pool can take writes at about 600 Mbytes/sec in
> > >> aggregate.
> > > ^^^^^^^^^
> > > Key word there.
> > >
> > > That's just 18MB/s per HDD (60MB/s with a replication of 3), a pretty
> > > disappointing result for the supposedly twice as fast BlueStore.
> > > Again, replication size and topology might explain that up to a point,
> > > but we don't know them (yet).
> > >
> > > Also exact methodology of your tests please, i.e. the fio command
> > > line, how was the RBD device (if you tested with one) mounted and
> > > where, etc...
> > >
> > >> However, it looks like the mechanism for cleaning the cache
> > >> down to the disk layer is being massively rate limited and I see about
> > >> 47 Mbytes/sec of read activity from each SSD while this is going on.
> > >>
> > > This number is meaningless w/o knowing home many NVMe's you have.
> > > That being said, there are 2 levels of flushing past Hammer, but if you
> > > push the cache tier to the 2nd limit (cache_target_dirty_high_ratio)
> > > you will get full speed.
> > >
> > >> This means that while I could be pushing data into the cache at high
> > >> speed, It cannot evict old content very fast at all, and it is very
> > >> easy to hit the high water mark and the application I/O drops
> > >> dramatically as it becomes throttled by how fast the cache can flush.
> > >>
> > >> I suspect it is operating on a placement group at a time so ends up
> > >> targeting a very limited number of objects and hence disks at any one
> > >> time. I can see individual disk drives going busy for very short
> > >> periods, but most of them are idle at any one point in time. The only
> > >> way to drive the disk based OSDs fast is to hit a lot of them at once
> > >> which would mean issuing many cache flush operations in parallel.
> > >>
> > > Yes, it is all PG based, so your observations match the expectations
> > > and what everybody else is seeing.
> > > See also the thread "Cache tier operation clarifications" by me,
> > > version 2 is in the works.
> > > There are also some new knobs in Jewel that may be helpful, see:
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.spinics.net_lists_ceph-2Dusers_msg25679.html&d=CwICAg&c=8S5idjlO_n28Ko3lg6lskTMwneSC-WqZ5EBTEEvDlkg&r=tA8AXp6f2QAGtnc-TrB3H1XZIqqTELvv3S6ZQGJZBLs&m=yIRKIZ4yBOSAr9O5lav-0J24ys7S21EhX394KIorQ-E&s=XV0tKumf_xV99IUKYgnbmJrvhbL9I5Fdk1eCwG-YUYQ&e=
> > >
> > > If you have a use case with a clearly defined idle/low use time and a
> > > small enough growth in dirty objects, consider what I'm doing,
> > > dropping the cache_target_dirty_ratio a few percent (in my case 2-3%
> > > is enough for a whole day) via cron job,wait a bit and then up again
> > > to it's normal value.
> > >
> > > That way flushes won't normally happen at all during your peak usage
> > > times, though in my case that's purely cosmetic, flushes are not
> > > problematic at any time in that cluster currently.
> > >
> > >> Are there any controls which can influence this behavior?
> > >>
> > > See above (cache_target_dirty_high_ratio).
> > >
> > > Aside from that you might want to reflect on what your use case,
> > > workload is going to be and how your testing reflects on it.
> > >
> > > As in, are you really going to write MASSIVE amounts of data at very
> > > high speeds or is it (like in 90% of common cases) the amount of small
> > > write IOPS that is really going to be the limiting factor.
> > > Which is something that cache tiers can deal with very well (or
> > > sufficiently large and well designed "plain" clusters).
> > >
> > > Another thing to think about is using the "readforward" cache mode,
> > > leaving your cache tier free to just handle writes and thus giving it
> > > more space to work with.
> > >
> > > Christian
> > > --
> > > Christian Balzer Network/Systems Engineer
> > > chibi@xxxxxxx Global OnLine Japan/Rakuten Communications
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gol.com_&d=CwICAg&c=8S5idjlO_n28Ko3lg6lskTMwneSC-WqZ5EBTEEvDlkg&r=tA8AXp6f2QAGtnc-TrB3H1XZIqqTELvv3S6ZQGJZBLs&m=yIRKIZ4yBOSAr9O5lav-0J24ys7S21EhX394KIorQ-E&s=D1RK9OOi5QmOFzGURxC82nkUr7mAe2-Ifo2FNgqYVQY&e=
> >
> >
> > ----------------------------------------------------------------------
> > The information contained in this transmission may be confidential. Any
> > disclosure, copying, or further distribution of confidential information
> > is not permitted unless such privilege is explicitly granted in writing
> > by Quantum. Quantum reserves the right to have electronic
> > communications, including email and attachments, sent across its
> > networks filtered through anti virus and spam software programs and
> > retain such messages in order to comply with applicable data security
> > and retention requirements. Quantum is not responsible for the proper
> > and complete transmission of the substance of this communication or for
> > any delay in its receipt.
>
>
> --
> Christian Balzer Network/Systems Engineer
> chibi@xxxxxxx Global OnLine Japan/Rakuten Communications
> http://www.gol.com/
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
