Be careful about dd, depending on the options you may be getting much larger physical writes than you think. Have you done a blktrace to make sure that you're physical I/O matches what you think you're asking for? Allen Samuels SanDisk |a Western Digital brand 2880 Junction Avenue, San Jose, CA 95134 T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@xxxxxxxxxxx > -----Original Message----- > From: Shehbaz Jaffer [mailto:shehbazjaffer007@xxxxxxxxx] > Sent: Thursday, May 26, 2016 10:18 AM > To: Allen Samuels <Allen.Samuels@xxxxxxxxxxx> > Cc: Sage Weil <sweil@xxxxxxxxxx>; ceph-devel@xxxxxxxxxxxxxxx > Subject: Re: SMR Benchmarking Results > > Hi Allen, > > I ran the SMR zone_open and zone_close commands to see what the > overhead was. zone_open and zone_close only take up 3% of the total time, > The rest of the time is consumed while zone_write is taking place. > > (open-close) (open-write-close) > 40 zones - 11.8 secs 6 min 41 secs > 400 zones - 1min 52 secs NA > > I looked at zbc_write command, it takes a chunk of file into a temporary > buffer (whose size is 1MB for my experiment) and writes it to disk. I assume > dd also works this way, taking block size (bs > parameter) number of bytes in memory at a time and writing to disk. I think > with this information we can infer that most of the time consumed is in > writing to disk. > > I observed a difference in the way zone was being opened while doing "dd > write" and "SMR aware zbc_write". for dd write, the zone is "implicitly > opened", and data is written to it. for zbc writes, when we issue a > zone_open command, the zone is "explicitly opened", followed by a write, > followed by zone_close. > > looking at documentation of SMR support for ext4 here: > https://github.com/Seagate/SMR_FS-EXT4 , it seems Implicit open zones are > those zones that the drive can close on its own. explicit zones cannot be > closed on their own - and needs to be closed by explicit commands like > zbc_close. > > From initial experiments it seems that implicitly opened zones perform > better that explicitly opened zones. Probably we will see a difference in > performance when the drive hits the maximum number of implicitly open > zones, when it will internally start closing zones - in that case we will need > smarter zone management - where I feel explicit zone open/close > commands will help. > > Thanks, > Shehbaz > > On Thu, May 26, 2016 at 1:20 AM, Allen Samuels > <Allen.Samuels@xxxxxxxxxxx> wrote: > > Sounds like the zone open and/or close commands are triggering I/O > operations. > > > > Just as an experiment, how long does it take to open and close a bunch of > zones with NO data being written to them? > > > > > > Allen Samuels > > SanDisk |a Western Digital brand > > 2880 Junction Avenue, San Jose, CA 95134 > > T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@xxxxxxxxxxx > > > > > >> -----Original Message----- > >> From: Shehbaz Jaffer [mailto:shehbazjaffer007@xxxxxxxxx] > >> Sent: Wednesday, May 25, 2016 9:44 PM > >> To: Sage Weil <sweil@xxxxxxxxxx> > >> Cc: Allen Samuels <Allen.Samuels@xxxxxxxxxxx>; ceph- > >> devel@xxxxxxxxxxxxxxx > >> Subject: SMR Benchmarking Results > >> > >> Hi Sage, > >> > >> I have been working on benchmarking SMR Drives using libzbc. It appears > >> that issuing ZBC commands for zone aware host is more inefficient as > >> compared to normal copy operations using 'dd' command. > >> > >> I created a 256 MB file and placed it in memory (so that we do not have > data > >> fetch overheads). I copy this file repeatedly on a Host Aware SMR drive in > 2 > >> scenarios : > >> > >> a) dd - I use dumb dd that takes 1MB chunks of file and keeps copying the > file > >> to SMR drive for <writeSize> bytes. Note that dd does not take the zones > >> into consideration. > >> > >> b) SMR_aware_copy - This copy takes file chunks 1MB in size, but issues > ZBC > >> commands to open each zone, write 256 MB data to the zone, close the > >> zone, and then move to another zone till <writeSize> bytes have been > >> written. > >> > >> performance results for 1GB, 10GB write sizes are 5x slower with "zone > >> aware" writing, as compared to normal dd writing: > >> > >> writeSize (in GB) dd time (in min:sec) smr_aware_copy (in min:sec) > >> 1 GB 0:7 0:34 > >> 10 GB 1:11 6:41 > >> 50 GB 5:51 NA > >> 100 GB 11:42 NA > >> > >> (all writes were followed by sync command) > >> > >> I was trying to see if there is an internal cache of some sort in the Host > Aware > >> SMR drive, which probably serializes all writes up to certain extent for dd > >> command, but the time for writes using the dd command for up to 100GB > >> follow a linear pattern. I will try to see if we hit a bottleneck with dd for > larger > >> file sizes or unaligned writes. > >> > >> Followup questions: > >> -------------------------- > >> > >> a) I think we should have some workload traces or patterns so that we can > >> benchmark SMR drives and make allocator more SMR friendly. In > particular - > >> i) size of files, > >> ii) alignment of files > >> iii) % read / write/ delete workloads > >> iv) degree of parallelism in writing. > >> > >> b) SMR Drive has a notion of parallel writes - i.e. multiple zones can be > kept > >> open and written to simultaneously. I do not think there are multiple > heads > >> involved but internally there is some form of "efficient parallel write to > zone" > >> mechanism in SMR. I am thinking about this because when we query SMR > >> drive information, it shows that most effieicnt number of zones can be > >> parallelly kept open = 128 . > >> Maybe this is something that we can take advantage of? > >> > >> Thanks and Regards, > >> Shehbaz > > > > -- > Shehbaz Jaffer > First Year Graduate Student > Sir Edward S Rogers Sr Department of Electrical and Computer Engineering > University of Toronto ��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
