Re: ubi/ubifs performance comparison on two NAND devices

[Tim Harvey <tharvey@xxxxxxxxxxxxx>]

On Wed, Feb 27, 2019 at 2:59 PM Richard Weinberger <richard@xxxxxx> wrote:
>
> Tim,
>
> Am Mittwoch, 27. Februar 2019, 23:43:17 CET schrieb Tim Harvey:
> > On Wed, Feb 27, 2019 at 2:13 PM Richard Weinberger
> > <richard.weinberger@xxxxxxxxx> wrote:
> > >
> > > On Wed, Feb 27, 2019 at 9:39 PM Tim Harvey <tharvey@xxxxxxxxxxxxx> wrote:
> > > >
> > > > Greetings,
> > > >
> > > > I've got two embedded boards (IMX6 using gpmi-nand driver) that are
> > > > identical except for the NAND FLASH and am trying to understand why
> > > > I'm seeing a massive performance difference when it comes to ubi and
> > > > ubifs (in particular ubi scanning and UBIFS resizing).
> > > >
> > > > Micron MT29F16G08ADACAH4:
> > > > - page size: 4320 bytes (4096 + 224 bytes)
> > > > - block size: 64 pages (256K + 14K bytes)
> > > > - device size: 16Gb
> > > > - 30ns per-block read
> > > > - 300us per-block program
> > > > - 2.0ms per-block erase
> > > >
> > > > Cypress S34ML16G202BHI000
> > > > - page size: 2176 bytes (2048+128)
> > > > - block size: 64 pages (128K + 8K)
> > > > - device size: 16Gb
> > > > - 25ns per-block read
> > > > - 300us per-block program
> > > > - 3.5ms per-block erase
> > > >
> > > > The parts are relatively close in per-block performance but because
> > > > the Micron has twice the block size I think of it being twice as fast
> > > > on a per-byte basis compared to the Cypress and I do see this relative
> > > > performance when testing read/write/erase both in modern U-Boot and
> > > > latest Linux. What doesn't add up is that the Cypress part takes about
> > > > 7x longer to complete the ubi attach (between 'attaching mtd' and
>
> Here you say attach takes 7 times longer.
>
> > > > 'scanning is finished' and about 100x longer to complete the UBIFS
> > > > free space fixup (between 'start fixing up free space' and 'free space
> > > > fixup complete') performed on the first mount of a ubifs filesystem
> > > > that was created with auto-resize enabled.
> > > >
> > > > Both parts are 2GiB and the ubi scanning on attach in the kernel goes
> > > > from ~4s on Micron to ~28s on Cypress and the UBIFS free space fixup
> > > > goes from 0.5s on Micron to a whopping 56s on Cypress.
> > > >
> > > > Any ideas why would I be seeing this kind of performance difference
> > > > when the raw erase/read/write performance is (both datasheet and Linux
> > > > tests) only 2x slower? Anywhere I can look to improve performance of
> > > > the Cypress part?
> > >
> > > Let's focus as first step on UBI attach by scanning. Here UBI reads
> > > the first two
> > > (sub)pages of each block.
> > >
> > > Please figure whether in both setup subpages are used or not. Then
> > > measure how long
> > > it takes to read a single page.
> > > Given this numbers we can start with the detective work.
> > >
> >
> > Hi Richard,
> >
> > I suppose I should have provided:
> >
> > [    6.438793] nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xd5
> > [    6.445240] nand: Micron MT29F16G08ADACAH4
> > [    6.449362] nand: 2048 MiB, SLC, erase size: 256 KiB, page size:
> > 4096, OOB size: 224
> > [    6.459037] Scanning device for bad blocks
> > [    8.784007] 3 fixed-partitions partitions found on MTD device gpmi-nand
> > [    8.790716] Creating 3 MTD partitions on "gpmi-nand":
> > [    8.795823] 0x000000000000-0x000001000000 : "uboot"
> > [    8.810841] 0x000001000000-0x000001100000 : "env"
> > [    8.819356] 0x000001100000-0x000080000000 : "rootfs"
> > [    9.355834] gpmi-nand 112000.gpmi-nand: driver registered.
> >
> > and
> >
> > [    6.014529] nand: device found, Manufacturer ID: 0x01, Chip ID: 0xd5
> > [    6.020907] nand: AMD/Spansion S34ML16G2
> > [    6.024917] nand: 2048 MiB, SLC, erase size: 128 KiB, page size:
> > 2048, OOB size: 128
> > [    6.033385] Scanning device for bad blocks
> > [   10.689345] 3 fixed-partitions partitions found on MTD device gpmi-nand
> > [   10.696056] Creating 3 MTD partitions on "gpmi-nand":
> > [   10.701143] 0x000000000000-0x000001000000 : "uboot"
> > [   10.720392] 0x000001000000-0x000001100000 : "env"
> > [   10.729253] 0x000001100000-0x000080000000 : "rootfs"
> > [   11.793715] gpmi-nand 112000.gpmi-nand: driver registered.
> >
> > So its taking 2.3s to scan the Micron and 5.4s to scan the Cypress
> > which is what I would expect (Micron 2x faster as it has similar read
> > time per block but half as many blocks)
>
> And now all is good.
> I'm confused.
>

Richard,

Whatever 'scan' mtd is doing achieves reasonable results in my opinion
as well as erase/read/write (tested with 'flash_erase' and 'dd' the
Micron part is about 2x faster MB/s than the Cypress part).

What doesn't provide reasonable results is the ubi scan and UBIFS
'free space fixup' (the fixup only occurs on first mount of a ubifs
created with the 'autoresize' flag):

With the Cypress FLASH:
[   10.141066] ubi0: default fastmap pool size: 256
[   10.145968] ubi0: default fastmap WL pool size: 128
[   10.150875] ubi0: attaching mtd2
[   10.206733] ubi0: scanning started
^^^ ubi scan takes ~12s compared to ~4s on Micron FLASH
[   22.539385] ubi0: scanning is finished
[   22.587780] ubi0 warning: ubi_eba_init: cannot reserve enough PEBs
for bad PEB handling, reserved 316, need 320
[   22.624958] ubi0: attached mtd2 (name "rootfs", size 2031 MiB)
[   22.630826] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[   22.637798] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[   22.644663] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[   22.651647] ubi0: good PEBs: 16248, bad PEBs: 0, corrupted PEBs: 0
[   22.657897] ubi0: user volume: 1, internal volumes: 1, max. volumes
count: 128
[   22.665188] ubi0: max/mean erase counter: 1/0, WL threshold: 4096,
image sequence number: 789781498
[   22.674302] ubi0: available PEBs: 0, total reserved PEBs: 16248,
PEBs reserved for bad PEB handling: 316
[   22.684179] ubi0: background thread "ubi_bgt0d" started, PID 233
...
[   24.326892] UBIFS (ubi0:0): start fixing up free space
[   51.460774] UBIFS (ubi0:0): free space fixup complete
[   51.473525] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 274
^^^ UBIFS free space fixup (first mount of fs created with
'autoresize' flag) takes ~27s compared to ~0.5s on Micron

So my question is why is the Cypress NAND taking way more than twice
as long compared to Micron with 'ubi' scan and 'ubifs resize' when the
read/write/erase performance is only 2x slower between the two.

> > > Did you also check what timings both NANDs are using?
> >
> > No, where would I see these? Would this be something that can be
> > provided in device-tree or a nand chip-specific driver?
>
> Usually the driver computes them based on various sources.
> See drivers/mtd/nand/raw/nand_timings.c
>

Ok, thanks for the pointer. Is there a sysfs node that contains all of
those? I didn't see anything obvious. I can printk them for comparison
but I don't see this as a raw nand issue, I see it as a ubi/ubifs
issue. There is something going on at the ubi/ubifs layer that makes
the Cypress FLASH very slow for the ubi scan that occurs on attach and
the UBIFS resize that occurs on first mount.

Thanks,

Tim

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/