RE: [PATCH v11 07/10] mtd: spi-nor: Add stacked memories support in spi-nor

["Mahapatra, Amit Kumar" <amit.kumar-mahapatra@xxxxxxx>]

> -----Original Message-----
> From: Mahapatra, Amit Kumar <amit.kumar-mahapatra@xxxxxxx>
> Sent: Wednesday, March 13, 2024 9:34 PM
> To: Tudor Ambarus <tudor.ambarus@xxxxxxxxxx>; broonie@xxxxxxxxxx;
> pratyush@xxxxxxxxxx; miquel.raynal@xxxxxxxxxxx; richard@xxxxxx;
> vigneshr@xxxxxx; sbinding@xxxxxxxxxxxxxxxxxxxxx; lee@xxxxxxxxxx;
> james.schulman@xxxxxxxxxx; david.rhodes@xxxxxxxxxx;
> rf@xxxxxxxxxxxxxxxxxxxxx; perex@xxxxxxxx; tiwai@xxxxxxxx
> Cc: linux-spi@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> michael@xxxxxxxx; linux-mtd@xxxxxxxxxxxxxxxxxxx;
> nicolas.ferre@xxxxxxxxxxxxx; alexandre.belloni@xxxxxxxxxxx;
> claudiu.beznea@xxxxxxxxx; Simek, Michal <michal.simek@xxxxxxx>; linux-
> arm-kernel@xxxxxxxxxxxxxxxxxxx; alsa-devel@xxxxxxxxxxxxxxxx;
> patches@xxxxxxxxxxxxxxxxxxxxx; linux-sound@xxxxxxxxxxxxxxx; git (AMD-
> Xilinx) <git@xxxxxxx>; amitrkcian2002@xxxxxxxxx; Conor Dooley
> <conor.dooley@xxxxxxxxxxxxx>; beanhuo@xxxxxxxxxx
> Subject: RE: [PATCH v11 07/10] mtd: spi-nor: Add stacked memories support
> in spi-nor
> 
> 
> 
> > -----Original Message-----
> > From: Tudor Ambarus <tudor.ambarus@xxxxxxxxxx>
> > Sent: Friday, February 9, 2024 9:44 PM
> > To: Mahapatra, Amit Kumar <amit.kumar-mahapatra@xxxxxxx>;
> > broonie@xxxxxxxxxx; pratyush@xxxxxxxxxx; miquel.raynal@xxxxxxxxxxx;
> > richard@xxxxxx; vigneshr@xxxxxx; sbinding@xxxxxxxxxxxxxxxxxxxxx;
> > lee@xxxxxxxxxx; james.schulman@xxxxxxxxxx; david.rhodes@xxxxxxxxxx;
> > rf@xxxxxxxxxxxxxxxxxxxxx; perex@xxxxxxxx; tiwai@xxxxxxxx
> > Cc: linux-spi@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> > michael@xxxxxxxx; linux-mtd@xxxxxxxxxxxxxxxxxxx;
> > nicolas.ferre@xxxxxxxxxxxxx; alexandre.belloni@xxxxxxxxxxx;
> > claudiu.beznea@xxxxxxxxx; Simek, Michal <michal.simek@xxxxxxx>;
> linux-
> > arm-kernel@xxxxxxxxxxxxxxxxxxx; alsa-devel@xxxxxxxxxxxxxxxx;
> > patches@xxxxxxxxxxxxxxxxxxxxx; linux-sound@xxxxxxxxxxxxxxx; git (AMD-
> > Xilinx) <git@xxxxxxx>; amitrkcian2002@xxxxxxxxx; Conor Dooley
> > <conor.dooley@xxxxxxxxxxxxx>; beanhuo@xxxxxxxxxx
> > Subject: Re: [PATCH v11 07/10] mtd: spi-nor: Add stacked memories
> > support in spi-nor

Hello Everyone,

I would like to propose another approach for handling stacked and 
parallel connection modes and would appreciate your thoughts on it. 
But before that, here is some background on what we are trying to achieve.

The AMD QSPI controller supports two advanced connection modes(Stacked and 
Dual Parallel) which allow the controller to treat two different flashes 
as one storage.

Stacked:
Flashes share the same SPI bus, but different CS line, controller asserts 
the CS of the flash to which it needs to communicate.

Dual Parallel:
Both the flashes have their separate SPI bus CS of both the flashes will 
be asserted/de-asserted at the same time. In this mode data will be split 
across both the flashes by enabling the STRIPE setting in the controller. 
If STRIPE is not enabled, then same data will be sent to both the devices.

For more information on the modes please feel free to go through the 
controller flash interface below
https://docs.amd.com/r/en-US/am011-versal-acap-trm/QSPI-Flash-Device-Interface

Mirochip QSPI controller also supports "Dual Parallel 8-bit IO mode", but 
they call it "Twin Quad Mode".
https://ww1.microchip.com/downloads/aemDocuments/documents/MPU32/ProductDocuments/DataSheets/SAMA7G5-Series-Data-Sheet-DS60001765.pdf

DT binding changes were added through the following commits:
https://github.com/torvalds/linux/commit/f89504300e94524d5d5846ff8b728592ac72cec4
https://github.com/torvalds/linux/commit/eba5368503b4291db7819512600fa014ea17c5a8
https://github.com/torvalds/linux/commit/e2edd1b64f1c79e8abda365149ed62a2a9a494b4

SPI core changes were adds through the following commit:
https://github.com/torvalds/linux/commit/4d8ff6b0991d5e86b17b235fc46ec62e9195cb9b

Based on the inputs/suggestions from Tudor, i am planning to add a new 
layer between the SPI-NOR and MTD layers to support stacked and parallel 
configurations. This new layer will be part of the spi-nor and located in 
mtd/spi-nor/

This layer would perform the following tasks:
 - During probing, store information from all the connected flashes, 
   whether in stacked or parallel mode, and present it as a single device 
   to the MTD layer.
 - Register callbacks through this new layer instead of spi-nor/core.c and 
   handle MTD device registration.
 - In stacked mode, select the appropriate spi-nor flash based on the 
   address provided by the MTD layer during flash operations.
 - Manage flash crossover operations in stacked mode.
 - Ensure both connected flashes are identical in parallel mode.
 - Handle odd byte count requests from the MTD layer during flash 
   operations in parallel mode.

For implementing this the current DT binding need to be updated as 
follows.

stacked-memories DT changes:
 - Flash size information can be retrieved directly from the flash, so it 
   has been removed from the DT binding.
 - Each stacked flash will have its own flash node. This approach allows 
   flashes of different makes and sizes to be stacked together, as each 
   flash will be probed individually.
 - The stacked-memories DT bindings will contain the phandles of the flash 
   nodes connected in stacked mode.

spi@0 {
  
  flash@0 {
    compatible = "jedec,spi-nor"
    reg = <0x00>;
    stacked-memories = <&flash@0 &flash@1>;
    spi-max-frequency = <50000000>;
    ...
              partition@0 { 
        label = "qspi-0";
        reg = <0x0 0xf00000>;
    };
                        

  }
  
  flash@1 {
    compatible = "jedec,spi-nor"
              reg = <0x01>;
    spi-max-frequency = <50000000>;
    ...
              partition@0 { 
        label = "qspi-1";
        reg = <0x0 0x800000>;
    };
  }
}

parallel-memories DT changes:
 - Flash size information can be retrieved directly from the flash, so it 
   has been removed from the DT binding.
 - Each flash connected in parallel mode will have its own flash node. 
   This allows us to verify that both flashes connected in parallel are 
   identical, as each flash node will be probed separately.
 - The parallel-memories DT bindings will contain the phandles of the 
   flash nodes connected in parallel.

spi@0 {
  
  flash@0 {
    compatible = "jedec,spi-nor"
    reg = <0x00>;
    parallel-memories = <&flash@0 &flash@1>;
    spi-max-frequency = <50000000>;
    ...
              partition@0 { 
        label = "qspi-0";
        reg = <0x0 0xf00000>;
    };
                        

  }
  
  flash@1 {
    compatible = "jedec,spi-nor"
              reg = <0x01>;
    spi-max-frequency = <50000000>;
    ...
              partition@0 { 
        label = "qspi-1";
        reg = <0x0 0x800000>;
    };
  }
}

Regards,
Amit

> >
> >
> >
> > On 2/9/24 11:06, Tudor Ambarus wrote:
> > >
> > >
> > > On 12/21/23 06:54, Mahapatra, Amit Kumar wrote:
> > >>> Something else to consider: I see that Micron has a twin quad mode:
> > >>> https://media-www.micron.com/-
> > >>> /media/client/global/documents/products/data-sheet/nor-flash/seria
> > >>> l-
> > >>> nor/mt25t/generation-
> > >>>
> > b/mt25t_qljs_l_512_xba_0.pdf?rev=de70b770c5dc4da8b8ead06b57c03500
> > >>>
> > >>> The micron's "Separate Chip-Select and Clock Signals" resembles
> > >>> the AMD's dual parallel 8-bit.
> > >> Yes, I agree.
> > >>
> > >>> Micron's "Shared Chip-Select and Clock Signals" differs from the
> > >>> AMD's stacked mode, as Micron uses DQ[3:0] and DQ[7:4], whereas
> > >>> AMD considers both as DQ[3:0].
> > >> Yes, correct.
> > >
> > > Amit, please help me to assess this. I assume Micron and Microchip
> > > is using the same concepts as AMD uses for the "Dual Parallel 8-bit
> > > IO mode", but they call it "Twin Quad Mode".
> > >
> > > I was wrong, the AMD datasheet [1] was misleading [2], it described
> > > the IOs for both flashes as IO[3:0], but later on in the "Table QSPI
> > > Interface Signals" the second flash is described with IO[7:4].
> > >
> > > The AMD's 8-bit Dual Flash Parallel Interface is using dedicated CS#
> > > and CLK# lines for each flash. As Micron does, isn't it?
> > >
> > > Micron says [3] that:
> > > "The device contains two quad I/O die, each able to operate
> > > independently for a total of eight I/Os. The memory map applies to
> > > each die. Each die has internal registers for status, configuration,
> > > and device protection that can be set and read independently from
> > > one
> > other.
> > > Micron recommends that internal configuration settings for the two
> > > die be set identically."
> >
> 
> Hello Tudor,
> 
> > Amit,
> >
> > I forgot to say my first conclusion about the quote from above. Even
> > if the dies are in the same physical package, micron asks users to
> > configure each die as it is a self-standing entity, IOW to configure
> > each die as it is a flash on its own. To me it looks like 2
> > concatenated flashes at first look. Thus identical to how AMD controller
> works.
> > Please clarify this.
> 
> That’s correct, the Micron flash that you referred can communicate with the
> AMD QSPI controller in both parallel and stacked mode.
> >
> > >
> > > it also says that:
> > > "When using quad commands in XIO-SPI or when using QIO-SPI,
> > > DQ[3:0]/DQ[7:4] are I/O."
> >
> > and this would be a parallel concatenation of two flashes.
> 
> That's correct.
> 
> Regards,
> Amit
> >
> > Then it would be good if you let us now the similarities and
> > differences between how amd and mchp controller work, I scrawled few
> ideas below.
> >
> > thanks,
> > ta
> > >
> > > So I guess the upper layers just ask for a chunk of memory to be
> > > written and the controller handles the cs# lines automatically. How
> > > is the AMD controller working, do you have to drive the cs# lines
> > > manually, or you just set the parallel mode and the controller takes
> > > care of
> > everything?
> > >
> > > I assume this is how mchp is handling things, they seem to just set
> > > a bit the protocol into the QSPI_IFR.PROTTYP register field and
> > > that's all [4]. They even seem to write the registers of both
> > > flashes at the same
> > time.
> > >
> > > In what regards the AMD's "dual stack interface", AMD is sharing the
> > > clock and IO lines and uses dedicated CS# lines for the flashes,
> > > whereas Micron shares the CS# and CLK# lines with different IO lines.
> > >
> > > Amit, please study the architectures used by mchp, micron and amd
> > > and let us know if they are the same or they differ, and if they
> > > differ what are the differences.
> > >
> > > I added Conor from mchp in cc, I see Nicolas is already there, and
> > > Bean from micron.
> > >
> > > Thanks,
> > > ta
> > >
> > > [1]
> > > https://docs.xilinx.com/r/en-US/am011-versal-acap-trm/QSPI-Flash-Int
> > > er
> > > face-Signals
> > > [2]
> > >
> >
> https://docs.xilinx.com/viewer/attachment/dwmjhDJGICdJqD4swyVzcQ/fD8nv
> > > 4ry78xM0_EF5kv4mA
> > > [3]
> > > https://media-www.micron.com/-
> > /media/client/global/documents/products/
> > > data-sheet/nor-flash/serial-nor/mt25t/generation-b/mt25t_qljs_l_512_
> > > xb
> > > a_0.pdf?rev=de70b770c5dc4da8b8ead06b57c03500
> > > [4]
> > >
> >
> https://ww1.microchip.com/downloads/aemDocuments/documents/MPU32/
> > Produ
> > > ctDocuments/DataSheets/SAMA7G5-Series-Data-Sheet-DS60001765.pdf