On Tue, Jan 10, 2023 at 05:54:10PM +0100, Michal Hocko wrote: > On Tue 10-01-23 17:23:58, Mike Rapoport wrote: > [...] > > +* ``ZONE_DMA`` and ``ZONE_DMA32`` represent memory suitable for DMA by > > + peripheral devices that cannot access all of the addressable memory. > > I think it would be better to not keep the historical DMA based menaning > and teach that future developers. You can say something like > > ZONE_DMA and ZONE_DMA32 have historically been used for memory suitable > for DMA. For many years there are better more robust interfaces to > get memory with DMA specific requirements (Documentation/core-api/dma-api.rst). But even today ZONE_DMA(32) means that the memory is suitable for DMA. This is nicely encapsulated with dma APIs and there should be no new GFP_DMA users, but still memory outside ZONE_DMA is not suitable for DMA. > > + Depending on the architecture, either of these zone types or even they both > > + can be disabled at build time using ``CONFIG_ZONE_DMA`` and > > + ``CONFIG_ZONE_DMA32`` configuration options. Some 64-bit platforms may need > > + both zones as they support peripherals with different DMA addressing > > + limitations. > > + > > +* ``ZONE_NORMAL`` is for normal memory that can be accessed by the kernel all > > + the time. DMA operations can be performed on pages in this zone if the DMA > > + devices support transfers to all addressable memory. ``ZONE_NORMAL`` is > > + always enabled. > > + > > +* ``ZONE_HIGHMEM`` is the part of the physical memory that is not covered by a > > + permanent mapping in the kernel page tables. The memory in this zone is only > > + accessible to the kernel using temporary mappings. This zone is available > > + only on some 32-bit architectures and is enabled with ``CONFIG_HIGHMEM``. > > + > > +* ``ZONE_MOVABLE`` is for normal accessible memory, just like ``ZONE_NORMAL``. > > + The difference is that most pages in ``ZONE_MOVABLE`` are movable. > > This is really confusing because those pages are not really movable. You > cannot move a page itself. I guess you meant to say something like > > The difference is that there are means to migrate memory via > migrate_pages interface. A typical example would be a memory mapped to > userspace which can be rellocate the underlying memory content and > update page tables so that userspace doesn't notice the physical data > placement has changed. I agree that this sentence is a bit confusing, but there's a clarification below. Also, I'd like to keep this at high level without going to the details about how exactly the pages can be migrated. > > That means > > + that while virtual addresses of these pages do not change, their content may > > + move between different physical pages. ``ZONE_MOVABLE`` is only enabled when > > + one of ``kernelcore``, ``movablecore`` and ``movable_node`` parameters is > > + present in the kernel command line. See :ref:`Page migration > > + <page_migration>` for additional details. > > This is not really true. The movable zone can be also enabled by memory > hotplug. In fact it is one of the more common usecases for the zone > because memory hot remove largerly depends on memory to be migrated for > offlining to succeed in most cases. Right. How about this version of ZONE_MOVABLE description: * ``ZONE_MOVABLE`` is for normal accessible memory, just like ``ZONE_NORMAL``. The difference is that the contents of most pages in ``ZONE_MOVABLE`` is movable. That means that while virtual addresses of these pages do not change, their content may move between different physical pages. Often ``ZONE_MOVABLE`` is populated during memory hotplug, but it may be also populated on boot using one of ``kernelcore``, ``movablecore`` and ``movable_node`` kernel command line parameters. See :ref:`Page migration <page_migration>` and :ref:`Memory Hot(Un)Plug <_admin_guide_memory_hotplug>` for additional details. > > +* ``ZONE_DEVICE`` represents memory residing on devices such as PMEM and GPU. > > + It has different characteristics than RAM zone types and it exists to provide > > + :ref:`struct page <Pages>` and memory map services for device driver > > + identified physical address ranges. ``ZONE_DEVICE`` is enabled with > > + configuration option ``CONFIG_ZONE_DEVICE``. > > + > > +It is important to note that many kernel operations can only take place using > > +``ZONE_NORMAL`` so it is the most performance critical zone. Zones are > > +discussed further in Section :ref:`Zones <zones>`. > > + > > +The relation between node and zone extents is determined by the physical memory > > +map reported by the firmware, architectural constraints for memory addressing > > +and certain parameters in the kernel command line. > > + > > +For example, with 32-bit kernel on an x86 UMA machine with 2 Gbytes of RAM the > > +entire memory will be on node 0 and there will be three zones: ``ZONE_DMA``, > > +``ZONE_NORMAL`` and ``ZONE_HIGHMEM``:: > > + > > + 0 2G > > + +-------------------------------------------------------------+ > > + | node 0 | > > + +-------------------------------------------------------------+ > > + > > + 0 16M 896M 2G > > + +----------+-----------------------+--------------------------+ > > + | ZONE_DMA | ZONE_NORMAL | ZONE_HIGHMEM | > > + +----------+-----------------------+--------------------------+ > > + > > + > > +With a kernel built with ``ZONE_DMA`` disabled and ``ZONE_DMA32`` enabled and > > +booted with ``movablecore=80%`` parameter on an arm64 machine with 16 Gbytes of > > +RAM equally split between two nodes, there will be ``ZONE_DMA32``, > > +``ZONE_NORMAL`` and ``ZONE_MOVABLE`` on node 0, and ``ZONE_NORMAL`` and > > +``ZONE_MOVABLE`` on node 1:: > > + > > + > > + 1G 9G 17G > > + +--------------------------------+ +--------------------------+ > > + | node 0 | | node 1 | > > + +--------------------------------+ +--------------------------+ > > + > > + 1G 4G 4200M 9G 9320M 17G > > + +---------+----------+-----------+ +------------+-------------+ > > + | DMA32 | NORMAL | MOVABLE | | NORMAL | MOVABLE | > > + +---------+----------+-----------+ +------------+-------------+ > > I think it is useful to note that nodes and zones can overlap in the > physical address range. It is not uncommong to interleave two nodes and > it is also possible that memory holes are memory hotplugged into MOVABLE > zone arbitrarily in the physical address range. Hmm, not sure I understand what you mean by "overlap". For interleaved nodes you mean that node 0 may span, say [0x0, 0x2000) and [0x4000, 06000) and node 1 spans [0x2000, 0x4000) and [0x6000, 0x8000)? And as for MOVABLE zone, you mean that it can appear between ranges of NORMAL zone? > Other than that looks good to me and thanks for taking care of filling > up these gaps! This is highly appreciated. Thanks! I'd appreciate more inputs ;-) > -- > Michal Hocko > SUSE Labs -- Sincerely yours, Mike.
