>On Tue, Jan 17, 2023 at 12:31 AM Jaewon Kim <jaewon31.kim@xxxxxxxxxxx> wrote: >> > Using order 4 pages would be helpful for many IOMMUs, but it could spend >> > quite much time in page allocation perspective. >> > >> > The order 4 allocation with __GFP_RECLAIM may spend much time in >> > reclaim and compation logic. __GFP_NORETRY also may affect. These cause >> > unpredictable delay. >> > >> > To get reasonable allocation speed from dma-buf system heap, use >> > HIGH_ORDER_GFP for order 4 to avoid reclaim. > >Thanks for sharing this! >The case where the allocation gets stuck behind reclaim under pressure >does sound undesirable, but I'd be a bit hesitant to tweak numbers >that have been used for a long while (going back to ion) without a bit >more data. > >It might be good to also better understand the tradeoff of potential >on-going impact to performance from using low order pages when the >buffer is used. Do you have any details like or tests that you could >share to help ensure this won't impact other users? > >TJ: Do you have any additional thoughts on this? > >thanks >-john Let me share what I tested with trace events of compaction, direct_reclaim. This log was captured within one dma-buf alloc for 1,957,888 byte size. It shows direct reclaim spent 4.973 msec total so that it could reclaim 209 pages total. But it spent 12.606 msec in compaction. I might miss preempted time but I think compaction is quite time consuming job. Sorry I don't know how much the current gfp is helpful to IOMMU. I think the default should be set like my change to improve allocation, and the current gfp could be set if need through other options like cmdline. -5972 [006] .... 7971.248176: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x40000 free_pfn=0xffc00 zone_end=0x100000, mode=async -5972 [006] .... 7971.248394: mm_compaction_end: zone_start=0x40000 migrate_pfn=0xac800 free_pfn=0xffc00 zone_end=0x100000, mode=async status=contended -5972 [006] .... 7971.248399: mm_vmscan_direct_reclaim_begin: order=4 gfp_flags=GFP_HIGHUSER|__GFP_NOWARN|__GFP_COMP|__GFP_ZERO -5972 [006] .... 7971.248922: mm_vmscan_direct_reclaim_end: nr_reclaimed=17 -5972 [006] .... 7971.248974: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0xacc00 free_pfn=0xffc00 zone_end=0x100000, mode=async -5972 [006] .... 7971.249063: mm_compaction_end: zone_start=0x40000 migrate_pfn=0xb5c00 free_pfn=0xffc00 zone_end=0x100000, mode=async status=contended -5972 [006] .... 7971.249067: mm_vmscan_direct_reclaim_begin: order=4 gfp_flags=GFP_HIGHUSER|__GFP_NOWARN|__GFP_COMP|__GFP_ZERO -5972 [006] .... 7971.249768: mm_vmscan_direct_reclaim_end: nr_reclaimed=18 -5972 [006] .... 7971.249843: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0xb6000 free_pfn=0xafc00 zone_end=0x100000, mode=async -5972 [006] .... 7971.249845: mm_compaction_end: zone_start=0x40000 migrate_pfn=0xb6000 free_pfn=0xafc00 zone_end=0x100000, mode=async status=partial_skipped -5972 [006] .... 7971.249849: mm_vmscan_direct_reclaim_begin: order=4 gfp_flags=GFP_HIGHUSER|__GFP_NOWARN|__GFP_COMP|__GFP_ZERO -5972 [006] .... 7971.250116: mm_vmscan_direct_reclaim_end: nr_reclaimed=22 -5972 [006] .... 7971.250152: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x40000 free_pfn=0xaf800 zone_end=0x100000, mode=async -5972 [006] .... 7971.250305: mm_compaction_end: zone_start=0x40000 migrate_pfn=0xac400 free_pfn=0xaf800 zone_end=0x100000, mode=async status=success -5972 [006] .... 7971.250375: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0xac800 free_pfn=0xaf800 zone_end=0x100000, mode=async -5972 [006] .... 7971.250380: mm_compaction_end: zone_start=0x40000 migrate_pfn=0xaf800 free_pfn=0xaf800 zone_end=0x100000, mode=async status=partial_skipped -5972 [006] .... 7971.250384: mm_vmscan_direct_reclaim_begin: order=4 gfp_flags=GFP_HIGHUSER|__GFP_NOWARN|__GFP_COMP|__GFP_ZERO -5972 [006] .... 7971.250597: mm_vmscan_direct_reclaim_end: nr_reclaimed=20 -5972 [006] .... 7971.251289: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x40000 free_pfn=0xaf000 zone_end=0x100000, mode=async -5972 [006] .... 7971.251497: mm_compaction_end: zone_start=0x40000 migrate_pfn=0xac400 free_pfn=0xaf000 zone_end=0x100000, mode=async status=contended -5972 [006] .... 7971.251502: mm_vmscan_direct_reclaim_begin: order=4 gfp_flags=GFP_HIGHUSER|__GFP_NOWARN|__GFP_COMP|__GFP_ZERO -5972 [006] .... 7971.251797: mm_vmscan_direct_reclaim_end: nr_reclaimed=22 -5972 [006] .... 7971.251890: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x5b400 free_pfn=0xb0400 zone_end=0x100000, mode=sync -5972 [006] .... 7971.252740: mm_compaction_end: zone_start=0x40000 migrate_pfn=0x63800 free_pfn=0xb0400 zone_end=0x100000, mode=sync status=success -5972 [006] .... 7971.252774: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x63c00 free_pfn=0xb0000 zone_end=0x100000, mode=async -5972 [006] .... 7971.253325: mm_compaction_end: zone_start=0x40000 migrate_pfn=0x74e20 free_pfn=0xade92 zone_end=0x100000, mode=async status=success -5972 [006] .... 7971.253333: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x83400 free_pfn=0xafc00 zone_end=0x100000, mode=async -5972 [006] .... 7971.253683: mm_compaction_end: zone_start=0x40000 migrate_pfn=0xac400 free_pfn=0xafc00 zone_end=0x100000, mode=async status=contended -5972 [006] .... 7971.253688: mm_vmscan_direct_reclaim_begin: order=4 gfp_flags=GFP_HIGHUSER|__GFP_NOWARN|__GFP_COMP|__GFP_ZERO -5972 [006] .... 7971.254134: mm_vmscan_direct_reclaim_end: nr_reclaimed=22 -5972 [006] .... 7971.254206: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0xad000 free_pfn=0xafc00 zone_end=0x100000, mode=async -5972 [006] .... 7971.254289: mm_compaction_end: zone_start=0x40000 migrate_pfn=0xae1c0 free_pfn=0xae000 zone_end=0x100000, mode=async status=partial_skipped -5972 [006] .... 7971.254304: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x40000 free_pfn=0xffc00 zone_end=0x100000, mode=async -5972 [006] .... 7971.254510: mm_compaction_end: zone_start=0x40000 migrate_pfn=0x74db0 free_pfn=0xadcc1 zone_end=0x100000, mode=async status=success -5972 [006] .... 7971.254519: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x75000 free_pfn=0xffc00 zone_end=0x100000, mode=async -5972 [006] .... 7971.254612: mm_compaction_end: zone_start=0x40000 migrate_pfn=0xac400 free_pfn=0xffc00 zone_end=0x100000, mode=async status=contended -5972 [006] .... 7971.254617: mm_vmscan_direct_reclaim_begin: order=4 gfp_flags=GFP_HIGHUSER|__GFP_NOWARN|__GFP_COMP|__GFP_ZERO -5972 [006] .... 7971.255303: mm_vmscan_direct_reclaim_end: nr_reclaimed=22 -5972 [006] .... 7971.255307: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x75000 free_pfn=0xffc00 zone_end=0x100000, mode=sync -5972 [006] .... 7971.260198: mm_compaction_end: zone_start=0x40000 migrate_pfn=0x751e5 free_pfn=0xaa83f zone_end=0x100000, mode=sync status=success -5972 [006] .... 7971.260236: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x40000 free_pfn=0xffc00 zone_end=0x100000, mode=async -5972 [006] .... 7971.260883: mm_compaction_end: zone_start=0x40000 migrate_pfn=0xb5c00 free_pfn=0xffc00 zone_end=0x100000, mode=async status=contended -5972 [006] .... 7971.260891: mm_vmscan_direct_reclaim_begin: order=4 gfp_flags=GFP_HIGHUSER|__GFP_NOWARN|__GFP_COMP|__GFP_ZERO -5972 [006] .... 7971.261174: mm_vmscan_direct_reclaim_end: nr_reclaimed=22 -5972 [006] .... 7971.261178: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x75000 free_pfn=0xffc00 zone_end=0x100000, mode=sync -5972 [006] .... 7971.264861: mm_compaction_end: zone_start=0x40000 migrate_pfn=0x75681 free_pfn=0xffc00 zone_end=0x100000, mode=sync status=success -5972 [006] .... 7971.264873: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0xbbc00 free_pfn=0xffc00 zone_end=0x100000, mode=async -5972 [006] .... 7971.264889: mm_compaction_end: zone_start=0x40000 migrate_pfn=0xc1400 free_pfn=0xffc00 zone_end=0x100000, mode=async status=contended -5972 [006] .... 7971.264891: mm_vmscan_direct_reclaim_begin: order=4 gfp_flags=GFP_HIGHUSER|__GFP_NOWARN|__GFP_COMP|__GFP_ZERO -5972 [006] .... 7971.265141: mm_vmscan_direct_reclaim_end: nr_reclaimed=22 -5972 [002] .... 7971.265585: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0xc1800 free_pfn=0xffc00 zone_end=0x100000, mode=async -5972 [002] .... 7971.265669: mm_compaction_end: zone_start=0x40000 migrate_pfn=0xc2c00 free_pfn=0xffc00 zone_end=0x100000, mode=async status=contended -5972 [002] .... 7971.265678: mm_vmscan_direct_reclaim_begin: order=4 gfp_flags=GFP_HIGHUSER|__GFP_NOWARN|__GFP_COMP|__GFP_ZERO -5972 [002] .... 7971.266987: mm_vmscan_direct_reclaim_end: nr_reclaimed=22 -5972 [006] .... 7971.267466: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x40000 free_pfn=0xffc00 zone_end=0x100000, mode=sync -5972 [006] .... 7971.267778: mm_compaction_end: zone_start=0x40000 migrate_pfn=0x5d000 free_pfn=0xffc00 zone_end=0x100000, mode=sync status=success -5972 [006] .... 7971.267927: mm_compaction_begin: zone_start=0x40000 migrate_pfn=0x5c000 free_pfn=0xffc00 zone_end=0x100000, mode=async -5972 [006] .... 7971.268092: mm_compaction_end: zone_start=0x40000 migrate_pfn=0x74d70 free_pfn=0xffc00 zone_end=0x100000, mode=async status=success Thank you Jaewon Kim
