Patch "mm: hugetlb pages should not be reserved by shmat() if SHM_NORESERVE" has been added to the 5.10-stable tree

[Sasha Levin <sashal@xxxxxxxxxx>]

This is a note to let you know that I've just added the patch titled

    mm: hugetlb pages should not be reserved by shmat() if SHM_NORESERVE

to the 5.10-stable tree which can be found at:
    http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary

The filename of the patch is:
     mm-hugetlb-pages-should-not-be-reserved-by-shmat-if-.patch
and it can be found in the queue-5.10 subdirectory.

If you, or anyone else, feels it should not be added to the stable tree,
please let <stable@xxxxxxxxxxxxxxx> know about it.



commit 248455f8b451d2ac07c8a295e2686f1524b80383
Author: Prakash Sangappa <prakash.sangappa@xxxxxxxxxx>
Date:   Tue Jan 23 12:04:42 2024 -0800

    mm: hugetlb pages should not be reserved by shmat() if SHM_NORESERVE
    
    [ Upstream commit e656c7a9e59607d1672d85ffa9a89031876ffe67 ]
    
    For shared memory of type SHM_HUGETLB, hugetlb pages are reserved in
    shmget() call.  If SHM_NORESERVE flags is specified then the hugetlb pages
    are not reserved.  However when the shared memory is attached with the
    shmat() call the hugetlb pages are getting reserved incorrectly for
    SHM_HUGETLB shared memory created with SHM_NORESERVE which is a bug.
    
    -------------------------------
    Following test shows the issue.
    
    $cat shmhtb.c
    
    int main()
    {
            int shmflags = 0660 | IPC_CREAT | SHM_HUGETLB | SHM_NORESERVE;
            int shmid;
    
            shmid = shmget(SKEY, SHMSZ, shmflags);
            if (shmid < 0)
            {
                    printf("shmat: shmget() failed, %d\n", errno);
                    return 1;
            }
            printf("After shmget()\n");
            system("cat /proc/meminfo | grep -i hugepages_");
    
            shmat(shmid, NULL, 0);
            printf("\nAfter shmat()\n");
            system("cat /proc/meminfo | grep -i hugepages_");
    
            shmctl(shmid, IPC_RMID, NULL);
            return 0;
    }
    
     #sysctl -w vm.nr_hugepages=20
     #./shmhtb
    
    After shmget()
    HugePages_Total:      20
    HugePages_Free:       20
    HugePages_Rsvd:        0
    HugePages_Surp:        0
    
    After shmat()
    HugePages_Total:      20
    HugePages_Free:       20
    HugePages_Rsvd:        5 <--
    HugePages_Surp:        0
    --------------------------------
    
    Fix is to ensure that hugetlb pages are not reserved for SHM_HUGETLB shared
    memory in the shmat() call.
    
    Link: https://lkml.kernel.org/r/1706040282-12388-1-git-send-email-prakash.sangappa@xxxxxxxxxx
    Signed-off-by: Prakash Sangappa <prakash.sangappa@xxxxxxxxxx>
    Acked-by: Muchun Song <muchun.song@xxxxxxxxx>
    Cc: <stable@xxxxxxxxxxxxxxx>
    Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
    Signed-off-by: Sasha Levin <sashal@xxxxxxxxxx>

diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index c3e9fa7ce75f7..bf3cda4989623 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -135,6 +135,7 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
 	loff_t len, vma_len;
 	int ret;
 	struct hstate *h = hstate_file(file);
+	vm_flags_t vm_flags;
 
 	/*
 	 * vma address alignment (but not the pgoff alignment) has
@@ -176,10 +177,20 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
 	file_accessed(file);
 
 	ret = -ENOMEM;
+
+	vm_flags = vma->vm_flags;
+	/*
+	 * for SHM_HUGETLB, the pages are reserved in the shmget() call so skip
+	 * reserving here. Note: only for SHM hugetlbfs file, the inode
+	 * flag S_PRIVATE is set.
+	 */
+	if (inode->i_flags & S_PRIVATE)
+		vm_flags |= VM_NORESERVE;
+
 	if (!hugetlb_reserve_pages(inode,
 				vma->vm_pgoff >> huge_page_order(h),
 				len >> huge_page_shift(h), vma,
-				vma->vm_flags))
+				vm_flags))
 		goto out;
 
 	ret = 0;