Hi Randy, On Sat, Sep 28, 2024 at 11:28 AM Randy Dunlap <rdunlap@xxxxxxxxxxxxx> wrote: > > > > On 9/27/24 11:52 AM, jeffxu@xxxxxxxxxxxx wrote: > > From: Jeff Xu <jeffxu@xxxxxxxxxxxx> > > > > Update doc after in-loop change: mprotect/madvise can have > > partially updated and munmap is atomic. > > > > Fix indentation and clarify some sections to improve readability. > > > > Signed-off-by: Jeff Xu <jeffxu@xxxxxxxxxxxx> > > --- > > Documentation/userspace-api/mseal.rst | 290 ++++++++++++-------------- > > 1 file changed, 136 insertions(+), 154 deletions(-) > > > > diff --git a/Documentation/userspace-api/mseal.rst b/Documentation/userspace-api/mseal.rst > > index 4132eec995a3..68986084e191 100644 > > --- a/Documentation/userspace-api/mseal.rst > > +++ b/Documentation/userspace-api/mseal.rst > > @@ -23,177 +23,159 @@ applications can additionally seal security critical data at runtime. > > A similar feature already exists in the XNU kernel with the > > VM_FLAGS_PERMANENT flag [1] and on OpenBSD with the mimmutable syscall [2]. > > > > -User API > > -======== > > -mseal() > > ------------ > > -The mseal() syscall has the following signature: > > - > > -``int mseal(void addr, size_t len, unsigned long flags)`` > > - > > -**addr/len**: virtual memory address range. > > - > > -The address range set by ``addr``/``len`` must meet: > > - - The start address must be in an allocated VMA. > > - - The start address must be page aligned. > > - - The end address (``addr`` + ``len``) must be in an allocated VMA. > > - - no gap (unallocated memory) between start and end address. > > - > > -The ``len`` will be paged aligned implicitly by the kernel. > > - > > -**flags**: reserved for future use. > > - > > -**return values**: > > - > > -- ``0``: Success. > > - > > -- ``-EINVAL``: > > - - Invalid input ``flags``. > > - - The start address (``addr``) is not page aligned. > > - - Address range (``addr`` + ``len``) overflow. > > - > > -- ``-ENOMEM``: > > - - The start address (``addr``) is not allocated. > > - - The end address (``addr`` + ``len``) is not allocated. > > - - A gap (unallocated memory) between start and end address. > > - > > -- ``-EPERM``: > > - - sealing is supported only on 64-bit CPUs, 32-bit is not supported. > > - > > -- For above error cases, users can expect the given memory range is > > - unmodified, i.e. no partial update. > > - > > -- There might be other internal errors/cases not listed here, e.g. > > - error during merging/splitting VMAs, or the process reaching the max > > - number of supported VMAs. In those cases, partial updates to the given > > - memory range could happen. However, those cases should be rare. > > - > > -**Blocked operations after sealing**: > > - Unmapping, moving to another location, and shrinking the size, > > - via munmap() and mremap(), can leave an empty space, therefore > > - can be replaced with a VMA with a new set of attributes. > > - > > - Moving or expanding a different VMA into the current location, > > - via mremap(). > > - > > - Modifying a VMA via mmap(MAP_FIXED). > > - > > - Size expansion, via mremap(), does not appear to pose any > > - specific risks to sealed VMAs. It is included anyway because > > - the use case is unclear. In any case, users can rely on > > - merging to expand a sealed VMA. > > - > > - mprotect() and pkey_mprotect(). > > - > > - Some destructive madvice() behaviors (e.g. MADV_DONTNEED) > > - for anonymous memory, when users don't have write permission to the > > - memory. Those behaviors can alter region contents by discarding pages, > > - effectively a memset(0) for anonymous memory. > > - > > - Kernel will return -EPERM for blocked operations. > > - > > - For blocked operations, one can expect the given address is unmodified, > > - i.e. no partial update. Note, this is different from existing mm > > - system call behaviors, where partial updates are made till an error is > > - found and returned to userspace. To give an example: > > - > > - Assume following code sequence: > > - > > - - ptr = mmap(null, 8192, PROT_NONE); > > - - munmap(ptr + 4096, 4096); > > - - ret1 = mprotect(ptr, 8192, PROT_READ); > > - - mseal(ptr, 4096); > > - - ret2 = mprotect(ptr, 8192, PROT_NONE); > > - > > - ret1 will be -ENOMEM, the page from ptr is updated to PROT_READ. > > - > > - ret2 will be -EPERM, the page remains to be PROT_READ. > > - > > -**Note**: > > - > > -- mseal() only works on 64-bit CPUs, not 32-bit CPU. > > - > > -- users can call mseal() multiple times, mseal() on an already sealed memory > > - is a no-action (not error). > > - > > -- munseal() is not supported. > > +SYSCALL > > +======= > > +mseal syscall signature > > +----------------------- > > + **int** mseal(**void \*** addr, **size_t** len, **unsigned long** flags) > > ugh. totally unreadable for people who just look at .rst files. > Right! The ** (bold) in rst collides with a pointer in c. it does make it unreadable. I will remove them. > Does other documentation go to this extreme? > > > + > > + **addr**/**len**: virtual memory address range. > > + The address range set by **addr**/**len** must meet: > > + - The start address must be in an allocated VMA. > > + - The start address must be page aligned. > > + - The end address (**addr** + **len**) must be in an allocated VMA. > > + - no gap (unallocated memory) between start and end address. > > + > > + The ``len`` will be paged aligned implicitly by the kernel. > > + > > + **flags**: reserved for future use. > > + > > + **Return values**: > > + - **0**: Success. > > + - **-EINVAL**: > > + * Invalid input ``flags``. > > + * The start address (``addr``) is not page aligned. > > + * Address range (``addr`` + ``len``) overflow. > > + - **-ENOMEM**: > > + * The start address (``addr``) is not allocated. > > + * The end address (``addr`` + ``len``) is not allocated. > > + * A gap (unallocated memory) between start and end address. > > + - **-EPERM**: > > + * sealing is supported only on 64-bit CPUs, 32-bit is not supported. > > + > > + **Note about error return**: > > + - For above error cases, users can expect the given memory range is > > + unmodified, i.e. no partial update. > > + - There might be other internal errors/cases not listed here, e.g. > > + error during merging/splitting VMAs, or the process reaching the max > > + number of supported VMAs. In those cases, partial updates to the given > > + memory range could happen. However, those cases should be rare. > > + > > + **Architecture support**: > > + mseal only works on 64-bit CPUs, not 32-bit CPU. > > CPUs. > Done, thanks. > > + > > + **Idempotent**: > > + users can call mseal multiple times, mseal on an already sealed memory > > times. mseal > Done, thanks. > > + is a no-action (not error). > > + > > + **no munseal** > > + Once mapping is sealed, it can't be unsealed. kernel should never > > + have munseal, this is consistent with other sealing feature, e.g. > > + F_SEAL_SEAL for file. > > + > > +Blocked mm syscall for sealed mapping > > +------------------------------------- > > + It might be imporant to note: **once the mapping is sealed, it will > > important > Done, thanks. > > + stay in the process's memory till the process terminates**. > > + > > + Example:: > > + > > + *ptr = mmap(0, 4096, PROT_READ, MAP_ANONYMOUS | MAP_PRIVATE, 0, 0); > > + rc = mseal(ptr, 4096, 0); > > + /* munmap will fail */ > > + rc = munmap(ptr, 4096); > > + assert(rc < 0); > > + > > + Blocked mm syscall: > > + - munmap > > + - mmap > > + - mremap > > + - mprotect and pkey_mprotect > > + - some destructive madvise behaviors: MADV_DONTNEED, MADV_FREE, > > + MADV_DONTNEED_LOCKED, MADV_FREE, MADV_DONTFORK, MADV_WIPEONFORK > > + > > + The first set of syscall to block is munmap, mremap, mmap. They can > > + either leave an empty space in the address space, therefore allow > > + replacement with a new mapping with new set of attributes, or can > > + overwrite the existing mapping with another mapping. > > + > > + mprotect and pkey_mprotect are blocked because they changes the > > + protection bits (rwx) of the mapping. > > preferably (RWX) > Done. > > + > > + Some destructive madvice behaviors (MADV_DONTNEED, MADV_FREE, > > madvise > Done. > > + MADV_DONTNEED_LOCKED, MADV_FREE, MADV_DONTFORK, MADV_WIPEONFORK) > > + for anonymous memory, when users don't have write permission to the > > + memory. Those behaviors can alter region contents by discarding pages, > > + effectively a memset(0) for anonymous memory. > > + > > + Kernel will return -EPERM for blocked syscalls. > > + > > + When blocked syscall return -EPERM due to sealing, the memory regions may or may not be changed, depends on the syscall being blocked: > > + - munmap: munmap is atomic. If one of VMAs in the given range is > > + sealed, none of VMAs are updated. > > + - mprotect, pkey_mprotect, madvise: partial update might happen, e.g. > > + when mprotect over multiple VMAs, mprotect might update the beginning > > + VMAs before reaching the sealed VMA and return -EPERM. > > + - mmap and mremap: undefined behavior. > > > > Use cases: > > ========== > > - glibc: > > The dynamic linker, during loading ELF executables, can apply sealing to > > - non-writable memory segments. > > + mapping segments. > > > > - Chrome browser: protect some security sensitive data-structures. > > data structures. > Done. Thanks. > > > > -Notes on which memory to seal: > > -============================== > > - > > -It might be important to note that sealing changes the lifetime of a mapping, > > -i.e. the sealed mapping won’t be unmapped till the process terminates or the > > -exec system call is invoked. Applications can apply sealing to any virtual > > -memory region from userspace, but it is crucial to thoroughly analyze the > > -mapping's lifetime prior to apply the sealing. > > +Don't use mseal on: > > Drop the ':'. Headings should not uses trailing colons. > (throughout this file) > Done everywhere. Thanks. > Maybe change the heading to "When not to use mseal". > > > +=================== > > +Applications can apply sealing to any virtual memory region from userspace, > > +but it is *crucial to thoroughly analyze the mapping's lifetime* prior to > > +apply the sealing. This is because the sealed mapping *won’t be unmapped* > > +till the process terminates or the exec system call is invoked. > > s/till/until/ preferably. > ok. > > > > For example: > > + - aio/shm > > + aio/shm can call mmap and munmap on behalf of userspace, e.g. > > + ksys_shmdt() in shm.c. The lifetime of those mapping are not tied to > > + the lifetime of the process. If those memories are sealed from userspace, > > + then munmap will fail, causing leaks in VMA address space during the > > + lifetime of the process. > > + > > + - ptr allocated by malloc (heap) > > + Don't use mseal on the memory ptr return from malloc(). > > + malloc() is implemented by allocator, e.g. by glibc. Heap manager might > > + allocate a ptr from brk or mapping created by mmap. > > + If app calls mseal on ptr returned from malloc(), this can affect the heap > > If an app calls mseal on a ptr > Done. thanks > > > + manager's ability to manage the mappings, the outcome is non-deterministic. > > mappings; the outcome > Done, thanks. > > + Example:: > > + > > + ptr = malloc(size); > > + /* don't call mseal on ptr return from malloc. */ > > + mseal(ptr, size); > > + /* free will success, allocator can't shrink heap lower than ptr */ > > + free(ptr); > > + > > +mseal doesn't block: > > +==================== > > +In a nutshell, mseal blocks certain mm syscall from modifying some of VMA's > > +attributes, such as protection bits (rwx). Sealed mappings doesn't mean the > > preferably (RWX). > Done, thanks. > > +memory is immutable. > > > > -- aio/shm > > - > > - aio/shm can call mmap()/munmap() on behalf of userspace, e.g. ksys_shmdt() in > > - shm.c. The lifetime of those mapping are not tied to the lifetime of the > > lifetimes OK. > ? > > > - process. If those memories are sealed from userspace, then munmap() will fail, > > - causing leaks in VMA address space during the lifetime of the process. > > - > > -- Brk (heap) > > - > > - Currently, userspace applications can seal parts of the heap by calling > > - malloc() and mseal(). > > - let's assume following calls from user space: > > - > > - - ptr = malloc(size); > > - - mprotect(ptr, size, RO); > > - - mseal(ptr, size); > > - - free(ptr); > > - > > - Technically, before mseal() is added, the user can change the protection of > > - the heap by calling mprotect(RO). As long as the user changes the protection > > - back to RW before free(), the memory range can be reused. > > - > > - Adding mseal() into the picture, however, the heap is then sealed partially, > > - the user can still free it, but the memory remains to be RO. If the address > > - is re-used by the heap manager for another malloc, the process might crash > > - soon after. Therefore, it is important not to apply sealing to any memory > > - that might get recycled. > > - > > - Furthermore, even if the application never calls the free() for the ptr, > > - the heap manager may invoke the brk system call to shrink the size of the > > - heap. In the kernel, the brk-shrink will call munmap(). Consequently, > > - depending on the location of the ptr, the outcome of brk-shrink is > > - nondeterministic. > > - > > - > > -Additional notes: > > -================= > > As Jann Horn pointed out in [3], there are still a few ways to write > > -to RO memory, which is, in a way, by design. Those cases are not covered > > -by mseal(). If applications want to block such cases, sandbox tools (such as > > -seccomp, LSM, etc) might be considered. > > +to RO memory, which is, in a way, by design. And those could be blocked > > +by different security measures. > > > > Those cases are: > > - > > -- Write to read-only memory through /proc/self/mem interface. > > -- Write to read-only memory through ptrace (such as PTRACE_POKETEXT). > > -- userfaultfd. > > + - Write to read-only memory through /proc/self/mem interface (FOLL_FORCE). > > + - Write to read-only memory through ptrace (such as PTRACE_POKETEXT). > > + - userfaultfd. > > > > The idea that inspired this patch comes from Stephen Röttger’s work in V8 > > CFI [4]. Chrome browser in ChromeOS will be the first user of this API. > > > > Reference: > > ========== > > -[1] https://github.com/apple-oss-distributions/xnu/blob/1031c584a5e37aff177559b9f69dbd3c8c3fd30a/osfmk/mach/vm_statistics.h#L274 > > - > > -[2] https://man.openbsd.org/mimmutable.2 > > - > > -[3] https://lore.kernel.org/lkml/CAG48ez3ShUYey+ZAFsU2i1RpQn0a5eOs2hzQ426FkcgnfUGLvA@xxxxxxxxxxxxxx > > - > > -[4] https://docs.google.com/document/d/1O2jwK4dxI3nRcOJuPYkonhTkNQfbmwdvxQMyXgeaRHo/edit#heading=h.bvaojj9fu6hc > > +- [1] https://github.com/apple-oss-distributions/xnu/blob/1031c584a5e37aff177559b9f69dbd3c8c3fd30a/osfmk/mach/vm_statistics.h#L274 > > +- [2] https://man.openbsd.org/mimmutable.2 > > +- [3] https://lore.kernel.org/lkml/CAG48ez3ShUYey+ZAFsU2i1RpQn0a5eOs2hzQ426FkcgnfUGLvA@xxxxxxxxxxxxxx > > +- [4] https://docs.google.com/document/d/1O2jwK4dxI3nRcOJuPYkonhTkNQfbmwdvxQMyXgeaRHo/edit#heading=h.bvaojj9fu6hc I will update and send v2. Appreciate the review comments. -Jeff
