+ Mimi, Dmitry, Integrity, FYI On Tue, Sep 17, 2024 at 11:54:17AM GMT, Maxwell Bland wrote: > On Tue, Sep 17, 2024 at 12:34:28AM GMT, Kees Cook wrote: > > On Thu, Sep 12, 2024 at 04:02:53PM +0000, Maxwell Bland wrote: > > > operated on around 0.1188 MB). But most importantly, third, without some degree > > > of provenance, I have no way of telling if someone has injected malicious code > > > into the kernel, and unfortunately even knowing the correct bytes is still > > > "iffy", as in order to prevent JIT spray attacks, each of these filters is > > > offset by some random number of uint32_t's, making every 4-byte shift of the > > > filter a "valid" codepage to be loaded at runtime. > > > > So, let's start here. What I've seen from the thread is that there isn't > > a way to verify that a given JIT matches the cBPF. Is validating the > > cBPF itself also needed? > > Yes(ish) but mostly no. Current kernel exploits, from what I have seen > and what is readily available consist of three stages: > > - Find a UAF > - Bootstrap this UAF into an unconstrained read/write > - Modify some core kernel resource to get arbitrary execution. > > Example dating back to 2019: > https://googleprojectzero.blogspot.com/2019/11/bad-binder-android-in-wild-exploit.html > > An adversary could modify the loaded cBPF program prior to loading in > order to, say, change the range of syscall _NR_'s accepted by the > seccomp switch statement in order to stage their escape from Chrome's > sandbox. > > However, JIT presents a more general issue, hence the mostly no, since > and exploited native system service could target the JITed code page > in order to exploit the kernel, rather than requiring something to > be staged within the modified seccomp sandbox in the "cBPF itself" > example. > > For example, Motorola has a few system services for hardware and other > things (as well as QCOM), written in C, for example, our native dropbox > agent. Supposing there were an exploit for this agent allowing execution > within that service's context, an adversary could find a UAF, and target > the page of Chrome's JITed seccomp filter in order to exploit the full > kernel. That is, they are not worried about escaping the sandbox so much > as finding a writable resource from which they can gain privileges in > the rest of the kernel. > > Admitted, there are ~29,000 other writable data structures (in > msm-kernel) they could also target, but the JIT'ed seccomp filter is the > only code page they could modify (since it is not possible to get > compile-time provenance/signatures). The dilemma is that opposed to > modifying, say, the system_unbound_wq and adding an entry to it that > holds a pointer to call_usermodehelper_exec_work, you could add some > code to this page instead, making the kernel the same level of > exploitable. > > The goal at the end of the day is to fix this and then try to build a > system to lock down the rest of the data in a sensible way. Likely an > ARM-MTE like, EL2-maintained tag system conditioned on the kernel's > scheduler and memory allocation infrastructure. At least, that is what I > want to be working on, after I figure out this seccomp stuff. > > > - The IMA subsystem has wanted a way to measure (and validate) seccomp > > filters. We could get more details from them for defining this need > > more clearly. > > You are right. I have added Mimi, Dmitry, and the integrity list. Their > work with linked lists and other data structures is right in line with > these concerns. I do not know if they have looked at building verifiers > for JIT'ed cBPF pages already. > > > - The JIT needs to be verified against the cBPF that it was generated > > from. We currently do only a single pass and don't validate it once > > the region has been set read-only. We have a standing feature request > > for improving this: https://github.com/KSPP/linux/issues/154 > > > Kees, this is exactly what I'm talking about, you are awesome! > > I'll share the (pretty straightforward) EL2 logic for this, though not the > code, since licensing and all that, but this public mailing list should > hopefully serve as prior art for any questionable chipset vendor attempting to > patent public domain security for the everyday person: > > - Marking PTEs null is fine > - If a new PTE is allocated, mark it PXN atomically using the EL2 > permission fault failure triggered from the page table lockdown (see > GPL-2.0 kernel module below). > - If a PTE is updated and the PXN bit is switched from 1 to 0, SHA256 > the page, mark it immutable, and let it through if it is OK. > > This lets the page be mucked with during the whole JIT process, but ensures > that the second the page wants to be priv-executable, no further modifications > happen. To "unlock" the page for free-ing, one just needs to set the PXN bit > back. Then if we ever want to execute from it again, the process repeats, so > on. This relies on my prior main.c vmalloc maintenance and the below ptprotect > logic (note, WIP, no warranty on this code). > > > For solutions, I didn't see much discussion around the "orig_prog" > > copy of the cBPF. Under CHECKPOINT_RESTORE, the original cBPF remains > > associated with the JIT. struct seccomp_filter's struct bpf_prog prog's > > orig_prog member. If it has value outside of CHECKPOINT_RESTORE, then > > we could do it for those conditions too. > > Unfortunately the Android GKI does not support checkpoint restore and makes the > orig_prog reference fail (at least in the case I'm trying to work towards for > cell phones). > > I could lock the orig_prog as immutable during the JIT, and given the resulting > code page, and then attempt to reproduce the code page in EL2 from the original > cBPF, but that seems dangerous and potentially buggy as opposed to checking the > reference addresses in the final machine code against knowledge of struct > seccomp_data (what I am working on right now). > > Maxwell > > // SPDX-License-Identifier: GPL-2.0 > /* > * Copyright (C) 2023 Motorola Mobility, Inc. > * > * Authors: Maxwell Bland > * Binsheng "Sammy" Que > * > * This program is free software; you can redistribute it and/or modify > * it under the terms of the GNU General Public License version 2 as > * published by the Free Software Foundation. > * > * This program is distributed in the hope that it will be useful, > * but WITHOUT ANY WARRANTY; without even the implied warranty of > * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > * GNU General Public License for more details. > * > * Initializes hypervisor-level protections for the kernel pagetables. In > * coordination with the moto_org_mem driver, which restricts executable code > * pages to a well defined region in-between > * > * stext <-> module_alloc_base + SZ_2G > * > * It is able to mark all page tables not corresponding to this virtual address > * range PXNTable. Mark the table these descriptors exist within as immutable. > * For all tables/descriptors which are marked privileged executable, these are > * marked permanently immutable, and their modifications are tracked directly. > */ > #ifndef _PTPROTECT_H > #define _PTPROTECT_H > > #include <linux/delay.h> > #include <linux/highmem.h> > #include <linux/kprobes.h> > #include <linux/list.h> > #include <linux/mm_types.h> > #include <linux/module.h> > #include <linux/of.h> > #include <linux/of_platform.h> > #include <linux/pagewalk.h> > #include <linux/types.h> > #include <asm/pgalloc.h> > #include <asm/pgtable-hwdef.h> > #include <asm/pgtable.h> > #include <mm/pgalloc-track.h> > #include <trace/hooks/fault.h> > #include <trace/hooks/vendor_hooks.h> > #include <fs/erofs/compress.h> > > uint64_t stext_vaddr = 0; > uint64_t etext_vaddr = 0; > uint64_t module_alloc_base_vaddr = 0; > > uint64_t last_pmd_range[2] = { 0, 0 }; > uint64_t pmd_range_list[1024][2] = { 0 }; > int pmd_range_list_index = 0; > > /** > * add_to_pmd_range_list - adds a range to the pmd range list > * @start: Start of the range > * @end: End of the range > * > * Used to implement a naive set of adjacent pmd segments to > * speed up protection code as otherwise we will treat each > * pmd (there are a lot of them, as a separate region to protect) > */ > static void add_to_pmd_range_list(uint64_t start, uint64_t end) > { > pmd_range_list[pmd_range_list_index][0] = start; > pmd_range_list[pmd_range_list_index][1] = end; > pmd_range_list_index++; > } > > void lock_last_pmd_range(void) > { > if (last_pmd_range[0] == 0 || last_pmd_range[1] == 0) > return; > split_block(last_pmd_range[0]); > mark_range_ro_smc(last_pmd_range[0], last_pmd_range[1], > KERN_PROT_PAGE_TABLE); > msleep(10); > } > > /** > * prot_pmd_entry - protects a range pointed to by a pmd entry > * > * @pmd: Pointer to the pmd entry > * @addr: Virtual address of the pmd entry > */ > static void prot_pmd_entry(pmd_t *pmd, unsigned long addr) > { > uint64_t pgaddr = pmd_page_vaddr(*pmd); > uint64_t start_range = 0; > uint64_t end_range = 0; > > /* > * Just found that QCOM's gic_intr_routing.c kernel module is getting > * allocated at vaddr ffffffdb87f67000, but modules code region should > * only be allocated from ffffffdb8fc00000 to ffffffdc0fdfffff... > * > * It seems to be because arm64's module.h defines module_alloc_base as > * ((u64)_etext - MODULES_VSIZE) But this module_alloc_base preprocesor > * define should be redefined/randomized by kernel/kaslr.c, however, it > * appears that early init modules get allocated before > * module_alloc_base is relocated, so c'est la vie, and the efforts of > * kaslr.c are for naught (_etext's vaddr is randomized though, so it > * does not matter, I guess). > */ > uint64_t module_alloc_start = module_alloc_base_vaddr; > uint64_t module_alloc_end = module_alloc_base_vaddr + SZ_2G; > > if (!pmd_present(*pmd) || pmd_bad(*pmd) || pmd_none(*pmd) || > !pmd_val(*pmd)) > return; > > /* Round the starts and ends of each region to their boundary limits */ > // module_alloc_start -= (module_alloc_start % PMD_SIZE); > // module_alloc_end += PMD_SIZE - (module_alloc_end % PMD_SIZE) - 1; > > start_range = __virt_to_phys(pgaddr); > end_range = __virt_to_phys(pgaddr) + sizeof(pte_t) * PTRS_PER_PMD - 1; > > /* If the PMD potentially points to code, check it in the hypervisor */ > if (!pmd_leaf(*pmd) && > ((addr <= etext_vaddr && (addr + PMD_SIZE - 1) >= stext_vaddr) || > (addr <= module_alloc_end && > (addr + PMD_SIZE - 1) >= module_alloc_start))) { > if (start_range == last_pmd_range[1] + 1) { > last_pmd_range[1] = end_range; > } else if (end_range + 1 == last_pmd_range[0]) { > last_pmd_range[0] = start_range; > } else if (last_pmd_range[0] == 0 && last_pmd_range[1] == 0) { > last_pmd_range[0] = start_range; > last_pmd_range[1] = end_range; > } else { > add_to_pmd_range_list(last_pmd_range[0], > last_pmd_range[1]); > lock_last_pmd_range(); > last_pmd_range[0] = start_range; > last_pmd_range[1] = end_range; > } > /* If the PMD points to data only, mark it PXN, as the caller will > * mark the PMD immutable after this function returns */ > } else { > if (!pmd_leaf(*pmd)) { > set_pmd(pmd, __pmd(pmd_val(*pmd) | PMD_TABLE_PXN)); > } else { > /* TODO: if block, ensure range is marked immutable */ > pr_info("MotoRKP: pmd block at %llx\n", start_range); > } > } > } > > pgd_t *swapper_pg_dir_ind; > void (*set_swapper_pgd_ind)(pgd_t *pgdp, pgd_t pgd); > > static inline bool in_swapper_pgdir_ind(void *addr) > { > return ((unsigned long)addr & PAGE_MASK) == > ((unsigned long)swapper_pg_dir_ind & PAGE_MASK); > } > > static inline void set_pgd_ind(pgd_t *pgdp, pgd_t pgd) > { > if (in_swapper_pgdir_ind(pgdp)) { > set_swapper_pgd_ind(pgdp, __pgd(pgd_val(pgd))); > return; > } > > WRITE_ONCE(*pgdp, pgd); > dsb(ishst); > isb(); > } > > /** > * prot_pgd_entry - protects a range pointed to by a pgd entry > * @pgd: pgd struct with descriptor values > * @addr: vaddr of start of pgds referenced memory range > */ > static int prot_pgd_entry(pgd_t *pgd, unsigned long addr, unsigned long next, > struct mm_walk *walk) > { > uint64_t pgaddr = pgd_page_vaddr(*pgd); > uint64_t start_range = 0; > uint64_t end_range = 0; > uint64_t module_alloc_start = module_alloc_base_vaddr; > uint64_t module_alloc_end = module_alloc_base_vaddr + SZ_2G; > uint64_t i = 0; > pmd_t *subdescriptor = 0; > unsigned long subdescriptor_addr = addr; > > if (!pgd_present(*pgd) || pgd_bad(*pgd) || pgd_none(*pgd) || > !pgd_val(*pgd)) > return 0; > > /* Round the starts and ends of each region to their boundary limits */ > // module_alloc_start -= (module_alloc_start % PGDIR_SIZE); > // module_alloc_end += PGDIR_SIZE - (module_alloc_end % PGDIR_SIZE) - 1; > > if (!pgd_leaf(*pgd)) { > start_range = __virt_to_phys(pgaddr); > end_range = __virt_to_phys(pgaddr) + > sizeof(p4d_t) * PTRS_PER_PGD - 1; > > /* If the PGD contains addesses between stext_vaddr and etext_vaddr or > * module_alloc_base and module_alloc_base + SZ_2G, then do not mark it > * PXN */ > if ((addr <= etext_vaddr && > (addr + PGDIR_SIZE - 1) >= stext_vaddr) || > (addr <= module_alloc_end && > (addr + PGDIR_SIZE - 1) >= module_alloc_start)) { > /* Protect all second-level PMD entries */ > for (i = 0; i < PTRS_PER_PGD; i++) { > subdescriptor = > (pmd_t *)(pgaddr + i * sizeof(pmd_t)); > prot_pmd_entry(subdescriptor, > subdescriptor_addr); > subdescriptor_addr += PMD_SIZE; > } > lock_last_pmd_range(); > > split_block(start_range); > mark_range_ro_smc(start_range, end_range, > KERN_PROT_PAGE_TABLE); > } else { > /* Further modifications protected by immutability from hyp_rodata_end to __inittext_begin in kickoff */ > set_pgd_ind(pgd, __pgd(pgd_val(*pgd) | 1UL << 59)); > } > } else { > /* TODO: Handle block case at this level? */ > pr_info("MotoRKP: pgd block at %llx\n", start_range); > } > return 0; > } > > /* > * Locks down the ranges of memory pointed to by all PGDs as read-only. > * Current kernel configurations do not bother with p4ds or puds, and > * thus we do not need protections for these layers (pgd points directly > * to pmd). > */ > static const struct mm_walk_ops protect_pgds = { > .pgd_entry = prot_pgd_entry, > }; > > #endif /* _PTPROTECT_H */ >
