On 9/20/21 3:17 PM, Kai Huang wrote: >> +The ioctl will often not able to remove SECS pages, in case their child >> +pages have not gone through ``EREMOVE`` yet; therefore, the ioctl returns the >> +number of pages that failed to be removed. ``SGX_IOC_VEPC_REMOVE_ALL`` should >> +first be invoked on all the ``/dev/sgx_vepc`` file descriptors mapped >> +into the guest; a second call to the ioctl will be able to remove all >> +leftover pages and will return 0. Any other return value on the second call >> +would be a symptom of a bug in either Linux or the userspace client. > Maybe also worth to mention userspace should guarantee there's no vcpu running > inside guest enclave when resetting guest's virtual EPC. Why, specifically? Is it because EREMOVE will also fail if there is a CPU running in the enclave? Maybe we should say something like: The ioctl() returns the number of pages where removal failed. Callers of the ioctl() need to handle two sources of failure: 1) Page removal will always fail when any thread is running in the enclave to which the page belongs. ioctl() users can avoid these failures by preventing execution of any vcpu which maps the virtual EPC. 2) Page removal will also fail for SGX "SECS" metadata pages which still have child pages. SECS pages can be removed by executing ``SGX_IOC_VEPC_REMOVE_ALL`` on all ``/dev/sgx_vepc`` file descriptors mapped into the guest. Yhis means that the ioctl() must be called twice: an initial set of calls to remove child pages and a subsequent set of calls to remove SECS pages. It indicates a bug in the kernel or the userspace client if any of the second round of ``SGX_IOC_VEPC_REMOVE_ALL`` calls fails (has a return code other than 0).
