On Tue, Feb 23, 2021 at 11:14:10AM -0800, Dave Hansen wrote:
> On 2/21/21 4:54 PM, Dave Hansen wrote:
> > Instead of having a for-each-section loop, I'd make it for-each-node ->
> > for-each-section. Something like:
> >
> > for (i = 0; i < num_possible_nodes(); i++) {
> > node = (numa_node_id() + i) % num_possible_nodes()
> >
> > if (!node_isset(nid, sgx_numa_mask))
> > continue;
> >
> > list_for_each_entry(section, &sgx_numa_nodes[nid],
> > section_list) {
> > __sgx_alloc_epc_page_from_section(section)
> > }
> > }
>
> OK, here's an almost completely fleshed-out loop:
>
> page = NULL;
> node = numa_node_id();
> start_node = node;
> while (1) {
> list_for_each_entry(section, &sgx_numa_nodes[nid],
> section_list) {
> page = __sgx_alloc_epc(section);
> if (page)
> break;
> }
> if (page)
> break;
>
> /*
> * EPC allocation failed on 'node'. Fall
> * back with round-robin to other nodes with
> * EPC:
> */
> node = next_node_in(node, sgx_numa_mask);
>
> /* Give up if allocation wraps back to the start: */
> if (node == start_node)
> break;
> }
>
> This will:
> 1. Always start close to the CPU that started the allocation
> 2. Always spread the allocations out among nodes evenly, never
> concentrating allocations on node 0, for instance. (This could also
> be node_random() and get a similar effect, but this probably has
> slightly better default NUMA behavior).
> 3. Efficiently look among all nodes because of 'sgx_numa_mask'
> 4. Have no special case for the first allocation. All allocations will
> be satisfied from this unified loop.
> 5. Compile down to no loop on CONFIG_NUMA=y systems.
> 6. Be guaranteed to make forward progress even if preempted and
> numa_node_id() changes in the loop.
>
> BTW, I think the name of __sgx_alloc_epc_page_from_section() can be
> shortened down. It's passed a section and returns a page, so both of
> those could be removed from the name.
I would start with what I have with minimal changes, and then continue
with a patch that completely wipes the section list.
Then fallback can just take a page from a flat FIFO of EPC pages.
/Jarkko
