Hi Gregory-
On 4/10/23 20:48, Gregory Price wrote:
On Mon, Apr 10, 2023 at 07:56:01PM -0500, Dragan Stancevic wrote:
Hi Gregory-
On 4/7/23 19:05, Gregory Price wrote:
3. This is changing the semantics of migration from a virtual memory
movement to a physical memory movement. Typically you would expect
the RDMA process for live migration to work something like...
a) migration request arrives
b) source host informs destination host of size requirements
c) destination host allocations memory and passes a Virtual Address
back to source host
d) source host initates an RDMA from HostA-VA to HostB-VA
e) CPU task is migrated
Importantly, the allocation of memory by Host B handles the important
step of creating HVA->HPA mappings, and the Extended/Nested Page
Tables can simply be flushed and re-created after the VM is fully
migrated.
to long didn't read: live migration is a virtual address operation,
and node-migration is a PHYSICAL address operation, the virtual
addresses remain the same.
This is problematic, as it's changing the underlying semantics of the
migration operation.
Those are all valid points, but what if you don't need to recreate HVA->HPA
mappings? If I am understanding the CXL 3.0 spec correctly, then both
virtual addresses and physical addresses wouldn't have to change. Because
the fabric "virtualizes" host physical addresses and the translation is done
by the G-FAM/GFD that has the capability to translate multi-host HPAs to
it's internal DPAs. So if you have two hypervisors seeing device physical
address as the same physical address, that might work?
Hm. I hadn't considered the device side translation (decoders), though
that's obviously a tool in the toolbox. You still have to know how to
slide ranges of data (which you mention below).
Hmm, do you have any quick thoughts on that?
The reference in this case is... the page tables. You need to know how
to interpret the data in the CXL memory region on the remote host, and
that's a "relative page table translation" (to coin a phrase? I'm not
sure how to best describe it).
right, coining phrases... I have been thinking of a "super-page" (for the
lack of a better word) a metadata region sitting on the switched CXL.mem
device that would allow hypervisors to synchronize on various aspects, such
as "relative page table translation", host is up, host is down, list of
peers, who owns what etc... In a perfect scenario, I would love to see the
hypervisors cooperating on switched CXL.mem device the same way cpus on
different numa nodes cooperate on memory in a single hypervisor. If either
host can allocate and schedule from this space then "NIL" aspect of
migration is "free".
The core of the problem is still that each of the hosts has to agree on
the location (physically) of this region of memory, which could be
problematic unless you have very strong BIOS and/or kernel driver
controls to ensure certain devices are guaranteed to be mapped into
certain spots in the CFMW.
Right, true. The way I am thinking of it is that this would be a part of
data-center ops setup which at first pass would be a somewhat of a
manual setup same way as other pre-OS related setup. But later on down
the road perhaps this could be automated, either through some pre-agreed
auto-ranges detection or similar, it's not unusual for dc ops to name
hypervisors depending of where in dc/rack/etc they sit etc..
After that it's a matter of treating this memory as incoherent shared
memory and handling ownership in a safe way. If the memory is only used
for migrations, then you don't have to worry about performance.
So I agree, as long as shared memory mapped into the same CFMW area is
used, this mechanism is totally sound.
My main concerns are that I don't know of a mechanism to ensure that. I
suppose for those interested, and with special BIOS/EFI, you could do
that - but I think that's going to be a tall ask in a heterogenous cloud
environment.
Yeah, I get that. But in my experience even heterogeneous setups have
some level of homogeneity, weather it's per rack, or per pod. As old
things are sunset and new things are brought in, it gives you these
segments of homogeneity with more or less advanced features. So at the
end of the day, if someone wants a feature X they will need to
understand the feature requirements or limitations. I feel like I deal
with hardware/feature fragmentation all the time, but doesn't preclude
bringing newer things in. You just have to plant it appropriately.
That's... complicated to say the least.
<... snip ...>
An Option: Make pages physically contiguous on migration to CXL
In this case, you don't necessarily care about the Host Virtual
Addresses, what you actually care about are the structure of the pages
in memory (are they physically contiguous? or do you need to
reconstruct the contiguity by inspecting the page tables?).
If a migration API were capable of reserving large swaths of contiguous
CXL memory, you could discard individual page information and instead
send page range information, reconstructing the virtual-physical
mappings this way.
yeah, good points, but this is all tricky though... it seems this would
require quiescing the VM and that is something I would like to avoid if
possible. I'd like to see the VM still executing while all of it's pages are
migrated onto CXL NUMA on the source hypervisor. And I would like to see the
VM executing on the destination hypervisor while migrate_pages is moving
pages off of CXL. Of course, what you are describing above would still be a
very fast VM migration, but would require quiescing.
Possibly. If you're going to quiesce you're probably better off just
snapshotting to shared memory and migrating the snapshot.
That is exactly my thought too.
Maybe that's the better option for a first-pass migration mechanism. I
don't know.
I definitely see your point, "canning" and "re-hydration" approach as a
first-pass. I'd be happy with even just a "Hello World" page migration
as a first pass :)
Anyway, would love to attend this session.
~Gregory
--
--
Peace can only come as a natural consequence
of universal enlightenment -Dr. Nikola Tesla
