On 15.10.20 10:28, Wei Yang wrote:
> On Mon, Oct 12, 2020 at 02:52:59PM +0200, David Hildenbrand wrote:
>> Let's check by traversing busy system RAM resources instead, to avoid
>> relying on memory block states.
>>
>> Don't use walk_system_ram_range(), as that works on pages and we want to
>> use the bare addresses we have easily at hand.
>>
>> Cc: "Michael S. Tsirkin" <mst@xxxxxxxxxx>
>> Cc: Jason Wang <jasowang@xxxxxxxxxx>
>> Cc: Pankaj Gupta <pankaj.gupta.linux@xxxxxxxxx>
>> Signed-off-by: David Hildenbrand <david@xxxxxxxxxx>
>> ---
>> drivers/virtio/virtio_mem.c | 19 +++++++++++++++----
>> 1 file changed, 15 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
>> index b3eebac7191f..6bbd1cfd10d3 100644
>> --- a/drivers/virtio/virtio_mem.c
>> +++ b/drivers/virtio/virtio_mem.c
>> @@ -1749,6 +1749,20 @@ static void virtio_mem_delete_resource(struct virtio_mem *vm)
>> vm->parent_resource = NULL;
>> }
>>
>> +static int virtio_mem_range_has_system_ram(struct resource *res, void *arg)
>> +{
>> + return 1;
>> +}
>> +
>> +static bool virtio_mem_has_memory_added(struct virtio_mem *vm)
>> +{
>> + const unsigned long flags = IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY;
>> +
>> + return walk_iomem_res_desc(IORES_DESC_NONE, flags, vm->addr,
>> + vm->addr + vm->region_size, NULL,
>> + virtio_mem_range_has_system_ram) == 1;
>> +}
>> +
>> static int virtio_mem_probe(struct virtio_device *vdev)
>> {
>> struct virtio_mem *vm;
>> @@ -1870,10 +1884,7 @@ static void virtio_mem_remove(struct virtio_device *vdev)
>> * the system. And there is no way to stop the driver/device from going
>> * away. Warn at least.
>> */
>> - if (vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE] ||
>> - vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL] ||
>> - vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE] ||
>> - vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL]) {
>> + if (virtio_mem_has_memory_added(vm)) {
>
> I am not sure this would be more efficient.
In general, no. However, this is a preparation for Big Block Mode, which
won't have memory block states.
(this path only triggers when unloading the driver - which most probably
only ever happens during my testing ... :) and we don't really care
about performance there)
>
>> dev_warn(&vdev->dev, "device still has system memory added\n");
>> } else {
>> virtio_mem_delete_resource(vm);
>
> BTW, I got one question during review.
>
> Per my understanding, there are 4 states of a virtio memory block
>
> * OFFLINE[_PARTIAL]
> * ONLINE[_PARTIAL]
>
> While, if my understanding is correct, those two offline states are transient.
> If the required range is onlined, the state would be change to
> ONLINE[_PARTIAL] respectively. If it is not, the state is reverted to UNUSED
> or PLUGGED.
Very right.
>
> What I am lost is why you do virtio_mem_mb_remove() on OFFLINE_PARTIAL memory
> block? Since we wait for the workqueue finish its job.
That's an interesting corner case. Assume you have a 128MB memory block
but only 64MB are plugged.
As long as we have our online_pages callback in place, we can hinder the
unplugged 64MB from getting exposed to the buddy
(virtio_mem_online_page_cb()). However, once we unloaded the driver,
this is no longer the case. If someone would online that memory block,
we would expose unplugged memory to the buddy - very bad.
So we have to remove these partially plugged, offline memory blocks when
losing control over them.
I tried to document that via:
"After we unregistered our callbacks, user space can online partially
plugged offline blocks. Make sure to remove them."
>
> Also, during virtio_mem_remove(), we just handle OFFLINE_PARTIAL memory block.
> How about memory block in other states? It is not necessary to remove
> ONLINE[_PARTIAL] memroy blocks?
Blocks that are fully plugged (ONLINE or OFFLINE) can get
onlined/offlined without us having to care. Works fine - we only have to
care about partially plugged blocks.
While we *could* unplug OFFLINE blocks, there is no way we can
deterministically offline+remove ONLINE blocks. So that memory has to
stay, even after we unloaded the driver (similar to the dax/kmem driver).
ONLINE_PARTIAL is already taken care of: it cannot get offlined anymore,
as we still hold references to these struct pages
(virtio_mem_set_fake_offline()), and as we no longer have the memory
notifier in place, we can no longer agree to offline this memory (when
going_offline).
I tried to document that via
"After we unregistered our callbacks, user space can no longer offline
partially plugged online memory blocks. No need to worry about them."
>
> Thanks in advance, since I may missed some concepts.
(force) driver unloading is a complicated corner case.
Thanks!
--
Thanks,
David / dhildenb
_______________________________________________
Virtualization mailing list
Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
