++adding @ceph-users-confirm+4555fdc6282a38c849f4d27a40339f1b7e4bde74@xxxxxxx <ceph-users-confirm+4555fdc6282a38c849f4d27a40339f1b7e4bde74@xxxxxxx> ++Adding dev@xxxxxxx Thanks,&, Regards Arihant Jain On Mon, 27 Nov, 2023, 7:48 am AJ_ sunny, <jains8550@xxxxxxxxx> wrote: > Hi team, > > After doing above changes I am still getting the issue in which machine > continuously went shutdown > > In nova-compute logs I am getting only this footprint > > Logs:- > 2023-10-16 08:48:10.971 7 WARNING nova.compute.manager > [req-c7b731db-2b61-400e-917f-8645c9984696 f226d81a45dd46488fb2e19515 848 > 316d215042914de190f5f9e1c8466bf0 default default] [instance: > 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] Received unexpected - vent > network-vif-plugged-f191f6c8-dff5-4c1b-94b3-8d91aa6ff5ac for instance with > vm_state active and task_state None. 2023-10-21 22:42:44.589 7 INFO > nova.compute.manager [-] [instance: 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] > VM Stopped (Lifecyc Event) > > 2023-10-21 22:42:44.683 7 INFO nova.compute.manager > [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] [instance: 4b04d3f1- > fbd-4b63-b693-a0ef316ecff3] During _sync_instance_power_state the DB > power_state (1) does not match the vm_power_state from ti e hypervisor (4). > Updating power_state in the DB to match the hypervisor. > > 2023-10-21 22:42:44.811 7 WARNING nova.compute.manager > [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d ----] [instance: 4b04d3f > 1-1fbd-4b63-b693-a0ef316ecff3] Instance shutdown by itself. Calling the > stop API. Current vm_state: active, current task_state : None, original DB > power_state: 1, current VM power_state: 4 2023-10-21 22:42:44.977 7 INFO > nova.compute.manager [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] > [instance: 4b04d3f1-1 > > fbd-4b63-b693-a0ef316ecff3] Instance is already powered off in the > hypervisor when stop is called. > > > And in this architecture we are using ceph is the backend storage for > Nova,glance & cinder > When machine auto goes down and if i try to start the machine it will go > in error i.e. in Vm console is show I/O ERROR during boot so first we need > to rebuild the volume from ceph side then I have to start the machine > Rbd object-map rebuild<volume-id> > Openstack server start <server-id> > > So this issue is showing two faces one from ceph side and another from > nova-compute log > can someone please help me out to fix out this issue asap > > Thanks & Regards > Arihant Jain > > On Tue, 24 Oct, 2023, 4:56 pm , <smooney@xxxxxxxxxx> wrote: > >> On Tue, 2023-10-24 at 10:11 +0530, AJ_ sunny wrote: >> > Hi team, >> > >> > Vm is not shutting off by owner from inside its automatically went to >> > shutdown i.e. libvirt lifecycle stop event triggering >> > In my nova.conf configuration I am using ram_allocation_ratio = 1.5 >> > And previously I tried to set in nova.conf >> > Sync_power_state_interval = -1 but still facing the same problem >> > OOM might be causing this issue >> > Can you please give me some idea to fix this issue if OOM is the cause >> the general answer is swap. >> >> nova should alwasy be deployed with swap even if you do not have over >> commit enabled. >> there are a few reason for this the first being python allocates memory >> diffently if >> any swap is aviable, even 1G is enough to have it not try to commit all >> memory. so >> when swap is aviable the nova/neutron agents will use much less resident >> memeory even with >> out usign any of the swap space. >> >> we have some docs about this downstream >> >> https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.1/html/configuring_the_compute_service_for_instance_creation/assembly_configuring-the-compute-service_osp#ref_calculating-swap-size_configuring-compute >> >> if you are being ultra conservative we recommend allocating (ram * >> allocation ratio) in swap so in your case allcoate >> 1.5 times your ram as swap. we woudl expect the actul useage of swap to >> be a small fraction of that however so we >> also provide a formula for >> >> overcommit_ratio = NovaRAMAllocationRatio - 1 >> Minimum swap size (MB) = (total_RAM * overcommit_ratio) + >> RHEL_min_swap >> Recommended swap size (MB) = total_RAM * (overcommit_ratio + >> percentage_of_RAM_to_use_for_swap) >> >> so say your host had 64G of ram with an allocation ratio of 1.5 and a min >> swap percentaiong of 25% >> the conserviver swap recommentation would be >> >> (64*(0.5+0.25)) + disto_min_swap >> (64*0.75) + 4G = 52G of recommended swap >> >> if your wondering why we add a min swap precentage and disto min swap its >> basically to acocund for the >> Qemu and host OS memory overhead as well as the memory used by the >> nova/neutron agents and libvirt/ovs >> >> >> if your not using memory over commit my general recommdation is if you >> have less then 64G of ram allcoate 16G if you >> have more then 256G of ram allocate 64G and you should be fine. when you >> do use memofy over commit you must >> have at least enouch swap to account for the qemu overhead of all >> instance + the over committed memory. >> >> >> the other common cause of OOM errors is if you are using numa affinity >> and the guest dont request >> hw:mem_page_size=<something> without setting a mem_page_size request we >> dont do numa aware memory placement. the kernel >> OOM system works >> on a per numa node basis, numa affintiy does not supprot memory over >> commit either so that is likly not your issue. >> i jsut said i woudl mention it to cover all basis. >> >> regards >> sean >> >> >> >> > >> > >> > Thanks & Regards >> > Arihant Jain >> > >> > On Mon, 23 Oct, 2023, 11:29 pm , <smooney@xxxxxxxxxx> wrote: >> > >> > > On Mon, 2023-10-23 at 13:19 -0400, Jonathan Proulx wrote: >> > > > >> > > > I've seen similar log traces with overcommitted memory when the >> > > > hypervisor runs out of physical memory and OOM killer gets the VM >> > > > process. >> > > > >> > > > This is an unusuall configuration (I think) but if the VM owner >> claims >> > > > they didn't power down the VM internally you might look at the local >> > > > hypevisor logs to see if the VM process crashed or was killed for >> some >> > > > other reason. >> > > yep OOM events are one common causes fo this. >> > > >> > > nova is bacialy just saying "hay you said this vm should be active >> but its >> > > not, im going to update the db to reflect >> > > reality." you can turn that off with >> > > >> > > >> https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.handle_virt_lifecycle_events >> > > or >> > > >> > > >> https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.sync_power_state_interval >> > > either disabel the sync via setign the interval to -1 >> > > or disable haneling the virt lifecycle events. >> > > >> > > i would recommend the sync_power_state_interval approach but again if >> vms >> > > are stopping >> > > and you dont know why you likely should discover why rahter then just >> > > turning if the update of the nova db to reflect >> > > the actual sate. >> > > >> > > > >> > > > -Jon >> > > > >> > > > On Mon, Oct 23, 2023 at 02:02:26PM +0100, smooney@xxxxxxxxxx wrote: >> > > > :On Mon, 2023-10-23 at 17:45 +0530, AJ_ sunny wrote: >> > > > :> Hi team, >> > > > :> >> > > > :> I am using openstack kolla ansible on wallaby version and >> currently I >> > > am >> > > > :> facing issue with virtual machine, vm is shutoff by itself and >> and >> > > from log >> > > > :> it seems libvirt lifecycle stop event triggering again and again >> > > > :> >> > > > :> Logs:- >> > > > :> 2023-10-16 08:48:10.971 7 WARNING nova.compute.manager >> > > > :> [req-c7b731db-2b61-400e-917f-8645c9984696 >> f226d81a45dd46488fb2e19515 >> > > 848 >> > > > :> 316d215042914de190f5f9e1c8466bf0 default default] [instance: >> > > > :> 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] Received unexpected - vent >> > > > :> network-vif-plugged-f191f6c8-dff5-4c1b-94b3-8d91aa6ff5ac for >> instance >> > > with >> > > > :> vm_state active and task_state None. 2023-10-21 22:42:44.589 7 >> INFO >> > > > :> nova.compute.manager [-] [instance: >> > > 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] >> > > > :> VM Stopped (Lifecyc Event) >> > > > :> >> > > > :> 2023-10-21 22:42:44.683 7 INFO nova.compute.manager >> > > > :> [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] [instance: 4b04d3f1- >> > > > :> fbd-4b63-b693-a0ef316ecff3] During _sync_instance_power_state >> the DB >> > > > :> power_state (1) does not match the vm_power_state from ti e >> > > hypervisor (4). >> > > > :> Updating power_state in the DB to match the hypervisor. >> > > > :> >> > > > :> 2023-10-21 22:42:44.811 7 WARNING nova.compute.manager >> > > > :> [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d ----] [instance: >> 4b04d3f >> > > > :> 1-1fbd-4b63-b693-a0ef316ecff3] Instance shutdown by itself. >> Calling >> > > the >> > > > :> stop API. Current vm_state: active, current task_state : None, >> > > original DB >> > > > :> power_state: 1, current VM power_state: 4 2023-10-21 >> 22:42:44.977 7 >> > > INFO >> > > > :> nova.compute.manager [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] >> > > > :> [instance: 4b04d3f1-1 >> > > > :> >> > > > :> fbd-4b63-b693-a0ef316ecff3] Instance is already powered off in >> the >> > > > :> hypervisor when stop is called. >> > > > : >> > > > :that sounds like the guest os shutdown the vm. >> > > > :i.e. somethign in the guest ran sudo poweroff >> > > > :then nova detected teh vm was stoped by the user and updated its >> db to >> > > match >> > > > :that. >> > > > : >> > > > :that is the expected beahvior wehn you have the power sync enabled. >> > > > :it is enabled by default. >> > > > :> >> > > > :> >> > > > :> Thanks & Regards >> > > > :> Arihant Jain >> > > > :> +91 8299719369 >> > > > : >> > > > >> > > >> > > >> >> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |