Re: qemu raw image file - qemu and grub2 can't find boot content from VM

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I updated to 7.9, rebooted everything, and it started working.

I will have QE try to break it again and report back. I couldn't break
it but they're better at breaking things (which is hard to imagine :)


On Fri, Jan 29, 2021 at 01:11:50PM -0600, Erik Jacobson wrote:
> Thank you.
> 
> We reproduced the problem after force-killing one of the 3 physical
> nodes 6 times in a row.
> 
> At that point, the grub2 loaded off the qemu virtual hard drive, but
> could not find partitions. Since there is random luck involved, we don't
> actually know if it was the force-killing that caused it to stop
> working.
> 
> When I start the VM with the image in this state, there is nothing
> interesting in the fuse log for the volume in /var/log/glusterfs on the
> node hosting the image.
> 
> No pending heals (all servers report 0 entries to heal).
> 
> The same VM behavior happens on all the physical nodes when I try to
> start with the same VM image.
> 
> Something from the gluster fuse mount log from earlier shows:
> 
> [2021-01-28 21:24:40.814227] I [MSGID: 114018] [client.c:2347:client_rpc_notify] 0-adminvm-client-0: disconnected from adminvm-client-0. Client process will keep trying to connect to glusterd until brick's port is available
> [2021-01-28 21:24:43.815120] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-adminvm-client-0: changing port to 49152 (from 0)
> [2021-01-28 21:24:43.815833] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-adminvm-client-0: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)
> [2021-01-28 21:24:43.817682] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-adminvm-client-0: Connected to adminvm-client-0, attached to remote volume '/data/brick_adminvm'.
> [2021-01-28 21:24:43.817709] I [MSGID: 114042] [client-handshake.c:930:client_post_handshake] 0-adminvm-client-0: 1 fds open - Delaying child_up until they are re-opened
> [2021-01-28 21:24:43.895163] I [MSGID: 114041] [client-handshake.c:318:client_child_up_reopen_done] 0-adminvm-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
> The message "W [MSGID: 114061] [client-common.c:2893:client_pre_lk_v2] 0-adminvm-client-0:  (94695bdb-06b4-4105-9bc8-b8207270c941) remote_fd is -1. EBADFD [File descriptor in bad state]" repeated 6 times between [2021-01-28 21:23:54.395811] and [2021-01-28 21:23:54.811640]
> 
> 
> But that was a long time ago.
> 
> Brick logs have an entry from when I first started the vm today (the
> problem was reproduced yesterday) all brick logs have something similar.
> Nothing appeared on the several other startup attempts of the VM:
> 
> [2021-01-28 21:24:45.460147] I [MSGID: 115029] [server-handshake.c:549:server_setvolume] 0-adminvm-server: accepted client from CTX_ID:613f0d91-34e6-4495-859f-bca1c9f7af01-GRAPH_ID:0-PID:6287-HOST:nano-1-PC_NAME:adminvm-client-2-RECON_NO:-0 (version: 7.2) with subvol /data/brick_adminvm
> [2021-01-29 18:54:45.455558] I [addr.c:54:compare_addr_and_update] 0-/data/brick_adminvm: allowed = "*", received addr = "172.23.255.153"
> [2021-01-29 18:54:45.455802] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 3b66cfab-00d5-4b13-a103-93b4cf95e144
> [2021-01-29 18:54:45.455815] I [MSGID: 115029] [server-handshake.c:549:server_setvolume] 0-adminvm-server: accepted client from CTX_ID:3774af6b-07b9-437b-a34e-9f71f3b57d03-GRAPH_ID:0-PID:45640-HOST:nano-3-PC_NAME:adminvm-client-2-RECON_NO:-0 (version: 7.2) with subvol /data/brick_adminvm
> [2021-01-29 18:54:45.494950] W [socket.c:774:__socket_rwv] 0-tcp.adminvm-server: readv on 172.23.255.153:48551 failed (No data available)
> [2021-01-29 18:54:45.494994] I [MSGID: 115036] [server.c:501:server_rpc_notify] 0-adminvm-server: disconnecting connection from CTX_ID:3774af6b-07b9-437b-a34e-9f71f3b57d03-GRAPH_ID:0-PID:45640-HOST:nano-3-PC_NAME:adminvm-client-2-RECON_NO:-0
> [2021-01-29 18:54:45.495091] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-adminvm-server: Shutting down connection CTX_ID:3774af6b-07b9-437b-a34e-9f71f3b57d03-GRAPH_ID:0-PID:45640-HOST:nano-3-PC_NAME:adminvm-client-2-RECON_NO:-0
> 
> 
> 
> Like before, if I halt the VM, kpartx the image, mount the giant root
> within the image, then unmount, unkpartx, and start the VM - it works:
> 
> nano-2:/var/log/glusterfs # kpartx -a /adminvm/images/adminvm.img
> nano-2:/var/log/glusterfs # mount /dev/mapper/loop0p31 /mnt
> nano-2:/var/log/glusterfs # dmesg|tail -3
> [85528.602570] loop: module loaded
> [85535.975623] EXT4-fs (dm-3): recovery complete
> [85535.979663] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
> nano-2:/var/log/glusterfs # umount /mnt
> nano-2:/var/log/glusterfs # kpartx -d /adminvm/images/adminvm.img
> loop deleted : /dev/loop0
> 
> VM WORKS for ONE boot cycle on one physical!
> 
> nano-2:/var/log/glusterfs # virsh start adminvm
> 
> However, this will work for a boot but later it will stop working again.
> (INCLUDING the physical node that booted once ok. The next boot fails
> again as does luanching it on the other two).
> 
> Based on feedback, I will not change the shard size at this time and
> will leave that for later. Some people suggest larger sizes but it isn't
> a universal suggestion. I'll also not attempt to make a logical volume
> out of a group of smaller images as I think it should work like this.
> Those are things I will try later if I run out of runway. Since we want
> a solution to deploy to sites, this would increase the maintenance of
> the otherwise simple solution.
> 
> I am leaving the state like this and will now proceed to update to the
> latest gluster 7.
> 
> I will report back after I get everything updated and services restarted
> with the newer version.
> 
> THANKS FOR ALL THE HELP SO FAR!!
> 
> Erik
> 
> On Wed, Jan 27, 2021 at 10:55:50PM +0300, Mahdi Adnan wrote:
> >  I would leave it on 64M in volumes with spindle disks, but with SSD volumes, I
> > would increase it to 128M or even 256M, but it varies from one workload to
> > another.
> > On Wed, Jan 27, 2021 at 10:02 PM Erik Jacobson <erik.jacobson@xxxxxxx> wrote:
> > 
> >     > Also, I would like to point that I have VMs with large disks 1TB and 2TB,
> >     and
> >     > have no issues. definitely would upgrade Gluster version like let's say
> >     at
> >     > least 7.9.
> > 
> >     Great! Thank you! We can update but it's very sensitive due to the
> >     workload. I can't officially update our gluster until we have a cluster
> >     with a couple thousand nodes to test with. However, for this problem,
> >     this is on my list on the test machine. I'm hoping I can reproduce it. So
> >     far
> >     no luck making it happen again. Once I hit it, I will try to collect more
> >     data
> >     and at the end update gluster.
> > 
> >     What do you think about the suggestion to increase the shard size? Are
> >     you using the default size on your 1TB and 2TB images?
> > 
> >     > Amar also asked a question regarding enabling Sharding in the volume
> >     after
> >     > creating the VMs disks, which would certainly mess up the volume if that
> >     what
> >     > happened.
> > 
> >     Oh I missed this question. I basically scripted it quick since I was
> >     doing it so often.. I have a similar script that takes it away to start
> >     over.
> > 
> >     set -x
> >     pdsh -g gluster mkdir /data/brick_adminvm/
> >     gluster volume create adminvm replica 3 transport tcp 172.23.255.151:/data/
> >     brick_adminvm 172.23.255.152:/data/brick_adminvm 172.23.255.153:/data/
> >     brick_adminvm
> >     gluster volume set adminvm group virt
> >     gluster volume set adminvm granular-entry-heal enable
> >     gluster volume set adminvm storage.owner-uid 439
> >     gluster volume set adminvm storage.owner-gid 443
> >     gluster volume start adminvm
> > 
> >     pdsh -g gluster mount /adminvm
> > 
> >     echo -n "press enter to continue for restore tarball"
> > 
> >     pushd /adminvm
> >     tar xvf /root/backup.tar
> >     popd
> > 
> >     echo -n "press enter to continue for qemu-img"
> > 
> >     pushd /adminvm
> >     qemu-img create -f raw -o preallocation=falloc /adminvm/images/adminvm.img
> >     5T
> >     popd
> > 
> > 
> >     Thanks again for the kind responses,
> > 
> >     Erik
> > 
> >     >
> >     > On Wed, Jan 27, 2021 at 5:28 PM Erik Jacobson <erik.jacobson@xxxxxxx>
> >     wrote:
> >     >
> >     >     > > Shortly after the sharded volume is made, there are some fuse
> >     mount
> >     >     > > messages. I'm not 100% sure if this was just before or during the
> >     >     > > big qemu-img command to make the 5T image
> >     >     > > (qemu-img create -f raw -o preallocation=falloc
> >     >     > > /adminvm/images/adminvm.img 5T)
> >     >     > Any reason to have a single disk with this size ?
> >     >
> >     >     > Usually in any
> >     >     > virtualization I have used , it is always recommended to keep it
> >     lower.
> >     >     > Have you thought about multiple disks with smaller size ?
> >     >
> >     >     Yes, because the actual virtual machine is an admin node/head node
> >     cluster
> >     >     manager for a supercomputer that hosts big OS images and drives
> >     >     multi-thousand-node-clusters (boot, monitoring, image creation,
> >     >     distribution, sometimes NFS roots, etc) . So this VM is a biggie.
> >     >
> >     >     We could make multiple smaller images but it would be very painful
> >     since
> >     >     it differs from the normal non-VM setup.
> >     >
> >     >     So unlike many solutions where you have lots of small VMs with their
> >     >     images small images, this solution is one giant VM with one giant
> >     image.
> >     >     We're essentially using gluster in this use case (as opposed to
> >     others I
> >     >     have posted about in the past) for head node failover (combined with
> >     >     pacemaker).
> >     >
> >     >     > Also worth
> >     >     > noting is that RHII is supported only when the shard size is 
> >     512MB, so
> >     >     > it's worth trying bigger shard size .
> >     >
> >     >     I have put larger shard size and newer gluster version on the list to
> >     >     try. Thank you! Hoping to get it failing again to try these things!
> >     >
> >     >
> >     >
> >     > --
> >     > Respectfully
> >     > Mahdi
> > 
> > 
> > 
> > --
> > Respectfully
> > Mahdi
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users



[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux