Also I see these 2 errors but not sure if they are preventing our host from starting any VMs. Any suggestions for action to correct this? root@pm3:~# fdisk -l Disk /dev/ram0: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram1: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram2: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram3: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram4: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram5: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram6: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram7: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram8: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram9: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram10: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram11: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram12: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram13: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram14: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/ram15: 64 MiB, 67108864 bytes, 131072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes The backup GPT table is corrupt, but the primary appears OK, so that will be used. Disk /dev/sda: 7.3 TiB, 8001563221504 bytes, 15628053167 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 33553920 bytes Disklabel type: gpt Disk identifier: 5F3DBC16-1812-43C5-9D5F-14411F92D73E Device Start End Sectors Size Type /dev/sda1 34 262177 262144 128M Microsoft reserved /dev/sda2 264192 15628052479 15627788288 7.3T Microsoft basic data Partition 2 does not start on physical sector boundary. Disk /dev/sdb: 111.8 GiB, 120034123776 bytes, 234441648 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: 74F24139-162F-4AB0-94C0-15C7D020C2F2 Device Start End Sectors Size Type /dev/sdb1 34 2047 2014 1007K BIOS boot /dev/sdb2 2048 262143 260096 127M EFI System /dev/sdb3 262144 234441614 234179471 111.7G Linux LVM Partition 2 does not start on physical sector boundary. Disk /dev/sdc: 223.6 GiB, 240057409536 bytes, 468862128 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: EA0BDABA-0E11-4CD8-B6D4-295C785073ED Device Start End Sectors Size Type /dev/sdc1 2048 10487807 10485760 5G unknown /dev/sdc2 10487808 20973567 10485760 5G unknown /dev/sdc3 20973568 31459327 10485760 5G unknown /dev/sdc4 31459328 41945087 10485760 5G unknown /dev/sdc5 41945088 52430847 10485760 5G unknown /dev/sdc6 52430848 62916607 10485760 5G unknown /dev/sdc7 62916608 73402367 10485760 5G unknown /dev/sdc8 73402368 83888127 10485760 5G unknown Disk /dev/sdd: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 56A60BE8-DE87-44BF-9AC1-F9BD8823E9AF Device Start End Sectors Size Type /dev/sdd1 2048 7814037134 7814035087 3.7T unknown Disk /dev/sde: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 6D8BDBA0-46A0-45D2-802B-0F8F64A83C7E Device Start End Sectors Size Type /dev/sde1 2048 7814037134 7814035087 3.7T unknown Disk /dev/sdf: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 42FE723A-499C-4B19-A86B-7E29D8EA17D2 Device Start End Sectors Size Type /dev/sdf1 2048 7814037134 7814035087 3.7T unknown Disk /dev/sdg: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 9F29158B-83B1-4C59-8ACE-4DC9418964D8 Device Start End Sectors Size Type /dev/sdg1 2048 7814037134 7814035087 3.7T unknown Disk /dev/mapper/pve-root: 27.8 GiB, 29796335616 bytes, 58195968 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/mapper/pve-swap: 13.9 GiB, 14898167808 bytes, 29097984 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/mapper/pve-data: 56.2 GiB, 60309897216 bytes, 117792768 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes root@pm3:~# ------------------------------------------------------------------------------- Our setup matches this one exactly for Proxmox and Ceph https://pve.proxmox.com/wiki/Ceph_Server. The brand of SSDs may not be the same but they are the same sizes or larger and are Enterprise quality. Filesystem Size Used Avail Use% Mounted on udev 10M 0 10M 0% /dev tmpfs 13G 1.1G 12G 9% /run /dev/dm-0 28G 28G 0 100% / tmpfs 32G 66M 32G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/mapper/pve-data 56G 655M 55G 2% /var/lib/vz tmpfs 100K 0 100K 0% /run/lxcfs/controllers cgmfs 100K 0 100K 0% /run/cgmanager/fs /dev/fuse 30M 28K 30M 1% /etc/pve /dev/sdc1 3.7T 1.3T 2.4T 36% /var/lib/ceph/osd/ceph-0 /dev/sdd1 3.7T 765G 2.9T 21% /var/lib/ceph/osd/ceph-1 /dev/sde1 3.7T 663G 3.0T 18% /var/lib/ceph/osd/ceph-2 /dev/sdf1 3.7T 677G 3.0T 19% /var/lib/ceph/osd/ceph-3 I can see /dev/dm-0 shows 100% full. Would this cause the error since this is just a vm ? Please advise what we can do to resolve this. -------- Hi Dan, Various proxmox daemons don't look happy on startup also. Are you using a single samsung SSD for your OSD journals on this host? Is that SSD ok? Brian On Tue, Mar 29, 2016 at 5:22 AM, Dan Moses <dan@xxxxxxxxxxxxxxxxxxx> wrote: > Any suggestions to fix this issue? We are using Ceph with proxmox and vms > won’t start due to these Ceph Errors. > > > > This in turn prevents any vm from starting up. This is a live server, please > advise. > > Mar 28 22:01:22 pm3 systemd[1]: Unit ceph.service entered failed state. > Mar 28 22:09:00 pm3 systemd[1]: Unit ceph-mon.2.1459218879.795083638.service > entered failed state. > Mar 28 22:10:49 pm3 console-setup[1642]: failed. > Mar 28 22:10:49 pm3 kernel: [ 2.605140] ata6.00: READ LOG DMA EXT failed, > trying unqueued > Mar 28 22:10:49 pm3 kernel: [ 2.605167] ata6.00: failed to get NCQ Send/Recv > Log Emask 0x1 > Mar 28 22:10:49 pm3 kernel: [ 2.605456] ata6.00: failed to get NCQ Send/Recv > Log Emask 0x1 > Mar 28 22:10:49 pm3 pmxcfs[1795]: [quorum] crit: quorum_initialize failed: 2 > Mar 28 22:10:49 pm3 pmxcfs[1795]: [confdb] crit: cmap_initialize failed: 2 > Mar 28 22:10:49 pm3 pmxcfs[1795]: [dcdb] crit: cpg_initialize failed: 2 > Mar 28 22:10:49 pm3 pmxcfs[1795]: [status] crit: cpg_initialize failed: 2 > Mar 28 22:10:49 pm3 pvecm[1798]: ipcc_send_rec failed: Connection refused > Mar 28 22:10:49 pm3 pvecm[1798]: ipcc_send_rec failed: Connection refused > Mar 28 22:10:49 pm3 pvecm[1798]: ipcc_send_rec failed: Connection refused > Mar 28 22:11:20 pm3 ceph[1891]: failed: 'timeout 30 /usr/bin/ceph -c > /etc/ceph/ceph.conf --name=osd.5 --keyring=/var/lib/ceph/osd/ceph-5/keyring > osd crush create-or-move -- 5 3.64 host=pm3 root=default' > Mar 28 22:11:20 pm3 ceph[1891]: ceph-disk: Error: ceph osd start failed: > Command '['/usr/sbin/service', 'ceph', '--cluster', 'ceph', 'start', > 'osd.5']' returned non-zero exit status 1 > Mar 28 22:11:50 pm3 ceph[1891]: failed: 'timeout 30 /usr/bin/ceph -c > /etc/ceph/ceph.conf --name=osd.7 --keyring=/var/lib/ceph/osd/ceph-7/keyring > osd crush create-or-move -- 7 3.64 host=pm3 root=default' > Mar 28 22:11:50 pm3 ceph[1891]: ceph-disk: Error: ceph osd start failed: > Command '['/usr/sbin/service', 'ceph', '--cluster', 'ceph', 'start', > 'osd.7']' returned non-zero exit status 1 > Mar 28 22:12:21 pm3 ceph[1891]: failed: 'timeout 30 /usr/bin/ceph -c > /etc/ceph/ceph.conf --name=osd.9 --keyring=/var/lib/ceph/osd/ceph-9/keyring > osd crush create-or-move -- 9 3.64 host=pm3 root=default' > Mar 28 22:12:21 pm3 ceph[1891]: ceph-disk: Error: ceph osd start failed: > Command '['/usr/sbin/service', 'ceph', '--cluster', 'ceph', 'start', > 'osd.9']' returned non-zero exit status 1 > Mar 28 22:12:51 pm3 ceph[1891]: failed: 'timeout 30 /usr/bin/ceph -c > /etc/ceph/ceph.conf --name=osd.11 > --keyring=/var/lib/ceph/osd/ceph-11/keyring osd crush create-or-move -- 11 > 3.64 host=pm3 root=default' > Mar 28 22:12:51 pm3 ceph[1891]: ceph-disk: Error: ceph osd start failed: > Command '['/usr/sbin/service', 'ceph', '--cluster', 'ceph', 'start', > 'osd.11']' returned non-zero exit status 1 > Mar 28 22:12:51 pm3 ceph[1891]: ceph-disk: Error: One or more partitions > failed to activate > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |