Bug 3011 - pve-qemu-kvm 5.1.0-2 secondary LVM disks not detected
Summary: pve-qemu-kvm 5.1.0-2 secondary LVM disks not detected
Status: RESOLVED DUPLICATE of bug 3010
Alias: None
Product: pve
Classification: Unclassified
Component: vzdump (show other bugs)
Version: 6
Hardware: PC Linux
: --- bug
Assignee: Stefan Reiter
URL:
: 3088 (view as bug list)
Depends on:
Blocks:
 
Reported: 2020-09-16 18:47 CEST by admin
Modified: 2020-10-22 16:11 CEST (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description admin 2020-09-16 18:47:30 CEST
Hello

There seems to be a bug or some change in the 5.1 version of pve-qemu-kvm which causes my Ubuntu 18 guests to fail on boot.

The exact error can be found here: https://redscr.net/vhcLMrABQL9r4vhvH8fB.png

I believe the UUID of the disk changes somehow and is no longer detected correctly by lvm.

If i downgrade back to 5.0.0-13 version, it seems to be working without problems.

The guest is installed with lvm, and the root lv is then extended to a secondary disk.

Steps to reproduce:
Install a Ubuntu 18 guest with LVM
Add a new virtio disk 
pvcreate and vgextend on it
lvresize the root partition.
Reboot

Thank you
Comment 1 Moayad Almalat 2020-09-17 14:08:21 CEST
Hello,

Please post output of `pveversion -v` and `config for Ubuntu VM`.
Comment 2 admin 2020-09-18 12:17:12 CEST
Ubuntu config:

root@ubuntu:~# lsblk
NAME                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                    11:0    1 1024M  0 rom
vda                   252:0    0   25G  0 disk
└─vda1                252:1    0   25G  0 part
  ├─ubuntu--vg-root   253:0    0   49G  0 lvm  /
  └─ubuntu--vg-swap_1 253:1    0  976M  0 lvm  [SWAP]
vdb                   252:16   0   25G  0 disk
└─ubuntu--vg-root     253:0    0   49G  0 lvm  /


#cat 100000.conf
agent: 1
balloon: 4096
bootdisk: virtio0
cores: 2
cpu: host
ide1: none,media=cdrom
memory: 4096
name: ubuntu18
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
shares: 400
smbios1: uuid=dd8a62db-f013-4f8e-84b1-ea3043e29c61
sockets: 1
vga: virtio
virtio0: local-lvm:vm-100000-disk-0,size=25G
virtio1: local-lvm:vm-100000-disk-1,size=25G



#pveversion -v:
proxmox-ve: 6.2-1 (running kernel: 5.3.18-2-pve)
pve-manager: 6.2-11 (running version: 6.2-11/22fb4983)
pve-kernel-5.4: 6.2-6
pve-kernel-helper: 6.2-6
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.60-1-pve: 5.4.60-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-6
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-12
pve-cluster: 6.1-8
pve-container: 3.2-1
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-2
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-14
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve1
Comment 3 Moayad Almalat 2020-09-18 13:53:09 CEST
Hi again,

I tested on my workstation with the same your config and all `pve-qemu-kvm` versions  everything works!

- Here is the config for VM

agent: 1
balloon: 1
bootdisk: virtio0
cores: 2
cpu: host
ide2: none,media=cdrom
memory: 4096
name: Ubuntu-18
net0: virtio=1E:11:22:E3:3D:47,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
shares: 400
smbios1: uuid=560dea65-59a7-477e-a9d3-aaee6b56299d
sockets: 1
virtio0: local-lvm:vm-115-disk-0,size=33G
virtio1: local-lvm:vm-115-disk-1,size=3G
vmgenid: 8942c770-ab87-479c-886a-e022cb8667e1



Can you check your syslog/journalctl if there any error during booting the Ubuntu VM?
Comment 4 admin 2020-09-18 15:45:00 CEST
Hello

There doesn't seem to be anything in syslog, the failure is at grub level.

Please try this ubuntu 18 image: http://185.141.60.106/vzdump-qemu-1000000-2020_09_18-16_22_20.vma.zst

It works if it is started in pve-qemu-kvm 5.0.0-13 and it fails if i it is started in pve-qemu-kvm to 5.1.x

The root password for the OS is 123456

Thank you
Comment 5 Stefan Reiter 2020-09-21 14:08:12 CEST
Hi,

I've managed to reproduce the issue (thanks for the detailed report!) and can confirm it's a regression in QEMU 5.1

The problem lies with the updated SeaBIOS version shipping with it, where an optimization was introduced to only initialize disks in BIOS if they are marked as bootable. It seems as though GRUB uses that information for itself, and decides that it shouldn't load the second disk, but then falls flat on its face when it realizes that the disk is part of the root LV.

I'll contact upstream about the issue and keep you up to date here.
Comment 6 Stefan Reiter 2020-09-22 09:39:39 CEST
Upstream has confirmed that this is indeed the wanted behaviour, and has only advised not to use such an LVM configuration[0]. While it is unusual, I don't think we should just ignore it though.

A workaround for SeaBIOS VMs is to add:

  args: -boot strict=off

to /etc/pve/qemu-server/<vmid>.conf

For OVMF this still doesn't work, and requires adding a bootindex= property to every affected drive, something not doable as an easy workaround ATM (same fix would work for SeaBIOS too though).

I'm not sure how common this issue is, but an accurate fix (for both SeaBIOS and OVMF) would change our bootorder model as we currently have it. I don't think we can fix this without either:
* Changing how bootorder works (i.e. all drives are bootable, even if not selected, albeit with lower prio)
* Require manual user intervention of some kind, in which case I'd honestly wait for someone with this problem on OVMF to show up since the workaround for SeaBIOS is so simple anyway

[0] https://mail.coreboot.org/hyperkitty/list/seabios@seabios.org/message/LUXUOBBHV3JHFBFAMHKWG6NH3AXNZ653/
Comment 7 Thomas Lamprecht 2020-09-22 11:06:50 CEST
Note that this is related to bug 3010 and the proposed solutions of bug 3010, comment 5 could also help here.
Comment 8 admin 2020-09-22 20:07:55 CEST
Thank you very much!
Comment 9 Stefan Reiter 2020-10-19 17:44:54 CEST
Closing this in favor of #3010, so we don't have to track progress on two fronts. The fix for that will also fix this problem (simply mark all required disks as bootable in the new GUI/config).

*** This bug has been marked as a duplicate of bug 3010 ***
Comment 10 Dominik Csapak 2020-10-22 16:11:59 CEST
*** Bug 3088 has been marked as a duplicate of this bug. ***