Jan 02

Good Morning Admin,

so you updated your proxmox installation and rebooted… waited patiently… and it didn’t come back? uh oh?
Hopefully you have some kind of console to see those nice errors..

Loading initial ramdisk ...
oading, please wait...
[ 2.394422] megaraid_sas 0000:02:00.0: INIT adapter done
modprobe: module unknown not found in modules.dep
mount: mounting ZFS=rpool/ROOT/pve-1 on /root failed: No such file or directory
Target filesystem doesn't have requested /sbin/init.
mount: mounting /dev on /root/dev failed: No such file or directory
No init found. Try passing init= bootarg.
modprobe: module ehci-orion not found in modules.dep

BusyBox v1.22.1 (Debian 1:1.22.0-9+deb8u1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/bin/sh: can't access tty; job control turned off

After lots of fuzzing around, booting one old kernel etc. 
it comes all down to broken -pve5 GRUB packages.

Solution (inside initramfs)

modprobe zfs
zpool import -R /rpool -N rpool
zpool mount -a
mount -t proc proc /rpool/proc
mount -t sysfs sys /rpool/sys/
mount -o bind /dev /rpool/dev/
mount devpts /dev/pts -t devpts
chroot /rpool
/etc/init.d/networking start
/etc/init.d/ssh start
#now you can finally get rid of this console (which seems to timeout every freakin 10seconds on those fujitsu boxes) and login via SSH.

now my tool of choice is aptitude, was searching for the package “grub-pc” using the “/” command. Then chose the previous -pve4 version of it and hit “!” a couple times till it said its was going to downgrade 5 packages. You can also hit “e” to examine which packages exactly. After hitting “g” for GO, it actually downgrades everything. I took the package maintainers version of grub config. After a sync & reboot, it booted the new kernel.

You might wanna block further upgrades of grub using “aptitude hold grub-pc”. Note this works only if aptitude is your tool of choice. Using apt-get will not consider those “holds”.

Hopefully this helps some of you.

preload preload preload