クラウドのインスタンスにリモート接続できなくなって焦った

先日、OracleCloudのインスタンスに、いつものようにSSH接続しようとしたら、それができなくなりました。インスタンスは実行中となっておりブラウザで再起動をしましたが接続できません。念のため、SSH接続だけの問題かもしれないので、他のトラフィックが許可してあるポートに接続を試みました。しかし、こちらも接続ができません。トラブルが発生する前に行った作業はOSの更新でした。

インスタンスのOS：OracleLinux8
トラブルが発生する前に自分で行った作業はOSの更新：dnf update

そこでコンソール接続をすることにしました。コンソール接続はトラブルシューティングの目的でのみ使用します。GoogleCloudでも同等な接続をしたことがあります。

インスタンスの詳細にあるコンソール接続に進み、「Cloud Shell接続の起動」をクリックします。

しかし、画面になにかしら表示されて次に進むことができません。

正常に起動しているインスタンスでCloud Shell接続を実行するとログイン画面が表示されます。出ない時はENTERキーを押します。このトラブルが発生したインスタンスはフリーズしているみたいです。

調査した結果、オラクルのブログらしきサイトに対処法がかいてあったので、これを試してみます。Cloud Shell接続をしている状態でインスタンスの再起動をします。同じページにどちらもあるので再起動できます。おそらくOSが起動していないのでインスタンスの強制再起動でいいと思います。普通に再起動すると15分くらいかかります。

再起動するとCloud ShellにOS選択画面が表示されます。

Oracle Linux Server 8 (5.4.17-2136.314.6.2.el8uek.aarch64)
Oracle Linux Server 8 (5.4.17-2136.313.6.el8uek.aarch64)
Oracle Linux Server 8 (5.4.17-2136.312.3.4.el8uek.aarch64)
Oracle Linux Server 8 (5.4.17-2136.309.5.el8uek.aarch64)
Oracle Linux Server 8 (0-rescue-529f7f58886140ffbc3504f0be288ae8)
Oracle Linux Server 8 (0-rescue-3bd3dafbece24d9fb8072975cd46f25d)
System setup

一番上のOSを選択してEnterキーを押すと次のエラーが発生しました。

error: ../../grub-core/fs/fshelp.c:258:file
`/initramfs-5.4.17-2136.314.6.2.el8uek.aarch64.img’ not found.

Press any key to continue…
EFI stub: Booting Linux Kernel…
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map…
[ 0.567151] Kernel panic – not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 0.568089] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.17-2136.314.6.2.el8uek.aarch64 #2
[ 0.568676] Hardware name: QEMU KVM Virtual Machine, BIOS 1.5.1 06/16/2021
[ 0.569172] Call trace:
[ 0.569357] dump_backtrace+0x0/0x1f0
[ 0.569623] show_stack+0x24/0x30
[ 0.569886] dump_stack+0xbc/0xe0
[ 0.570139] panic+0x15c/0x36c
[ 0.570358] mount_block_root+0x250/0x304
[ 0.570660] mount_root+0x84/0x90
[ 0.570900] prepare_namespace+0x13c/0x184
[ 0.571212] kernel_init_freeable+0x2d4/0x304
[ 0.571536] kernel_init+0x18/0x110
[ 0.571794] ret_from_fork+0x10/0x18
[ 0.572066] SMP: stopping secondary CPUs
[ 0.572550] Kernel Offset: disabled
[ 0.572812] CPU features: 0x01000e,2a00a238
[ 0.573127] Memory Limit: none
[ 0.573356] —[ end Kernel panic – not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]—

カーネルパニックと最後に表示され、次に進みません。
次はレスキューと表示があるOSを選択します。

Welcome to Oracle Linux Server 8.6 dracut-049-202.git20220511.0.1.el8_6 (Initramfs)!

[ OK ] Listening on Journal Socket (/dev/log).
[ OK ] Started Dispatch Password Requests to Console Directory Watch.
[ OK ] Reached target Paths.
[ OK ] Reached target Slices.
[ OK ] Listening on udev Control Socket.
[ OK ] Reached target Swap.
[ OK ] Listening on udev Kernel Socket.
[ OK ] Reached target Local Encrypted Volumes.
[ OK ] Listening on Open-iSCSI iscsiuio Socket.
[ OK ] Reached target Timers.
[ OK ] Listening on Open-iSCSI iscsid Socket.
[ OK ] Listening on Journal Socket.
Starting iSCSI UserSpace I/O driver…
Starting Journal Service…
Starting Setup Virtual Console…
Starting Load Kernel Modules…
Starting Create list of required st…ce nodes for the current kernel…
[ OK ] Reached target Sockets.
[ OK ] Started iSCSI UserSpace I/O driver.
[ OK ] Started Create list of required sta…vice nodes for the current kernel.
[ OK ] Started Setup Virtual Console.
Starting dracut ask for additional cmdline parameters…
Starting Create Static Device Nodes in /dev…
[ OK ] Started Load Kernel Modules.
[ OK ] Started Journal Service.
[ OK ] Started Create Static Device Nodes in /dev.
Starting Apply Kernel Variables…
[ OK ] Started dracut ask for additional cmdline parameters.
Starting dracut cmdline hook…
[ OK ] Started Apply Kernel Variables.
[ OK ] Started dracut cmdline hook.
Starting dracut pre-udev hook…
[ OK ] Started dracut pre-udev hook.
Starting udev Kernel Device Manager…
[ OK ] Started udev Kernel Device Manager.
Starting dracut pre-trigger hook…
[ OK ] Started dracut pre-trigger hook.
Starting udev Coldplug all Devices…
Mounting Kernel Configuration File System…
[ OK ] Mounted Kernel Configuration File System.
[ OK ] Started udev Coldplug all Devices.
Starting udev Wait for Complete Device Initialization…
[ OK ] Started udev Wait for Complete Device Initialization.
Starting Device-Mapper Multipath Device Controller…
[ OK ] Started Device-Mapper Multipath Device Controller.
[ OK ] Reached target Local File Systems (Pre).
[ OK ] Reached target Local File Systems.
Starting Create Volatile Files and Directories…
Starting Open-iSCSI…
[ OK ] Started Create Volatile Files and Directories.
[ OK ] Reached target System Initialization.
[ OK ] Reached target Basic System.
[ OK ] Started Open-iSCSI.
Starting dracut initqueue hook…
[ 2.194701] dracut-initqueue[634]: RTNETLINK answers: File exists
[ OK ] Found device /dev/mapper/ocivolume-root.
[ OK ] Reached target Initrd Root Device.
[ 4.678600] dracut-initqueue[806]: RTNETLINK answers: File exists
Stopping Open-iSCSI…
[ OK ] Stopped Open-iSCSI.
Stopping iSCSI UserSpace I/O driver…
[ OK ] Stopped iSCSI UserSpace I/O driver.
Starting iSCSI UserSpace I/O driver…
[ OK ] Started iSCSI UserSpace I/O driver.
Starting Open-iSCSI…
[ OK ] Started Open-iSCSI.
[ OK ] Started dracut initqueue hook.
Starting dracut pre-mount hook…
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
[ OK ] Started dracut pre-mount hook.
Starting File System Check on /dev/mapper/ocivolume-root…
[ OK ] Started File System Check on /dev/mapper/ocivolume-root.
Mounting /sysroot…
[ OK ] Mounted /sysroot.
[ OK ] Reached target Initrd Root File System.
Starting Reload Configuration from the Real Root…
Stopping Device-Mapper Multipath Device Controller…
[ OK ] Started Reload Configuration from the Real Root.
[ OK ] Stopped Device-Mapper Multipath Device Controller.
Starting dracut mount hook…
[ OK ] Reached target Initrd File Systems.
[ OK ] Reached target Initrd Default Target.
[ OK ] Started dracut mount hook.
Starting dracut pre-pivot and cleanup hook…
[ 11.382350] dracut-pre-pivot[1002]: Dec 16 14:37:39 | /etc/multipath.conf does not exist, blacklisting all devices.
[ 11.383981] dracut-pre-pivot[1002]: Dec 16 14:37:39 | You can run “/sbin/mpathconf –enable” to create
[ 11.386388] dracut-pre-pivot[1002]: Dec 16 14:37:39 | /etc/multipath.conf. See man mpathconf(8) for more details
[ OK ] Started dracut pre-pivot and cleanup hook.
Starting Cleaning Up and Shutting Down Daemons…
[ OK ] Stopped dracut pre-pivot and cleanup hook.
[ OK ] Stopped target Remote File Systems.
[ OK ] Stopped dracut mount hook.
[ OK ] Stopped dracut pre-mount hook.
[ OK ] Stopped target Remote File Systems (Pre).
[ OK ] Stopped target Timers.
[ OK ] Stopped target Initrd Default Target.
[ OK ] Stopped target Initrd Root Device.
[ OK ] Stopped target Basic System.
[ OK ] Stopped target System Initialization.
[ OK ] Stopped Apply Kernel Variables.
[ OK ] Stopped target Local Encrypted Volumes.
[ OK ] Stopped Load Kernel Modules.
[ OK ] Stopped target Swap.
[ OK ] Stopped Create Volatile Files and Directories.
[ OK ] Stopped target Local File Systems.
[ OK ] Stopped target Local File Systems (Pre).
[ OK ] Stopped target Slices.
[ OK ] Stopped target Paths.
[ OK ] Stopped Dispatch Password Requests to Console Directory Watch.
[ OK ] Stopped target Sockets.
[ OK ] Stopped dracut initqueue hook.
Stopping Open-iSCSI…
[ OK ] Stopped udev Wait for Complete Device Initialization.
[ OK ] Stopped udev Coldplug all Devices.
[ OK ] Stopped dracut pre-trigger hook.
Stopping udev Kernel Device Manager…
[ OK ] Stopped Open-iSCSI.
Stopping iSCSI UserSpace I/O driver…
[ OK ] Closed Open-iSCSI iscsid Socket.
[ OK ] Stopped iSCSI UserSpace I/O driver.
[ OK ] Stopped udev Kernel Device Manager.
[ OK ] Started Cleaning Up and Shutting Down Daemons.
[ OK ] Stopped dracut pre-udev hook.
[ OK ] Stopped dracut cmdline hook.
[ OK ] Stopped dracut ask for additional cmdline parameters.
[ OK ] Stopped Create Static Device Nodes in /dev.
[ OK ] Stopped Create list of required sta…vice nodes for the current kernel.
[ OK ] Closed udev Kernel Socket.
[ OK ] Closed udev Control Socket.
Starting Cleanup udevd DB…
[ OK ] Closed Open-iSCSI iscsiuio Socket.
[ OK ] Started Cleanup udevd DB.
[ OK ] Reached target Switch Root.
Starting Switch Root…

Welcome to Oracle Linux Server 8.7!

[ OK ] Stopped Switch Root.
[ OK ] Stopped Journal Service.
Starting Journal Service…
[ OK ] Listening on udev Control Socket.
[ OK ] Created slice system-serial\x2dgetty.slice.
[ OK ] Listening on Process Core Dump Socket.
[ OK ] Listening on udev Kernel Socket.
[ OK ] Set up automount Arbitrary Executab…rmats File System Automount Point.
[ OK ] Stopped File System Check on Root Device.
Starting Remount Root and Kernel File Systems…
[ OK ] Reached target rpc_pipefs.target.
[ OK ] Listening on Device-mapper event daemon FIFOs.
Starting Monitoring of LVM2 mirrors…ng dmeventd or progress polling…
[ OK ] Created slice system-sshd\x2dkeygen.slice.
[ OK ] Started Forward Password Requests to Wall Directory Watch.
[ OK ] Created slice system-getty.slice.
[ OK ] Created slice User and Session Slice.
[ OK ] Reached target Slices.
[ OK ] Stopped target Switch Root.
[ OK ] Stopped target Initrd File Systems.
[ OK ] Stopped target Initrd Root File System.
Starting udev Coldplug all Devices…
Mounting Huge Pages File System…
[ OK ] Listening on initctl Compatibility Named Pipe.
Starting Read and set NIS domainname from /etc/sysconfig/network…
[ OK ] Created slice system-systemd\x2dfsck.slice.
[ OK ] Listening on RPCbind Server Activation Socket.
[ OK ] Reached target RPC Port Mapper.
[ OK ] Listening on LVM2 poll daemon socket.
Starting Apply Kernel Variables…
[ OK ] Started Dispatch Password Requests to Console Directory Watch.
[ OK ] Reached target Paths.
[ OK ] Reached target Local Encrypted Volumes.
Mounting Kernel Debug File System…
Mounting POSIX Message Queue File System…
[ OK ] Started Journal Service.
[ OK ] Started Remount Root and Kernel File Systems.
[ OK ] Started Monitoring of LVM2 mirrors,…sing dmeventd or progress polling.
[ OK ] Mounted Huge Pages File System.
[ OK ] Started Read and set NIS domainname from /etc/sysconfig/network.
[ OK ] Mounted Kernel Debug File System.
[ OK ] Mounted POSIX Message Queue File System.
Activating swap /.swapfile…
Starting Load/Save Random Seed…
Starting Create Static Device Nodes in /dev…
Starting Flush Journal to Persistent Storage…
[ OK ] Started Apply Kernel Variables.
[ OK ] Started udev Coldplug all Devices.
[ OK ] Started Load/Save Random Seed.
[ OK ] Activated swap /.swapfile.
[ OK ] Started Flush Journal to Persistent Storage.
[ OK ] Reached target Swap.
Starting udev Wait for Complete Device Initialization…
[ OK ] Started Create Static Device Nodes in /dev.
Starting udev Kernel Device Manager…
[ OK ] Started udev Kernel Device Manager.
[ OK ] Created slice system-lvm2\x2dpvscan.slice.
Starting LVM event activation on device 8:3…
[ OK ] Started udev Wait for Complete Device Initialization.
[ OK ] Reached target Local File Systems (Pre).
Mounting /boot…
Mounting /var/oled…
Starting File System Check on /dev/disk/by-uuid/F2D1-BCCB…
[ OK ] Started LVM event activation on device 8:3.
[ OK ] Started File System Check on /dev/disk/by-uuid/F2D1-BCCB.
[ OK ] Mounted /boot.
[ OK ] Mounted /var/oled.
Mounting /boot/efi…
[FAILED] Failed to mount /boot/efi.
See ‘systemctl status boot-efi.mount’ for details.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Mark the need to relabel after reboot.
[DEPEND] Dependency failed for Oracle Cloud …ent for management and monitoring.
[DEPEND] Dependency failed for Oracle Cloud Infrastructure agent updater.
Starting Apply Ksplice updates…
[ OK ] Reached target NFS client services.
[ OK ] Reached target Timers.
[ OK ] Reached target Cloud-init target.
[ OK ] Reached target Network (Pre).
[ OK ] Reached target Network.
Starting Initial cloud-init job (metadata service crawler)…
[ OK ] Reached target Sockets.
Starting Import network configuration from initramfs…
[ OK ] Reached target User and Group Name Lookups.
[ OK ] Reached target Login Prompts.
[ OK ] Started Import network configuration from initramfs.
[ OK ] Started Emergency Shell.
[ OK ] Reached target Emergency Mode.
Starting Create Volatile Files and Directories…
[ OK ] Started Create Volatile Files and Directories.
Starting Security Auditing Service…
Starting RPC Bind…
[ OK ] Started RPC Bind.
[ OK ] Started Security Auditing Service.
Starting Update UTMP about System Boot/Shutdown…
[ OK ] Started Update UTMP about System Boot/Shutdown.
Starting Update UTMP about System Runlevel Changes…
[ OK ] Started Update UTMP about System Runlevel Changes.
[ OK ] Started Apply Ksplice updates.
[ 15.823129] cloud-init[1372]: Cloud-init v. 22.1-5.0.1.el8 running ‘init’ at Fri, 16 Dec 2022 14:37:44 +0000. Up 15.56 seconds.
[ 15.836296] cloud-init[1372]: ci-info: ++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++
[ 15.848455] cloud-init[1372]: ci-info: +——–+——+————————-+—————+——–+——————-+
[ 15.870321] cloud-init[1372]: ci-info: +——–+——+————————-+—————+——–+——————-+
[ 15.900577] cloud-init[1372]: ci-info: +——–+——+————————-+—————+——–+——————-+
[ 15.903245] cloud-init[1372]: ci-info: +++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++
[ 15.907236] cloud-init[1372]: ci-info: +——-+————-+———-+—————+———–+——-+
[ 15.913897] cloud-init[1372]: ci-info: +——-+————-+———-+—————+———–+——-+
[ 15.927949] cloud-init[1372]: ci-info: +——-+————-+———-+—————+———–+——-+
[ 15.931469] cloud-init[1372]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[ 15.933134] cloud-init[1372]: ci-info: +——-+————-+———+———–+——-+
[ 15.939560] cloud-init[1372]: ci-info: +——-+————-+———+———–+——-+
[ 15.952105] cloud-init[1372]: ci-info: +——-+————-+———+———–+——-+
[ OK ] Started Initial cloud-init job (metadata service crawler).
[ OK ] Reached target Cloud-config availability.
[ OK ] Reached target Network is Online.
Starting Open-iSCSI…
Starting Notify NFS peers of a restart…
[ OK ] Started Notify NFS peers of a restart.
[ OK ] Started Open-iSCSI.
Starting Login and scanning of iSCSI devices…
Starting Logout off all iSCSI sessions on shutdown…
[ OK ] Started Logout off all iSCSI sessions on shutdown.
[ OK ] Started Login and scanning of iSCSI devices.
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
Starting Crash recovery kernel arming…
[FAILED] Failed to start Crash recovery kernel arming.
See ‘systemctl status kdump.service’ for details.
You are in emergency mode. After logging in, type “journalctl -xb” to view
system logs, “systemctl reboot” to reboot, “systemctl default” or “exit”
to boot into default mode.