>_nexthopreachable

Broke My Server Trying to Clean Up Files

Ali Bahadır Coşkun5 min read

BACKGROUND

It all started with me wanting to do some labs a on my new containerlab server after midnight. This is a lesson why you should never do critical tasks at midnight. I run a cloud VM hosting a few docker containers for testing. It had been running fine but I noticed that the disk had accumulated about 20GB of Docker debris over months of testing and wanted to use it for containerlab. There were old containers, dangling volumes, build cache, unused images not much space left for lab work. I wanted to clean it up before it got worse. I was about to run out of space. I cleaned it up alright but the server wasn’t very happy about it. CPU spiked up to 100. Then I panicked and shut the server down. That was the moment I knew I messed up. Ended up with a corrupted filesystem and a server that wouldn’t boot. It took me hours to figure out what went wrong and my lab session was ruined yet again.

THE CLEANUP

bash
docker builder prune -af          # 15.89 GB of build cachedocker volume prune                # 2.056 GB of dangling volumesdocker volume rm snipeit_db_data_v623 snipeit_storage_test [... 11 more]docker image prune -a              # 1.749 GB of unused images

I did not think that deleting 20GB of Docker data is a significant disk operation and could cause serious issues. Apparently docker does not just delete files it has to clean up overlayfs layer directories, remove whiteout files, free inodes, update directory entries all kinds of mambo jambo. All of it hits the underlying ext4 filesystem directly. On a busy disk this generates a lot of I/O and a corresponding CPU spike. Everything was fine actually until I got scared seeing the high CPU number and shut it down.

THE NEXT BOOT

The server did not come back up. Rest in peace. The VNC console showed kernel boot messages and then froze. SSH was not available because the system never finished booting. The screen was stuck on a wall of kernel output going nowhere. I started searching for a fix like crazy. The boot log showed ata1 through ata6: SATA link down. At first my search came up with hypervisor-level storage mapping failure. The virtual disk had become disconnected from the virtual SATA controller. Recommended fix was to stop the instance, detach and reattach the volume, or escalate to the cloud provider. This was a reasonable-sounding diagnosis. It was also wrong and it wasted 3 hours of my time.

Turns out those SATA link down messages are completely normal on a QEMU VM that does not use SATA storage. The VM boots from virtio block devices. The kernel reports the empty SATA ports as "down" because nothing is attached to them not because anything broke. This is just how QEMU works. But the troubleshooting went down the hypervisor path anyway. I stopped the instance. Restarted it Tried detaching volumes. Restoring a new vm from backup. None of it helped, because the problem was obviously not the hypervisor.

WHAT WAS ACTUALLY WRONG

Apparently, when you shut down a server while it is mid-write on 20GB of data, the filesystem does not get a chance to finish what it was doing. Things get left in a broken state. The server had a corrupted filesystem. That is it. Not a storage hardware failure. Not a hypervisor bug. Just a filesystem that got cut off at the worst possible moment because I turned the power off. The e2fsck output confirmed it when we eventually got there:

text
Inode 1048663 was part of the orphaned inode list. IGNORED.Inode 1049946 was part of the orphaned inode list. IGNORED.[... 40 more orphaned inodes ...]Directory inode 1619035, block #0, offset 0: directory corrupted

All of those orphaned inodes were Docker's overlayfs layer files temporary data stranded when the shutdown interrupted the delete operation. The corrupted directory block was somewhere inside /var/lib/docker, a directory entry half-written before the power went.

GETTING TO THE FIX

Went into rescue mode, the server booted further this time and dropped to a maintenance shell with:

text
EXT4-fs (vdb1): warning: mounting fs with errors, running e2fsck is recommended

That was the actual clue. For the past 3 hours I had been chasing the wrong thing.

THE ACTUAL FIX

After hours of rescue mode, GRUB editing, kernel parameters, and two wrong diagnoses, the fix was two commands.

bash
echo u > /proc/sysrq-triggere2fsck -fy /dev/vdb1

The first one put the filesystem into a state where it can be repaired. The second one repaired it. A few minutes later the server booted normally. I only needed 2 commands. Fscheck should have been the first thing I tried. I should have known that a sudden shutdown during heavy disk writes can cause filesystem corruption. I should have known that the SATA link down messages are normal for a QEMU VM. But I panicked and went down the wrong path. I wasted hours on a wild goose chase. The lesson is to stay calm, understand what you are seeing, and try the simplest fix first.

THE REAL CHAIN OF EVENTS

text
Docker cleanup (20GB of overlayfs deletions)→ CPU spike (normal, server was working)→ shutdown during heavy disk writes (not normal)→ ext4 journal interrupted → filesystem corruption→ kernel couldn't mount root cleanly → boot hang→ misdiagnosed as hypervisor SATA mapping failure→ misdiagnosed as kernel regression→ e2fsck fixed it in under a minute