Fork Bomb Part 2: Controlled Chaos - Monitoring Resource Exhaustion in QEMU

I ran a forkbomb in Debian VM, run on Hypervisor: QEMU

Specs of VM-

  • Cores: 2
  • RAM Memory: 20GB

I had set up 3 monitoring environments beside it in my host OS-
1) htop (monitor CPU + Processes)
2) dmesg -w (monitor kernel messages)
3) vmstat 1 (monitor system level metrics - process stats, memory stats, swap, io, cpu breakdown)

within 4 seconds, all 8 cores in my CPU were all running at full speed.
Note: The cores that I assigned to the VM were virtual cores, but in reality, the all 8 cores of my machine were being used

The Resource temporatily unavailable message has already started popping up. This means that the system has hit the process limit and cannot create new processes.
The fork bomb's exponential growth:

  • 1 second: ~32 processes
  • 2 seconds: ~1,024 processes
  • 3 seconds: ~32,768 processes (system-wide PID exhaustion)
  • 4 seconds: Complete scheduler collapse
    I suspect that Ubuntu is reaching is cgroups limit. I will look into this more in the future.

Memory usage shows 215.2M, 161M, 161M, 161M for different processes

The processes are running in /usr/bin and `qemu paths, confirming QEMU involvement

In the dmesg, we observe that-

  • Authentication and association attempts with access point a8:49:4d:5d:xx:xx
  • Connection being lost and re-established
    This suggests the system load was so high that it affected network stability, causing WiFi disconnections

We can see that the context switches per second had climbed up to a 6000-8000

The number of user processes are also amazing (98, 97).
Note: the forkbomb here counts as a user process since it was run within a VM - which is a user process. however, running the forkbomb for the host system would have categorized as a system process.

Since the forkbomb was running in a VM, the processes and kernel threads couldn't be observed properly.

The Load Average measures the amount of processes waiting for CPU/disk. The current load is 0.57. On average 1 process wants CPU every 2 minutes. However, you can see in the next SS, the load average has increased significantly. The 5 min and 15 min load averages are low, since the forkbomb had only just started and had not started affecting the system yet.

This was the last screenshot taken after 3:40 mins.

The load average here is 11.13. This means that on average 11 processes were wanting CPU, but only 8 cores are there, so 3 were in waiting stage.
This indicates CPU saturation and buildup.

The 5 minute load is 6.14.. this means that the load increased drastically within the past 5 minutes. And the 15 minute load is 2.88. This means that the load was spiked recently and wasn't as high before.

The Swap memory is still empty. Swap memory is the virtual memory used when the RAM is full. It's empty because the RAM wasnt full yet. We can observe that the RAM used is 6.5GB/14GB.
(note that this is my host system stats.)

I let the forkbomb run for a few minutes in the background without monitoring it, just to see if my host system would crash... and after around 15-20 mins, this is what i observed 

[ 1167.918946] Out of memory: Killed process 62327 (bash) total-vm:8800kB, anon-rss:2060kB, file-rss:240kB, shmem-rss:0kB, UID:1000 pgtables:56kB oom_score_adj:0 

[ 1180.594847] [ 72293] 1000 72293 2200 453 453 0 0 53248 65 0 bash [ 1180.596087] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,task=bash,pid=50984,uid=1000 

[ 1180.597303] Out of memory: Killed process 50984 (bash) total-vm:8800kB, anon-rss:2020kB, file-rss:232kB, shmem-rss:0kB, UID:1000 pgtables:56kB oom_score_adj:0 

[ 1182.650550] oom_reaper: reaped process 50984 (bash), now anon-rss:76kB, file-rss:232kB, shmem-rss:0kB

My system did not crash. Instead, the linux kernel's OOM (out of memory) Killer was activated.
The forkbomb had exhausted the available memory for the VM, triggering the defence mechanish ,i.e OOM Killer.
The OOM killer is designed to prevent total system failure by selectively terminating processes to free up memory

From the above log, we can see that the OOM killer first killed the process, then the OOM reaper immediately reclaimed the memory.

This is how my forkbomb was successfully dismantled by the OOM killer, that protected my main system.

Inference
From this experiment, I infer that running the forbomb experiment in my VM put quite a bit of load on my machine.

This can be confirmed when i checked the past journalctl logs for the machine next day and saw logs saying that the system was becoming slow and was behind time. 

Feb 12 01:22:18 laptop gnome-shell[3967]: libinput error: event1 - Power Button: client bug: event processing lagging behind by 144ms, your system is too slow

Feb 12 01:22:18 laptop gnome-shell[3967]: libinput error: event2 - Power Button: client bug: event processing lagging behind by 101ms, your system is too slow

Feb 12 01:27:04 laptop gnome-shell[3967]: libinput error: event1 - Power Button: client bug: event processing lagging behind by 106ms, your system is too slow

Feb 12 01:27:04 laptop gnome-shell[3967]: libinput error: event1 - Power Button: WARNING: log rate limit exceeded (5 msgs per 60min). Discarding future messages.

Feb 12 01:27:04 laptop gnome-shell[3967]: libinput error: event2 - Power Button: client bug: event processing lagging behind by 105ms, your system is too slow

Feb 12 01:27:04 laptop gnome-shell[3967]: libinput error: event2 - Power Button: WARNING: log rate limit exceeded (5 msgs per 60min). Discarding future messages.

furthermore, my journalctl had also become corrupted at this time, which i saw in dmesg. 

[   12.906499] systemd-journald[440]: File /var/log/journal/0b9f13fbbe114b26929155c72177e0a1/user-1000.journal corrupted or uncleanly shut down, renaming and replacing.

What to look in my next experiment-

  • after how many forks does this fork fail, and at what time
  • cgroup limits enforced by ubuntu to limit number of processes

Comments

Post a Comment