This is how one 128TB file misled my high disk usage investigation.

The reasoning behind this article

Recently, I was dealing with a rather weird issue: an EC2 instance which ran a dockerized environment was starting to consume a lot of space on the root partition. It usually is a normal thing if the project uses a lot of data, such as static images, dependencies, etc. although this kind of data is stored on an EFS. Logs also weren't an issue, since log rotation (and implicitly retention) is in place, and also container logs wouldn't be an issue since it runs in a swarm environment and the project-related containers are recreated once a new commit is deployed and the docker system is cleared after the deployment is done.

Basically, this means that on the root partition, other than a few utilities and docker images for the latest iteration of the project, no additional space is used. I've also used ncdu to visualize the disk usage since I had no other ideea what could be the cause, but the math wasn't adding up. ~70GB that were nowhere to be found.

Q: So since this project barely uses any space on the root partition, what was the cause of the high disk usage?
A: One long-running process.

To understand this, let's first briefly explore how processes on a Linux system work.

A brief explanation of Linux processes

By now, you've probably heard of the phrase "In Linux, everything is a file", and it is true. Everything from hardware to software on the system is a file, including processes. They live under /proc, under a specific structure:

$ tree /proc
/proc
├── 1
│   ├── attr
│   │   ├── apparmor
│   │   │   ├── current
│   │   │   ├── exec
│   │   │   └── prev
│   │   ├── current
│   │   ├── display
│   │   ├── exec
│   │   ├── fscreate
│   │   ├── keycreate
│   │   ├── prev
│   │   ├── smack
│   │   │   └── current
│   │   └── sockcreate
│   ├── autogroup
│   ├── auxv
│   ├── cgroup
│   ├── clear_refs
│   ├── cmdline
...

Each file is responsible for a part of the process, like cmdline shows what command the process runs, cwd is a symlink to the directory in which the process is currently working, fd contains symlinks to file descriptors (when you open a file, the kernel has to save somewhere the information about it being opened, this where it does) , and so on and so forth. Also, each folder in /proc is named by the PID of the process. It is actually easy to understand what each file does in a process does, the structure is rather simple, but what about the files in the root of /proc?

Here are the files you would usually find:

$ find . -type f -maxdepth 1 | xargs ls -alh
-r--r--r--. 1 root root    0 Dec 14 19:32 ./bootconfig
-r--r--r--. 1 root root    0 Dec 14 19:32 ./buddyinfo
-r--r--r--. 1 root root    0 Dec 13 23:42 ./cgroups
-r--r--r--. 1 root root  136 Dec 14 15:46 ./cmdline
-r--r--r--. 1 root root    0 Dec 14 19:32 ./consoles
-r--r--r--. 1 root root    0 Dec 13 23:56 ./cpuinfo
-r--r--r--. 1 root root    0 Dec 14 19:32 ./crypto
-r--r--r--. 1 root root    0 Dec 13 17:37 ./devices
-r--r--r--. 1 root root    0 Dec 14 19:32 ./diskstats
-r--r--r--. 1 root root    0 Dec 14 19:32 ./dma
-r--r--r--. 1 root root    0 Dec 14 19:32 ./execdomains
-r--r--r--. 1 root root    0 Dec 14 19:32 ./filesystems
-r--r--r--. 1 root root    0 Dec 14 19:32 ./interrupts
-r--r--r--. 1 root root    0 Dec 14 19:32 ./iomem
-r--r--r--. 1 root root    0 Dec 14 19:32 ./ioports
-r--r--r--. 1 root root    0 Dec 14 19:32 ./kallsyms
-r--------. 1 root root 128T Dec 14 00:19 ./kcore
-r--r--r--. 1 root root    0 Dec 14 19:32 ./keys
-r--r--r--. 1 root root    0 Dec 14 19:32 ./key-users
-r--------. 1 root root    0 Dec 14 19:32 ./kmsg
-r--------. 1 root root    0 Dec 14 19:32 ./kpagecgroup
-r--------. 1 root root    0 Dec 14 19:32 ./kpagecount
-r--------. 1 root root    0 Dec 14 19:32 ./kpageflags
-rw-r--r--. 1 root root    0 Dec 14 19:32 ./latency_stats
-r--r--r--. 1 root root    0 Dec 13 15:40 ./loadavg
-r--r--r--. 1 root root    0 Dec 14 19:32 ./locks
-r--r--r--. 1 root root    0 Dec 14 01:01 ./mdstat
-r--r--r--. 1 root root    0 Dec 13 17:37 ./meminfo
-r--r--r--. 1 root root    0 Dec 14 19:32 ./misc
-r--r--r--. 1 root root    0 Dec 14 19:32 ./modules
-rw-r--r--. 1 root root    0 Dec 13 15:39 ./mtrr
-r--------. 1 root root    0 Dec 14 19:32 ./pagetypeinfo
-r--r--r--. 1 root root    0 Dec 14 19:32 ./partitions
-r--r--r--. 1 root root    0 Dec 14 19:32 ./schedstat
-r--------. 1 root root    0 Dec 14 19:32 ./slabinfo
-r--r--r--. 1 root root    0 Dec 14 19:32 ./softirqs
-r--r--r--. 1 root root    0 Dec 13 15:40 ./stat
-r--r--r--. 1 root root    0 Dec 13 17:37 ./swaps
--w-------. 1 root root    0 Dec 14 19:32 ./sysrq-trigger
-r--------. 1 root root    0 Dec 14 19:32 ./timer_list
-r--r--r--. 1 root root    0 Dec 14 19:32 ./uptime
-r--r--r--. 1 root root    0 Dec 14 19:32 ./version
-r--------. 1 root root    0 Dec 14 19:32 ./vmallocinfo
-r--r--r--. 1 root root    0 Dec 14 19:32 ./vmstat
-r--r--r--. 1 root root    0 Dec 13 19:10 ./zoneinfo

Most of the files you find here actually represent the output for utilities you might frequently use:

• version contains the output of what uname -a would parse
• cpuinfo contains the output of what lscpu would parse
• same goes for uptime, partitions, swaps, etc.

So a lot of actually useful information which you can retrieve by yourself outputting the contents of the file. If your GNU/Linux distribution doesn't include utilities such as lscpu or lshw, most of the stuff you want you can find it here.

There is a specific file which really caught my attention since it seems like it uses a ton of disk space:

-r--------. 1 root root 128T Dec 14 00:19 ./kcore

Wait, 128 terabytes? How is that even possible? Your disk may not even be 128T (if it is, then you most likely store a ton of "Linux ISOs").

My instinct was to instantly blame this file, until I digged into what it is, what it does and if it is the cause of my problem.

Hear me out, this isn't actual space taken on the disk, because all the stuff you see in /proc, isn't actually on the disk.

Enter procfs

All of the stuff you see in /proc isn't actually on the disk, because /proc is a pseudo-filesystem. Like any filesystem, it isn't necessary to mount it on /proc, you can mount it anywhere you wish to, although typically it is automatically mounted at boot by the system with most of the files are read-only. Those files which are also writable allow the system administrator to change kernel variables.

So, back to kcore. According to the Linux man-pages: It is a pseudo-file which represents your system's virtual memory layout. If you have an unstripped kernel binary, you can use gdb to examine the current state of any kernel data structures.

Although there is one phrase in the man-page which doesn't make sense:

The total length of the file is the size of physical memory (RAM) plus 4 KiB.

This doesn't make any sense. That EC2 instance has only 8GB RAM. Where do the 128TB come from?

To understand this, lets look at the virtual memory map of the kernel. You can find it here.

A typical modern system uses 64bit address spacing. This means that allocated integers are 64bit wide. If we look at the virtual memory map, we can see the start address and the end address:

========================================================================================================================
    Start addr    |   Offset   |     End addr     |  Size   | VM area description
========================================================================================================================
                  |            |                  |         |
 0000000000000000 |    0       | 00007fffffffffff |  128 TB | user-space virtual memory, different per mm
__________________|____________|__________________|_________|___________________________________________________________

You can see that the size of the virtual memory map from the start address to the end address on a 64bit system running a Linux kernel, is 128TB. To understand why, you need to understand base-16 calculation (which I haven't done in years), but there is a pattern:

00007fffffffffff is basically 0x7fffffffffff, because the leading zeros are ignored. The F represents the number 15 in decimal, there is one 7 and 11 F's so the formula would be:

$$15x16^{11} + 15x16^{10} + 15x16^{9} + 15x16^{8} + 15x16^{7} + 15x16^{6} + ... + 15x16^{0}$$

which equals to:
$$140,737,488,355,327$$

And represented in powers of two:

$$2^{47} − 1$$

That $^{47}$ means that the user-space virtual memory is 47 address bits wide, stored in 64-bit addresses. This is the length for a vmap with 4-level page tables. If you look at the Documentation/x86/x86_64/mm.txt you can also see that there are 5-level page tables which have 56-bit layouts.

This is why /proc/kcore is 128TB, because it represents the kernel’s virtual address space. Remember that this is a pseudo file, more precisely an ELF core filesystem file. Most of it is unmapped memory.

So was kcore the issue?

In hindsight, it was tempting to blame /proc/kcore since it is indeed 128TB in size, knowing that most of the virtual address space is unmapped and the size of the file doesn't change, I was so sure it could be it. However, this is where correlation can easily be mistaken for causation.

Another important detail is that after checking the process, it was running since 13Sep, and it most likely kept a lot of file descriptors (remember fd) open. Hence, why after killing it, the disk usage went down dramatically. Those file descriptors were released. Even though I knew this detail, I was so intrigued by /proc/kcore that I was confident it was the issue.

Conclusion

If you remember one thing from this investigation, let it be this:

When disk usage is high but files are nowhere to be found, the problem is almost always processes, not filesystems.

Analyze your processes and see which ones were up for the longest and consumed a lot of virtual memory, they can keep the high disk usage. Check their file descriptors, if a process opens and deletes files, they are not really deleted unless the process dies. You can check this using lsof +L1 .