JVM Anatomy Quark #12: Native Memory Tracking

"JVM Anatomy Quarks" is the on-going mini-post series, where every post is describing some elementary piece of knowledge about JVM. The name underlines the fact that the single post cannot be taken in isolation, and most pieces described here are going to readily interact with each other.

The post should take about 5-10 minutes to read. As such, it goes deep for only a single topic, a single test, a single benchmark, a single observation. The evidence and discussion here might be anecdotal, not actually reviewed for errors, consistency, writing 'tyle, syntaxtic and semantically errors, duplicates, or also consistency. Use and/or trust this at your own risk.

Aleksey Shipilëv, JVM/Performance Geek
Shout out at Twitter: @shipilev; Questions, comments, suggestions: aleksey@shipilev.net

I have 512 MB of available memory, so I set -Xms512m -Xmx512m, and my VM fails with "not enough memory to proceed". Why?

JVM is the native application, and it also needs memory to maintain its internal data structures that represent application code, generated machine code, heap metadata, class metadata, internal profiling, etc. This is not accounted in Java heap, because most of those things are native, allocated in C heap, or mmap-ed to memory. JVM also prepares lots of things expecting the active long-running application with decent number of classes loaded, enough generated code created at runtime, etc. The defaults may be too high for short-lived applications in memory-constrained scenarios.

OpenJDK 8 onwards has a nice internal VM feature, called "Native Memory Tracking" (NMT): it instruments all internal VM allocations and lets categorize them, get the idea where they are coming from, etc. This feature is invaluable for understanding what VM uses memory for.

NMT can be enabled with -XX:NativeMemoryTracking=summary. You can get jcmd to dump the current NMT data, or you may request the data dump at JVM termination with -XX:+PrintNMTStatistics. Saying -XX:NativeMemoryTracking=detail would get the memory map for mmaps and callstacks for mallocs.

Most of the time, "summary" suffices for the overview. But we can also read the "detail"-ed log, see where allocations are coming from or for what, read the VM source code, and/or play with VM options to see what affects what. For example, take a simple "Hello World" application, like this:

It is obvious that Java heap takes a significant part of allocated memory, let’s trim -Xmx16m -Xms16m, and see our baseline:

Okay. 75 MB for 16 MB Java heap is certainly unexpected.

Let’s roll over different parts of that NMT output to see if those parts are tunable.

Start with something familiar:

This accounts for GC native structures. The log says GC malloc-ed around 10 MB and mmap-ed around 0.6 MB. One should expect this to grow with increasing heap size, if those structures describe something about the heap — for example, marking bitmaps, card tables, remembered sets, etc. Indeed it does so:

Quite probably malloc-ed parts are the C heap allocations of task queues for parallel GC, and mmap-ed regions are the bitmaps. Not surprisingly, they grow with heap size, and take around 3-4% from the configured heap size. This raises deployment questions, like in the original question: configuring the heap size to take all available physical memory will blow the memory limits, possibly swapping, possibly invoking OOM killer.

But that overhead is also dependent on the GC in use, because different GCs choose to represent Java heap differently. For example, switching back to the most lightweight GC in OpenJDK, -XX:+UseSerialGC, yields this dramatic change in our test case:

Note this improved both "GC" parts, because less metadata is allocated, and "Thread" part, because there are less GC threads needed when switching from Parallel (default) to Serial GC. This means we can get partial improvement by tuning down the number of GC threads for Parallel, G1, CMS, Shenandoah, etc. We’ll see about the thread stacks later. Note that changing the GC or the number of GC threads will have performance implications — by changing this, you are selecting another point in space-time tradeoffs.

It also improved "Class" parts, because metadata representation is slightly different. Can we squeeze out something from "Class"? Let us try Class Data Sharing (CDS), enabled with -Xshare:on:

There we go, saved another 0.5 MB in internal symbol tables by loading the pre-parsed representation from the shared archive.

Now let’s focus on threads. The log would say:

Looking into this, you can see that most of the space taken by threads are the thread stacks. You can try to trim the stack size down from the default (which appears to be 1M in this example) to something less with -Xss. Note would yield a greater risk of StackOverflowException-s, so if you do change this option, be sure to test all possible configurations of your software to look out for ill effects. Adventurously setting this to 256 KB with -Xss256k yields:

Not bad, another 4 MB is gone. Of course, the improvement would be more drastic with more application threads, and it will quite probably be the second largest consumer of memory after Java heap.

Continuing on threading, JIT compiler itself also has threads. This partially explains why we set stack size to 256 KB, but the data above says the average stack size is still 7517 / 11 = 683 KB. Trimming the number of compiler threads down with -XX:CICompilerCount=1 and setting -XX:-TieredCompilation to enable only the latest compilation tier yields:

Not bad, three threads are gone, and their stacks gone too! Again, this has performance implications: less compiler threads means slower warmup.

Trimming down Java heap size, selecting appropriate GC, trimming down the number of VM threads, trimming down the Java stack thread sizes and thread counts are the general techniques for reducing VM footprint in memory-constrained scenarios. With these, we have trimmed down our 16 MB Java heap test case to:

Moving to insane parts, which involve tuning down internal VM settings. This is not guaranteed to work, and may crash and burn unexpectedly. For example, we can control the stack sizes required for our Java application by coding it carefully. But we don’t know what is going on inside the JVM itself, so trimming down the stack size for VM threads is dangerous. Still, hilarious to try with -XX:VMThreadStackSize=256:

Ah yes, another 2 MB are gone along with compiler and GC thread stacks.

Let’s continue abusing the compiler code: why don’t we trim down the initial code cache size — the size of area for generated code? Enter -XX:InitialCodeCacheSize=4096 (bytes!):

Ho-ho! This will balloon up once we hit heavy compilation, but so far so good.

Looking closer to "Class" again, we can see that most of the 4 MB committed for our Hello World is the initial metadata storage size. We can trim it down with -XX:InitialBootClassLoaderMetaspaceSize=4096 (bytes!):

Overall, after all this madness, we came even closer to 16 MB of Java heap size, wasting only 8.5 MB on top of that:

We can probably come even closer if we start yanking parts of JVM in our custom build.

For fun, we can see how the native overhead changes with heap size on our test workload:

This confirms our gained intuition that GC overheads are the constant factor of Java heap size, and native VM overheads starts to matter only on lower heap sizes when the absolute values for VM overhead start to become a factor in overall footprint. This picture omits the second most important thing though: thread stacks.

Default JVM configuration is usually tailored to long-running server-class applications, and so its initial guesses about the GCs, the initial sizes for internal data structures, stack sizes, etc. may be not appropriate for short-running memory-constrained applications. Understanding what are the major memory hogs in current JVM configuration helps cramming more JVMs on the host.

Using NMT to discover where VM spends memory is usually an enlightening exercise. It almost immediately leads to insights where to get memory footprint improvements for the particular application. Hooking up online NMT monitor to performance management systems would help to adjust the JVM parameters when running actual production applications. This is much, much, much easier than trying to figure out what JVM is doing by parsing the opaque memory maps from e.g. /proc/$pid/maps.

Also see "OpenJDK and Containers" by Christine Flood.