Please Test Your (Computer) Memory

JVMs are colloquially known to be memory hungry. But in many cases, that is the side effect of heap sizing policy that prefers to expand the heap instead of trying more aggressive (often stop-the-world) GCs. In concurrent GCs, this gets easier, because GC does not stop the JVM all that much. But there are throughput tradeoffs anyway: keeping CPUs busy with GC instead of whatever application is doing does have costs.

Many configurations choose to set large -Xmx (max heap size) to plan for extended capacity, accept allocation spikes, etc. Many other configurations additionally set large -Xms (initial/min heap size) to avoid heap resizing hiccups, that might involve going to OS for memory and/or pre-zeroing it. Some configurations run with -XX:+AlwaysPreTouch to get OS to commit all heap pages to physical memory right away.

When JVM is the sole process running on a host, it is common to see -Xmx to be close to the physical memory limits, minus native JVM overheads. Because apart from leaving the space for I/O caches, why keep that memory unused?

This has implications for memory errors. If a JVM process takes a lot of memory, it means its Java heap is mapped over the most of the physical memory. Which means that if there is a bad memory cell somewhere, it has a very high chance to be used for the Java heap.

If you haven’t read anything about how JVM represents objects, consider that JVM have the metadata with every object that describes its system state, including sizes, types, GC metadata, etc. Interested readers are advised to read "Java Objects Inside Out" for more discussion.

There are two notable statistical properties of common Java objects: they are usually small, and they are usually reference-rich. This translates to having a lot of object metadata overhead (mark and class words) and lots of pointers in memory (as the straight-forward representation of the Java reference).

To see how dense metadata/pointers are, let us consider the idiomatic example:

If we initialize it with something like new Person("Aleksey", "Shipilev"), then the memory representation of the entire ensemble of objects behind it would be something like:

Look at this contrived case: out of 120 bytes total, there are 68 bytes of metadata and 16 bytes of references. In total, there are 84 out of 120 bytes carrying the "system" data. And these "system" pieces are carried along with the rest of objects' "user" data, which means system data permeates the entire Java heap.

Again, this has implications for memory errors. The bit flip somewhere in user data — for example, in a large byte[] array — would probably go unnoticed unless it breaks specially checked applications invariants, or the application logic puts the additional verifications.

A bit flip in system data breaks the JVM’s own invariants. A bit flip in object metadata breaks the Java heap integrity: change bits in markword and GC gets confused about object state; change bits in classword, and the whole runtime is confused about the type of the object; change bits in arraylength, and generated code, runtime, GCs access something beyond the array.

To our benefit, most of those failures are more or less easily detectable: corrupting the heap usually leads to GC crashes when it walks the heap, bad memory accesses usually access something at odd pointer values, outside the mapped memory (thus, "Segmentation Fault"), etc.

Combined with the previous observation, not only you have a high chance to have a bad memory cell holding the Java heap, there is also a high chance it holds the system bits of data, and the failure breaks the program in an more or less observable way.

Why would the bit-flips in metadata/pointers be readily observable?

Sure, the application itself uses the data, for example it goes through the references every so often. But you might be lucky to never visit the broken object with the application code — for example if your application has lots of "cold" data, or it is under light load, etc.

But the tracing GCs themselves do visit a lot of objects. In our toy example above, these are the locations that a usual tracing GC would touch:

Outgoing references are routinely touched during the traversal of the object graph: we need to know which other objects the current object points to. Class words would be used to figure out the object sizes for either heap walks (due to heap parsability) or to see how much data to copy (for evacuation). Mark words would be used to see if the object has a forwarded copy already.

If any of those pieces of system data are bad, that is the (lucky!) opportunity for GC to crash.

There are two mitigating factors:

So we frequently have a "perfect storm":

BAM! GC crash.

And since GC code is usually small but visiting large swaths of physical memory, you would see roughly the same crash, even if the memory errors are actually random. The only difference would be subtle changes in the failing memory addresses.

As the industry, we know that memory errors are common. An old, but widely cited paper ballparks the incidence rate at 25 events per Gbit per year. If we boldly assume the uniformity, then on a 128 GB machine, that amounts to about 3 events per hour!

In fact, we do regularly protect L1/L2 caches with ECC:

For years, ECC for main memory was strictly in the realm of enterprise servers. But today, you can get decent ECC support for desktop platforms as well, and sometimes you can even see vendors qualifying their desktop systems for ECC use. These days, I never even consider going for high-density DIMMs on any important system without going ECC. All my "production" home machines run with ECC modules.

Since ECC is online most of the time, it can help against soft errors, that are temporary upsets that do not indicate the hardware failure (usual example: background radiation upsets). These soft errors are very hard to reliably reproduce due to their physical nature. The continuous memory tests catch the hard errors more or less reliably: once something is broken in hardware, it has a much higher chance to replicate well. Catching the soft errors with continuous memory tests might be problematic.

Running memory scrubbing on ECC systems is also recommended, since error correction codes are usually SECDED (Single Error Correction, Double Error Detection). Which means, if you have the single-bit upset in cold data somewhere, you want ECC to detect it very early, while it is still a single-bit error. The risk of adjacent memory error might be low, as these failures are rarely correlated, but better be safe than sorry.

While memtest86+ is the go-to tool for testing memory, I like memtester(8) much better. Mostly because it does not require to compiletely offline the machines while testing is performed. This is very convenient for lab servers that do not have an easily attachable physical/virtual console. It is nice to go to a lab server, shut down all services like build agents, and then run memtester under a screen, until it completes or you get bored.

You would need to have a trial/error session to figure out how much memory the test can take and still leave the responsive system. On my 128G graphical desktop, 120G seems to fit. On my 128G bare lab server, 125G can be tested.

The whole thing takes many hours, depending on memory size, memory speed, etc. This round took about 6 hours on my desktop:

I personally do this after any substantial hardware change or every 3 months, whatever comes first.

As noted before, these tests are not very useful to catch soft errors, but they do catch hard errors reasonably well.

The gold standard for memory testing is memtest86+. The thing I like about running it separately is multi-threading support. This might uncover more bugs. In practice, most of the faulty memory modules I had were failing in the single threaded mode too.

Every vendor I bought the memory from accepted the RMA with a short description like "memtest86+ is failing on this module". This seems to be a lingua franca in the testing world: if the module does not pass memtest86+, that is a real problem. In very rare cases the module itself is not a problem: mainboard is flaky, PSU is flaky, CPU is bad, etc.

memtest86+ seems to be the part of every serious pre-deployment checklist. I think it should be the part of every suspicious crash report checklist too, or at least it should be ran periodically to catch the memory errors early without running long memtest after any crash.