AMD Threadripper Build Server Efficiency

This new system had replaced the build server producing OpenJDK builds. Previous build server is i7-4790K, which is loaded ~60%, and sometimes flatlines at 100% load, even without aggressive test suites. So, whatever additional capacity the new build server brings, it can consume it all: we can add more testing if capacity is available.

Therefore, I am expecting to run the machine at full blast 24/7/365. Fast-forward, this is the machine 3 months in:

With residential electricity prices in Germany, power efficiency becomes the issue,^[1] and this is why I am interested in downclocking the Threadripper to improve efficiency at the cost of some throughput. Building full OpenJDK build in less than 4 minutes would be considered acceptable.^[2]

This is what we have:

Sensors:

Tests:

Video card takes around ~5 W, but it was inconvenient to run without it. So, both idle and active power figures include it. Wireless, Bluetooth, and Audio seem to take another 5 W combined, but those are disabled explicitly, and do not affect either idle or active power.

The discussion below references the lines in raw data, like this "[1]". The time and power alone do not tell much about the efficiency, although we would not like to raise either time to complete or power requirements all too much. The thing we want to optimize is amount of energy spent doing the job. The table above conveniently has it in kilojoules.

First of all, Tctl offset of +27 ^oC made me really concerned, when both bios and old Ubuntu installation reporting idle temperatures at +60 ^oC. I frankly thought I screwed up the thermal grease (this is a large CPU!), regreased, and the temperatures stayed the same. Then I remembered about this whole temp offset thing, and sure enough, recent Fedora installations report both Tctl and Tdie. Chip idles at 29 ^oC in 21 ^oC ambient, which is good.

With default settings, full load consumes quite a lot of power: up to 235 W at wall socket [1, 11]! Running it for a year at full load was going to cost me around 600 EUR, which I can spend better elsewhere. It also runs quite loud (more on this later), but not too hot.

Precision Boost settings in BIOS allows to manually override the wattage the platform is allowed to pump to the CPU package: Package Power Threshold (PPT). While I have seen most people increasing it to provide the headroom for overclocking, it can also be tuned down to deprive the package of power. In that case, the system starts to throttle. This can be seen with effective frequency going down.

Before messing with PPT, it seems that disabling Core Boost kicks out some steam from CPU, by disallowing it to get to frequencies above the nominal. It seems to correlate with significant reduce in power and the subjective loudness of the cooling system. Compare [1] with [6]: there are some benefits in allowing Core Boost for workloads with varying parallelism like OpenJDK build, and less benefit for flatlined. But, both power goes down, and efficiency goes up when CoreBoost is disabled. With 1.48x less power, we get only 1.10x longer builds.

Further trimming the PPT cuts the power and reaches the peak efficiency at around PPT=75..100W, see [2..5]. This adds up to the CB disabling nicely [7-10]. It would seem that running the workloads that exibit significant low-parallelism phases would be fine with CB enabled: while full package power is still limited, it can boost up individual cores to get the extra kick. Unfortunately, that seems to correlate to significantly higher Tdie, even though the power itself rises not very significantly, see for example [4] and [9].

Zen microarchitecture was reported to bind the internal interconnect frequency to the memory speed. Which apparently means that running at higher XMP profile makes Threadripper faster? This corroborated by experiments, see [1] and [11]. Running with XMP on stock settings does improve performance more than it degrades power, so overall efficiency gets higher.

However, this starts to bite us back as we try to downclock the thing. There, it seems that elevated power requirements for internal interconnect eat up the PPT budget and CPU starts to throttle much more aggressively, see [13..16]. The funny data point is [16] in comparison with [10], where CPU cores were apparently so deprived of power to run, the build times tanked 2.3x. Which, in turn, foobared the efficiency, because interconnect still consumed the same relatively huge amount of power for much longer time.

XMP profile seems to also increase idle power considerably.

After the first iteration of this experiment, I configured the server to run in configuration [9]: it runs decently fast (1.17x slower than stock) with much less power (2.1x lower than stock), which means 1.8x improvement in overall efficiency. Not only it makes the system whisper-quiet, even in absolute silence of the night, this also saves me around 300 EUR/yr in running costs, and keeps enough headroom for higher ambient temperatures in summer, and in denser racks. In fact, I was buying 2950X thinking to keep the case in the study room for better airflow, but now I wonder if with 110W power I might as well tuck it in the same storage closet as my old server, hm.

Other lessons learned. First, 750W PSU is probably way too much for this kind of system — and lower rated PSUs would reach their efficiency peaks at my target load. I think current PSU wastes around 5..7W right now, probably no biggie. Second, next time I can save up and not buy fast memory — I think I overpaid around 250 EUR on current one, without running it with XMP. Or I can find some solitude in believing that running memory below what’s it capable of would give me some reliability benefits.