On Binary Translation and its Consequences

extract

The PC market is one of the toughest areas for a CPU designer to compete in. Consumers in the PC segment expect high performance across a wide range of applications, and famously expect their devices to reach beyond a constrained, curated set of use cases. Microsoft’s own Surface RT prominently failed more than a decade ago because it could not run the programs that PC users expect to just work on Windows. Recent 64-bit Arm cores from Qualcomm and others are fast enough to satisfy the performance component of the PC equation, but software compatibility presents a tougher roadblock. PC software is traditionally built for x86-64. Getting developers to offer 64-bit Arm (aarch64) versions of their programs is a slow and gradual process. Some programs may never get aarch64 ports because they’re no longer under active development and were only distributed in binary form. Arm’s PC market chances therefore ride on Microsoft’s efforts to ensure x86-64 binaries can run seamlessly on aarch64 hosts. Windows 11’s newest binary translator, dubbed Prism, enables this by translating x86-64 instructions to aarch64 ones. Binary translation is challenging because x86-64 instructions sometimes don’t map to aarch64 ones in a straightforward manner. Translation also has to be fast to minimize program launch delays and avoid consuming excessive CPU time for doing the translation. All of this means binary translation comes with a performance penalty compared to running equivalent native code. Here, I’m checking out what that penalty looks like using Geekbench 7. Geekbench 7 comes in both aarch64 and x86-64 versions, providing an opportunity to compare performance with binary translation against native aarch64 performance. John Poole (Founder of Primate Labs, Creator of Geekbench) has kindly provided a pro key, which lets me profile individual workloads. Of course, one benchmark suite can’t reflect behavior across a wide range of applications, but I consider this a good starting point. As for hardware, I’m testing with the Snapdragon X2 Elite Extreme X2E-96-100 in an Asus Zenbook A16 laptop that was sampled by Asus as well as Arm instances available on Microsoft Azure. Geekbench 7 Instruction Characteristics Like Geekbench 6, Geekbench 7 has a number of workloads that leverage vector extensions when available. The suite is slanted towards high IPC, compute-bound workloads. Running workloads through Intel’s Software Development Emulator (SDE) set to expose Haswell’s feature set shows more than half the workloads using AVX and 256-bit vector width. I’m running each workload with 200 iterations to ensure the vast majority of counted instructions come from the workload rather than Geekbench’s test harness. With 200 iterations, each workload executes roughly a few trillion instructions. Performance counter data from various aarch64 cores show roughly similar instruction counts when executing Geekbench 7’s native aarch64 binary, with occasional exceptions. Qualcomm’s cores curiously report higher retired instruction counts than Neoverse N1 and N2 in many tests, which suggests there may be some inaccuracy in hardware performance monitoring. Executing Geekbench 7’s x86-64 version under binary translation results in dramatically inflated instruction counts across the board, except in PDF Viewer. A Geekbench 7 workload will execute roughly twice as many aarch64 instructions as x86-64 ones when going through binary translation, as a rule of thumb. Microsoft hints that binary translation doesn’t work the same way across all aarch64 CPUs. Different CPUs support different instruction set extensions, some of which may let Prism more closely map x86-64 instructions to aarch64 ones. An aarch64 CPU can also hypothetically provide guarantees beyond what the ISA requires, like maintaining store ordering even though aarch64 lets one core observe stores from another out of program order. Prism is optimized and tuned specifically for Qualcomm Snapdragon processors. Some performance features within Prism require hardware features only available in the Snapdragon X series, but Prism is available for all supported Windows 11 on Arm devices with Windows 11 24H2. If Prism is optimized specifically for Qualcomm’s cores, it doesn’t make a difference from the instruction count side. Ampere Altra’s Neoverse N1 is the oldest core I’ve tested on, and performance counters show very similar executed instruction counts. Similar instruction counts can of course hide large differences. Performance is ultimately what matters in the end, and that’s affected both by the core architecture and the nature of the generated instructions. Score Impact Geekbench 7’s score shows harsh penalties from binary translation. Every core, including Qualcomm’s, suffers several generations worth of performance loss. However, the cores in the Snapdragon X2 Elite have much higher baseline performance than Neoverse N1 and N2. The Snapdragon X2 Elite has a 6-core cluster of “performance” cores, and two 6-core cluster of “prime” cores. “Performance” cores are 6-wide and run at 3.6 GHz. Since they fill the same role as efficiency cores in other chips, I’ll call them E-Cores for simplicity. “Prime” cores are 9-wide and run at 5 GHz. They aim for maximum performance, so I’ll call them P-Cores. Qualcomm’s E-Cores manage to outperform Neoverse N1 even when taking a binary translation penalty. The same applies to the P-Cores against Neoverse N2. There’s really no arguing with much wider cores running at very high clock speeds, even with binary translation penalties in play. Qualcomm’s cores also take a slightly lower penalty from binary translation. It’s hard to tell whether this is down to Qualcomm-specific optimizations in Prism, or whether it’s down to Qualcomm’s cores having much higher throughput. Instruction count increases from binary translation remind me of playing with Claude’s C Compiler (CCC). CCC’s case involved creating a lot of extra instructions off the critical path, which an out-of-order CPU can often absorb. Recent high performance cores usually leave most of their core width unused, so extra core width can mitigate higher instruction counts to some extent. But I’m not sure that’s the case here, because Qualcomm’s 4-wide E-Core comes off with a lighter penalty than 5-wide Neoverse N2. Individual workloads suffer to varying degrees when run under binary translation. Navigation is a low IPC workload that’s heavily bound by branch prediction and backend memory latency, and takes a comparatively low penalty when run under binary translation. At the other end of the spectrum, Video Player is a well vectorized workload that takes advantage of AVX instructions. Score differences in that test are absolutely massive. Photo Editor and Photo Library are in a similar situation. Workload characteristics under binary translation stay mostly the same on Arm’s Neoverse N1 and N2. Core Throughput (IPC) Performance counters show Qualcomm’s cores putting a dent in binary translation overhead by dipping into unused core width. However, the IPC increase is minor compared to the instruction count overhead, which explains the large binary translation penalty. Snapdragon X2 Elite Extreme’s P-Core manages a huge IPC increase in the Video Player workload, but gains elsewhere tend to be muted. Qualcomm’s E-Core is in a similar situation, though as a narrower core it’s able to make much better use of its core width. Text Processing, HDR, and Video Player basically have the core running up against its 4-wide width limitation when running under binary translation. However, the meager IPC increases seen on Qualcomm’s P-Core in Text Processing and HDR suggest that widening the E-Core might do little, as the workload might soon run into other limitations. Neoverse N2 starts off at lower IPC and curiously fails to gain much IPC when running x86-64 under binary translation. The core performs fine in an absolute sense when looking at IPC for native workloads, but it doesn’t look great next to Qualcomm’s latest efficiency optimized core. No workload on Neoverse N2 can average above 3 IPC, with or without binary translation in play. Neoverse N1 is the oldest core in this set, and unsurprisingly has the hardest time. N1 in Ampere Altra spearheaded Arm’s push to get a foothold in the server market, and succeeded because the core could deliver adequate performance while being more density optimized than its x86-64 contemporaries. But that was years ago. Qualcomm’s E-Core is also 4-wide, and leaves it in the dust. Comparatively good performance from Qualcomm’s E-Core serves as a reminder that core width means relatively little compared to other architecture characteristics. Qualcomm gives their modern E-Core a much larger out-of-order engine than Neoverse N1. Doing so lets the core keep more instructions in flight, reducing the impact of short duration delays on individual instructions. Top-Down Analysis Accounting for dispatch-stage pipeline throughput shows a largely similar picture for workloads running natively and under binary translation, which isn’t a surprise because they should be doing the same high level work. On Qualcomm’s P-Core, the biggest difference is more utilized pipeline slots, along with some movement towards being less frontend bound. Qualcomm’s narrower E-Core uses a huge portion of its core width in many tests, especially the x86-64 versions running under binary translation. A narrower core is much easier to feed, and pipeline slots lost to frontend reasons are far fewer compared to on the wider P-Core. Detailed performance monitoring is harder on Azure because the platform doesn’t pass through top-down counters that account at the pipeline slot granularity. Roughly sketching things out with cycle-level accounting (STALL_FRONTEND and STALL_BACKEND) however shows that Neoverse N2 loses more pipeline slots to both backend and frontend reasons than Qualcomm’s E-Core. On one hand, feeding a 5-wide pipeline is harder than a 4-wide one. Also, Qualcomm has larger core-private caches and a sizeable 12 MB L2 for its E-Core cluster with 21 cycle latency. Missing the 1 MB L2 on Neoverse N2 exposes the core to ~100 cycle L3 latency, which will be difficult for any core to cope with. Binary translation doesn’t change much on that front. Backend stalls are even more of a problem on Neoverse N1, which also has to contend with >100 cycle L2 miss latency. N1 is more severely affected because of its smaller out-of-order engine. A comparison with Qualcomm’s E-Core shows just how rough the situation is for Neoverse N1. Qualcomm’s core is designed to keep more than twice as many instructions in flight, even though it has the same core width and similar execution resources. Extra instructions from binary translation increase pressure on out-of-order resources, and Qualcomm’s beefier core has a better chance of absorbing that overhead. Translated Code Prism stores translated code in C:\Windows\XtaCache. Writing generated code to disk serves two functions. First, it lets Prism avoid the overhead of re-translating code every time a binary gets launched. Second, it lets Windows users examine translated code to understand how x86-64 binaries are able to run on aarch64 hosts. This is important because it lets them empathize with the hardware. XtaCache’s efficacy on the latter point has been limited because Microsoft forgot to document their .jc cache format, but thankfully others have stepped in to do so. Unfortunately, digging through translated code is still a very time consuming exercise. So here, I’ll stick to giving a couple of examples to give an idea of what Prism’s translation looks like. Geekbench 7’s Video Player workload stands out because it runs at much higher IPC when under binary translation, and also suffers from a huge executed instruction count increase. Profiling Video Player on Skylake with VTune shows that the hottest basic block is a 17 instruction loop. This loop traverses an array and performs fused multiply-add operations using AVX FMA3 instructions. Prism translates this to a 69 instruction sequence on both Neoverse N1 and Snapdragon X2 Elite. AVX instructions get mapped onto NEON ones, which only have 128-bit vector width. Qualcomm’s latest cores support SVE, but using it likely wouldn’t make much difference because Qualcomm’s cores implement SVE with 128-bit vector length. Also, fixed width x86-64 instructions wouldn’t map well to SVE’s variable vector length anyway. Instruction overhead comes from performing 256-bit vector operations using multiple NEON instructions, replicating x86-64’s more flexible addressing modes, and having to perform memory accesses with separate instructions rather than specifying both a load and a math operation with a single instruction. For example, vbroadcastss ymm8, [r8+r10*4] loads a single precision scalar (32-bit) value and broadcasts it across all vector lanes. Prism uses three aarch64 instructions. One loads the value, another replicates it across a 128-bit NEON register’s lanes, and a final one copies the value to another NEON register that represents the upper 128 bits of ymm8. Fused multiply-add instructions get even more expansion. vfmadd231ps ymm7, ymm8, [rax + r9*4 - 0x13c] uses x86-64’s most complex addressing mode, with a base, scaled index, and immediate offset. Prism does the addressing calculation with two instructions, and is able to mash both the index scaling and addition to the array base address into a single add-with-scale. Then, a load-pair (LDP) instruction specifies a 256-bit load into a pair of 128-bit NEON registers. Prism carries out the FMA operation with a pair of FMLA instructions, but curiously spills NEON register holding the high half of ymm7 to the stack. It does this again for all eight of the loop’s FMA instructions, which doubles cache bandwidth demands. None of the spilled registers are modified before the next loop iteration, so spilling and reloading them is completely unnecessary, and potentially harmful in a loop that already has high bandwidth demands. Toward the end of the loop body, Prism faithfully translates software prefetch instructions into aarch64 equivalents, including extra ALU instructions to generate addresses. Then, loop counter increment and control flow instructions translate directly into aarch64 equivalents, though of course with a different jump target address. A human translator would avoid Prism’s unnecessary register spills. They could also omit prefetch instructions, because linearly traversing an array creates a predictable access pattern that a hardware prefetcher would likely pick up on. They could also eliminate a few instructions here and there, like by using a single register to hold both halves of ymm8 because the broadcast load’s destination is never modified. Prism does deserve credit though for simplifying address generation for six out of the eight FMA operations by emitting just a subtraction, recognizing that rax + r9*4 has already been generated. I also took a peek at Geekbench 7’s HDR workload because its binary translated version barely gains any IPC. The hottest block from profiling on Skylake is a 36 instruction loop with scalar AVX operations. That should make it easier to translate because Prism won’t have to map AVX’s wider vector width onto NEON. Prism translates this into somewhere over 60 instructions. The complete translation didn’t seem to be available in the translation cache when I grabbed it from my Neoverse N1 VM. Prism introduces significant instruction overhead by trying to maintain a register mapping where x86 SSE registers map to the equivalently numbered aarch64 NEON register. For example, within the instruction sequence Geekbench 7 does a FP add (vaddss xmm5, xxm4, [rdi+rbx*4]) and stores the result to memory (vmovss [rdi+rbx*4], xmm5). Prism’s aarch64 translation puts the FP add result in s18, and then moves it to v5 before storing it in order to use v5 to represent xmm5. Additional overhead comes from inexplicable stores to the stack, as well as extra instructions that seemingly generates the carry flag for a right shift instruction. Many x86-64 instructions set flags, and not all have flag-setting equivalents in aarch64. Prism seems to prefer preserving the flag generating behavior rather than expend compute power seeing if the flags are used. Prism generates mostly the same sequence for Qualcomm’s Snapdragon X2 Elite, but with differences around loads and stores. Qualcomm’s newer cores support FEAT_LPCPC2, which lets it use ldapurb (load-acquire with an immediate offset) instead of ldaprb (load-acquire). Being able to take an immediate offset simplifies address generation, cutting out some extra instructions. In both cases, Prism translates loads to load-acquire instructions rather than aarch64’s basic LDR to ensure memory ordering. Final Words Binary translation is key to making Arm on Windows viable in the PC market. Prism’s translations are mostly straightforward. x86-64 instructions get their behavior faithfully replicated by equivalent aarch64 sequences, so much that it’s possible to look through translated code and easily see which sequences map to x86-64 instructions. I suspect Prism does this to avoid consuming excessive compute power, because what it’s doing looks achievable with a linear pass through each block with minimal state kept in memory. However, doing the translation in such a straightforward manner misses potential optimizations that a human could carry out. Replicating x86-64 addressing in aarch64 often requires extra instructions. x86-64 compilers also take advantage of instructions that combine a load and math operation. Those have to be broken into two instructions in aarch64, and may require using an extra temporary register to hold the loaded value. On the hardware side, high performance cores can often absorb dead code with unused core width. That sometimes happens when Prism generates dead code, but Prism doesn’t generate excessive amounts of dead code the way Claude’s C Compiler does. Instruction overhead from replicating address generation or copying values between registers to maintain Prism’s register mapping can easily lengthen critical dependency chains. There’s not much an out-of-order engine can do about that. The ultimate answer to binary translation’s consequences is to have a high performance architecture with few weaknesses in the first place. Qualcomm seems to be going for that with their latest Snapdragon X2 Elite. A 9-wide core running at 5 GHz is nothing to sneeze at. Large instruction caches (128 KB on the E-Core, 192 KB on the P-Core) and ample out-of-order resources are good for performance in general, and crucially help mitigate the instruction overhead from binary translation. I’m still trying different use cases on the Asus Zenbook A16 UX3607OA laptop, but it feels surprisingly normal to use outside of a few corner case scenarios like gaming. I assume the Windows on Arm situation will only get better over time as more applications get aarch64 ports, and higher performance cores mean performance gets more acceptable even when applications have to run through binary translation.