loading
loading
In 1990, Eugene Brooks of Lawrence Livermore told the Supercomputing conference that nothing would withstand the attack of the killer micros. “Micro” referred to microprocessor, meaning a general-purpose chip that did a good job at most types of computation. They were also being sold by the millions per year into desktop PCs, delivering faster and faster performance each year. Around that same time, competing specialized architectures, such as vector processors, massively parallel processors (MPPs), and Lisp machines started dying off en masse. ETA systems closed in 1989, Multiflow in 1990, Alliant in 1992, Symbolics in 1993, Kendall Square Research and Thinking Machines (not the same one as today) in 1994, and Cray Computer in 1995. Danny Hillis, the founder of Thinking Machines, remarked years later that “…ultimately these things were just going to be boxes of microprocessors.” Interestingly, these specialized chips didn’t lose because they were superior at unimportant problems. The general-purpose microprocessor was actually outperforming them, not just on a wide spectrum of workloads, but on the very workloads the specialized chips were designed for. And although the companies that made them didn’t make it, the ideas they brought lived on inside other chips. CPUs absorbed vector processing as SSE and AVX, and massively parallel designs influenced both multi-core CPU and GPU architectures. The pattern at the time was that the specialists came up with good ideas, but the generalist chips absorbed those ideas into a single architecture. Today, we are in the middle of another heyday for the industry. AI workloads are different enough from traditional software workloads that the industry is in the midst of rethinking what computing should actually look like, with many different new chips and ideas behind them. There is a massive explosion of new accelerators to serve AI workloads, and GPUs generate more revenue for chip companies than CPUs do (although CPUs are also in their own demand crunch right now). In times like this, it’s interesting to look back on the last hardware boom and see what has changed since last time (and what remains the same), to better predict what will happen next. Is this time different for specialized chips, or will they ultimately be absorbed into general purpose chips like before? Why general-purpose chips won (for a time) Why were the “killer micros” beating specialized hardware - at the very workloads the specialists were designed for - back in the 90s? The general-purpose microprocessor was able to leverage a virtuous cycle: generality drove volume, volume drove performance, which in turn drove generality. 1. Generality drives volume The microprocessor was sold by the tens of millions into desktop PCs, being general-purpose hardware that could run every workload a PC had. The specialist hardware, on the other hand, had much lower volumes because it only ran one. 2. Volume drives performance Those volumes translated into much lower costs, more buyers, and ultimately more money to fund development for the microprocessor. From 1986 to 2003, Hennessy and Patterson estimate that single-processor performance grew over 50% year over year, most of which was delivered by faster clock speeds. Moore’s law scaling delivered a free lunch during that period - that is, free for anyone who can continually fund leading edge fabs and large chip design teams to take advantage of every new generation. The specialists with their lower volumes could not afford the same cadence, and therefore fell behind. In addition, the generalists were able to absorb some of the best ideas of the specialists once the specialist workloads were better understood (see vector processing becoming a capability of CPUs). 3. Performance drives generality The faster clock speeds largely enabled by transistor scaling were a rising tide that lifted all boats in the micro camp. The same program moved from a 386 to a 486 would automatically get faster since they shared an instruction set. For programs that required recompiles or new libraries based on new instructions or hardware capabilities, Intel and software vendors could afford to invest heavily in the operating system and compiler side and scale those out to their large pool of users. Specialized hardware, on the other hand, required its own software stacks and it was much harder for those companies to keep up with limited funds and lower scale. Each time the microprocessor got faster and cheaper, applications that previously weren’t feasible were unlocked. One interesting example was PCs with DVD drives that were originally shipped with specialist video decoder cards. Within about two years, these cards went extinct when the Pentium II could decode video in software. Losing our free lunch Unfortunately, the “automatic” speed gains from Dennard scaling were not to last forever. In 1974, Robert Dennard at IBM published that a smaller transistor was faster, cheaper, and lower power than its larger equivalent, so the clock speed could increase each generation while power stayed constant. This matters for the dilemma around specialization, because we can drive performance improvements for workloads in different ways (for example algorithmic improvements, chips getting specialized/better at a specific tasks at the hardware level, and fundamental efficiency gains in process technology can all drive better performance). The first piece of Dennard scaling to fall was the ability to scale smaller transistors to lower voltages, which was what unlocked the ability to maintain the same power level for more, denser transistors in the same chip area. This happened around 2005, as you can see from the chart below. As a result, smaller transistors couldn’t benefit from proportionally lower voltages. This voltage scaling was the thing that allowed us to scale clock frequency, so you can see frequencies also flattened out at this time. We can see that number of logical cores started rising around this time as well, to compensate for the flattening of clock speeds, however not every application can be efficiently parallelized, so it wasn’t the same “free lunch” as before. The next thing to fall was cost per transistor, which happened in about 2012. Transistors did continue to shrink, but the cost of the fab and extra process costs grew faster than the transistor count per wafer, so each shrink still gave more transistors per chip but at the same or higher price per transistor. There were other scaling challenges as well. Going back in time to 1995, experts at the time named the “memory wall” after the observation that memory bandwidth was increasing much more slowly than compute (and it never caught up). David Patterson estimates that NVIDIA GPU compute rose 80x from 2012 to 2022, while memory bandwidth rose 17x. One key factor for this difference is that compute scales with the area of the chip, whereas the bandwidth you can get into a chip scales with its perimeter. Back in 2012, when transistor costs stalled out, I was a graduate student at Mark Horowitz’s group at Stanford. One of my cofounders, Omid Azizi, worked with Mark on research into sources of inefficiency in general-purpose chips, finding that a 720p encoder running on a general-purpose chip was about 500x less energy efficient than running it on a specialized ASIC. Now that the free lunch of Dennard scaling was over, was it time to return to specialized chips? The rise of GPUs: revenge of the specialized, or the new general-purpose chip? The GPU was one of the original specialist survivors of the CPU era. The GeForce 256 in 1999 did a single job very well: rendering triangles fast on a screen for gaming use cases. Unlike other specialized chips like vector processors and Lisp machines, it had a mainstream audience and therefore large volume, with millions shipping into gaming PCs. Rendering is massively parallel, demanding massive numbers of cores, something that 50% YoY improvement in CPU performance couldn’t alone deliver. Once clock speeds stopped scaling in 2005, chips had to pivot from doing the same thing faster to doing more things at once. GPUs were already designed to do this, with their thousands of simple cores and very high memory bandwidth, a tradeoff over single-threaded speed. This made it a very attractive platform for very large scale workloads, from early machine learning to crypto to high-performance computing. Without the dramatic efficiency and performance that GPUs offer over CPUs for AI workloads, we couldn’t have trained and scaled trillion or even billion parameter LLMs, and would probably still be stuck limited to tiny regression models. The GPU was able to dramatically scale its performance, but in a different way than its CPU predecessors did in the height of Dennard scaling. Bill Dally from Nvidia broke down where performance gains have come from in the past 10-15 years, at Hot Chips 2023 via the following graphic, showing inference performance gains over time. For single-GPU inference performance, about 16X of the 1000X speedup game came from simpler number formats (e.g. FP32 to BF16). 12.5X came from new instructions, 2X came from leveraging sparsity (skipping weights in the computation), and only 2.5X came from process improvements from 28nm to 5nm. This meant that the process, which had driven massive performance scaling in the industry over 40 years, contributed less than the other major factors! Note that this analysis doesn’t include the massive performance scaling that also happened simultaneously around large-scale systems of many GPUs (since it is limited to a single GPU’s inference performance). As AI continues to scale, GPU sales have actually risen past CPU sales. Perhaps at this point we should call GPUs the general purpose accelerator and CPUs the specialized chip! Why the future will be even more heterogeneous (for a time) Zooming back to today, inference is growing incredibly fast. According to a 2026 Goldman Sachs report, demand for tokens will increase by 24x from 2026 to 2030, yet available power will only increase by about 2x. We need to run these workloads more efficiently and performantly, because we will likely still have scarcity in the available physical infrastructure. So what levers do we have left? Dennard scaling is dead, and each lever that produced the GPU’s 1000x gain in 10 years will yield diminishing returns. Going from 32 to 4 bits of precision was hard, but there isn’t that much room below 4 comparatively. Network scaling and algorithms will remain hugely important but also challenging in their own ways. The memory wall is a major limiting factor for inference workloads, especially in the decode phase which is highly memory bandwidth bound. To help boost performance of inference, the GPU has been following in the footsteps of the CPU, by incorporating inference-specific hardware concepts directly inside the GPU, similarly to how the CPU once incorporated floating point units. Tensor cores were added in 2017, MXFP4 and NVFP4 in 2024 which are predominantly used by inference. Is it time for specialized chips to make their return, delivering dramatically more efficient performance for specific workloads? There is a Cambrian explosion of new chips focused on inference, but one of the most interesting moves in this direction came from Nvidia in its large-scale deal with Groq. Groq, now known as Nvidia’s LPU, is an SRAM-centric chip which has a fundamentally different architecture from a GPU. GPUs keep their memory far from compute in HBM where they can store a lot of data, whereas near-memory compute chips like Groq, Cerebras, and d-Matrix put the memory directly on die next to the compute, which provides extremely fast memory bandwidth when data fits on the available chip memory. Both designs have advantages depending on the slice of the workload, and NVIDIA is now orchestrating inference across GPUs and LPUs in tandem in order to continue to push inference performance. Inference is a highly heterogeneous workload, and different phases of inference benefit from different tradeoffs in the underlying hardware. Given the scaling challenges across transistors, memory bandwidth, and network bandwidth, selecting the right chip architecture for each segment of the workload offers large performance improvements, even if the underlying resources are no longer scaling like before. Orchestrating inference workloads across diverse accelerators is what my company Gimlet is most focused on. By splitting workloads across different types of chips - such as NVIDIA GPUs and LPUs, or other hardware like Google’s TPU, Cerebras wafers, d-Matrix Corsair, Trainium, AMD GPUs, and other accelerators - we can see dramatically better performance per watt than homogeneous stacks alone can achieve. Although specialized hardware failed in the past, the circumstances are different today. It’s easier than ever to develop optimized code for new accelerator types using agents. We have significantly more expertise and infrastructure for building new chips than we did decades prior (most chip companies don’t need to handle their own fabrication, for example). We are severely power limited with a workload (inference) that continues to explode in scale, with many of the low hanging fruit of performance already picked off the tree. Splitting inference workloads across different types of chips is not an easy feat, and as much of the complexity is in the physical infrastructure as it is in the software layer (but on the plus side, physical infrastructure complexity might be a more durable “moat” than software can be). It’s important to select and deploy hardware in a very modular, configurable way, with a software stack that can leverage all available hardware and not leave stranded capacity earmarked for particular tasks. I am bullish on the benefits of bringing together diverse silicon, because in this era of inference, delivering dramatically more or faster tokens with limited power is such a massive advantage that it’s worth the complexity. This phase will last years, although it may not last forever. The pendulum of specialization Tsugio Makimoto, who ran Hitachi’s semiconductor business, noted that general versus specialized parts are a pendulum that swings cyclically from side to side. While time to market, cost effectiveness, and operational efficiency benefit standardized parts, differentiation and supply-demand imbalance cause the pendulum to swing to the other side. So where is the pendulum in 2026? In spite of the move toward heterogeneity, there are still tailwinds for the GPU. Fab and HBM allocations are very scarce and difficult for new players to acquire. AI accelerators (GPU and otherwise) need to be deployed for multiple years to be economical, while the workload continues to evolve. And GPUs will continue to absorb new inference features as they prove their worth. At the same time, the specialists have many factors in their favor. One of the scarcest resources is power and specifically performance per watt of AI workloads, and specialized chips can maximize performance for a given workload. It’s getting easier to develop software for new hardware, as agents are shrinking both the time it takes to write kernels or even design new chips. We expect to see a continued increase of specialized chips to deliver better performance per watt for inference, especially in conjunction with GPUs for the portions of the workload that are not easily offloaded to new accelerators. Although I expect the pendulum will likely eventually swing back to generalized architectures, the time of specialization is upon us and a very exciting time to work in infrastructure. Each era of computing (desktop, cloud, mobile, now AI) brings a new set of dominant workloads, and a need to rethink what computing actually looks like. Even after a pendulum swing, the novel, specialized hardware of today has the potential to become the mainstream hardware of tomorrow (and even if it doesn’t, its contributions can still continue to live on in future generalist chips). Thanks for reading, and if you are interested in these topics, send us a note at hello@gimletlabs.ai! We are hiring for all roles and scaling out our inference cloud to deliver radically faster tokens to more users in the next few months. This newsletter is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. Furthermore, this content is not investment advice, nor is it intended for use by any investors or prospective investors in any a16z funds. This newsletter may link to other websites or contain other information obtained from third-party sources - a16z has not independently verified nor makes any representations about the current or enduring accuracy of such information. If this content includes third-party advertisements, a16z has not reviewed such advertisements and does not endorse any advertising content or related companies contained therein. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z; visit https://a16z.com/investment-list/ for a full list of investments. Other important information can be found at a16z.com/disclosures. You’re receiving this newsletter since you opted in earlier; if you would like to opt out of future newsletters you may unsubscribe immediately.
In 1990, Eugene Brooks of Lawrence Livermore told the Supercomputing conference that nothing would withstand the attack of the killer micros. “Micro” referred to microprocessor, meaning a general-purpose chip that did a good job at most types of computation. They were also being sold by the millions per year into desktop PCs, delivering faster and faster performance each year. Around that same time, competing specialized architectures, such as vector processors, massively parallel processors (MPPs), and Lisp machines started dying off en masse. ETA systems closed in 1989, Multiflow in 1990, Alliant in 1992, Symbolics in 1993, Kendall Square Research and Thinking Machines (not the same one as today) in 1994, and Cray Computer in 1995. Danny Hillis, the founder of Thinking Machines, remarked years later that “…ultimately these things were just going to be boxes of microprocessors.” Interestingly, these specialized chips didn’t lose because they were superior at unimportant problems. The general-purpose microprocessor was actually outperforming them, not just on a wide spectrum of workloads, but on the very workloads the specialized chips were designed for. And although the companies that made them didn’t make it, the ideas they brought lived on inside other chips. CPUs absorbed vector processing as SSE and AVX, and massively parallel designs influenced both multi-core CPU and GPU architectures. The pattern at the time was that the specialists came up with good ideas, but the generalist chips absorbed those ideas into a single architecture. Today, we are in the middle of another heyday for the industry. AI workloads are different enough from traditional software workloads that the industry is in the midst of rethinking what computing should actually look like, with many different new chips and ideas behind them. There is a massive explosion of new accelerators to serve AI workloads, and GPUs generate more revenue for chip companies than CPUs do (although CPUs are also in their own demand crunch right now). In times like this, it’s interesting to look back on the last hardware boom and see what has changed since last time (and what remains the same), to better predict what will happen next. Is this time different for specialized chips, or will they ultimately be absorbed into general purpose chips like before? Why general-purpose chips won (for a time) Why were the “killer micros” beating specialized hardware - at the very workloads the specialists were designed for - back in the 90s? The general-purpose microprocessor was able to leverage a virtuous cycle: generality drove volume, volume drove performance, which in turn drove generality. 1. Generality drives volume The microprocessor was sold by the tens of millions into desktop PCs, being general-purpose hardware that could run every workload a PC had. The specialist hardware, on the other hand, had much lower volumes because it only ran one. 2. Volume drives performance Those volumes translated into much lower costs, more buyers, and ultimately more money to fund development for the microprocessor. From 1986 to 2003, Hennessy and Patterson estimate that single-processor performance grew over 50% year over year, most of which was delivered by faster clock speeds. Moore’s law scaling delivered a free lunch during that period - that is, free for anyone who can continually fund leading edge fabs and large chip design teams to take advantage of every new generation. The specialists with their lower volumes could not afford the same cadence, and therefore fell behind. In addition, the generalists were able to absorb some of the best ideas of the specialists once the specialist workloads were better understood (see vector processing becoming a capability of CPUs). 3. Performance drives generality The faster clock speeds largely enabled by transistor scaling were a rising tide that lifted all boats in the micro camp. The same program moved from a 386 to a 486 would automatically get faster since they shared an instruction set. For programs that required recompiles or new libraries based on new instructions or hardware capabilities, Intel and software vendors could afford to invest heavily in the operating system and compiler side and scale those out to their large pool of users. Specialized hardware, on the other hand, required its own software stacks and it was much harder for those companies to keep up with limited funds and lower scale. Each time the microprocessor got faster and cheaper, applications that previously weren’t feasible were unlocked. One interesting example was PCs with DVD drives that were originally shipped with specialist video decoder cards. Within about two years, these cards went extinct when the Pentium II could decode video in software. Losing our free lunch Unfortunately, the “automatic” speed gains from Dennard scaling were not to last forever. In 1974, Robert Dennard at IBM published that a smaller transistor was faster, cheaper, and lower power than its larger equivalent, so the clock speed could increase each generation while power stayed constant. This matters for the dilemma around specialization, because we can drive performance improvements for workloads in different ways (for example algorithmic improvements, chips getting specialized/better at a specific tasks at the hardware level, and fundamental efficiency gains in process technology can all drive better performance). The first piece of Dennard scaling to fall was the ability to scale smaller transistors to lower voltages, which was what unlocked the ability to maintain the same power level for more, denser transistors in the same chip area. This happened around 2005, as you can see from the chart below. As a result, smaller transistors couldn’t benefit from proportionally lower voltages. This voltage scaling was the thing that allowed us to scale clock frequency, so you can see frequencies also flattened out at this time. We can see that number of logical cores started rising around this time as well, to compensate for the flattening of clock speeds, however not every application can be efficiently parallelized, so it wasn’t the same “free lunch” as before. The next thing to fall was cost per transistor, which happened in about 2012. Transistors did continue to shrink, but the cost of the fab and extra process costs grew faster than the transistor count per wafer, so each shrink still gave more transistors per chip but at the same or higher price per transistor. There were other scaling challenges as well. Going back in time to 1995, experts at the time named the “memory wall” after the observation that memory bandwidth was increasing much more slowly than compute (and it never caught up). David Patterson estimates that NVIDIA GPU compute rose 80x from 2012 to 2022, while memory bandwidth rose 17x. One key factor for this difference is that compute scales with the area of the chip, whereas the bandwidth you can get into a chip scales with its perimeter. Back in 2012, when transistor costs stalled out, I was a graduate student at Mark Horowitz’s group at Stanford. One of my cofounders, Omid Azizi, worked with Mark on research into sources of inefficiency in general-purpose chips, finding that a 720p encoder running on a general-purpose chip was about 500x less energy efficient than running it on a specialized ASIC. Now that the free lunch of Dennard scaling was over, was it time to return to specialized chips? The rise of GPUs: revenge of the specialized, or the new general-purpose chip? The GPU was one of the original specialist survivors of the CPU era. The GeForce 256 in 1999 did a single job very well: rendering triangles fast on a screen for gaming use cases. Unlike other specialized chips like vector processors and Lisp machines, it had a mainstream audience and therefore large volume, with millions shipping into gaming PCs. Rendering is massively parallel, demanding massive numbers of cores, something that 50% YoY improvement in CPU performance couldn’t alone deliver. Once clock speeds stopped scaling in 2005, chips had to pivot from doing the same thing faster to doing more things at once. GPUs were already designed to do this, with their thousands of simple cores and very high memory bandwidth, a tradeoff over single-threaded speed. This made it a very attractive platform for very large scale workloads, from early machine learning to crypto to high-performance computing. Without the dramatic efficiency and performance that GPUs offer over CPUs for AI workloads, we couldn’t have trained and scaled trillion or even billion parameter LLMs, and would probably still be stuck limited to tiny regression models. The GPU was able to dramatically scale its performance, but in a different way than its CPU predecessors did in the height of Dennard scaling. Bill Dally from Nvidia broke down where performance gains have come from in the past 10-15 years, at Hot Chips 2023 via the following graphic, showing inference performance gains over time. For single-GPU inference performance, about 16X of the 1000X speedup game came from simpler number formats (e.g. FP32 to BF16). 12.5X came from new instructions, 2X came from leveraging sparsity (skipping weights in the computation), and only 2.5X came from process improvements from 28nm to 5nm. This meant that the process, which had driven massive performance scaling in the industry over 40 years, contributed less than the other major factors! Note that this analysis doesn’t include the massive performance scaling that also happened simultaneously around large-scale systems of many GPUs (since it is limited to a single GPU’s inference performance). As AI continues to scale, GPU sales have actually risen past CPU sales. Perhaps at this point we should call GPUs the general purpose accelerator and CPUs the specialized chip! Why the future will be even more heterogeneous (for a time) Zooming back to today, inference is growing incredibly fast. According to a 2026 Goldman Sachs report, demand for tokens will increase by 24x from 2026 to 2030, yet available power will only increase by about 2x. We need to run these workloads more efficiently and performantly, because we will likely still have scarcity in the available physical infrastructure. So what levers do we have left? Dennard scaling is dead, and each lever that produced the GPU’s 1000x gain in 10 years will yield diminishing returns. Going from 32 to 4 bits of precision was hard, but there isn’t that much room below 4 comparatively. Network scaling and algorithms will remain hugely important but also challenging in their own ways. The memory wall is a major limiting factor for inference workloads, especially in the decode phase which is highly memory bandwidth bound. To help boost performance of inference, the GPU has been following in the footsteps of the CPU, by incorporating inference-specific hardware concepts directly inside the GPU, similarly to how the CPU once incorporated floating point units. Tensor cores were added in 2017, MXFP4 and NVFP4 in 2024 which are predominantly used by inference. Is it time for specialized chips to make their return, delivering dramatically more efficient performance for specific workloads? There is a Cambrian explosion of new chips focused on inference, but one of the most interesting moves in this direction came from Nvidia in its large-scale deal with Groq. Groq, now known as Nvidia’s LPU, is an SRAM-centric chip which has a fundamentally different architecture from a GPU. GPUs keep their memory far from compute in HBM where they can store a lot of data, whereas near-memory compute chips like Groq, Cerebras, and d-Matrix put the memory directly on die next to the compute, which provides extremely fast memory bandwidth when data fits on the available chip memory. Both designs have advantages depending on the slice of the workload, and NVIDIA is now orchestrating inference across GPUs and LPUs in tandem in order
document fetched 2026-10-24 11:25 · canonical https://a16z.news/p/why-hardware-specialization-is-back