Nvidia is moving to commercialize technology from its largest acquisition, announcing on Monday that its Groq 3 LPX rack has entered full production as the chipmaker seeks to capture a growing market for low-latency AI inference.
The Groq 3 LPX will be deployed alongside Nvidia’s Vera central processors and Rubin graphics processors at cloud provider Nebius, with the systems expected to come online later this year, Nvidia senior director Dion Harris told reporters.
The production milestone comes less than a year after Nvidia agreed to acquire assets from AI-chip startup Groq for $20 billion, making it the company’s largest acquisition on record. The rapid transition from acquisition to commercial deployment highlights how strategically important inference has become as AI systems move beyond generating answers to performing tasks continuously through agents.
Register for the next Tekedia Mini-MBA.
Register for Tekedia AI in Business Masterclass.
Join Tekedia Capital Syndicate and co-invest in great global startups.
Inference is the stage at which a trained AI model generates responses for users. For applications such as coding agents, real-time assistants, and other autonomous systems, the speed at which those responses are generated can materially affect the user experience.
Nvidia is therefore betting that the future of AI computing will require more than powerful GPUs. It will require specialized processors capable of handling particular parts of the inference workload at much lower latency.
“For folks who are serving tokens, it unlocks the ability to offer premium tiers of service for those users and those customers who actually demand the most latency-sensitive” service agreements, Harris said.
The Groq architecture is designed around that requirement.
Each Groq chip contains 500 megabytes of high-speed SRAM directly on the chip, reducing the need to repeatedly move data between processors and external memory. Nvidia packages 256 Groq 3 chips into an LPX rack.
Nvidia says the resulting Groq 3 LPX rack can generate 3,400 tokens per second, citing a benchmark from Artificial Analysis.
The technology is particularly relevant to AI agents, which can require models to generate large numbers of tokens rapidly as they reason, write code, call tools, and respond to changing information.
For users, lower latency can make the difference between an AI agent feeling instantaneous and one that appears to pause repeatedly during a task. That creates an opportunity for cloud providers to charge premium prices for faster inference.
But Nvidia is careful to position Groq as complementary to its dominant GPU business rather than a replacement for it.
“This isn’t about replacing GPUs,” Harris said. “It’s about using the right price, right processor for the right part of the workload.”
That matters because Nvidia’s GPUs remain the primary general-purpose engines for AI workloads. They can be used for both training and inference and offer the flexibility required as models, software frameworks, and AI architectures evolve.
Groq’s technology is more specialized. It is primarily designed to accelerate the “decode” phase of inference, when an AI model generates output tokens one after another.
But analysts have questioned whether Nvidia’s specialized inference processors can expand the company’s addressable market without undermining demand for its core GPUs.
The company appears to believe that both markets can grow together.
Nvidia is already ramping up production of its Vera Rubin systems, which entered production earlier this year. At the unveiling of the Vera Rubin and Groq 3 LPX systems in March, CEO Jensen Huang forecast $1 trillion in cumulative sales from Blackwell and Vera Rubin systems through 2027.
Huang also said Nvidia would allocate about a quarter of data-center capacity intended for coding applications to Groq processors.
“The rest of my data center is all 100% Vera Rubin,” Huang said at the time.
That allocation illustrates how Nvidia views specialized inference within its broader data-center strategy. The company does not need Groq to displace its GPUs across the data center. It needs specialized chips to handle workloads where the economics of speed and latency justify a different architecture.
Competition is already developing rapidly.
AMD announced earlier this year that it would integrate its rack-scale systems with chips from Cerebras, another company targeting low-latency inference. Cerebras recently went public and has positioned its technology around rapid AI inference.
OpenAI has also highlighted the demand for faster inference. Its newly announced Ultrafast mode promises speeds of up to 750 tokens per second and is powered by Cerebras technology. The competition matters because AI inference is likely to become a much larger part of total AI computing demand as businesses deploy agents at scale.
Training a frontier model is an enormous but relatively periodic computing exercise. Inference is continuous. Every user request, coding task, search query, or autonomous action requires computation after the model has been trained. As AI agents become more widely deployed, the amount of inference required could therefore increase dramatically.
That could change the economics of AI infrastructure.
The market is moving from a model in which companies primarily compete to build the most powerful training hardware toward one in which they also compete over the cost, speed, and efficiency of serving AI models to millions of users.
Nvidia’s Groq acquisition gives it exposure to that transition. The company is effectively attempting to cover both ends of the AI infrastructure stack: highly flexible GPUs and CPUs for broad workloads, alongside specialized inference hardware for applications where latency is critical.
The acquisition also gives Nvidia access to Groq’s architectural approach without requiring it to build the technology entirely from scratch.
Groq chips are manufactured by Samsung, while Nvidia’s GPUs are manufactured by Taiwan Semiconductor Manufacturing Co. The differing manufacturing relationships also demonstrate that Nvidia’s expanding AI hardware portfolio is increasingly dependent on a broader semiconductor supply chain.
The Groq production announcement comes as Nvidia is due to report quarterly earnings on Wednesday. Investors will be looking not only at revenue and profit, but also for evidence that demand for AI infrastructure remains strong enough to sustain the extraordinary expectations embedded in Nvidia’s valuation.
The company has become the central beneficiary of the AI data-center spending cycle, but that position increasingly depends on the industry continuing to expand beyond model training.
Inference could be the next major phase of that expansion.
If AI agents become widely used for coding, enterprise automation and other real-time applications, the demand for low-latency inference could rise sharply. That would create a new source of demand for specialized processors while also increasing overall demand for the networking, memory, power and data-center infrastructure surrounding them.
Nvidia’s strategy is therefore broader than simply selling another AI chip. It is positioning itself for a market in which different AI workloads are handled by different processors, with GPUs remaining the workhorse while specialized chips handle tasks where speed, efficiency or cost make them more attractive.
The commercialization of Groq 3 LPX is the first major test of that strategy. If Nebius and other cloud providers can demonstrate that customers are willing to pay a premium for substantially faster inference, Nvidia’s $20 billion bet could become an important part of its next phase of growth.
The bigger prize, however, is the emerging AI agent economy. As AI shifts from answering questions to continuously executing tasks, latency becomes a product feature rather than simply a technical specification. Nvidia is betting that the companies able to deliver those responses fastest will be able to charge more, and that the hardware powering that speed will become a major new battleground in the AI infrastructure race.