The move reflects a broader industry push toward purpose-built silicon for different stages of AI development. Training models requires large-scale compute and memory bandwidth, while inference focuses on speed and efficiency when responding to user requests. By splitting these functions, Google is aiming to optimize performance for both use cases rather than relying on a single, general-purpose chip.
Google said its new training processor delivers 2.8 times the performance of its prior-generation Ironwood TPU at the same price, while the inference chip offers an 80% performance increase. The inference processor, called TPU 8i, uses 384 megabytes of static random-access memory, triple the amount in the previous version.
The architecture is designed “to deliver the massive throughput and low latency needed to concurrently run millions of agents cost-effectively,” Alphabet CEO Sundar Pichai wrote in a blog post.
The update comes as competition in AI hardware intensifies. Nvidia continues to dominate the market for AI accelerators, but companies including Google, Microsoft, Amazon, and Meta are investing in custom chips to reduce reliance on third-party suppliers and tailor hardware to their own systems. Nvidia recently highlighted new silicon designed to improve response times for AI models, following its acquisition of chip startup Groq.
Google has been developing its own AI processors for more than a decade, first deploying them internally in 2015 and later offering them to cloud customers starting in 2018. Adoption has been increasing, with clients such as Citadel Securities using TPUs for research applications and U.S. Energy Department laboratories running software built on the chips. Anthropic has also committed to using multiple gigawatts of Google’s TPU capacity.
Despite these gains, Google is not positioning its chips as direct replacements for Nvidia’s hardware. The company did not compare performance with competing processors, underscoring how entrenched Nvidia remains as the primary supplier for AI workloads.
Still, the shift to separate training and inference chips highlights how AI infrastructure is evolving as workloads become more complex. As companies deploy systems that must handle large-scale model development alongside real-time responses, hardware design is increasingly being tailored to those distinct demands.
For Google, the strategy ties closely to its cloud business. Offering specialized chips through Google Cloud gives customers an alternative to Nvidia-based systems, while allowing the company to align its hardware more closely with its own software stack and AI models.
This analysis is based on reporting from CNBC.
Image courtesy of Google.
This article was generated with AI assistance and reviewed for accuracy and quality.
