The release addresses a longstanding obstacle for companies considering alternatives to Nvidia's AI processors. Developers have traditionally needed to adapt software built around Nvidia's CUDA ecosystem when moving to different hardware, a process that can require extensive engineering work and additional testing.
Amazon designed TorchNeuron to eliminate much of that effort while preserving compatibility with familiar development tools. In a demonstration, engineers executed the same model training program on conventional hardware and Trainium, changing only the hardware configuration. Amazon reported that both versions demonstrated equivalent learning behavior.
“If you know PyTorch, then you know Trainium,” Colin Brace, vice president of technology software at Amazon's Annapurna Labs, told GeekWire.
The software supports both Trainium2 and Trainium3, allowing developers to use Amazon's custom processors for AI training and inference. Companies operating large AI workloads can also move computing capacity between those tasks, potentially improving hardware utilization when demand changes.
TorchNeuron has been undergoing private testing since late last year, with more than 30 companies and internal Amazon teams participating. Early users include organizations working on speech generation, scientific research, and video applications.
Cartesia, which develops conversational AI technology, used TorchNeuron to run its existing inference pipeline on Trainium with minimal changes. Deepgram, another speech AI company, reported that Trainium processed approximately 60% more audio per dollar than its previous GPU configuration. According to Amazon, Deepgram has since deployed the technology in production.
The Lieber Institute for Brain Development is also exploring Trainium for research involving biological, chemical, and language models. Its work includes investigating AI techniques that could accelerate computationally demanding molecular simulations used in drug discovery.
Amazon has applied TorchNeuron internally as well. Its Prime Video team used the technology to develop Prime 360, an AI-powered feature that reconstructs basketball plays in three dimensions. The system enables NBA viewers to watch replays from perspectives unavailable through conventional camera footage and is scheduled to appear during select broadcasts this season.
Beyond individual applications, Amazon is working to make Trainium compatible with established AI development infrastructure. Hugging Face has incorporated native support for the processors into its tools, allowing developers to use Trainium without adding Amazon-specific programming instructions.
TorchNeuron also integrates with TorchTitan, PyTorch's distributed training framework. Developers can use familiar techniques such as Fully Sharded Data Parallelism to distribute model training across multiple Trainium instances.
For more specialized workloads, Amazon provides options for writing low-level code and optimizing performance across multiple chips. The company has also contributed to vLLM, an open-source inference framework, and released performance kernels that developers can use when building their own applications.
Amazon says it evaluates TorchNeuron against a large collection of AI models to identify compatibility issues. Its testing includes nightly checks involving more than 100,000 models from Hugging Face, alongside broader validation conducted during development.
The company is contributing changes directly to PyTorch to reduce dependencies on particular hardware architectures. These improvements are intended to make the framework more compatible with different processors, extending beyond Amazon's own products.
TorchNeuron arrives as major cloud providers develop alternatives to Nvidia's hardware and software ecosystem. Google is working on TorchTPU to support PyTorch applications on its Tensor Processing Units, while Microsoft has outlined plans to expand the use of its internally developed Maia processors.
Amazon already has substantial AI infrastructure commitments involving Trainium. Anthropic uses the processors to train and operate Claude models, while OpenAI agreed in February to use 2 gigawatts of Trainium capacity as part of a broader partnership with Amazon.
The company's custom chip business has also grown considerably. Amazon reported that its silicon operations, which include Trainium and Graviton processors, exceeded a $20 billion annual revenue run rate earlier this year. A more recent company announcement placed the figure above $25 billion.
Amazon's investment in custom processors dates to its acquisition of Annapurna Labs in 2015. The company has since developed several chip families designed for different cloud computing workloads, including AI processing and general-purpose computing.
With TorchNeuron, Amazon is extending that investment into the software developers use to build AI systems. By making Trainium compatible with standard PyTorch workflows, the company aims to reduce the technical barriers that have historically made switching AI hardware difficult.
The public beta and upcoming open-source release give developers a way to evaluate Trainium using their existing applications, without first committing to extensive software modifications.
This analysis is based on reporting from Amazon & Geek Wire.
Image courtesy of Amazon.
This article was generated with AI assistance and reviewed for accuracy and quality.
