Those results are central to OpenAI’s pitch for the hardware. Inference systems often have to prioritize either serving large numbers of requests efficiently or minimizing the delay experienced by an individual user. OpenAI says Jalapeño was built to improve both within the same architecture. “The bottom line is that the results show a very, very significant performance advance over state of the art,” Richard Ho, OpenAI’s head of hardware, told TechCrunch. “Jalapeño can serve more AI work per unit of power, while also returning responses more quickly. It’s very efficient to serve a lot of customers, but it can also be very low latency.”
On GPT-OSS 120B, Jalapeño reached peak mixed-token throughput of 85,448 tokens per second per kilowatt, compared with 44,960 for the system used as its benchmark. OpenAI also measured peak per-user decoding speeds of 1,459 tokens per second, versus 535 for the comparison platform.
The differences were also visible on larger models. With DeepSeek R1, Jalapeño reached 19,641 mixed tokens per second per kilowatt at peak efficiency, compared with 11,781 for the comparison system. Its maximum per-user decoding throughput reached 700 tokens per second, versus 169.
On Kimi K2.5, the largest public model included in the testing, Jalapeño produced about 1.5 times greater peak performance per watt and reduced end-to-end latency by roughly 3.4 times. Peak per-user decoding reached 694 tokens per second, compared with 182 for the benchmark system.
OpenAI evaluated the hardware across different operating points rather than focusing on performance under a single workload. The company says Jalapeño remained competitive as the balance shifted between maximizing overall throughput and delivering highly interactive responses.
Power consumption was normalized using each accelerator’s published chip rating. Jalapeño carries a 700-watt rating, although OpenAI says sustained consumption stayed at or below 550 watts during the workloads it tested.
The architecture was designed specifically around language-model inference. Processing an AI request involves phases with different constraints: handling the initial prompt places heavier demands on computation, while generating tokens can depend more heavily on memory bandwidth. Moving data between components can create additional delays.
OpenAI designed Jalapeño to reduce that movement. Model state, including the KV cache used during generation, can remain local while compute, memory and networking resources are coordinated around each stage of inference. “We designed Jalapeño to minimize data movement and communication delays,” OpenAI said. “This means that model state, including the KV cache used while generating a response, can be explicitly placed and kept local while the system activates the right combination of compute, memory, and networking for each inference phase.”
The company also used its own AI models during development. OpenAI says AI-assisted design helped the team move from its initial design to tapeout in nine months, while models were also used to improve arithmetic circuits and shorten parts of the design and verification process.
That approach has continued into software development. Using Codex with GPT-Astra, the team optimized three open-weight models that were not originally part of Jalapeño’s production plan within two months. For selected GPT-OSS attention and mixture-of-experts blocks, AI-generated implementations performed between 1.5 and 1.8 times faster than versions written by human experts. OpenAI noted that those measurements apply only to the selected components rather than the entire model.
Jalapeño is intended to become the first generation of a larger hardware platform rather than a one-off processor. OpenAI says its second-generation chip is already deep in development, while work on a third generation has begun.
The company plans to start deploying Jalapeño in its own compute infrastructure by the end of the year. Ho told TechCrunch that initial deployment is expected in “very small volumes,” with broader deployment following later.
OpenAI said it will continue using accelerators from NVIDIA and other partners alongside its own silicon for both training and inference. Before Jalapeño enters deployment, the company is continuing production qualification, expanding software support and testing the hardware across additional models.
This analysis is based on reporting from TechCrunch & OpenAI.
Image courtesy of OpenAI.
This article was generated with AI assistance and reviewed for accuracy and quality.
