OpenAI details Jalapeño inference chip benchmark results

OpenAI has disclosed initial benchmark results for Jalapeño, its inference system developed with Broadcom, at the Hot Chips conference. In Semianalysis’s InferenceX benchmark, the company said Jalapeño delivered more tokens per user and higher throughput per kilowatt than currently available state-of-the-art inference processors.

Richard Ho, OpenAI’s head of hardware, described the outcome as a significant performance advance. He said the system can serve more AI work per unit of power while returning responses more quickly, combining high customer capacity with low latency.

Focus on inference bottlenecks

Jalapeño is intended to be a multigenerational platform in which AI products, models, chips and memory are developed in concert. That full-stack approach is designed to address particular stages of inference processing that create friction rather than treating compute, memory and networking as separate layers.

OpenAI identified the prefill and communication phases as frequent bottlenecks. Its design seeks to reduce data movement and communication delays by explicitly placing model state locally, including the KV cache used when a response is generated. The system then activates the appropriate combination of compute, memory and networking for each inference phase.

Benchmark comparison and deployment timetable

The published comparison was against an Nvidia Blackwell system. That context matters because Jalapeño is not expected to reach full deployment immediately, and competing inference hardware may advance before volumes rise.

Ho estimated that Jalapeño would be deployed in very small volumes at the end of 2026, followed by more significant deployment in 2027. The programme builds on OpenAI and Broadcom’s Jalapeño inference accelerator and frames the accelerator as part of OpenAI’s longer-term effort to align its hardware with the requirements of its models.

What enterprises should take from the announcement

The results highlight how inference efficiency depends on more than peak compute capability. For organisations planning AI services, the practical implication is to assess latency, power use, memory locality and communication overhead together when evaluating infrastructure for large-scale model serving.