OpenAI publishes first Jalapeño inference performance results
OpenAI has released its first performance results for Jalapeño, the company’s custom AI inference chip, reporting 1.5 to 1.9 times more AI work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency than the comparison systems across three public models. The figures cover GPT-OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T.
The company tested the system using InferenceX, a public benchmark from SemiAnalysis that measures the full serving process for an AI request. OpenAI said Jalapeño delivered a stronger combination of throughput, power efficiency and latency across the tested range, from high-throughput serving to highly interactive, low-latency use.
Power efficiency and response time in one design
OpenAI evaluates performance at a matched user experience, measuring useful AI work completed per unit of power while meeting latency requirements. That approach is particularly relevant to agents, where a task may require many sequential steps and delays can accumulate over the course of a request.
For the tested workloads, Jalapeño was normalized using its published 700-watt chip power rating. OpenAI said its measured sustained power remained at or below 550 watts. On Kimi K2.5 1T, the largest public model in the tests, Jalapeño delivered about 1.5 times higher peak performance per watt and 3.4 times lower end-to-end latency than the comparison system.
The first results build on Jalapeño accelerator for efficient LLM inference and show how the accelerator is being positioned for energy-efficient, rack-scale language-model inference. OpenAI describes Jalapeño as sitting on the Pareto frontier across the three tested models, meaning it combined power-normalized performance and latency without the trade-off often seen in other systems.
Architecture designed around inference phases
OpenAI said Jalapeño was designed across chip, memory, networking, software and rack-scale infrastructure as one system. Inference has different constraints at different stages: prompt processing, or prefill, is compute-intensive, while token-by-token decoding is more limited by memory bandwidth. Communication between cores and chips can introduce further latency.
To reduce those delays, the design explicitly places and keeps model state, including the KV cache, local while activating the needed combination of compute, memory and networking. OpenAI said the network forms an integral part of the architecture, allowing the complete workload to remain within one connected system and reducing data movement.
AI-assisted engineering and deployment plans
OpenAI used AI during Jalapeño’s development and said the team moved from initial design to tapeout in nine months. The company also used Codex with GPT-Astra to bring three open-weight models not included in the original production plan to high performance within two months. For selected GPT-OSS attention and mixture-of-experts blocks, AI-generated implementations ran 1.5 to 1.8 times faster than existing human-expert-written implementations.
OpenAI plans to begin deploying Jalapeño in its compute infrastructure by the end of the year while continuing production qualification, software maturation and validation across additional models. For businesses assessing AI infrastructure, the practical implication is to evaluate serving platforms against the throughput, latency and power profile required by their own workloads rather than treating accelerator performance as a single headline metric.