Open Source AI Inference Benchmark | InferenceX by SemiAnalysis Kimi K3 benchmarks are liveNew First inference numbers across NVIDIA and AMD chips, click to explore. Explore Open Source Continuous Inference Benchmark Trusted by GigaWatt Token Factories MiniMaxMoonshot KimiAlibaba QwenZhipu GLMOpenAIMicrosoftMetaOraclevLLMGPU ModePyTorchCoreWeaveTensorWaveSGLangWEKAStanfordHugging FaceLambdaRed HatSambaNovaTileRT “Vendor-neutral, continuously updated benchmarking is essential as models and inference stacks co-evolve. MiniMax M3 was built with both frontier capability and real-world deployment efficiency in mind, and the day-one vLLM support from the community reflects the collaborative spirit we're proud to be part of. InferenceX provides the kind of transparent, reproducible data the ecosystem needs.” “At Moonshot AI, we are dedicated to supporting the open-source ecosystem by advancing frontier open models. As the Kimi K2 series evolves, we are glad to see its performance tracked in InferenceX™’s open and reproducible benchmarks. InferenceX™ helps the community better understand industry-level performance and encourages the ecosystem to keep improving and optimizing.” “Qwen has always been about putting capable models into the hands of as many developers as possible, and real-world inference efficiency is what makes that scale. InferenceX™ brings rigorous, vendor-neutral measurement to exactly the questions that matter: how models like Qwen3.5 actually perform across accelerators. Independent, reproducible benchmarks on real hardware give the community the clarity it needs to deploy with confidence, and we're glad to see that level of transparency driving the inference ecosystem forward.” “GLM was built for agentic coding and long-horizon autonomous execution, the kind of workloads where real inference performance is everything. Developers are running GLM-5.2 inside coding agents and multi-step tool-calling pipelines every day, so transparent data on how it actually performs across accelerators matters enormously. InferenceX™ gives the community exactly that: open, reproducible, vendor-neutral benchmarks on real hardware. We’re proud to see rigorous measurement helping developers deploy open models with confidence.” “As we build systems at unprecedented scale, it's critical for the ML community to have open, transparent benchmarks that reflect how inference really performs across hardware and software. InferenceMAX™'s head-to-head benchmarks cut through the noise and provide a living picture of token throughput, performance per dollar, and tokens per Megawatt. This kind of open source effort strengthens the entire ecosystem and helps everyone, from researchers to operators of frontier datacenters, make smarter decisions.” “Our mission at Azure is to give customers the most performant, efficient, and cost-effective cloud for AI. SemiAnalysis InferenceMAX™ supports that mission by providing transparent, reproducible benchmarks that track inference performance across GPUs and software stacks under realistic workloads. This continuous data on throughput, efficiency, and cost per watt strengthens our ability to tune Azure's inference platform for scale, helping customers build with confidence on Microsoft Cloud.” “PyTorch was built on the belief that open tools accelerate the entire AI ecosystem. InferenceX™ embodies that same philosophy—open, reproducible, and vendor-neutral benchmarks that give the community real data on real hardware. As inference workloads scale to serve billions of users, having a continuously updated, transparent performance baseline across accelerators is essential for practitioners and platform teams making critical infrastructure decisions.” “Oracle Cloud Infrastructure is built to give frontier labs & enterprises flexibility and choice, with many GPU SKUs available for AI at scale. InferenceMAX strengthens that mission by delivering open source, reproducible benchmarks that reflect real-world performance, efficiency, and cost on the latest hardware and software. With this transparency, customers can confidently select the platforms that best align with their AI strategies.” “The industry needs many public, reproducible benchmarks of inference performance. We're excited to collaborate with InferenceMAX™ from the vLLM team. More diverse workloads and scenarios that everyone can trust and reference will help the ecosystem move forward. Fair, transparent measurements drive progress across every layer of the stack, from model architectures to inference engines to hardware.” “Arguably the most important OSS benchmark suite out today InferenceX” “InferenceMAX™ demonstrates how an open ecosystem can operate in practice. Many leading inference stacks such as vLLM, SGLang, and TensorRT-LLM are built on PyTorch, and benchmarks like this show how innovations across kernels, runtimes, and frameworks translate into measurable performance on a range of hardware platforms, including NVIDIA and AMD GPUs. By being open source and running nightly, InferenceMAX™ offers a transparent, community-driven approach to tracking progress and providing PyTorch users with data-driven insights.” “InferenceMAX™ raises the bar by delivering open, transparent benchmarks that track how inference really performs across the latest GPUs and software stacks. For customers, having reproducible data that measures real world tokens per dollar & tokens per watt, turns abstract marketing numbers into actionable insight. At CoreWeave, we support this effort because it brings clarity to a fast-moving space and helps the entire ecosystem build with confidence.” “At TensorWave, we're building a next-generation cloud on AMD GPUs because we believe innovation thrives when customers have strong alternatives. InferenceMAX™ reinforces that vision by providing open source, reproducible benchmarks that track throughput, efficiency, and cost across the latest hardware and software. By cutting through synthetic numbers and highlighting real-world inference performance, it helps customers see the full potential of AMD platforms for AI at scale.” “SGLang is the inference engine behind many production inference factories such as xAI's Grok, earning its recognition as THE Inference King. At scale, we see firsthand how much performance varies across hardware, models, and configurations. InferenceX™ benchmarks SGLang across every major GPU platform nightly, capturing that variance in a way no other benchmark does, continuously, & reproducibly.” “InferenceX™ ensembles precisely that — open, reproducible benchmarks that are continuously updated as xPU accelerators (GPUs/TPUs/LPUs), memory, storage, and software stacks evolve. I'm excited to see the InferenceX benchmarking roadmap include agentic coding workloads that stress CPU KV Cache offloading & soon NVMe KV Cache offloading from xPUs. As WEKA helps scale the Memory Wall by building the KV Cache infrastructure that feeds these xPUs, having this level of visibility into inference performance helps the entire ecosystem make smarter decisions about where to invest.” “For researchers working on inference optimizations, understanding how new techniques interact across the software and hardware stack is critical yet incredibly hard to measure. InferenceX™ provides much-needed insights into how inference performance evolves across major hardware platforms, moving the field forward with open, reproducible data that makes the gaps and progress visible.” “Hugging Face exists to make AI open and accessible to everyone. InferenceX™ extends that mission to ai chip performance, pulling models directly from the Hub and benchmarking them across every major accelerator, continuously and transparently. When the community can see exactly how frontier open models perform on real hardware in real time, it raises the bar for the entire ecosystem.” “Lambda exists to make GPU compute simple and accessible for AI teams, from individual researchers to the largest labs. InferenceX™ aligns with that mission by giving the community open, reproducible benchmarks that measure what actually matters: real-world throughput, cost efficiency, and performance per watt across the latest hardware and software stacks. Teams can make informed compute choices grounded in transparent, continuously updated data.” “The benchmark is good sir” “Premium Inference is a new category, driven by agents that need fast, interactive tokens with the best per-chip throughput on intelligent frontier models. InferenceX™ is one of the few benchmarks measuring this chip performance in the open, across every major accelerator. We look forward to participating in the official InferenceX with SN50 and showing our premium decode performance on frontier open models tracked transparently.” “TileRT's ultra-low-latency inference requires abolishing the kernel as the unit of execution, not just launching kernels faster. InferenceX™'s open, reproducible, and continuously updated benchmarks gave us fair & neutral stage to verify what a persistent Engine Kernel can deliver — 500 tokens/s/user on frontier open models from a single B200 node. We look forward to continuing this collaboration as InferenceX expands into long context Agentic benchmarks.” See more supporters → Explore InferenceX Start with a concise cost overview across active models and key platforms, or open the full dashboard for every model, chip, framework, and metric. Compare NVIDIA GB300 NVL72, GB200 NVL72, B300, B200, H200, H100, AMD MI355X, MI325X, MI300X and soon VR200 NVL72, AMD MI455X UALoE72, TPUv7 Ironwood, etc across DeepSeekv4 Pro, Qwen, Kimi, GLM, MiniMax, gpt-oss, Llama and other models. OverviewFull Dashboard Every Result Is Transparently done through Public GitHub Actions Automation Every data point on the dashboard is produced by a public GitHub Actions workflow run. The recipe lives in the repo, the run executes on the actual target hardware, and the full logs and artifacts are publicly viewable. Click any point on a chart to jump straight to the run that produced it. All reproducible, auditable, and open source. 1,000+ new benchmark datapoints added per week on average. Browse every new model, chip, framework, and configuration as it lands. Public Actions runs Every benchmark executes on GitHub Actions with full logs visible while the run is in progress. Open recipes Every model, framework, precision, and parallelism setting is committed to the public repo as a shell script. Weekly DB snapshots The full benchmark database is published as a public GitHub Release every week so the historical dataset stays auditable. Browse submissionsView benchmark runs on GitHub ActionsHow it works Quick Comparisons Jump straight into the most popular chip inference benchmark comparisons, curated and ready to explore. Kimi K3 — First LookNew First benchmarks of Kimi K3 across every available chip. New configurations appear here as they come online. KimiK3 GB200 NVL72 vs B200 — Multi vs Single Node GB200 NVL72 Dynamo TRTLLM vs B200 Dynamo TRTLLM on DeepSeek R1 (8k/1k) at FP4. DeepSeekGB200B200DynamoFP4NVL72 B200 vs H200 — Blackwell vs Hopper Blackwell B200 vs Hopper H200 Dynamo TRTLLM throughput per chip on DeepSeek R1 (8k/1k) at FP8. DeepSeekB200H200DynamoFP8 AMD MI300X → MI325X → MI355X Three generations of AMD Instinct on SGLang at FP8. Generational throughput scaling on DeepSeek R1 (8k/1k). DeepSeekMI300XMI325XMI355XSGLangFP8 H100 vs GB300 Disagg — DeepSeek H100 FP8 disagg vs GB300 FP8 disagg vs GB300 FP4 disagg on DeepSeek R1 (8k/1k). DeepSeekH100GB300DisaggFP8FP4 Disagg B200 SGLang vs MI355X vs B200 TRTLLM Disaggregated B200 Dynamo SGLang vs MI355X MoRI SGLang vs B200 Dynamo TRTLLM on DeepSeek R1 (8k/1k) at FP8. DeepSeekB200MI355XDynamoMoRIFP8Disagg MI355X SGLang Disagg Over Time — DeepSeek (FP8) MI355X SGLang dis…