Epoch AI launches Inference Performance Index
Epoch AI: Inference Performance Index: changing the way we measure inference capacity
Epoch AI's Inference Performance Index, which goes beyond the underestimates of FLOP/s measures and scores chips on memory and communication
Get smart on it
The Inference Performance Index is a new measurement tool that compares how much artificial intelligence work different computer chips can handle, going beyond traditional measures of raw computing speed. It matters because memory and communication between chips often have a bigger impact on inference performance than arithmetic speed alone, so comparing chips by one metric can be misleading by a factor of ten or more. The index measures capacity by tracking how many simultaneous requests a chip can serve across different required output speeds, accounting for the tradeoff between handling many requests at once versus generating responses quickly for each individual request. Version 1 of the index focuses on agentic inference workloads and uses a specific chip as a reference point to express all other chips' capacity in comparable terms.
Relevant courses
Want to go deeper than the news? Explore live, cohort-based AI courses taught by practitioners.
Browse AI courses on MavenRelated news
Google Unveils TPU 8i Max and TPU 8tx: A Dual-Chip TPU and a Confidential TPU
Google has added two new systems to its eighth-generation TPU lineup. TPU 8i Max targets massive context windows and rack density. TPU 8tx targets secure, confidential inference. Google TPU 8i Max is a dual-chip package built from 2 TPU 8i chips. Both arrive alongside TPU 8t and TPU 8i, announced in April. TL;DR Size: TPU 8i Max = 2 TPU 8i chips in 1 package. TPU 8tx = TPU 8t silicon plus 5th Gen AMD EPYC hosts. Runs on: Google Cloud TPU systems. Performance: 8i Max doubles TPU 8
Also reported by Data Center Knowledge
Anthropic joins AI’s natural gas frenzy, buys $3.4B worth of turbines
Anthropic bought billions of dollars worth of natural gas turbines in a deal that could give it buyer's remorse.
Protesters break into Microsoft data center site in Amsterdam
Members of the action group Geef Tegengas put a banner on a crane during a protest. | Image: Getty Images Protesters have broken into a construction site of a new Microsoft data center in Amsterdam this morning. Members of the Geef Tegengas group helped organize the protest against the construction of three 85-meter data center towers in the outskirts of Amsterdam. Reuters reports that around 100 people made it into the construction site, with some scaling a crane to place a "fu
Also reported by Wired AI
We Visited China’s Stargate in Inner Mongolia
Stargate Abilene has become the symbol of America’s AI buildout. It is a gigawatt-scale campus on the Texas plains built for OpenAI. China has campuses on a similar scale, though few people outside the country have had a look inside. In September, we visited one, and we found the speed, scale, ultra low cost, and deep vertical integration astonishing.During the trip, we toured Inner Mongolia, China’s fastest-growing AI datacenter region. It's the “West Texas” of China, bu
Crusoe sells troubled Wyoming data center project to Google
The deal highlights the challenges of developing massive AI data centers.
What Huawei’s Horinger data center reveals about China’s compute gap
Huawei’s AI data center in Horinger, Inner Mongolia, uses Ascend chips and has an estimated AI computing capacity that is just 7% that of the world’s largest AI data center, SpaceX’s Colossus 2 in Tennessee.