GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained (2026)
First, separate 2 ideas: containers vs. quantization methods Most confusion comes from mixing 2 layers. A container defines how tensors are stored on disk. A quantization method defines how weights are squeezed into fewer bits. Containers: safetensors, GGUF, PyTorch pickle (.bin / .pt). Methods: GPTQ, AWQ, bitsandbytes NF4, llama.cpp K-quants and I-quants. Both at once: EXL2 and EXL3 are a method plus a storage layout tied to one inference library. A quick memory rule of thu
Get smart on it
Large language models can be stored and compressed using several different formats and methods, each making different tradeoffs between file size, speed, and accuracy. The formats define how model data is physically stored on disk, while quantization methods determine how to reduce the precision of weights to save space. Understanding these formats matters because they determine which software can run a model and how much memory and computing power it requires. The main formats discussed are GGUF for the llama.cpp ecosystem, GPTQ and AWQ for general use with various inference libraries, and EXL2 which combines both a quantization method and storage format.
Related news
A Coding Guide to TypeSafe AI Jev: Typed Decisions, Calibrated Confidence, and Speculative Fan-Out with a System One Model
In this tutorial, we work with Jev, TypeSafe AI’s first System One model, which does not generate text at all: we send it a piece of program state and a set of typed questions, and it returns choices, scores, and yes/no probabilities that our code can branch on directly. We install the official Python SDK, make a first call that uses all three question primitives at once, and look at how the shape of the state changes what the model can know. We then recompute the published confidence sta
NVIDIA Releases Nemotron 3 Diarization: A 100M-Parameter Open-Weight Model That Tracks 8 Speakers in Real Time
NVIDIA has released Nemotron 3 Diarization, an open-weight speaker diarization model on Hugging Face. It answers one question about any conversation: who spoke when. The 100M-parameter model tracks up to 8 speakers, including when voices overlap. One checkpoint handles both offline recordings and real-time streaming. Is it deployable? Yes. The weights are released under the OpenMDW License 1.1, which permits commercial use. It runs on Linux through NVIDIA NeMo, using Ampere, Ada Lovelace, Ho
