Transformers now runs llama.cpp quants
Running quantized models locally reduces costs and latency while keeping data private, making AI more accessible to developers without expensive hardware.
Get smart on it
Hugging Face has added support for running GGUF models, a widely used file format for local AI inference developed by the llama.cpp team, through its transformers library. GGUF packages model weights and metadata in a single file and supports different quantization levels that allow users to reduce file size by trading some precision for smaller memory requirements. This integration allows users to load these models locally on their machines using familiar transformers APIs and reuses underlying kernels from llama.cpp to achieve comparable performance. The initial focus is on Apple Silicon devices, with the ability to serve models through an OpenAI-compatible interface for use with various client applications.
Related news
A Coding Guide to TypeSafe AI Jev: Typed Decisions, Calibrated Confidence, and Speculative Fan-Out with a System One Model
In this tutorial, we work with Jev, TypeSafe AI’s first System One model, which does not generate text at all: we send it a piece of program state and a set of typed questions, and it returns choices, scores, and yes/no probabilities that our code can branch on directly. We install the official Python SDK, make a first call that uses all three question primitives at once, and look at how the shape of the state changes what the model can know. We then recompute the published confidence sta
NVIDIA Releases Nemotron 3 Diarization: A 100M-Parameter Open-Weight Model That Tracks 8 Speakers in Real Time
NVIDIA has released Nemotron 3 Diarization, an open-weight speaker diarization model on Hugging Face. It answers one question about any conversation: who spoke when. The 100M-parameter model tracks up to 8 speakers, including when voices overlap. One checkpoint handles both offline recordings and real-time streaming. Is it deployable? Yes. The weights are released under the OpenMDW License 1.1, which permits commercial use. It runs on Linux through NVIDIA NeMo, using Ampere, Ada Lovelace, Ho
