LFM2.5 Q4\_0 Checkpoints from Quantization-Aware Distillation
Smaller, more efficient model versions could make advanced AI more accessible to researchers and developers with limited computing resources.
Get smart on it
Quantization-Aware Distillation (QAD) is a technique that compresses AI models into smaller 4-bit versions by training a smaller "student" model using knowledge from a larger "teacher" model, rather than shrinking an already-trained model. This matters because the resulting compressed models run on edge devices like phones and laptops with the same speed and memory efficiency as standard compression methods, but recover 97 percent of the accuracy lost in the compression process. The LFM2.5 models in four sizes have been released as QAD checkpoints that can be used with standard AI inference software.
Related news
NVIDIA Releases TensorRT Model Connect in Public Preview: Hugging Face Checkpoint to Native C++ Inference in Two Commands
NVIDIA has released TensorRT Model Connect (TRTMC) in public preview, an open-source project that takes a supported Hugging Face or local checkpoint to end-to-end TensorRT inference in two commands. There is no intermediate ONNX export step. The build produces a versioned .bundle artifact that runs through native C++ task APIs, so inference can execute in a C++ service, embedded application, or robotics stack without PyTorch in the runtime path. The project is Apache-2.0 licensed and ships as a
Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
These models improve search accuracy by capturing multiple semantic meanings per word, helping systems better match user intent with relevant results.
