Context is king: How Avride uses cloud VLMs as a safety net for delivery robots

Delivery robots that operate autonomously on city streets use onboard sensors and local neural networks to handle basic navigation and object detection, but they struggle to understand complex real-world context, such as distinguishing a police officer on routine business from an active crime scene. To address this gap, one company has integrated cloud-based vision-language models that analyze camera snapshots every few seconds to detect unusual or sensitive situations and alert human remote assistants when intervention may be needed. The system preserves privacy by anonymizing faces and license plates on the robot before data leaves the device, and the company uses a plug-and-play architecture to test multiple state-of-the-art models rather than relying on a single provider. This approach evolved from the company's existing practice of using cloud VLMs to automatically filter video data for rare, valuable training scenarios, a technique that proved effective enough to extend into live safety operations.