Skip to content
[ 01 ]Ventures

OpenEye

Open-source, CLI-first perception engine that turns raw video into structured world models for robots and autonomous agents — like Ollama, but for vision AI.

OpenEye
  • Safety-first: human detection with a sub-50ms robot halt
  • Model-agnostic: 8 vision models, one CLI
  • Self-hosted — all inference runs locally, data never leaves your network

OpenEye — The Perception Layer for the Agent Era

Turn raw video into structured world models for robots and autonomous systems.

AI can now reason, plan, and act. But without vision, it cannot truly understand the world.

Today's machines see pixels—not meaning. Robotics, automation, and agent systems lack a unified perception layer that transforms visual input into structured, actionable intelligence. Safety, context, and real-world awareness remain unsolved challenges.

OpenEye was created to change that.

OpenEye is an open-source perception engine that converts raw visual data into real-time world models—enabling machines to see, understand, and act safely in physical environments.

Think Ollama, but for vision AI: model-agnostic, self-hosted, and driven entirely from the command line. Pull a model, run inference, serve it.

OpenEye: open-source eyes for the agent era, with the interactive CLI demo
OpenEye: open-source eyes for the agent era, with the interactive CLI demo

The Problem: Vision Without Understanding

Modern computer vision is fragmented.

  • Detection models identify objects but lack context
  • Systems fail to reason about spatial relationships and hazards
  • Safety is reactive, not built into perception
  • Integration across models, hardware, and workflows is complex

The result: machines that see—but don't understand.

The Solution: Structured Perception for Real-World Intelligence

OpenEye transforms visual input into structured, agent-ready outputs.

It combines real-time detection, spatial reasoning, and multimodal AI to build a continuous perception loop—detecting objects, understanding context, identifying risks, and generating actions in real time.

This is not just vision. It's machine perception.

Six Layers, One Pipeline

  1. Camera feed. Hardware-agnostic ingestion from USB cameras, RTSP streams, video files or simulated feeds.
  2. Vision engine. YOLO26, Grounding DINO, SAM 2 and RF-DETR detect, segment and localize every object in frame.
  3. Scene understanding. Objects, a spatial map and flagged hazards are assembled into a structured scene graph.
  4. LLM reasoning. A VLM (Qwen2.5-VL via OpenRouter or the Nebius Token Factory) interprets context: what changed, what is unsafe, what to do.
  5. Action planner. Structured task decomposition turns intent into executable robot actions.
  6. Robot adapter. Plans are dispatched to the connected controller: Unitree G1, SmolVLA or simulation.

Key Features

Real-Time Object Detection

Process live video streams with low-latency detection and spatial awareness.

Hazard Detection & Safety Layer

Identify risks, enforce safety zones, and halt operations instantly when danger is detected.

Agentic Perception Loop

Continuously perceive, reason, and act—creating a closed-loop system for autonomous agents.

Model-Agnostic Architecture

Integrate with leading vision and multimodal models (YOLO, DINO, SAM, VLMs, and more).

Self-Hosted & Privacy-First

Run entirely on local hardware—your data never leaves your environment.

Open-Source & Extensible

Apache 2.0 licensed—inspect, modify, and deploy freely across any system.

The Safety Guardian

As robots enter homes and workplaces, OpenEye acts as a visual safety layer, monitoring any robot's workspace in real time and intervening before accidents happen.

  • Fast layer: YOLO runs every frame for pure-geometry checks, such as whether a hand is inside the danger zone, giving a low-latency halt with no LLM overhead
  • Smart layer: a VLM analyzes the scene periodically to catch what geometry cannot: a knife that was not there before, an unstable stack, context-dependent risk
  • Halt protocol: on danger, OpenEye freezes the connected robot in under 50ms and resumes only once the workspace is confirmed clear
Detect to halt: under 100ms, end to end.
Safety Guardian dashboard: live status, scene graph and VLM reasoning
Safety Guardian dashboard: live status, scene graph and VLM reasoning

Model-Agnostic by Design

Swap models, combine pipelines and extend with custom adapters. Eight models ship integrated, spanning real-time detection, segmentation, depth and vision-language reasoning, all behind one unified API with ONNX and TensorRT runtimes.

  • YOLOv8 and YOLO26: real-time object detection across 80 COCO classes (Ultralytics)
  • RF-DETR: an end-to-end detection transformer with no NMS (Roboflow)
  • Grounding DINO: open-set detection driven by text prompts (IDEA Research)
  • SAM 2: zero-shot segment-anything for any object in any image (Meta AI)
  • Depth Anything: monocular depth estimation at any resolution
  • SmolVLA: a compact vision-language-action model for robot control
  • Qwen2.5-VL: multimodal reasoning with native visual grounding (Alibaba Qwen)
Model Registry: browse, benchmark and manage adapters
Model Registry: browse, benchmark and manage adapters

A Plug-and-Play CLI

Every capability is one command away, like ffmpeg for machine perception.

  • openeye run: inference on any image with any model, unified JSON out
  • openeye watch: a live feed with real-time detection, safety zones and hazard alerts
  • openeye serve: a FastAPI server with REST endpoints, WebSocket streams and a browser dashboard
  • openeye agent: an agentic perceive, reason, act loop with observation memory and VLM reasoning
  • openeye fleet: register, monitor and deploy to edge devices with canary rollouts and OTA updates
  • openeye mlops and openeye govern: A/B testing, automated retraining, audit trails and policy enforcement
Agentic Loop: YOLO every frame (under 100ms), VLM reasoning every 3 seconds, synthesized into a plan
Agentic Loop: YOLO every frame (under 100ms), VLM reasoning every 3 seconds, synthesized into a plan

Built for the Physical World

  • Robotics: safety-zone enforcement, human-robot coexistence, hazard identification and scene graphs for planning
  • Application debugging: visual regression detection, layout validation and VLM-powered UI analysis in CI/CD
  • Desktop agents: UI element detection and multi-window screen understanding as the eyes of computer-use agents

The Advantage: From Pixels to Decisions

Most systems process images. OpenEye understands environments.

By transforming visual data into structured scene graphs, hazard maps, and action plans, it enables machines to operate safely and intelligently in the real world.

Open Infrastructure, Open Models

Your cameras, your hardware, your data. Every frame is processed on-device or on-prem, so camera feeds never leave the network: deploy on laptops, edge devices or air-gapped servers.

OpenEye is released publicly under aw3-technology on GitHub (v0.2.0, Apache 2.0), installable with pipx install openeye-sh, with an interactive demo and MkDocs documentation. It was built at the Nebius.Build SF: OpenClaw + Robotics Hackathon in 2026.

  • Stack: Python, FastAPI, Typer, Pydantic, React, TypeScript, ONNX and TensorRT
  • Scope: CLI, inference server, web dashboard, fleet and MLOps
  • Repo: github.com/aw3-technology/openeye.sh

Strategic Vision

OpenEye is more than a vision tool—it is the perception infrastructure for autonomous systems.

Its foundation enables:

  • Safe human-robot interaction in homes and workplaces
  • Intelligent robotics across logistics, manufacturing, and consumer environments
  • Visual intelligence for desktop agents and automation systems
  • A standardized perception layer across the agent ecosystem

Within the AW3 ecosystem, OpenEye represents the perception layer—where machines gain the ability to see, interpret, and act on reality.

Final Word

Intelligence without perception is incomplete.

OpenEye gives machines the ability to understand the world—turning vision into awareness, and awareness into action.

See clearly. Act safely. Build the future.