BackActive development
Active development2025 to presentAuthor

Kurat

Autonomous conversational robot

A companion robot split into three brains, perception, cognition, action, so that any one of them can fail and recover without taking the robot down with it.

Problem

A robot that talks, sees and moves has three subsystems with completely different failure characteristics and timing requirements. Speech recognition stalling should not stop the robot navigating; a lost localization fix should not end a conversation.

Coupling them into one pipeline means the slowest and least reliable component sets the behaviour of the whole robot.

What I built

Three subsystems with independent failure and recovery, running on one embedded platform.

  • Perception, YOLOv8 detection and ORB-SLAM3 localization on RealSense D435i RGB-D
  • Cognition, speech via Whisper, and the conversation system on top
  • Action, Nav2 navigation on the mechanical platform
  • Modular autonomy boundaries so each brain degrades on its own

Architecture

The three-brain split is the architecture: perception, cognition and action communicate through defined interfaces rather than a shared pipeline, so a failure in one is a degraded capability rather than a stopped robot. All three run on a Jetson Orin Nano with a RealSense D435i as the primary sensor.

Three brains, so one can fail

The split is the architecture. Each brain degrades on its own rather than taking the robot down, speech stalling should not stop navigation, and a lost localization fix should not end a conversation.

  1. Perception

    YOLOv8 detection and ORB-SLAM3 localization on RealSense D435i RGB-D.

    Localization lost, navigation degrades, conversation continues

  2. Cognition

    Whisper speech recognition and the conversation system.

    Speech stalls, the robot keeps moving

  3. Action

    Nav2 navigation on the mechanical platform.

    Path blocked, recovery, while the other two brains carry on

  4. Each recovers independently

What runs where

All of it on a single Jetson Orin Nano, which is the constraint that shapes what can run concurrently.

BrainModelsSensor
PerceptionYOLOv8, ORB-SLAM3RealSense D435i RGB-D
CognitionWhisper, conversation systemMicrophone
ActionNav2Odometry and the chassis

Limitations and failure modes

  • Active development, the conversation and navigation subsystems are not yet validated together at length.
  • Running four models on a single Jetson Orin Nano constrains what can run concurrently.

Gallery

Kurat, RealSense D435i and Jetson Orin Nano on a four-wheel chassis
Kurat, RealSense D435i and Jetson Orin Nano on a four-wheel chassis
Live detection running on the robot
Live detection running on the robot

Attribution

Built by me
Kurat is mine, architecture, integration, hardware and software.
Based on external research
YOLOv8, Whisper, ORB-SLAM3 and Nav2 are upstream, integrated here.

Related