Ludi₀.₁: An Agentic System for Socially Intelligent Robots
Published: August 19, 2026
Author:
Report: Download PDF | arXiv (coming soon)
Contact: research@ludorobotics.ai
Robotics has advanced rapidly in recent years. Action foundation models, including models based on pretrained large language models (LLMs) and world models, have demonstrated increasingly capable perception, navigation, and manipulation.
Much of the field has focused on expanding what robots can physically accomplish. At Ludo Robotics, we believe that physical capability alone is not enough. Robots also need social intelligence: the ability to let everyday people guide and collaborate with them naturally through speech, gesture, and shared understanding.
We believe now is the right time to address this problem directly.
Social Intelligence in Action
It's a weekday evening. Jake is at the kitchen sink, sleeves pushed up, working through a stack of dishes. Chloe has been at her desk in the bedroom for the past two hours. Being the thoughtful partner that he is, Jake calls out to Ludi, a social robot standing nearby:
“Hey Ludi, could you bring Chloe a Coke?”
On the table are two soda cans, a Coke and a Pepsi. Ludi says:
“Sure.”
It turns toward the table, and reaches for the red can. But Jake got it wrong, and he realizes it several seconds later:
“Oh, wait. She actually prefers Pepsi.”
The robot should promptly interpret this correction and revise its behavior. Depending on its current physical state, it might redirect its reach toward the Pepsi can or safely set down the Coca-Cola before retrieving the Pepsi.
Despite rapid progress in robotic perception and control, current systems cannot yet perform this kind of interaction reliably across varied environments and tasks. Doing so requires a robot to understand spoken input while acting, track the user’s evolving intent, reconcile long-term preferences with immediate requests, revise an ongoing action when circumstances change, communicate the revised plan, and carry it out safely and smoothly. These capabilities must operate together seamlessly and in real time.
Ludo Robotics was founded to make this kind of fluid,
natural human–robot collaboration possible.
Ludi₀.₁: An Agentic Approach
Ludi₀.₁ is our first system designed to integrate robot perception, navigation, and manipulation with interactive speech, dialogue, memory, and social reasoning in a functioning robot. We deliberately chose an agentic architecture rather than relying on a single end-to-end model.
At the center of Ludi₀.₁ is the Ludi Brain, a fine-tuned vision-language model that provides the agent’s perception, reasoning, dialogue, and tool-selection capabilities. We trained and fine-tuned this model with simulated demonstrations of complex tasks and human–robot interactions, teaching it not only to recognize scenes and follow instructions, but to reason about evolving situations, choose among available tools, and respond appropriately as the interaction unfolds. The resulting VLM perceives the current scene, interprets the user’s request, reasons over the interaction history, and decides what the robot should say or do next. It can respond through speech, wait for more information, navigate, or invoke specialized vision-language-action (VLA) policies to carry out manipulation skills such as picking and placing objects.
The agent operates within a harness that manages the interaction loop. The harness assembles the current context, calls the VLM, validates and executes its selected tool, records the result, and calls the VLM again with the updated context. A single user request can therefore produce several cycles of reasoning, tool use, and observation before the interaction returns to the user. The VLM decides what should happen next; the harness manages how that decision is executed and incorporated into the ongoing interaction.
User speech initiates a new agent turn, giving the user direct control over the interaction as it unfolds. When the VLM decides to speak, its response is converted to speech asynchronously, allowing reasoning and action to continue while the robot talks. New user input can arrive at any time and redirect the robot even while it is speaking or acting.
The harness maintains a shared interaction record containing dialogue, visual observations, model decisions, tool calls, and outcomes. The VLM reasons over this record and contributes new responses and actions. As the interaction grows, an auxiliary model compacts older context in the background without disrupting an active turn. All inference and system operation can run locally, without relying on external model APIs or cloud-based inference. In our demos, Ludi₀.₁ ran on an onsite GPU workstation connected to the robot.
We refer to the complete system (including the VLM, control harness, speech components, navigation system, and manipulation tools) as Ludi₀.₁.
What Ludi₀.₁ Can Do
Within its current set of supported environments and skills, Ludi₀.₁ can engage in spoken interaction, interpret the surrounding scene, ask clarifying questions, navigate indoors, and carry out manipulation tasks using its underlying VLA models. It can maintain context across an extended interaction and revise an ongoing plan when the user provides new information.
Together, these capabilities allow Ludi₀.₁ to handle interactions that cannot be reduced to isolated perception, dialogue, navigation, or control tasks. The robot can move fluidly between understanding a request, gathering missing information, explaining its intent, acting in the physical world, and revising its behavior in response to the user.
The goal is not simply to make robots more conversational. Speech must remain grounded in what the robot sees, remembers, plans, and does, so that communication and physical action unfold as one coherent, ongoing interaction in the real world.
Demo — Hand Over Interaction
Demo — Checking Status of the House Environment
Demo — Communication Scenarios
The Ludi₀.₁ Brain: A VLM for Interactive Robotics
The Ludi Brain is a fine-tuned vision-language model that serves as the decision-making core of Ludi₀.₁. At each turn, it combines the robot’s ego-view and panoramic camera images with transcribed speech, interaction history, and results from previous actions to decide what the robot should do next—speak, ask for clarification, navigate, manipulate an object, or wait.
Rather than relying only on prompting a general-purpose VLM, we fine-tuned the model on synthetic multi-turn interaction traces designed around the situations that make human–robot collaboration difficult: ambiguous requests, corrections, interruptions, multi-step tasks, and conversations that mix social and task-oriented dialogue. This training teaches the model not just to recognize scenes and follow instructions, but to reason about how an interaction is evolving and adapt its behavior accordingly.
Generating the training data. The training corpus is fully synthetic. Scenarios are composed combinatorially over tasks, objects, and interaction events: a delivery that gets corrected mid-way, a request that matches two objects, a question that arrives while the robot is walking. Each scenario is rendered into a multi-turn tool-call trace with intermediate reasoning, paired with scene imagery from generated images and simulator renders of environments that mirror the deployment space.
Fine-tuning. We fine-tuned Qwen3.5 at the 4B and 9B scale with supervised learning on these traces. Plain supervised fine-tuning outperformed the distillation- and prompt-optimization-based alternatives we evaluated. The fine-tuned 4B stays close to the fine-tuned 9B on our benchmark, so we deploy the smaller, lower-latency model on the robot.
Evaluate the model. Interaction Core28 is our closed-loop simulation benchmark: 28 scenarios where the agent serves a user end-to-end and both the dialog and the task must succeed. Fine-tuning doubles success, from 10 to 17 of 28, near a prompt-optimized frontier model at 26.
The Ludi₀.₁ Harness and Tools: From Decisions to Physical Action
Ludi₀.₁’s harness is designed to turn the Ludi Brain’s decisions into responsive, reliable interaction. It manages context and tool execution while compacting history in the background, using a fast primary VLM with a stricter fallback, pre-filling visual input to reduce latency, and queuing speech asynchronously so talking does not block reasoning or action.
The harness connects the Ludi Brain to specialized physical skills. For navigation, KISS-ICP with an EKF provides localization, named keypoints support reliable movement between important locations in the home, and a custom continuous planner adds finer humanoid-specific motion when needed. Manipulation is handled by fine-tuned GR00T N1.7 VLA policies trained on real-robot teleoperation data for household pick-and-place tasks. The Ludi Brain decides what should happen next; the harness and tools carry those decisions into the physical world.
The Limits (and Promise) of Agentic Robotics
Ludi₀.₁ is modular at the execution layer but integrated at the interaction layer: the VLM reasons over shared context and chooses what to do next, while the harness coordinates specialized speech, navigation, and manipulation tools. This enables clarification, interruption, memory-informed behavior, and tighter coordination between conversation and physical action.
The limitation is that this integration is orchestrated rather than fully learned. We are therefore pursuing two paths in parallel: continuing to improve the agentic system today, while using the multimodal interaction traces it produces to train a more deeply integrated foundation model for robots and people.
Toward Ludi 1.0: A Foundation Model for Robots and People
Ludi 1.0 is the second part of our effort: a foundation model for robots and people that more deeply integrates perception, dialogue, memory, reasoning, and control. Rather than treating social intelligence as an interface layered on top of robot behavior, Ludi 1.0 aims to ground social understanding and physical action in a shared representation, so each can continually inform the other.
Such a model must understand what the robot is doing and why, what the user expects, how their intent is changing, and how the robot’s physical state constrains what it can do next. It must also know when to explain, clarify, revise, pause, or stop.
The goal is to move more of the intelligence now achieved through orchestration into the learned model itself, integrating social understanding, memory, reasoning, perception, and physical action within a shared representation. Ludi₀.₁ provides both a working system and the interaction traces needed to develop this direction, toward robots that understand people, communicate naturally, and act as collaborative partners through one coherent, ongoing interaction.
We plan to release Ludi 1.0 later this year. Stay tuned.