RobotX Research Program

Teaching mobile robots long horizon tasks that involve locomotion and manipulation, such as moving to a table to pick up an object and delivering it to a target location, is challenging. We formulate this task as a reinforcement learning problem. We propose a framework that learns from a large set of uncurated demonstrations of the robot interacting with different environments and a few task-specific expert demonstrations. Our method consists of three modules. First we learn to extract meaningful behaviors from the uncurated demonstrations. Second, we introduce a teacher-student framework that learns to interact with the environment to solve the desired manipulation task. To guide the learning, the teacher rewards the student agent to act similarly to the expert demonstrations. However, when encountered with novel states, the student receives suggestions from the previously extracted behavioral prior. Our end goal is to demonstrate the desired task on a real robotic system, where we do not have access to privileged information as we do in simulation. Crucially, our proposed framework allows an easy integration of real world sensing capabilities that still leverages the benefits of easily accessible, privileged information in simulation. Due to the modularity of our framework, only the student agent’s sensors have to be adjusted, e.g., to an RGB-D camera, whereas the rest of the framework can still benefit from additional information during training. Therefore, our framework will be able to learn complex control tasks and transfer them to the real world, bringing learning-based mobile manipulation robots closer to deployment.

The ALLIES project aims to develop an intelligent legged robot that can understand the environment, perform complex sequences of mobile manipulation tasks, and assist people autonomously with only linguistic instruction as the interface. Such a technology could free people from tedious household chores and improve people’s quality of life, especially for those with motor impairments since it can greatly reduce their dependence on a caregiver.

The legged manipulator developed at RSL can already perform specific predefined tasks but falls short of reasoning about the environment and completing complex sequences of tasks. In this project, we aim to build on top of our initial investigations on semantic understanding and robotic planning and exploration to achieve a high autonomy of the robot allowing it to better comprehend and interact with its surroundings. Large language models (LLMs) will be used to understand verbal commands from a human operator and deconstruct them into a plan that the robot is able to execute. In addition, visual language models (VLMs) and open-vocabulary object detection methods will be used to develop a representation of the environment around the robot and provide it with the perception and localization capabilities it needs to execute the plan successfully.

Novel aerial robotic systems, such as tilt-rotor platforms, have made meaningful in-flight physical interaction possible, enabling a wealth of novel applications in industrial inspection and maintenance. In contrast to regular drones, they use individually servo-actuated arms with propellers to perform thrust vectoring and generate forces in arbitrary directions.

The precise and robust control in flight is highly challenging and prone to modeling mismatches and unknown environmental factors. Classic control approaches applied to tilt-rotor platforms have reached limits in both precision and robustness, mainly due to complex non-linear dynamics and the lack of accurate models. While accurate propeller thrust models exist, the unknown dynamics and inaccuracies (e.g. backlash) of servomotors are a significant impediment to better flight performance. Additionally, in current approaches, the actuator allocation, which distributes desired forces and torques to the different actuators, is blind to the context of flight - e.g. the closeness to obstacles and airflow disturbances or precision/smoothness requirements.

We are convinced that control and modeling mismatches are the main impediments to robust and precise aerial manipulation. We propose to tackle these problems by i) applying data-driven modeling techniques for actuator identification, ii) integrating the resulting model in a comprehensive simulation environment, iii) and subsequently train novel context-dependent control and allocation policies.

This will enable the system to leverage the full actuation capabilities and redundancies of tilt-rotor platforms, with the ultimate aim to create a new generation of aerial workers, capable of accurate force generation for precise flight and physical interaction with the environment.

The main objective of this project is to investigate the benefits and challenges of targeted 3D and semantic reconstruction and to develop quality-adaptive semantically guided Simultaneous Localization and Mapping (SLAM) algorithms. The goal is to make an agent (e.g., Boston Dynamics Spot robot) able to navigate and find a target object (or other semantics) in an unknown or partially known environment while reconstructing the scene in a quality-adaptive manner. We interpret being quality-adaptive as making the reconstruction accuracy and detailedness dependent on finding the target class – i.e., reconstruct only until we are certain that the observed object does not belong to the target class. We divide the project into three work packages (WPs).

First, we will develop a scene representation that is suitable for targeted quality-adaptive reconstruction. The representation has to accommodate 3D in different qualities and scene semantics in a hierarchical manner while allowing efficient exploration and map updates. Second, we will develop targeted SLAM algorithms that allow an agent to find the sought semantic such that both the path traveled and the reconstruction time is close to minimal. In the third WP, we aim at creating shorter loops between learning in simulation and testing in the real world. We want to update learned representations and navigation models with information from how the agent is performing in the real world, toward bridging the gap of successfully applying the agent in the latter. Moreover, this work package includes data collection and it runs in parallel with the other WPs.