In visibility-based pursuit-evasion tasks, a team of mobile pursuer robots with limited sensing capabilities is tasked with detecting all evaders in a multiply-connected planar environment, whose map may or may not be known to pursuers beforehand.
This requires tight coordination among multiple agents to ensure that the omniscient and potentially arbitrarily fast evaders are guaranteed to be detected by the pursuers.
Whereas existing methods typically rely on a relatively large team of agents to clear the environment, we propose ViPER, a neural solution that leverages a graph attention network to learn a coordinated yet distributed policy via multi-agent reinforcement learning (MARL).
We experimentally demonstrate that ViPER significantly outperforms other state-of-the-art non-learning planners, showcasing its emergent coordinated behaviors and adaptability to more challenging scenarios and various team sizes, and finally deploy its learned policies on hardware in an aerial search task.
@inproceedings{wang2024viper,title={ViPER: Visibility-based Pursuit-Evasion via Reinforcement Learning},author={Wang*, Yizhuo and Cao*, Yuhong and Chiun, Jimmy and Subhadeep, Koley and Mandy, Pham and Sartoretti, Guillaume},booktitle={Conference on Robot Learning},year={2024}}
Privileged Reinforcement and Communication Learning for Distributed, Bandwidth-limited Multi-robot Exploration
Yixiao
Ma, Jingsong
Liang, Yuhong
Cao, Derek Ming Siang
Tan, and Guillaume
Sartoretti
In International Symposium on Distributed Autonomous Robotic Systems (DARS) , 2024
Communication bandwidth is an important consideration in multi-robot exploration, where information exchange among robots is critical. While existing methods typically aim to reduce communication throughput, they either require significant computation or significantly compromise exploration efficiency. In this work, we propose a deep reinforcement learning framework based on communication and privileged reinforcement learning to achieve a significant reduction in bandwidth consumption, while minimally sacrificing exploration efficiency. Specifically, our approach allows robots to learn to embed the most salient information from their individual belief (partial map) over the environment into fixed-sized messages. Robots then reason about their own belief as well as received messages to distributedly explore the environment while avoiding redundant work. In doing so, we employ privileged learning and learned attention mechanisms to endow the critic (i.e., teacher) network with ground truth map knowledge to effectively guide the policy (i.e., student) network during training. Compared to relevant baselines, our model allows the team to reduce communication by up to two orders of magnitude, while only sacrificing a marginal 2.4% in total travel distance, paving the way for efficient, distributed multi-robot exploration in bandwidth-limited scenarios.
@inproceedings{ma2024privileged,title={Privileged Reinforcement and Communication Learning for Distributed, Bandwidth-limited Multi-robot Exploration},author={Ma, Yixiao and Liang, Jingsong and Cao, Yuhong and Tan, Derek Ming Siang and Sartoretti, Guillaume},booktitle={International Symposium on Distributed Autonomous Robotic Systems (DARS)},year={2024}}
IR2: Implicit Rendezvous for Robotic Exploration Teams under Sparse Intermittent Connectivity
Derek Ming Siang
Tan, Yixiao
Ma, Jingsong
Liang, Yi Cheng
Chng, Yuhong
Cao, and Guillaume
Sartoretti
In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2024
Information sharing is critical in time-sensitive and realistic multi-robot exploration, especially for smaller robotic teams in large-scale environments where connectivity may be sparse and intermittent. Existing methods often overlook such communication constraints by assuming unrealistic global connectivity. Other works account for communication constraints (by maintaining close proximity or line of sight during information exchange), but are often inefficient. For instance, preplanned rendezvous approaches typically involve unnecessary detours resulting from poorly timed rendezvous, while pursuit-based approaches often result in short-sighted decisions due to their greedy nature. We present IR2, a deep reinforcement learning approach to information sharing for multi-robot exploration. Leveraging attention-based neural networks trained via reinforcement and curriculum learning, IR2 allows robots to effectively reason about the longer-term trade-offs between disconnecting for solo exploration and reconnecting for information sharing. In addition, we propose a hierarchical graph formulation to maintain a sparse yet informative graph, enabling our approach to scale to large-scale environments. We present simulation results in three large-scale Gazebo environments, which show that our approach yields 6.6-34.1% shorter exploration paths and significantly improved mapped area consistency among robots when compared to state-of-the-art baselines.
@inproceedings{tan2024IR2,title={IR2: Implicit Rendezvous for Robotic Exploration Teams under Sparse Intermittent Connectivity},author={Tan, Derek Ming Siang and Ma, Yixiao and Liang, Jingsong and Chng, Yi Cheng and Cao, Yuhong and Sartoretti, Guillaume},booktitle={2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},year={2024}}
STAR: Swarm Technology for Aerial Robotics Research
Jimmy
Chiun, Yan Rui
Tan, Yuhong
Cao, John
Tan, and Guillaume
Sartoretti
The International Conference on Control, Automation, and Systems, 2024
In recent years, the field of aerial robotics has witnessed significant progress, finding applications in diverse domains, including post-disaster search and rescue operations. Despite these strides, the prohibitive acquisition costs associated with deploying physical multi-UAV systems have posed challenges, impeding their widespread utilization in research endeavors. To overcome these challenges, we present STAR (Swarm Technology for Aerial Robotics Research), a framework developed explicitly to improve the accessibility of aerial swarm research experiments. Our framework introduces a swarm architecture based on the Crazyflie, a low-cost, open-source, palm-sized aerial platform, well suited for experimental swarm algorithms. To augment cost-effectiveness and mitigate the limitations of employing low-cost robots in experiments, we propose a landmark-based localization module leveraging fiducial markers. This module, also serving as a target detection module, enhances the adaptability and versatility of the framework. Additionally, collision and obstacle avoidance are implemented through velocity obstacles. The presented work strives to bridge the gap between theoretical advances and tangible implementations, thus fostering progress in the field.
@article{chiun2024star,title={STAR: Swarm Technology for Aerial Robotics Research},author={Chiun, Jimmy and Tan, Yan Rui and Cao, Yuhong and Tan, John and Sartoretti, Guillaume},journal={The International Conference on Control, Automation, and Systems},year={2024}}
Deep Reinforcement Learning-based Large-scale Robot Exploration
In this work, we propose a deep reinforcement learning (DRL) based reactive planner to solve large-scale Lidar-based autonomous robot exploration problems in 2D action space. Our DRL-based planner allows the agent to reactively plan its exploration path by making implicit predictions about unknown areas, based on a learned estimation of the underlying transition model of the environment. To this end, our approach relies on learned attention mechanisms for their powerful ability to capture long-term dependencies at different spatial scales to reason about the robot’s entire belief over known areas. Our approach relies on ground truth information (i.e., privileged learning) to guide the environment estimation during training, as well as on a graph rarefaction algorithm, which allows models trained in small-scale environments to scale to large-scale ones. Simulation results show that our model exhibits better exploration efficiency (12% in path length, 6% in makespan) and lower planning time (60%) than the state-of-the-art planners in a 130mx100m benchmark scenario. We also validate our learned model on hardware.
@article{cao2024deep,title={Deep Reinforcement Learning-based Large-scale Robot Exploration},author={Cao, Yuhong and Zhao, Rui and Wang, Yizhuo and Xiang, Bairan and Sartoretti, Guillaume},journal={IEEE Robotics and Automation Letters},year={2024},publisher={IEEE}}
2023
Context-Aware Deep Reinforcement Learning for Autonomous Robotic Navigation in Unknown Area
Jingsong
Liang*, Zhichen
Wang*, Yuhong
Cao*, Jimmy
Chiun, Mengqi
Zhang, and Guillaume Adrien
Sartoretti
Mapless navigation refers to a challenging task where a mobile robot must rapidly navigate to a predefined destination using its partial knowledge of the environment, which is updated online along the way, instead of a prior map of the environment. Inspired by the recent developments in deep reinforcement learning (DRL), we propose a learning-based framework for mapless navigation, which employs a context-aware policy network to achieve efficient decision-making (ie, maximize the likelihood of finding the shortest route towards the target destination), especially in complex and large-scale environments. Specifically, our robot learns to form a context of its belief over the entire known area, which it uses to reason about long-term efficiency and sequence show-term movements. Additionally, we propose a graph rarefaction algorithm to enable more efficient decision-making in large-scale applications. We empirically demonstrate that our approach reduces average travel time by up to 61.4% and average planning time by up to 88.2% compared to benchmark planners (D* lite and BIT) on hundreds of test scenarios. We also validate our approach both in high-fidelity Gazebo simulations as well as on hardware, highlighting its promising applicability in the real world without further training/tuning.
@inproceedings{liang2023context,title={Context-Aware Deep Reinforcement Learning for Autonomous Robotic Navigation in Unknown Area},author={Liang*, Jingsong and Wang*, Zhichen and Cao*, Yuhong and Chiun, Jimmy and Zhang, Mengqi and Sartoretti, Guillaume Adrien},booktitle={Conference on Robot Learning},pages={1425--1436},year={2023},organization={PMLR}}
Intent-based Deep Reinforcement Learning for Multi-agent Informative Path Planning
Tianze
Yang, Yuhong
Cao, and Guillaume
Sartoretti
In 2023 International Symposium on Multi-Robot and Multi-Agent Systems (MRS) , 2023
In multi-agent informative path planning (MAIPP), agents must collectively construct a global belief map of an underlying distribution of interest (e.g., gas concentration, light intensity, or pollution levels) over a given domain, based on measurements taken along their trajectory. They must frequently replan their path to balance the distributed exploration of new areas and the collective, meticulous exploitation of known high-interest areas, to maximize the information gained within a predefined budget (e.g., path length or working time). A common approach to achieving such cooperation relies on planning the agents’ paths reactively, conditioned on other agents’ future actions. However, as the agent’s belief is updated continuously, these predicted future actions may not end up being the ones executed by agents, introducing a form of noise/inaccuracy in the system and often decreasing performance. In this work, we propose a decentralized deep reinforcement learning (DRL) approach to MAIPP, which relies on an attention-based neural network, where agents optimize long-term individual and cooperative objectives by explicitly sharing their intent (i.e., medium-/long-term future positions distribution, obtained from their individual policy) in a reactive, asynchronous manner. That is, in our work, intent sharing allows agents to learn to claim/avoid broader areas of the world. Moreover, since our approach relies on learned attention over these shared intents, agents are able to learn to recognize the useful portion(s) of these (imperfect) predictions to maximize cooperation even in the presence of imperfect information. Our comparison experiments demonstrate the performance of our approach compared to its variants and high-quality baselines over a large set of MAIPP simulations.
@inproceedings{yang2023intent,title={Intent-based Deep Reinforcement Learning for Multi-agent Informative Path Planning},author={Yang, Tianze and Cao, Yuhong and Sartoretti, Guillaume},booktitle={2023 International Symposium on Multi-Robot and Multi-Agent Systems (MRS)},pages={71--77},year={2023},organization={IEEE}}
Spatio-temporal attention network for persistent monitoring of multiple mobile targets
Yizhuo
Wang, Yutong
Wang, Yuhong
Cao, and Guillaume
Sartoretti
In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2023
This work focuses on the persistent monitoring problem, where a set of targets moving based on an unknown model must be monitored by an autonomous mobile robot with a limited sensing range. To keep each target’s position estimate as accurate as possible, the robot needs to adaptively plan its path to (re-)visit all the targets and update its belief from measurements collected along the way. In doing so, the main challenge is to strike a balance between exploitation, i.e., re-visiting previously-located targets, and exploration, i.e., finding new targets or re-acquiring lost ones.
Encouraged by recent advances in deep reinforcement learning, we introduce an attention-based neural solution to the persistent monitoring problem, where the agent can learn the inter-dependencies between targets, i.e., their spatial and temporal correlations, conditioned on past measurements.
This endows the agent with the ability to determine which target, time, and location to attend to across multiple scales, which we show also helps relax the usual limitations of a finite target set with prior positional information.
We experimentally demonstrate that our method outperforms other baselines in terms of number of targets visits and average estimation error in complex environments.
Finally, we implement and validate our model in a drone-based simulation experiment to monitor mobile ground targets in a high-fidelity simulator.
@inproceedings{wang2023spatio,title={Spatio-temporal attention network for persistent monitoring of multiple mobile targets},author={Wang, Yizhuo and Wang, Yutong and Cao, Yuhong and Sartoretti, Guillaume},booktitle={2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},pages={3903--3910},year={2023},organization={IEEE}}
Ariadne: A reinforcement learning approach using attention-based deep networks for exploration
In autonomous robot exploration tasks, a mobile robot needs to actively explore and map an unknown environment as fast as possible. Since the environment is being revealed during exploration, the robot needs to frequently re-plan its path online, as new information is acquired by onboard sensors and used to update its partial map. While state-of-the-art exploration planners are frontier- and sampling-based, encouraged by the recent development in deep reinforcement learning (DRL), we propose ARiADNE, an attention-based neural approach to obtain real-time, non-myopic path planning for autonomous exploration. ARiADNE is able to learn dependencies at multiple spatial scales between areas of the agent’s partial map, and implicitly predict potential gains associated with exploring those areas. This allows the agent to sequence movement actions that balance the natural trade-off between exploitation/refinement of the map in known areas and exploration of new areas. We experimentally demonstrate that our method outperforms both learning and non-learning state-of-the-art baselines in terms of average trajectory length to complete exploration in hundreds of simplified 2D indoor scenarios. We further validate our approach in high-fidelity Robot Operating System (ROS) simulations, where we consider a real sensor model and a realistic low-level motion controller, toward deployment on real robots.
@inproceedings{cao2023ariadne,title={Ariadne: A reinforcement learning approach using attention-based deep networks for exploration},author={Cao, Yuhong and Hou, Tianxiang and Wang, Yizhuo and Yi, Xian and Sartoretti, Guillaume},booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)},pages={10219--10225},year={2023},organization={IEEE}}
2022
CAtNIPP: Context-aware attention-based network for informative path planning
Informative path planning (IPP) is an NP-hard problem, which aims at planning a path allowing an agent to build an accurate belief about a quantity of interest throughout a given search domain, within constraints on resource budget (e.g., path length for robots with limited battery life). IPP requires frequent online replanning as this belief is updated with every new measurement (i.e., adaptive IPP), while balancing short-term exploitation and longer-term exploration to avoid suboptimal, myopic behaviors. Encouraged by the recent developments in deep reinforcement learning, we introduce CAtNIPP, a fully reactive, neural approach to the adaptive IPP problem. CAtNIPP relies on self-attention for its powerful ability to capture
dependencies in data at multiple spatial scales. Specifically, our agent learns to form a context of its belief over the entire domain, which it uses to sequence local movement decisions
that optimize short- and longer-term search objectives. We experimentally demonstrate that CAtNIPP significantly outperforms state-of-the-art non-learning IPP solvers in terms of solution quality and computing time once trained, and present experimental results on hardware.
@inproceedings{cao2023catnipp,title={CAtNIPP: Context-aware attention-based network for informative path planning},author={Cao, Yuhong and Wang, Yizhuo and Vashisth, Apoorva and Fan, Haolin and Sartoretti, Guillaume Adrien},booktitle={Conference on Robot Learning},pages={1928--1937},year={2022},organization={PMLR}}
Distributed reinforcement learning for robot teams: A review
Recent advances in sensing, actuation, and computation have opened the
door to multi-robot systems consisting of hundreds/thousands of robots, with promising applications
to automated manufacturing, disaster relief, harvesting, last-mile delivery, port/airport operations,
or search and rescue. The community has leveraged model-free multi-agent reinforcement learning
(MARL) to devise efficient, scalable controllers for multi-robot systems (MRS). This review aims
to provide an analysis of the state-of-the-art in distributed MARL for multi-robot cooperation. Decentralized MRS face fundamental challenges, such as non-stationarity and
partial observability. Building upon the “centralized training, decentralized execution” paradigm,
recent MARL approaches include independent learning, centralized critic, value decomposition, and
communication learning approaches. Cooperative behaviors are demonstrated through AI benchmarks and fundamental real-world robotic capabilities such as multi-robot motion/path planning. This survey reports the challenges surrounding decentralized model-free MARL
for multi-robot cooperation and existing classes of approaches. We present benchmarks
and robotic applications along with a discussion on current open avenues for research.
@article{wang2022distributed,title={Distributed reinforcement learning for robot teams: A review},author={Wang, Yutong and Damani, Mehul and Wang, Pamela and Cao, Yuhong and Sartoretti, Guillaume},journal={Current Robotics Reports},volume={3},number={4},pages={239--257},year={2022},publisher={Springer}}
Dan: Decentralized attention-based neural network for the minmax multiple traveling salesman problem
Yuhong
Cao, Zhanhong
Sun, and Guillaume
Sartoretti
In International Symposium on Distributed Autonomous Robotic Systems , 2022
The multiple traveling salesman problem (mTSP) is a well-known NP-hard problem with numerous real-world applications. In particular, this work addresses MinMax mTSP, where the objective is to minimize the max tour length among all agents. Many robotic deployments require recomputing potentially large mTSP instances frequently, making the natural trade-off between computing time and solution quality of great importance. However, exact and heuristic algorithms become inefficient as the number of cities increases, due to their computational complexity. Encouraged by the recent developments in deep reinforcement learning (dRL), this work approaches the mTSP as a cooperative task and introduces DAN, a decentralized attention-based neural method that aims at tackling this key trade-off. In DAN, agents learn fully decentralized policies to collaboratively construct a tour, by predicting each other’s future decisions. Our model relies on attention mechanisms and is trained using multi-agent RL with parameter sharing, providing natural scalability to the numbers of agents and cities. Our experimental results on small- to large-scale mTSP instances (50 to 1000 cities and 5 to 20 agents) show that DAN is able to match or outperform state-of-the-art solvers while keeping planning times low. In particular, given the same computation time budget, DAN outperforms all conventional and dRL-based baselines on larger-scale instances (more than 100 cities, more than 5 agents), and exhibits enhanced agent collaboration.
@inproceedings{cao2022dan,title={Dan: Decentralized attention-based neural network for the minmax multiple traveling salesman problem},author={Cao, Yuhong and Sun, Zhanhong and Sartoretti, Guillaume},booktitle={International Symposium on Distributed Autonomous Robotic Systems},pages={202--215},year={2022},organization={Springer}}
