WildCap
Funding Source: Cyber Valley
Motivation: Understanding animal behavior, i.e., how they move and interact among each other and with their environment, is a fundamental requirement for addressing the most important ecological problems today. Fast and accurate understanding of animal behavior depends on their accurate 3D body pose and shape estimated over time. This estimation is called '4D motion capture', or MoCap. State-of-the-art methods for animal MoCap either require sensors or markers on the animals (e.g., GPS collars and IMU tags), or rely on camera traps fixed in the animal's environment. Not only do these methods pose danger to the animals due to tranquilization and physical interference, but their scope is also difficult to extend to a larger number of animals in vast environments. In WildCap, we are developing autonomous methods for MoCap of endangered wild animals, which will address the aforementioned issues. Our methods will not require any physical interference with the animals. Our novel approach is to develop a team of intelligent, autonomous and vision-based aerial robots which will detect, track, follow and perform MoCap of wild animals.
Goals and Objectives:
WildCap's goal is to achieve continuous, accurate and on-board 4D MoCap of domestic and endangered wild animal species from multiple, unsynchronized and close-range aerial images acquired in the animal's natural habitat, without any sensors or markers on the animal, and without modifying the environment.
In pursuit of the above goal, the key objectives of this project are
Methodology: Aerial robots with longer autonomy time and payload are critical for continuous and long distance tracking of animals in the wild. To this end, we are developing novel systems, particularly lighter than air vehicles that could potentially address these issues. Furthermore, we are developing formation control strategies for such vehicles to maximize the visual coverage of animals and accuracy in their state estimates. Finally, we are leveraging learning-in-simulation methods to develop algorithms for 4D pose and shape estimation of animals. The methodology will be updated once initial results from the project are obtained.
Publications:
[1] Liu, Y.T., Price, E., Black, M.J., Ahmad, A. (2022) Deep Residual Reinforcement Learning based Autonomous Blimp Control, Accepted to IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
[2] Saini, N., Bonetto, E., Price, E., Ahmad, A., & Black, M. J. (2022). AirPose: Multi-View Fusion Network for Aerial 3D Human Pose and Shape Estimation. IEEE Robotics and Automation Letters, 7(2), 4805–4812.https://doi.org/10.1109/LRA.2022.3145494
[3] Price, E., Liu, Y.T., Black, M.J., Ahmad, A. (2022). Simulation and Control of Deformable Autonomous Airships in Turbulent Wind. In: Ang Jr, M.H., Asama, H., Lin, W., Foong, S. (eds) Intelligent Autonomous Systems 16.IAS 2021. Lecture Notes in Networks and Systems, vol 412. Springer, Cham. https://doi.org/10.1007/978-3-030-95892-3_46
AirCap
Funding Source: Max Planck Institute Grassroots Funding
Motivation: Human pose tracking and full body pose estimation and reconstruction in outdoor, unstructured environments is a highly relevant and challenging problem. Its wide range of applications includes search and rescue, managing large public gatherings, and coordinating outdoor sports events. In indoor settings, similar applications usually make use of body-mounted sensors, artificial markers and static cameras. While such markers might still be usable in outdoor scenarios, dynamic ambient lighting conditions and the impossibility of having environment-fixed cameras make the overall problem difficult. On the other hand, body-mounted sensors are not feasible in several situations (e.g., large crowds of people). Therefore, our approach to the aforementioned problem involves a team of micro aerial vehicles (MAVs) tracking subjects by using only on-board monocular cameras and computational units, without any subject-fixed sensor or marker.
Goals and Objectives:
AirCap's goal is to achieve markerless, unconstrained, human motion capture (mocap) in unknown and unstructured outdoor environments. To that end, our objectives are
Methodology:
Publications:
[1] Deep Neural Network-based Cooperative Visual Tracking through Multiple Micro Aerial Vehicles, Price, E., Lawless, G., Ludwig, R., Martinovic, I., Buelthoff, H. H., Black, M. J., Ahmad, A., IEEE Robotics and Automation Letters, Robotics and Automation Letters, 3(4):3193-3200, IEEE, October 2018, Also accepted and presented in the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
[2] Markerless Outdoor Human Motion Capture Using Multiple Autonomous Micro Aerial Vehicles, Saini, N., Price, E., Tallamraju, R., Enficiaud, R., Ludwig, R., Martinović, I., Ahmad, A., Black, M., Proceedings 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages: 823-832, IEEE, October 2019
[3] Active Perception based Formation Control for Multiple Aerial Vehicles, Tallamraju, R., Price, E., Ludwig, R., Karlapalem, K., Bülthoff, H. H., Black, M. J., Ahmad, A., IEEE Robotics and Automation Letters, Robotics and Automation Letters, 4(4):4491-4498, IEEE, October 2019
[4] AirCap – Aerial Outdoor Motion Capture, Ahmad, A., Price, E., Tallamraju, R., Saini, N., Lawless, G., Ludwig, R., Martinovic, I., Bülthoff, H. H., Black, M. J.
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019), Workshop on Aerial Swarms, November 2019.
[5] Decentralized MPC based Obstacle Avoidance for Multi-Robot Target Tracking Scenarios, Tallamraju, R., Rajappa, S., Black, M. J., Karlapalem, K., Ahmad, A.
2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), pages: 1-8, IEEE, August 2018
[6] An Online Scalable Approach to Unified Multirobot Cooperative Localization and Object Tracking, Ahmad, A., Lawless, G., Lima, P., IEEE Transactions on Robotics (T-RO), 33, pages: 1184 - 1199, October 2017
AeRoShip
Funding Source: University of Stuttgart
Motivation: Lighter-than-air vehicles (LTAVs) such as rigid and nonrigid dirigibles, or airships, are clearly a superior choice for some applications like wildlife monitoring, sports event capture and continuous patrolling of forest regions for anti-poaching tasks. LTAVs are uniquely suited as aerial communication relays, for wildlife monitoring and conservation tasks, which we address in one of our other projects, WildCap. Airships produce little noise, have high energy efficiency and long flight times, display benign collision and crash characteristics, pose low danger and cause little environmental impact. However, their comparably high handling complexity, size, lifting gas requirements and cost create an entry barrier for researchers. Unlike for heavier-than-air drones, there have been no off the shelf flight controllers that support autonomous dirigible flight. Therefore, guidance and control algorithms have to be implemented for each vehicle - even though various suitable control strategies can be found in the literature. Similar to both rotor craft and fixed wing UAVs, dirigibles come in many types of actuator arrangements: Fixed or vectoring main thrusters, differential thrust, different tail fin arrangements and auxiliary thrusters, single or double hull, etc. Thus, a control algorithm for a specific vehicle might not always be applicable to others.
Goals and Objectives:
In project AeRoShip our goal is to develop a team autonomous airships which is capable of monitoring wildlife for long durations. To this end, our objectives are
Methodology:
Publications:
[1] Price, E., Liu, Y. T., Black, M. J., & Ahmad, A. (2022). Simulation and Control of Deformable Autonomous Airships in Turbulent Wind. In M. H. Ang Jr, H. Asama, W. Lin, & S. Foong (Hrsg.), Intelligent Autonomous Systems 16 (S. 608--626). Springer International Publishing. https://doi.org/10.1007/978-3-030-95892-3_46
[2] Liu, Y.T., Price, E., Black, M.J., Ahmad, A. (2022) Deep Residual Reinforcement Learning based Autonomous Blimp Control, 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct 2022.
Autonomous Soaring
Funding Source: Cyber Valley
Motivation: Aeronautics is currently undergoing a major transition worldwide. The pandemic has shown that airline traffic can be drastically reduced with an extremely beneficial effect on atmospheric emissions. At the same time, there are large investments and developments in the field of individual mobility, such as urban air mobility vehicles and unmanned aircraft applications, e.g. for medical transportation or rescue purposes. The majority of these aircraft or aerial vehicles are based on electric propulsion and thus, range and endurance are quite limited. In the case of fixed-wing aircraft, these parameters can be significantly improved by exploiting (harvesting) energy from the atmospheric environment. An extreme example is conventional soaring, which is a glider flight, where a pilot combines experience, skills, knowledge, and perception in a decision-making process in such a way, that updrafts are detected and exploited to maximize flight range while keeping situational awareness at all times. These tasks are very complex and can only be accomplished by highly trained pilots. The objective of this work is to find systematic approaches to autonomously maximize the exploitation of environmental energy for flights with small fixed-wing aircraft (unmanned or manned, <2t), while, at the same time, minimizing the flight duration for a required distance. The underlying problem is the trade of short-term rewarding actions (covering some distance) against actions that are expected to pay off in the long term (mapping and exploiting atmospheric updrafts) while navigating in a complex, particularly hard-to-model environment. This constitutes a challenging decision-making problem. Autonomous soaring serves as a test scenario.
Goals and Objectives:
Publications:
[1] Stefan Notter, Fabian Schimpf, Gregor Müller, and Walter Fichter, Hierarchical Reinforcement Learning Approach for Autonomous Cross-Country Soaring, Journal of Guidance, Control, and Dynamics 2023 46:1, 114-126
AirCapRL
Funding Source: Max Planck Institute Grassroots Funding
Motivation: Realizing an aerial motion capture (MoCap) system for humans or animals involves several challenges. The system's robotic front-end [1,3] must ensure that the subject is i) accurately and continuously followed by all aerial robots (UAVs), and ii) within the field of view (FOV) of the cameras of all robots. The back-end [2] of the system estimates the 3D pose and shape of the subject, using the images and other data acquired by the front-end. The front-end poses a formation control problem for multiple MAVs, while the back-end requires an estimation method.
In existing solutions for outdoor MoCap systems, including ours [1] [2] [3], the front and back end are developed independently. The formation control algorithms of the existing aerial MoCap front ends consider that their objective should be to center the person in every UAV's camera image and she/he should be within a threshold distance to each MAV. These assumptions are intuitive and important. Also, experimentally it has been shown that it leads to a good MoCap estimate. However, it remains sub-optimal without any feedback from the human pose estimation back-end of the MoCap system. This is because the estimated human pose is strongly dependent on the viewpoints of the UAVs.
Goals and Objectives:
In this project, our goal is to develop an end-to-end approach for human and animal motion capture - a MoCap system where the UAV-based front-end is completely driven by the back-end's accuracy.
To this end, our objectives are
Methodology: In our work [4], we introduce a deep reinforcement learning (RL) based multi-robot formation controller for the task of autonomous aerial human motion capture (MoCap). We focus on vision-based MoCap, where the objective is to estimate the trajectory of body pose and shape of a single moving person using multiple micro aerial vehicles. State-of-the-art solutions to this problem are based on classical control methods, which depend on hand-crafted system and observation models. Such models are difficult to derive and generalize across different systems. Moreover, the non-linearity and non-convexities of these models lead to sub-optimal controls. We formulate this problem as a sequential decision making task to achieve the vision-based motion capture objectives, and solve it using a deep neural network-based RL method. We leverage proximal policy optimization (PPO) to train a stochastic decentralized control policy for formation control. The neural network is trained in a parallelized setup in synthetic environments. We performed extensive simulation experiments to validate our approach. Finally, real-robot experiments demonstrate that our policies generalize to real world conditions.
Publications:
[1] Deep Neural Network-based Cooperative Visual Tracking through Multiple Micro Aerial Vehicles, Price, E., Lawless, G., Ludwig, R., Martinovic, I., Buelthoff, H. H., Black, M. J., Ahmad, A., IEEE Robotics and Automation Letters, Robotics and Automation Letters, 3(4):3193-3200, IEEE, October 2018, Also accepted and presented in the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
[2] Markerless Outdoor Human Motion Capture Using Multiple Autonomous Micro Aerial Vehicles, Saini, N., Price, E., Tallamraju, R., Enficiaud, R., Ludwig, R., Martinović, I., Ahmad, A., Black, M., Proceedings 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages: 823-832, IEEE, October 2019
[3] Active Perception based Formation Control for Multiple Aerial Vehicles, Tallamraju, R., Price, E., Ludwig, R., Karlapalem, K., Bülthoff, H. H., Black, M. J., Ahmad, A., IEEE Robotics and Automation Letters, Robotics and Automation Letters, 4(4):4491-4498, IEEE, October 2019
[4] AirCapRL: Autonomous Aerial Human Motion Capture Using Deep Reinforcement Learning, Tallamraju, R., Saini, N., Bonetto, E., Pabst, M., Liu, Y. T., Black, M., Ahmad, A., IEEE Robotics and Automation Letters, IEEE Robotics and Automation Letters, 5(4):6678 - 6685, IEEE, October 2020, Also accepted and presented in the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
ActiveSLAM
Funding Source: Max Planck Institute and the University of Stuttgart
Motivation: Robots that are capable to help humans in everyday tasks, either in workplaces or homes, are becoming rapidly popular. To be fully functional companions, robots should be able to navigate and map unknown environments in a seamless and efficient way -- quickly, using less energy and without navigating unnecessarily. Simultaneous localization and mapping, popularly called SLAM, has been developed mainly as a passive process where robots are only required to follow external control inputs, are controlled directly by humans or where previous knowledge is being exploited through predefined waypoints or landmarks. Active SLAM, on the other hand, refers to an approach in which robots exploit their sensors measurements and based on that take control decisions to increase map information, while simultaneously performing other user-defined tasks in an energy-efficient way.
Goals and Objectives:
Methodology: In our most recent work in ActiveSLAM, we introduce an active visual SLAM approach for our omnidirectional robot 'Robotino' [1]. It focuses both on the exploration of the unknown parts of the environment and on the re-observation of already mapped areas to improve the so called 'coverage information' for a better overall map quality. We employ activeness at two different levels -- the first one acts on the global planning, through informative path planning. This is done by selecting the best path and headings at every waypoint on that path using the information provided by the global occupancy map. The second one influences only the short-term movement using the real-time local distribution of 3D visual features. Inside the utility function, we use Shannon's entropy measure and balance between exploration and coverage behaviours. By exploiting all the available information to drive the camera direction (since our robot is omni directional), we are able to maximize the amount of information gathered during the robot's movement between waypoints [2].
Publications:
[1] Bonetto, E., Goldschmid, P., Pabst, M., Black, M. J., & Ahmad, A. (2022). iRotate: Active visual SLAM for omnidirectional robots. Robotics and Autonomous Systems. https://doi.org/10.1016/j.robot.2022.104102
[2] Bonetto, E., Goldschmid, P., Black, M. J., & Ahmad, A. (2021). Active Visual SLAM with Independently Rotating Camera. 2021 European Conference on Mobile Robots (ECMR), 1–8. https://doi.org/10.1109/ECMR50962.2021.9568791