Deep Reinforcement Learning for Robot Navigation: Concepts, Current Trends, Challenges, and Future Directions

2.1 Transition of path planning from static to dynamic environments

Robot navigation initially focused on static environments before advancing to dynamic, real-time scenarios, with various path planning algorithms proposed over time [15, 16]. In situations without prior knowledge of the environment, a robot can determine its local path reactively by relying on sensor information, an approach known as behavior-based navigation [17, 18] or the artificial potential field method [19]. These are considered reactive approaches in the early evolution of mobile robot local path planning [20, 21]. However, such systems have limitations. While they can help a robot avoid obstacles in basic dynamic environments within a local domain, they are prone to issues such as the local minima problem and cannot guarantee the optimality of the global path.

By utilizing information about a known or partially known environment where a complete or partial map is available to the robot, global path planning can be applied in static environments [20, 22]. In such cases, appropriate algorithms are used to determine the robot’s global path on the given map. Common and well-known classical and heuristic-based algorithms include A* [23, 24], D*, distance wave transform, Dijkstra’s algorithm [25], RRT (rapidly exploring random trees) [26, 27, 28], and PRM (Probabilistic Roadmap). With the evolution of artificial intelligence, other optimization algorithms have been introduced, such as Genetic Algorithm (GA) [29, 30], Ant Colony Optimization (ACO) [31, 32], and Bee Colony Optimization [33], among others. These algorithms have been shown to produce optimal paths with the shortest travel distance. This pre-route computation allows the robot system to rely more on offline processing rather than online computation. Additionally, various intelligent controllers can be embedded into the system to enhance performance, including fuzzy logic [34, 35, 36], neural networks [37, 38], model predictive control (MPC) [39], and proportional-integral-derivative (PID) control [40], and many others.

Recognizing the advantages of having two separate systems, one for global path planning and another for local path planning, many researchers have sought alternative approaches to combine both into a single hybrid system. Such a system enables greater flexibility for robots operating in dynamic environments. In this approach, both systems complement each other: the optimal path to the goal is pre-calculated offline using the global planner before the robot begins moving, while the local reactive method takes over during execution to handle unexpected dynamic obstacles within the environment. As a result, various hybrid methods have been applied in robot navigation, such as improved A* with the Fuzzy Dynamic Window Approach (FDWA) [41], Particle Swarm Optimization (PSO) combined with Simulated Annealing (SA) [42], and Fuzzy Logic integrated with Artificial Potential Field (APF)) [43]. These methods have demonstrated improvements in optimizing travel time and distance through the global planner while ensuring safe obstacle avoidance via the local planner. Despite these successes, challenges remain, particularly the high computational cost in large-scale dynamic environments, where mapping choices and algorithm selection significantly affect performance. Furthermore, the local minima problem may still occur as obstacle complexity increases, especially in real-time dynamic scenarios.

With the introduction of deep reinforcement learning (DRL), robotics navigation has gained significant advantages over conventional path planning and rule-based methods. DRL is well-suited for real-world applications as it can handle continuous action spaces and does not require complete prior knowledge of the environment. This adaptability makes it particularly effective in complex and uncertain scenarios. Over recent years, DRL has been successfully applied to various robot navigation case studies, demonstrating strong performance in challenging environments [44]. It has proven effective in complete navigation systems [45, 46], path planning tasks [47, 48], and obstacle avoidance [10], ensuring safety-aware navigation that prevents collisions and unsafe paths. DRL has also been applied to address specific motion planning problems [49] and to navigate different terrain types [50]. Several examples highlight DRL’s versatility. In a static environment, Geetha Sharma et al. [5] showed that autonomous unmanned aerial vehicles (UAVs) using the Soft Actor-Critic (SAC) algorithm could efficiently plan paths, avoid obstacles, and optimize trajectories. In dynamic environments with varying numbers of obstacles, TurtleBot 3 employed the Deep Deterministic Policy Gradient (DDPG) algorithm combined with Voronoi diagrams, demonstrating high effectiveness [51]. In highly dynamic scenarios, UAVs equipped with a layered Deep Q-Network (DQN) achieved faster convergence, improved time and speed performance, and better generalization capabilities [52].

The traditional robot navigation framework is the approach where the robot creates a map of its surroundings using LIDAR or camera inputs through simultaneous localization and mapping (SLAM) [53]. Based on the generated local and global maps, the path to the goal point is planned while avoiding obstacles in the environment. However, deep reinforcement learning (DRL) has emerged as a powerful alternative due to its ability to handle high-dimensional problems and process large amounts of input data through deep neural networks. As a result, DRL has been used to replace or integrate with traditional navigation frameworks, particularly in navigation tasks. Figure 3 demonstrates the interaction process between an agent and the environment in a typical DRL-based navigation framework. In this setup, the agent assumes roles traditionally handled by separate modules such as robot localization, map building, and local path planning.

In summary, the evolution of path planning algorithms has progressed in proportion to the complexity of the environments in which robots operate. The more challenging and dynamic the environment, especially in real-time scenarios with moving objects and dense surroundings, the more advanced and adaptable the algorithms need to be. Such algorithms must be flexible enough to adapt to changes over time, learn from experience, generalize to unseen scenarios, and handle dynamics and uncertainty without requiring explicit mapping. This contrasts with conventional algorithms, which rely on predefined maps and world models and perform well only in known, static environments. Thanks to these capabilities, DRL has made significant contributions to the field. Krishna Teja et al. [16] presents a graph showing the number of cases and algorithms applied across different environmental domains. According to the reviewed literature, DRL is the most widely applied method for robot path planning in dynamic and hybrid environments, outperforming many conventional algorithms in terms of both effectiveness and efficiency [44].

2.2 Sample inefficiency

To learn an effective policy, deep reinforcement learning (DRL) requires extensive sampling and interaction with the environment. However, sample efficiency is often low due to the large number of samples needed. Several strategies can improve sample efficiency, such as using off-policy algorithms (e.g., DQN, DDPG, SAC) that reuse past experiences [55]. Compared to on-policy algorithms like PPO, SAC often performs better. Experience replay [56, 57, 58] further enhances learning by enabling more training per sample. Improving exploration can also prevent sample wastage; for example, SAC leverages entropy regularization to encourage more effective exploration [57].

In addition, imitation learning, model-based reinforcement learning, and improved reward design [3, 59] can further boost efficiency. Imitation learning and model-based DRL allow agents to avoid starting from scratch, reducing the number of interactions required. Proper reward design significantly affects the learning process, as sparse or poorly shaped rewards hinder efficiency. Metaheuristic algorithms can be integrated with DRL to optimize reward shaping or path parameters [59, 60, 61] fine-tuning reward weights or structures to make them more informative. This accelerates convergence, improves generalization, and reduces wasted samples from misleading rewards—benefits that are particularly valuable in real-world robotics and simulation-to-real transfer, where sample collection is costly.

To address instability in DRL for continuous control, Gwang Jong Ko et al. [62] proposed a two-phase optimization framework, where a swarm intelligence algorithm is applied at the end of the learning phase. Experiments demonstrated that this approach achieved superior and more stable performance than conventional DRL methods in robot locomotion tasks. Finally, simulators such as GAZEBO [14, 63] and NVIDIA Isaac SIM [9] can significantly improve sample efficiency by allowing extensive pre-training in virtual environments before deployment in the real world. This enables agents to generalize better with fewer real-world samples.

2.3 Integration with perception and mapping

To improve the robustness and efficiency of deep reinforcement learning (DRL) in robotics, tighter integration with core robot subsystems such as perception and mapping is essential. For example, snake robots navigating in unknown dynamic environments can leverage multiple onboard sensors to reach a goal while avoiding obstacles [64]. A deep Q-Learning-based path planning method combined with simultaneous localization and mapping (SLAM) and visual perception from 2D LiDAR and IMU enables end-to-end navigation. Results show that active SLAM improves the success rate and reduces collisions compared with traditional A* and sampling-based RRT* algorithms.

Rui Wang et al. [65] demonstrated this with an aerial robot exploring an unknown environment. SLAM was employed to map the area, while an improved frontier-based exploration strategy integrated with DRL and a neural network architecture guided the robot toward target points. The proposed method consistently explored environments of varying sizes, requiring shorter travel distances than competing approaches.

Hybrid models also show significant benefits. In a simulated indoor maze with moving obstacles, a mobile robot using LiDAR and IMU inputs achieved higher speed and goal-reaching success rates than standard DRL. In this case, Proximal Policy Optimization (PPO) learned obstacle avoidance strategies, while a Genetic Algorithm optimized hyperparameters such as discount factors, reward weights, and network layer sizes before training. This integration not only improved sample efficiency and robustness but also enhanced dynamic obstacle avoidance.

2.4 Multi-agent robot research

A current research trend focuses on multi-agent systems, where the number of robots is increasing. Commonly referred to as swarm or cooperative navigation tasks as well as collaborative and competitive multi-robot systems [66, 67, 68, 69]. When multiple agents work cooperatively, tasks can be distributed among them, improving overall performance performances [70]. To enable agents to learn effectively and adapt to various domains, DRL has been applied in multi-agent case studies to enhance scalability as the number of agents increases [45].

Multi-robot path finding (MRPF) [47] has been proposed to achieve efficient coordination and collaboration in high-dimensional multi-agent systems. The framework is built on DRL using a two-layer concept, and by integrating this with a hierarchical reward mechanism, the success rate improves by more than 20% compared to existing algorithms, leading to better generalization and efficiency.

An innovative DRL variant, known as the DRL-MPCGNNs model, integrates DRL with model predictive control (MPC) and graph neural networks (GNN) for optimized path planning and task allocation in multi-agent systems. Proposed by ZhiXian Li et al. [71], this approach significantly improves path planning efficiency, task allocation effectiveness, and inter-robot collaboration performance.

2.5 Multi-objective navigation

As robot navigation tasks grow increasingly complex, research focus has shifted from achieving single objectives to addressing multiple objectives through advanced variations of Deep Reinforcement Learning (DRL). For example, Chien-Lun Cheng et al. [53] proposed a crowd-aware, multi-objective DRL-based navigation system called multi-objective dual-selection reinforcement learning (MODSRL), designed to navigate efficiently in crowded environments. By employing an appropriately designed reward function, MODSRL simultaneously achieves multiple objectives, including safety, time efficiency, collision avoidance, and smooth trajectory generation, demonstrating strong system robustness.

In the domain of aerial robotics, Jiahao Wu et al. [71] introduced a multi-objective navigation reinforcement learning (MONRL) framework for drones to navigate and avoid obstacles in unknown environments. Leveraging DRL, the drone learns navigation policies that optimize path planning while mitigating wind disturbances using camera input. The method was evaluated in both a virtual environment and a real-world model of New York City, achieving effective and adaptive navigation performance.

2.6 Simulation-to-real transfer

Agents are typically trained in simulation to avoid the high cost, risk, and inefficiency associated with real-world training. However, generalization often fails when transferring policies from simulation to real-world environments due to the simulation-to-reality gap. To address this challenge, several approaches can be adopted, including domain adaptation, domain randomization, world models, and simulation fidelity improvement.

For domain adaptation, methods such as feature-level alignment [72], adversarial training, or fine-tuning can be applied to map sensor inputs such as camera or LiDAR data from simulated to real-world domains. Domain randomization improves robustness and generalization by exposing the robot to randomized simulation conditions during training, including variations in lighting, obstacle properties, and dynamics.

Using high-fidelity simulators such as NVIDIA Omniverse, Gazebo, MuJoCo, and Webots [73] which incorporate realistic noise, latency, and sensor inaccuracies helps bridge the gap by better matching real-world physics and sensor data. In practice, most researchers train all policies in simulation before transferring them to real robots, and this approach has proven effective in achieving generalization across quadruped, mobile, and humanoid robot platforms.

3.1 Challenge working in continuous action spaces

The real environment, which matches continuous action spaces, poses several challenges for deep reinforcement learning, such as high computational complexity, instability of control, exploration difficulties, and low sampling efficiency [3]. A few algorithms have been designed to handle continuous actions without discretization, including Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC). In these methods, the policy is modeled using a continuous distribution such as a Gaussian, from which the agent samples an action and then deterministically outputs that action.

3.2 Time intensive

Training robots in real-world environments is time-consuming due to the complexity of the process. Transfer learning offers an alternative by enabling models to be trained in simulations before fine-tuning in the real world, thereby accelerating learning and reducing data requirements. This approach also improves the adaptability and responsiveness of DRL models to dynamic and changing real-world scenarios. Consequently, many researchers focus on simulation-based training [10, 48, 75, 76] or a combination of simulation and experimental setups [45] to ensure robust generalization. Studies have shown that DRL models trained in simulations can perform effectively in real-world environments [10, 14], demonstrating that good generalization is achievable. To bridge the simulation-to-reality gap, validation must be conducted in both domains to ensure effective domain adaptation.

3.3 Transfer from simulation to the real world

Testing in simulation differs significantly from real-world deployment due to variations in sensors, noise, dynamics, and environment. These discrepancies can cause learned policies to fail when applied to real robots, potentially leading to unsafe and unpredictable conditions. Additionally, latency issues in real-time hardware and controllers may impair DRL policy performance. Policies trained in one environment often struggle to generalize to new or slightly altered settings, necessitating transfer learning [77] and domain adaptation, both challenging for DRL.

To address these challenges, several techniques are commonly employed and should be continuously refined. These include training robust policies that are less sensitive to small environmental changes [78], using high-fidelity simulators [73], initiating training in simulation followed by fine-tuning with real-world data, and applying domain randomization during simulation to improve policy generalization across diverse robotic settings.

3.4 Exploration vs. exploitation: Complexities in reward design

In reinforcement learning, an agent must balance exploiting known strategies to maximize rewards while exploring new actions to discover better ones. Deep reinforcement learning (DRL) trains agents by maximizing cumulative rewards received from the environment, enabling them to learn desired behaviors. However, balancing exploration and exploitation is challenging, especially in high-dimensional or sparse-reward environments [58]. Poorly designed rewards can lead to suboptimal learning, unintended behaviors, or agents getting stuck in local optima due to inadequate exploration. For example, in navigation tasks, rewards may be higher when the agent is near the goal and penalized when far away [14].

In Soft Actor-Critic (SAC), the reward function directly influences exploitation and indirectly affects exploration [79]. SAC’s trade-off is based on maximum entropy reinforcement learning, where the agent maximizes both expected returns and policy entropy (encouraging stochastic actions). Fine-tuning techniques such as adaptive and dynamic entropy adjustment have been developed to optimize SAC’s performance. Improper entropy tuning can cause excessive randomness, undermining learning efficiency.

While entropy encourages exploration, it may also lead to suboptimal actions in some applications. A well-shaped reward function accelerates learning by providing clear signals for exploitation, driving the agent toward desired behavior. Without meaningful rewards, the agent relies heavily on entropy-driven exploration, slowing progress. In summary, designing appropriate reward functions is a complex but crucial aspect of effective DRL training.

3.5 Dynamic environments: Dense and disaster-focused scenarios

Dynamic environments, characterized by changing obstacles, lighting, or terrain pose significant challenges for deep reinforcement learning (DRL) [79]. The agent must continuously re-explore as the environment evolves, complicating exploration and increasing sample requirements. This leads to longer, more expensive training due to the greater number of interactions needed to learn effective policies across varying conditions.

Adapting quickly to such changes is difficult; policies may diverge instead of converging, especially for off-policy algorithms like deep Q-networks (DQN), whose replay buffers can contain outdated experiences that no longer represent the current dynamics [80]. The increased variance in reward signals caused by environmental changes further complicates stable policy learning. Unlike static settings, the same state in a dynamic environment may yield different outcomes at different times, causing the agent to struggle in adapting to new conditions.

This unpredictability increases the risk of outdated or mislearned policies, potentially leading to unsafe robot behaviors such as collisions or damage. Dense environments such as crowded pedestrian areas [75, 76] and disaster scenarios [46, 80] amplify these challenges, requiring robust and adaptive DRL solutions to ensure safe and effective navigation.

3.6 Scalability

Scalability refers to a DRL system’s ability to learn efficiently as the number of agents, tasks, or environmental complexities increase, while maintaining reasonable time and resource requirements—especially in high-dimensional action and complex state spaces. In swarm robotics, navigation and task allocation become increasingly challenging due to dynamic environments [81]. Niyazi Furkan Bar et al. [6] proposed a DRL framework based on distributed architecture combined with adaptive reward mechanisms, demonstrating improved navigation success rates as the swarm size grows exponentially.

Martinez et al. [82] evaluated DRL performance in highly dynamic, real-world-like environments and confirmed its robustness in crowded and previously unseen scenarios. In contrast, Qiang Gao et al. [83] introduced Multi-Agent Deep Deterministic Policy Gradient (MADDPG) for multi-unmanned surface vehicle (USV) path planning, showing enhanced training effectiveness and scalability, yielding more efficient and practical path planning.

To further improve scalability in algorithms like Soft Actor-Critic, strategies such as minimizing communication overhead, employing complex policy optimization, decentralized learning, graph-based policies, and federated reinforcement learning have been proposed [84]. These approaches help DRL handle growing system complexity while preserving performance and learning efficiency.

4.1 Dynamic environment

The ultimate goal of autonomous robotic systems is to navigate unknown and dynamic real-world environments. Deep reinforcement learning (DRL) has been widely applied in this domain to achieve adaptive and intelligent control. However, significant gaps remain, particularly in disaster [80] and emergency scenarios [85], where unpredictable and continuous changes demand highly adaptive robot control systems. To prevent accidents and ensure practical efficiency, these systems must be further enhanced to cope with such challenging environments. Approaches like adversarial training and uncertainty-aware policy learning show promise for improving robustness in unpredictable settings.

In disaster environments, where human access is limited, robots are expected to navigate complex and hazardous terrain efficiently. Their role includes real-time monitoring, hazard identification, and survivor detection, supported by artificial intelligence and advanced onboard sensors. Such capabilities contribute to more resilient and responsive disaster management. DRL particularly deep Q-networks has demonstrated significantly higher rescue success rates across various natural disaster scenarios compared to traditional methods [80].

4.2 Time-varying obstacles environment

Highly dynamic environments are characterized by frequently changing obstacle configurations, where both the shape and position of obstacles vary continuously over time. For instance, pedestrian environments [75] exhibit constantly shifting movements of people [76] while logistics and intelligent transportation settings feature dynamic obstacles such as both pedestrians and vehicles. The high level of dynamism in these environments demands safe and efficient navigation strategies. The risk of failure increases significantly in densely populated dynamic spaces, where obstacle density is high. Therefore, intensive research efforts should focus on these complex scenarios rather than on simpler obstacle configurations when applying deep reinforcement learning (DRL) [75].

4.3 Increasing learning efficiency through sample and data optimization

Learning with deep reinforcement learning (DRL) demands substantial computational resources due to the vast amounts of data generated from interactions among environment, actions, and rewards. In high-dimensional state and action spaces, the exponential growth of possible states leads to increased data usage, longer training times, and higher computational costs. To mitigate this complexity, future strategies could include improved exploration methods, model-based reinforcement learning, integration of prior knowledge, and extensive simulation training.

Exploration remains a critical challenge, especially in complex environments where extensive exploration is necessary to discover effective strategies, directly impacting learning efficiency. For example, exploration in Soft Actor-Critic (SAC) can be enhanced by increasing target entropy, augmenting reward terms, and incorporating model-based approaches for sparse-reward tasks [57].

4.4 Advanced robotics application

The application of deep reinforcement learning (DRL) has demonstrated significant success in enhancing the adaptability and efficiency of robot navigation in challenging environments. Integrating DRL within robotic systems improves perception [3], instruction interpretation, and symbolic reasoning for high-level planning. For instance, Li BoXin and Wang Zhaokui [86] proposed a two-stage DRL control method combining Vision-TD3 and Force-TD3 networks, which leverage hybrid visual and force feedback for end-to-end lunar robot assembly operations, enabling autonomous, intelligent task execution without relying on model-based control.

To accelerate learning and align robot behavior with human intentions, incorporating human feedback such as reward shaping, demonstrations, and corrections has proven effective [87]. Chang Chiu Liu et al. [88] introduced a mixed perception-based human-robot collaborative maintenance system employing a hierarchical structure, where online DRL aids decision-making. Experimental results demonstrated competitive performance compared to state-of-the-art approaches.

Hybridizing Soft Actor-Critic (SAC) with blockchain technology further enhances decentralized decision-making, security, and computational efficiency. By integrating SAC with distributed cloud frameworks, real-time deployment becomes feasible on autonomous ground vehicles, drones, and robotic arms, even on low-power processors [3].

As reviewed by Saba Waseem et al. [89], the future of advanced robotic manipulators depends on intelligent processing technologies and expert systems. Emphasis should be placed on robust controllers capable of adapting to unforeseen scenarios, improving human-robot collaboration, and ensuring operational safety and efficiency. Mohsen Soori et al. [90] highlighted applications of AI, machine learning, and DRL across advanced robotics industries, underscoring the need to continually enhance system robustness and performance.

Beyond navigation, DRL is increasingly applied to collaborative manipulation [91, 92], warehouse automation [93, 94], and swarm robot exploration [6], with future efforts focused on extending DRL to multi-robot cooperative and competitive scenarios to meet complex real-world demands.

4.5 Bridging the gap between simulation and real robot transfer

Bridging the reality gap between simulation and physical robots remains a critical challenge in robotics research. Future directions are likely to emphasize strategies such as learning invariant representations, domain randomization, domain adaptation, and the incorporation of real-world data into simulated training [45] Given the high computational cost of DRL algorithms and their limited feasibility on embedded robotic platforms, developing efficient DRL methods that can run on constrained hardware is an important priority.

For example, the COBOT robot using Soft Actor-Critic (SAC) for obstacle avoidance has demonstrated efficiency in both simulation and real-world experiments [10]. To further bridge the simulation-to-reality gap, the official Unified Robot Description Format (URDF) is employed to enhance simulation fidelity. Physical parameters such as maximum joint speeds, accelerations, rotational limits, momentum, and center of mass are calibrated to match real robotic arm characteristics, significantly improving the accuracy of the simulation.

In another study, SAC was tested on the TurtleBot3 Waffle Pi, equipped with a Raspberry Pi, using the Gazebo emulator within the Robot Operating System (ROS) framework. The hardware included an NVIDIA GeForce GTX 1080 GPU and Intel Core i7 CPU (2.9 GHz). Wen et al. further advanced DRL-based path planning in unknown environments through Twin Long Short-Term Memory Delayed Deep (TLSMD) reinforcement learning, validated via numerous 3D simulations [14, 63].