Paper #1

Using PER-DQN for Intelligent Control of Dynamic Warehousing and Sorting Systems

Xizhuo Han and Chuanchuan Teng

Abstract: This paper aims to address the lag problem of traditional control strategies caused by order fluctuations and equipment coordination complexity in dynamic warehousing and sorting systems, specifically addressing the limitations of the Deep Q-Network (DQN) algorithm, notably its poor sample efficiency and instability in dynamic environments. It uses PER-DQN (Prioritized Experience Replay-DQN) to build an efficient and stable intelligent control method, which enhances timing adaptability through dynamic replanning and a time-varying reward mechanism to solve existing problems. First, a 10-dimensional state vector is constructed, covering real-time orders and robot status information, and a dynamic environment model is established using the discrete grid division method to achieve a comprehensive perception of the system's operating status. Subsequently, a structured action space was designed that includes task assignment priorities and robot path directions to support multi-dimensional collaborative control. Then, a multi-objective composite reward function is proposed to comprehensively consider the task completion efficiency, dynamically adjust the optimization direction through time-varying weights, and accelerate the convergence of the policy network. Finally, a dual-network training architecture combining the PER mechanism and Double DQN is adopted to improve the utilization of key experience and suppress Q-value overestimation. An online learning mechanism is applied to achieve incremental strategy updates, ensuring that the system can maintain stable performance in sudden order scenarios. Experiments show that PER-DQN has an average delay of 14.35 seconds for small-batch orders, 24.21 seconds for mixed orders, and 34.82 seconds for large-batch orders. PER-DQN also has a path conflict rate of only 1.2% and a sorting accuracy of 99.1%. In the robustness test, PER-DQN's device utilization increases by 25.4% and 27.8% compared to DQN under device failure and high load pressure conditions, respectively. This study verifies the efficiency and robustness of PER-DQN, which improves task scheduling accuracy and resource coordination capabilities through prioritized experience replay and structured action space design.

Keywords: Deep Q-Network; Intelligent Control; Multi-objective Composite Reward Function; Prioritized Experience Replay; Warehousing and Sorting System.