A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning

Abstract The formulation of high-efficient energy management strategy (EMS) for hybrid electric vehicles (HEVs) becomes the most crucial task owing to the variation of electrified powertrain topology and uncertainty of driving scenarios. In this study, a deep reinforcement learning (DRL) algorithm, namely TD3, is leveraged to derivate intelligent EMS for HEV. A heuristic rule-based local controller (LC) is embedded within the DRL loop to eliminate irrational torque allocation with considering the characteristics of powertrain components. In order to resolve the influence of environmental disturbance, a hybrid experience replay (HER) method is proposed based on a mixed experience buffer (MEB) consisting of offline computed optimal experience and online learned experience. The results indicate that improved TD3 based EMS obtained the best fuel optimality, fastest convergence speed and highest robustness in comparison to typical value-based and policy-based DRL EMSs under various driving cycles. LC leads to a boosting effect on the convergence speed of TD3-based EMS wherein a “warm” start of exploring is exhibited. Meanwhile, by incorporating HER coupled with MEB, the impact of environmental disturbance including load mass and road gradient, as an increase of input observations, can be negligible to the performance of TD3-based EMS.