Energy-Aware and Sustainable UAV Selection in 6G Inter Base Station Communications using Hierarchical Set Attention Reinforcement Learning

Abstract

The transition from fifth-generation to sixth-generation wireless networks brings stringent requirements on reliability, coverage, and resilience under infrastructure failure. Unmanned Aerial Vehicles (UAVs) have emerged as a key enabler of these properties, and one important deployment scenario is inter-base-station data relaying when terrestrial backhaul is disrupted by disaster or congestion. The operational question that arises in this scenario is which UAV from a large fleet should be assigned to a given relay task. Existing approaches optimize an instantaneous metric — for example, the highest signal-to-noise ratio, the lowest energy cost for the immediate task, or a fixed multi-objective utility function. They reach a reasonable per-task performance but ignore an important long-horizon effect. Repeatedly selecting the same high Scoring UAVs over many decision epochs progressively depletes their batteries until they must be excluded from selection altogether. The effective fleet shrinks, the candidate pool for future tasks degrades, and the per-task performance also degrades over time. This thesis formulates the UAV selection problem as a finite-horizon Markov Decision Process under explicit energy scarcity. UAVs whose battery falls below a threshold become dormant: they cannot be selected, but they continue to recharge, and they re-enter the active pool only after sufficient recovery. We propose a novel deep reinforcement learning framework called HSA-PPO (Hierarchical Set Attention with Proximal Policy Optimization). The architecture exploits the fact that the UAV fleet is a permutation-invariant set, and uses a two-level hierarchy. A lightweight fleet encoder summarises the global state of all UAVs at linear cost, while expensive self-attention and cross-attention operate only on the small subset of UAVs that are actually selectable for the current task. A sustainability-shaped reward, dominated by a near-dormancy penalty, drives the policy to leave low-battery UAVs alone so that they can recover, while still meeting per-task utility goals. We validate the framework on a 25-base-station, 1500-UAV testbed over 15,000 training epochs and five independent random seeds. The proposed method matches the strongest greedy baseline on per-task utility (0.403 vs 0.399 ) while reducing the number of dormant UAVs by 54coefficient by 13five seeds. An ablation study confirms that the near-dormancy penalty in the reward function is the specific component responsible for the long-horizon advantage.