Millimeter-wave (mmWave) networks have the potential to support high throughput and low-latency requirements of 5G-and-beyond communication standards. But transmissions in this band are highly vulnerable to attenuation and blockages from humans, buildings, and foliage, which increase end-to-end packet delays. This work designs dynamic scheduling policies that minimize end-to-end packet delays while keeping packet transmission costs low. Specifically, we consider a mmWave network that consists of a transmitter that transmits data packets over an unreliable communication channel modeled as a Gilbert-Elliott channel.The transmitter operates under an ACK/NACK feedback model and does not observe the channel state unless it attempts a transmission. The objective is to minimize a weighted average cost consisting of end-to-end packet delays and packet transmission costs. We pose this dynamic optimization problem as a partially observable Markov decision process (POMDP). To the best of our knowledge, this is the first POMDP formulation for mmWave network with partial channel state information that considers delay minimization. We show that the POMDP admits a solution that has a threshold structure, i.e., for each queue length, the belief (the conditional probability that the channel is in a good state) is partitioned into intervals, and the transmitter sends j packets when the belief lies in the j-th interval. We then consider the case when the system parameters such as the packet arrival rate, and the transition probabilities of the channel are not known, and leverage these structural results in order to use the actor-critic algorithm to efficiently search for a policy that is locally optimal.

翻译：毫米波网络具备支持5G及未来通信标准高吞吐量与低延迟需求的潜力。然而，该频段的传输极易受人体、建筑物和植被造成的衰减与阻塞影响，从而增加端到端数据包延迟。本研究设计了动态调度策略，旨在最小化端到端数据包延迟的同时保持较低的包传输成本。具体而言，我们考虑一个毫米波网络，其中发射机通过不可靠通信信道（建模为吉尔伯特-埃利奥特信道）传输数据包。发射机在ACK/NACK反馈模型下运行，且仅在尝试传输时才能观测信道状态。目标是最小化由端到端数据包延迟和包传输成本构成的加权平均代价。我们将此动态优化问题建模为部分可观测马尔可夫决策过程。据我们所知，这是首个针对具有部分信道状态信息的毫米波网络且考虑延迟最小化的POMDP建模。我们证明该POMDP存在具有阈值结构的解：对于每个队列长度，置信度（信道处于良好状态的条件概率）被划分为若干区间，当置信度位于第j个区间时，发射机发送j个数据包。随后，我们考虑当系统参数（如数据包到达率、信道转移概率）未知的情况，并利用这些结构特性，通过演员-评论家算法高效搜索局部最优策略。