Simulation of Pan-Tilt-Zoom Tracking for Augmented Reality Air Traffic Control

Using Augmented Reality (AR) technology for Air Traffic Control (ATC) holds great promise but comes with a number of technical challenges. In addition to displaying the position of surrounding aircraft, a zoomed-in view of a certain aircraft captured with a Pan-Tilt-Zoom (PTZ) camera is also very useful in practice. This is to allow the ATC officer to perform the visual checks that they typically do with large binoculars, directly with the AR headset. Therefore, the PTZ camera has to be able to track the aircraft in a fast and robust manner to produce images suitable to be projected on the AR headset. Unfortunately, PTZ tracking algorithms are notoriously hard to implement, since the captured images depend on the PTZ controls, which depend on the outputs of the tracking algorithm, which depend on the images. In this paper, we describe our generic framework which leverages 3D simulation to design PTZ tracking algorithms and offer an in-depth explanation of how we use it in the context of AR ATC.