Validation of the test Methodology

This deliverable reports on the results from the activities conducted under the scope of COURAGEOUS
WP4 Task 4.4 (validation of test methodology). The work that is carried out in this task, builds upon
results reported in D2.3, D3.1, D3.2, D4.1, D4.2 and D4.3. In this task, a validation method for the
COURAGEOUS test methodology and corresponding performance evaluations is presented.
Based on the test methodology in D4.2, a validation method has been defined using a simulation-based
approach to validate the developed test methodology and corresponding performance evaluation.
This is done in a simulation environment that facilitates carrying out sensitivity tests.
A test framework has been defined and implemented building upon the generic test methodology
presented in [4]. This test framework contains a number of key building blocks that together form the
validation method:
• Test environment dealing with scenario’s including objects to secure, environmental aspects
(e.g., trees, buildings) and objects of interest (drones).
• System under test representing the DTI system that is to be tested and evaluated. A basic nonphysical
model of a DTI system has been created with detection, tracking and alarming
capability. Different DTI strategies have been implemented in this model facilitating sensitivity
analysis.
• Test suite covering the test methodology and corresponding performance evaluation using a
number of defined metrics.
In addition, the insights that have been gained during the definition and the development of the test
framework are used as an input to the trial-based validation activities. Furthermore, a complete
processing and evaluation pipeline for the COURAGEOUS trial data has been defined. With this pipeline
a seamless processing of COURAGEOUS trial data can be achieved.
Finally, analysis of results emanating from several simulation runs that were repeated for three
different DTI model strategies and different characteristics (such as field of view, track association
capabilities) are presented. All evaluated metrics were aggregated (optionally using weights that can
be set by the end-user) to come up with an overall evaluation score for a DTI on its core functionalities.