Space Evac: Development and user testing of an online game for selection of high-risk professionals

In the EDA project SEGARES (Serious Gaming for Resilience Screening), an international consortium is working on the development of a serious game to screen military and police personnel on critical competencies for dealing with challenging situations. This report describes the development of the demonstrator Space Evac, a serious game for the selection and assessment of high-risk professionals. It also describes the user experience research conducted across all organizations participating in the consortium.

Goal
To quickly and efficiently test candidates on military-critical competencies under challenging circumstances, the organisations in the consortium (Defense organisations of the Netherlands, Belgium, Finland, Sweden and the Netherlands Police Academy) expressed a need for a new (online) screening tool that uses serious gaming techniques, suitable for the new generation of employees. Such an instrument is not available on the market. In this project we therefore developed a knowledge base to support the development of serious
games for selection on and assessment of military-critical competencies, we designed a proof of concept of a serious game for selection and asssment of high-risk professionals, and we performed user tests with this proof of concept.

Description of Activities
In the first phase of the project a framework was developed for the design of serious games (see report TNO R10999). In the second phase of the project, this framework was applied to create a serious game concept that innovatively measures competencies for successfully managing challenging and stressful situations, which are difficult to assess with the current selection instruments. This concept was then developed into the Space Evac 1.0 demonstrator, in collaboration with the game development company IX-technologies. All
consortium partners conducted user research within their own organizations using this demonstrator. This research focused on the players’ (the intended candidates) experience of the game. Specifically, we investigated the perceived usability, the level of immersion and experienced stress, the extent to which players exhibited behavior consistent with their ‘natural’ behavior, their opinions on using this game for selection purposes, and the influence of personal characteristics not relevant to selection decisions on the gaming experience.

Results and Conclusions
The developed demonstrator Space Evac 1.0 is an online game that can be played in approximately 30-40 minutes. In the game, the player acts as the deputy commander of a space station, ensuring that the station’s processes run smoothly. Throughout the game, setbacks and unexpected events occur, requiring the player to respond. The game is designed to measure coping flexibility, adaptability, situational awareness, and cognitive flexibility through various methods and modules (behavioral measurements, situational judgment-like tests, gamified cognitive tests). By conducting the measurements under both
normal and stressful conditions, the game can also provide insights into players’ stress resilience. The user research results indicate that the developed game is in essence suitable for its intended purpose. Players find the game sufficiently user-friendly, feel immersed in the game, experience some stress while playing, and report behaving in the game as they would in real life. Players rate the suitability of the game as a selection tool as moderately high. Additionally, the research shows that the game is largely perceived similarly by players with different characteristics (such as gaming experience, technological readiness, age, or gender), suggesting minimal risk of the game unfairly favoring or disadvantaging certain groups.

Applicability
The results of the user study provide a sufficient basis to further develop the Space Evac 1.0 demonstrator into a usable product. In the follow-up project SEGARES II, Space Evac 1.0 will be further developed into Space Evac 2.0, a measurement instrument for the intended competencies, with built-in scoring methodologies in the form of a game. Subsequently, a large-scale study will be conducted with Space Evac 2.0 by all consortium partners involved, focusing on construct validity and, if possible, incremental validity. If the competencies can
be validly measured, the game can be technically implemented and then used by the consortium parties in the selection process or for training purposes.