Project reference: 653590 Funded under: H2020-EU.3.7. - Secure societies - protecting freedom and security of Europe and its citizens
Serious Game Scenario Generator for Mixed Reality
Terrorism is a major threat to
human life and critical infrastructure in Europe. The
intruders continuously devise new strategies and execute
highly sophisticated plans that are increasingly
unfamiliar to the security forces and therefore more
difficult to predict, prepare and defend against. In
principle, in order to prepare for terrorist acts or
critical incidents, security forces are trained using
traditional methods such as live scenario training
though full-scale field exercises. This type of training
can be very costly, time-consuming and dangerous to
those participating. In addition, it can only offer a
limited number of scenarios that individually require
major resources to plan and carryout. Thus, limiting the
amount and variety of situations for which security
forces are currently trained and making it difficult to
maintain uniform levels of skill, judgement and
preparedness across teams and individuals.
Virtual Training Environments (VTEs) and Serious Games
(SGs) provide the opportunity to prepare security forces
for events that cannot be, or are very difficult to
train for using live scenario exercises (e.g. an attack
on fully populated transport terminal). However, the
nature of these systems allows for the simulation of a
wide range of situations and settings in a quicker and
more cost-efficient manner. These software tools can be
tailored to offer trainers and trainees a broad
selection of scenarios, environments, characters, props
and possible outcomes. VTEs have been widely used in
military, surgery and law-enforcement training, where
the latter has focussed on training interview and
interrogation techniques and range shooting.
AUGGMED (AUtomated serious Game scenario Generator
for MixED reality training) is part of the European
Commission’s Horizon 2020 programme. The aim of AUGGMED
is to develop a serious game platform to enable single
and team-based training of end-users (e.g. police,
counter-terrorism, medics, firefighters) with different
level of expertise from different organisations
responding to terrorist and organised crime threats. The
platform will automatically generate non-linear
scenarios tailored to suit the needs of individual
trainees with learning outcomes that will improve the
acquisition of emotional management, analytical
thinking, problem solving and decision making skills.
The game scenarios will include advanced simulations of
operational environments, agents, telecommunications and
threats, and will be delivered through Virtual Reality
(VR) and Mixed Reality (MR) environments with multimodal
interfaces. This will result in highly realistic
training scenarios allowing advanced interactivity while
encouraging security staff and first responders to
engage and actively participate in the training process.
In addition, the AUGGMED platform will include tools for
trainers enabling them to set learning objectives,
define scenarios, monitor training sessions, modify
scenarios and provide feedback in real-time, as well as
evaluate trainee performance and set training curricula
for individual personnel in the post-training session
phase. Finally, the platform will be offered in
affordable and cost-effective Modes including Basic Mode
(low VR fidelity and interactivity through mobile
devices), Intermediate Mode (immersive multimodal VR)
and Full Mode (immersive multimodal MR On-Site).
Training Requirements and AUGGMED Platform
The objective of this work package is to define the security training requirements for several transport terminals and determine the training platform specifications. The overall system specification, design of the architecture indicating how the three main elements of the system (serious game scenario, VR environment and training tools) will communicate and interact with each other will be specified.
FSEG involvement in this work package consists of defining the requirements for the modelled geometry, agent and crowd simulations. New behavioural features will need to be developed for the EXODUS crowd simulation engine e.g. ability for agents representing authority figures to interact with other agents, providing them with instructions to modify their behaviour. This will allow simulated authority figures to issue voice commands or make hand gestures to other agents instructing them to move in a particular direction, to indicate to stop or to run or to crawl, etc. The capabilities of the EXODUS software will be expanded further to include the capability for an external person (i.e. live player) to take control of one of the agents within the simulation controlling their movement through the simulated environment and controlling the nature of their interaction with the environment and with other simulated agents or other agents guided by other real players.
Work Package 2:
Automated Game Scenario Engine for Training
The objective of this work package is to design and develop an engine that will automatically generate game scenarios on-demand to suit the individual training needs of the first responders and security personnel. The engine will adapt the generated scenarios to the different levels of VR and MR environments while maximising the training objectives and learning outcomes.
FSEG will be involved in the development of appropriate game scenarios and gameplay logic. Several game scenarios will be evaluated and the most relevant scenario and game play will be realised. During this task FSEG will focus on modelling and developing the environments of the game. Algorithms will be developed to automatically generate a scenario based on selected training criteria and user profiles. Game scenarios will involve specification of: the geometry, the crowd population, the perpetrators (number and location), the security forces (number and location) and the hazards (e.g. smoke, heat, toxic gases, etc). The appropriate scenario factors will be represented within the EXODUS simulation environment. This information will be communicated to EXODUS via the communications layer and interpreted via a scenario layer within EXODUS.
Work Package 3:
VR and MR Environments
The objective of this work package is to design and develop the VR and MR environments that will be used to deliver the serious game to the end-users. The physical and behavioural modelling and technologies of the VR environments which will support the gaming platform will be developed in this work package. In addition, the framework for interfaces and devices allowing effective interaction between the end-user and the VR/MR environments will be developed here.
FSEG involvement in this task will include the modelling of three pilot infrastructures including appropriate agent behaviours and threats. Support for multimodal interfaces and devices delivering the interactive virtual environment to the end-users will also be developed. The agents present within the models will be simulating using the EXODUS environment. Agents will include counter-terrorist forces, first responders as well as general public and intruders. Important aspects of agent’s physical (e.g., resting, walking, running), emotional (e.g., friendly, assertive, aggressive, scared), and physiological (e.g., healthy, injured, dead) status will be simulated through modelling of interactive behaviour as well as through direct control from a user in real time during the running of the simulation (e.g., a security agent instructed other agents to perform a certain action such stop, go, change direction or change stance, i.e. crawl.) The simulations within EXODUS will be enhanced to include simulations of threats like fire spreading within a building, or explosives detonated by the intruders. In addition, sophisticated simulations of the telecommunication capabilities of the security units and first responders, the public and the intruders will be included.
Circulation scenario modelled in buildingEXODUS and represented in vrEXODUS. This will help provide the initial conditions for the scenario involving the terrorist attack to the airport.
Smoke hazard from SMARTFIRE fire simulations: Smoke at head height spreading in airport terminal, RED showing 10m visibility or less.
Heat hazard from SMARTFIRE fire simulations: Temperature at head height spreading in the airport terminal. Exposure to air Temp. above 121°C causes pain.
Heat hazard from SMARTFIRE fire simulations: Temperature iso-surfaces showing the evolving spread of the heat in the airport terminal.
Work Package 4:
The objective of this work package is to design and develop the tools that will enable trainers to improve the training outcome for all personnel involved in training programmes. The objective of this work package is to design and develop the trainer tools that will allow trainers to access the functionality of game scenario generation.
Work Package 5:
AUGGMED platform integration
The objective of this work package is to integrate the game engine developed in WP2, the VR/MR technologies developed in WP3 and the training tools developed in WP4. Device management and security issues will also be addressed in this work packages.
FSEG will be involved in implementing a communication layer that controls all access between EXODUS and the other AUGGMED components. This will be a conduit for EXODUS-AUGGMED communication (e.g. agent actions, simulated results, communication with the VR shell, instructions passed on to the client interface) and AUGGMED-EXODUS communication (user operations, SMARTFIRE environmental data, and communication with server).
Work Package 6:
Pilots and Evaluation
The objective of this work package is to evaluate the effectiveness of the AUGGMED platform as a training tool. The activities in this work package will focus on the pilot plans including logistics as well as evaluation protocols as defined in WP1. In addition, the analysis of the results with conclusions and recommendations for further improvements will take place in this WP.
FSEG will contribute in the formulation of detailed plans for the pilot studies including timelines, recruitment of volunteers, trainees and trainers (and a small number of public if necessary).
Prior to testing, the prototypes will be installed. Depending on the scenario and/or Mode that is evaluated, the work may include software and/or hardware installation. Once the installation has been completed a rigorous test will be carried out to ensure reliable functionality of all prototypes during evaluation.
Prof. Ed Galea
Fire Safety Engineering Group
University of Greenwich
Greenwich Maritime Campus
Old Royal Naval College
Queen Mary Building
Greenwich SE10 9LS
Tel: +44 (020) 8331 8730
The AUGGMED project is funded by
the European Union’s HORIZON 2020 Framework Programme
for Research and Innovation