The state of affairs in education and training: AR, VR, MR and AI

Trying is more important than studying – in fact, that has always been the motto if something is to succeed. Nowadays this is possible through training 4.0 with Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR) and Artificial Intelligence (AI) without direct consequences in the ‘real’ professional world itself. This is exactly what more and more employers are using in the training and further education of their employees.

In the Netherlands today, many different educational institutions, but also start-ups, use these resources for training and educational purposes. Education 4.0 is also becoming commonplace in Germany. In 2016, the new possibilities were already reason enough for the German Ministry of Education and Research (BMBF) to launch the initiative Vocational Training 4.0. In cooperation with the Bundesinstitut für Berufsbildung (BIBB) (Federal Institute for Vocational Education and Training), the Ministry is looking for new legislation for future relevant, attractive and competitive vocational education. That is why numerous projects have been developed in collaboration with universities, institutes and IT companies.

Application

One of the applications of this type of technology we see around applications. Thanks to the research project Empat, young people can prepare themselves specifically for interactions during a job interview. Using avatars in the form of a virtual, empathic training system, the situation is practiced and one’s own behaviour trained. To adapt the behavior of the interactive avatar to the socio-emotional situation of the user, a real-time analysis of social signals is linked to an emotional real-time user model. For researchers at the University of Augsburg (Department of Computer Science), the Universität des Saarlandes (Psychology) and companies in the field of 3D graphics, the greatest challenge is to realistically design emotional feedback through the avatar.

Highly sensitive sensors register social communication signals, such as eye, hand and body movements, simulating emotional-social interaction patterns. The results are also used to calculate possible emotional states of a user. The project is supervised by an ethical advisory board in the field of coaching, youth work, law and education.

Of course, professional conversations can also be trained in other situations. VR glasses, for example, are used for training to become a bank advisor. A research project started in 2018 and will be expanded in the first half of 2019. Approximately 80 potential customer advisors receive training units with virtual reality glasses.

Virtual emergencies

The Epicsave research project has also been running since 2017. Here different scenarios are tested by using VR in paramedical training. The aim is “… to make life-threatening emergencies that are too rare for adequate training, such as allergic shock in children, practically trainable”, according to the initiators on their website. Besides VR, other digital tools are used in the field of gaming, eye-tracking, virtual agents and 3D interaction. The project is supervised by experts from different fields. Project management and media technology in the research into experience-based Serious Games and Virtual Reality is the responsibility of the Hochschule Bonn-Rhein-Sieg. The Fraunhofer IESE is responsible for the media didactic conception and evaluation of the learning modules and the technical implementation of the Serious Games and the VR environment came from Tricat. The Ulm-based company is regarded worldwide as a pioneer in the field of virtual 3D learning and working environments. The simulation training was implemented at the Malteser Hilfsdienst and the Academy for Emergency Medicine. In the long term, Epicsave can also be transferred to trainees in healthcare professions and to doctors who are obliged to undergo further training. In a third step, the development of an educational portal on health emergencies for the general public is also being considered.

A similar approach is followed by virtual fire training in the Cham district, which is now also used throughout Germany. Together with the University of Kassel, a system has been developed with which emergency forces can be deployed in virtual emergency situations using 3D glasses. The big advantage: trainers can observe, analyze and correct the behavior.

Social Virtual Reality expands the real world with a digital one. The app stores the design data of a Heidelberg SX 74 series press, which is most commonly used in training workshops at schools ©Heidelberger Druckmaschinen.

Future workshop

The Zukunftswerkstatt der Heidelberger Druckmaschinen AG was founded with the aim of developing employees who are able to deal with dynamic changes in a sustainable way and at the same time make a more traditional industry interesting for digital natives. In this course, trainees get to know the company’s digitization strategy from the very beginning. Mechatronics trainees can, for example, use Virtual Reality (VR) to take a closer look at the inner workings of a printing press. And they learn to virtually simulate assembly, maintenance and repair work. “Here we have an example of a successful transition from a funded project to an application for an industrial company,” says Dr. Lutz Goertz, e-learning researcher and head of educational research at mmb Institut. The walk-on printing press was previously developed and tested in the BMBF project Social Virtual Learning.

Digital help

The BMW plant in Landshut is now also using digital assistance systems in the training. VR goggles are used to train pupils, especially in the field of production. For example, they first learn virtually how to paint a part with a paint spray gun or how to drill a workpiece. According to BMW, the gaming atmosphere is not only appreciated by younger employees. The VR goggles give a real image of the workstation and also guide the user to the right design. At the end of the training, the result is evaluated, so that opportunities for improvement can also be looked at immediately. MRK technology (‘Human-robot collaboration’) has been used at the Bavarian car manufacturer’s Dingolfingen plant since 2012. Potential electronics engineers, mechatronics engineers and IT specialists gain their first experience in programming MRK robots for production here. The robots can be used flexibly. They move directly next to people and often perform monotonous work steps that require high reliability or constant strength and endurance.

Virtual flight students

The European Flight Academy Lufthansa Aviation Training (LAT) has existed since 2017. Recently the training module Visual Flight Rules (VFR) was voted best e-learning tool 2019 in the category Virtual Reality. The application makes it possible for flight students to interactively experience the structure of airspace and the recognition of landmarks from the air before they make their first cross-country flight in a VR environment. After a virtual briefing with 360° film shots with 3D animated objects, the students experience an immersive, location- and time-independent experience of the airspace and its structure. After that they will fly virtually around the Goodyear/USA training airfield. They need to recognise certain points and focus on landmarks. The project was developed in collaboration with psychologists from the Technical University of Berlin and can already show a nice result: the flight students showed some 15 percent better overall performance in real flight missions, while at the same time reducing their stress levels.

Mixed reality at the university

Thanks to the combination of augmented reality and actual conditions, students of Civil Engineering at the University of Weimar and the University of Ulm can realistically represent spatial information in three dimensions. At the same time, they receive background information about existing objects so that interaction becomes possible. Application examples are the construction of roads, bridges or buildings with additional computer-generated information, such as formulas and explanations. In addition, there are hidden parameters such as soil structure, CO2 emissions or thermodynamics, which can be seen and experienced thanks to Mixed Reality (MR), making good planning possible.

Didactics

This is only part of where ‘the journey to training 4.0’ goes. The online magazine Immersive Learning offers an up-to-date overview of the theme Learning in virtual worlds.

E-learning researcher Goertz also sees limits in relation to digital learning modules. “The use of VR may not be an end in itself. The new tools must serve to achieve learning goals,” he says. For example, “Virtual Reality Learning can be particularly well used for understanding complex processes, practicing behavioural patterns, training movements and acquiring soft skills, such as communication skills”.

Sünne Eichler, education management consultant, predicts an increasing importance of VR in learning situations thanks to continuously optimised computer performance, improved graphics and decreasing hardware costs. “In this way we offer the learner a learning experience that is close to reality and yet without risk to the real world. Using AI, we can tailor specific learning offers,” says Eichler. That means individual ‘learning on demand’ instead of shooting with hail. Using AI, we can give the student a quick answer to his specific question in a work process, for example when operating a machine, via chatbots. He gets the exact knowledge he needs and the machine immediately learns where the typical sources of operating errors lie.

Main picture: The VR glasses show a real image of the workplace. It also leads the user to the right design. It makes it possible to train new activities at an early stage and independently of the real workplace. © BMW.