It was a long road to this research breakthrough for the team led by Prof. Jens Haueisen, EMBRACE project leader and director of the Institute for Biomedical Engineering and Computer Science (BMTI) at the University of Thuringia. The highly interdisciplinary team—comprising biomedical engineers, computer scientists, neuroscientists, and psychologists from both research and industry—developed a system that allows for the simultaneous, real-time recording of three different signal groups from test subjects who were physically interacting, such as playing table tennis or dancing together: the neural signals; physiological reactions, i.e., those reflecting biological processes in the body; and kinematic signals, which pertain to body movements. In total, the researchers recorded ten signals synchronously and in real time: from the brain, heart, lungs, muscles, and body movements. The synchronization of all these signals was essential for the research, not only to obtain meaningful multimodal data but also so that this data, originating from different sources, could subsequently be reliably analyzed.
To achieve this, the researchers synchronized independent systems from different manufacturers wirelessly—a crucial step for conducting analyses on people who are moving, such as during physical activities. Using a seemingly simple strategy, they reduced the number of devices that needed to be synchronized by grouping them into individual intermediate devices: The EEG cap for recording brain waves, the ECG electrodes for monitoring the heart, and each subject’s respiratory belt were wirelessly connected to the same mobile EEG recording device for each subject, while the EMG sensors, which measure the electrical activity of the muscles, and the subjects’ motion markers were connected to a stationary motion capture system for all subjects. The researchers then developed a system to synchronize the mobile and stationary device groups.
Software tools for multimodal data fusion and analysis
Equally important for the new analytical method are the in-house software tools for multimodal data fusion and analysis, which make it possible to interpret the complex signals through a single interactive process. These include, in particular, new methods for reducing so-called motion artifacts—that is, disturbances or changes in the recorded signals caused by the participants’ movements during recording. The scientists also developed methods for analyzing the functional connectivity of the participants’ brains, which provide insights into the synchrony of brain activity during their interaction.
