The RUBIN alliance Advanced Multimodal Imaging (AMI), coordinated by TU Ilmenau and Steinbeis Qualitätssicherung und Bildverarbeitung GmbH Ilmenau, has demonstrated the potential of multimodal imaging combined with AI processes for small and medium-sized enterprises (SMEs): Together, eleven research and industry partners in the project funded by the German Federal Ministry of Research, Technology and Space developed innovative systems to precisely measure vital data such as heart rate and respiratory rate without contact, even during movement and in difficult conditions, and to improve the recycling of plastics and construction waste.
To develop high-performance imaging technologies that combine sensory information such as color, depth, spectral and thermal data and make them usable for SMEs in the course of digitalization: This was the goal of the RUBIN alliance AMI, launched in 2022 and coordinated by Steffen Lübbecke from Steinbeis Qualitätssicherung und Bildverarbeitung GmbH Ilmenau and Prof. Gunter Notni, Head of Quality Assurance and Industrial Image Processing at TU Ilmenau. Prof. Notni:
Multimodal imaging allows us to visualize objects and substances as well as their properties that remain unseen by individual sensors or the naked eye. The intelligent combination of this data creates real added value for applications in industry and medicine.
The project has resulted in three innovative system solutions for applications in industrial growth areas that are particularly relevant for Thuringia: a single-origin return system for plastic containers such as plastic cups, drinks or shampoo bottles, an analysis device for construction waste recyclates and a system for the miniaturized sensing of vital signs and activity in individuals.
The demonstrators developed are based on multispectral camera systems using the latest sensor technology, such as SenSWIR technology, real-time 3D sensor systems that can also be used for 3D detection of transparent objects, or infrared and thermal cameras in the SWIR and LWIR spectral range.
These innovative multimodal imaging systems were supplemented and combined with AI-based algorithms in order to evaluate the highly complex and memory-intensive multimodal image data and make it usable for material classification in recycling, robust object recognition under difficult environmental conditions and the contactless recording of vital and activity parameters.
The developed demonstrators are based on multispectral camera systems using state-of-the-art sensor technology, such as the so-called SenSWIR technology; real-time-capable 3D sensor systems that can also be used for the 3D acquisition of transparent objects; as well as infrared and thermal cameras in the SWIR and LWIR spectral ranges.
These novel multimodal imaging systems were complemented and combined with AI-based algorithms to analyze the highly complex and memory-intensive multimodal image data and to make them usable for material classification in recycling, robust object detection under challenging environmental conditions, and the contact-free acquisition of vital and activity parameters.
Intelligent sensors for better recycling
The SenSWIR sensor technology makes it possible for the first time to capture materials simultaneously from the visible spectrum through to the short-wave infrared (SWIR) range. This has opened up new pathways for recycling, as materials can now be identified far more accurately. In combination with artificial intelligence, plastics could be reliably distinguished into up to eight different types – an important prerequisite for single-stream take-back systems in retail.
Construction waste could also be analyzed with high precision and classified into 24 different material categories. The reliable identification of construction and demolition waste such as concrete, bricks, tiles, or ceramics forms the basis for reusing these materials as high-quality recycled construction materials in line with the principles of the circular economy – for example, as aggregates for new concrete in building and civil engineering, or in road construction.
Contact-free, precise acquisition of health data
The precise acquisition of vital signs without direct physical contact is considered a key technology for many future applications – from health monitoring to assistive systems. Although camera-based methods are already available today, they often work reliably only under ideal conditions. In real-world scenarios, they quickly become unreliable: movements, changing facial expressions, occluded skin areas, or fluctuating lighting conditions interfere with the already very weak physiological signals. As a result, many existing systems require long measurement times or deliver inaccurate results in dynamic situations. For stable everyday use – including in darkness or changing environments – new measurement and analysis approaches are therefore required that can reliably separate relevant vital information from external sources of interference.
The camera-based vital sensor developed at TU Ilmenau within the RUBIN consortium AMI is based on miniaturized multi-aperture camera systems with application-specific, tailored spectral filters. Among the parameters recorded are heart rate and heart rate variability, respiratory rate, oxygen saturation, activity and movement patterns, as well as indicators for pain detection.
Robust detection even during movement and under changing environmental conditions
Data analysis is performed using AI-based algorithms that were specifically developed for highly complex, multimodal image data and enable robust detection even in the presence of movement or changing environmental conditions. The measurement results achieve a high level of accuracy: deviations of less than 1 bpm for respiratory rate and around 3 bpm for heart rate compared to contact-based reference systems demonstrate, according to Prof. Notni, that contact-free methods are increasingly reaching clinically relevant quality:
Our goal was to reliably and practically capture vital parameters for everyday use—without sensors on the skin. The results show that multimodal imaging and AI can meet this demand.
In total, eleven partners from industry and academia collaborated within the consortium. The outcomes are reflected in more than 30 scientific publications, one patent application, and numerous qualification theses.
Prof. Notni adds:
The vital sensor developed at TU Ilmenau exemplifies how applied research, artificial intelligence, and interdisciplinary collaboration can enable solutions to key societal challenges—from contact-free health monitoring to new assistive systems in care, the workplace, and medicine.
About the RUBIN alliance AMI
Eleven alliance partners, consisting of eight small and medium-sized companies primarily from Thuringia and one partner from Saxony, as well as three research partners, the TU Ilmenau with the Department of Quality Assurance and Industrial Image Processing, the Fraunhofer Institute for Applied Optics and Precision Engineering IOF in Jena and the non-university research institute MFPA Materialforschungs- und -prüfanstalt in Weimar, conducted research and development in the RUBIN AMI alliance. The companies include SQB GmbH in Ilmenau, Vision & Control GmbH in Suhl, Sielaff GmbH & Co. KG Automatenbau in Ilmenau, Zentrum für Bild- und Signalverarbeitung e. V. in Ilmenau, LUCAS instruments GmbH in Jena, InfraTec GmbH in Dresden, TechnoTeam Bildverarbeitung GmbH in Ilmenau and 3plusplus GmbH in Suhl.
The alliance was funded by the Federal Ministry of Research, Technology and Space within the "RUBIN - Regional Entrepreneurial Alliances for Innovation" program for a period of three years with a total of nine million Euros.
More information on the RUBIN-AMI alliance
Contact us
Prof. Gunther Notni
Leiter Fachgebiet Qualitätssicherung und industrielle Bildverarbeitung (QBV)