Human uncertainty in interaction with a machine: establishing a reference dataset. - In: Engineering for a changing world, (2023), 5.2.127, S. 1-6
We investigate the task of malformed object classification in an industrial setting, where the term ‘malformed’ encompasses objects that are misshapen, distorted, corroded or broken. Recognizing whether such an object can be repaired, taken apart so that its components can be used otherwise, or dispatched for recycling, is a difficult classification task. Despite the progress of artificial intelligence for the classification of objects based on images, the classification of malformed objects still demands human involvement, because each such object is unique. Ideally, the intelligent machine should demand expert support only when it is uncertain about the class. But what if the human is also uncertain? Such a case must be recognized before being dealt with. Goal of this research thread is to establish a reference dataset on human uncertainty for such a classification problem and to derive indicators of uncertainty from sensory inputs. To this purpose, we designed an experiment for an object classification scenario where the uncertainty can be directly linked to the difficulty of labelling each object. By thus controlling uncertainty, we intend to build up a reference dataset and investigate how different sensory inputs can serve as uncertainty indicators for these data.
https://doi.org/10.22032/dbt.58928
Investigations of closed source registration method of depth sensor technologies for human-robot collaboration. - In: Engineering for a changing world, (2023), 5.2.070, S. 1-15
Productive teaming is the new form of human-robot interaction. The multimodal 3D imaging has a key role in this to gain a more comprehensive understanding of production system as well as to enable trustful collaboration from the teams. For a complete scene capture, the registration of the image modalities is required. Currently, low-cost RGB-D sensors are often used. These come with a closed source registration function. In order to have an efficient and freely available method for any sensors, we have developed a new method, called Triangle-Mesh-Rasterization-Projection (TMRP). To verify the performance of our method, we compare it with the closed-source projection function of the Azure Kinect Sensor (Microsoft). The qualitative comparison showed that both methods produce almost identical results. Minimal differences at the edges indicate that our TMRP interpolation is more accurate. With our method, a freely available open-source registration method is now available that can be applied to almost any multimodal 3D/2D image dataset and is not like the Microsoft SDK optimized for Microsoft products.
https://doi.org/10.22032/dbt.58927
Event-based sensor fusion in human-machine teaming. - In: Engineering for a changing world, (2023), 5.1.132, S. 1-8
Realizing intelligent production systems where machines and human workers can team up seamlessly demands a yet unreached level of situational awareness. The machines' leverage to reach such awareness is to amalgamate a wide variety of sensor modalities through multisensor data fusion. A particularly promising direction to establishing human-like collaborations can be seen in the use of neuro-inspired sensing and computing technologies due to their resemblance with human cognitive processing. This note discusses the concept of integrating neuromorphic sensing modalities into classical sensor fusion frameworks by exploiting event-based fusion and filtering methods that combine time-periodic process models with event-triggered sensor data. Event-based sensor fusion hence adopts the operating principles of event-based sensors and even exhibits the ability to extract information from absent data. Thereby, it can be an enabler to harness the full information potential of the intrinsic spiking nature of event-driven sensors.
https://doi.org/10.22032/dbt.58924
Investigation into the implementation of a multimodal 3D measurement system for a forestry harvesting process. - In: Engineering for a changing world, (2023), 2.2.090, S. 1-13
In the context of digitalization, monitoring and traceability are also becoming increasingly important in the forestry sector. An essential component of the most efficient value creation is the recording of relevant characteristics right from the start. The optical and tactile recording of characteristics, such as diameter and volume, have been solved to a large extent in the harvesting of heavy timber, but differs significantly from that of small timber. This paper is about an investigation on the implementation of a multimodal 3D sensor system, which is used for the stable detection of biomass directly in the harvesting process of weak wood. System technical possibilities are shown how biomass can be determined directly during the harvesting process by means of multimodal 3D measurement technology. Considerations regarding possible measurement principles and methods result in two methods, which are discussed within this thesis regarding their advantages and disadvantages. The development stages are presented in detail up to the practical tests, which also includes the acquisition of empirical a priori information. Finally, data are determined by means of test scenarios, which prove the principle functionality and make the methods evaluable.
https://doi.org/10.22032/dbt.58858
Suitability study for real-time depth map generation using stereo matchers in OpenCV and Python. - In: Engineering for a changing world, (2023), 2.2.075, S. 1-11
Stereo imaging provides an easy and cost-effective method to measure 3D surfaces, especially due to the availability of extensive free program libraries like OpenCV. An extension of the application to the field of forestry was aimed at here in the context of a project to capture the elevation profile of forest roads by means of stereo imaging. For this purpose, an analysis of the methods contained in OpenCV for the successful generation of depth maps was carried out. The program sections comprised the reading of the image stream, the image correction on the basis of calibrations carried out in advance as well as the generation of the disparity maps by the stereo matchers. These are then converted back into depth maps and stored in suitable memory formats. A data set of the image size 1280x864 pixels consisting of 30 stereo image pairs was used. The aim was to design an evaluation program which allows the processing of the described steps within one second for 30 image pairs. With a sequential processing of all steps under the used test system and the usage of a local stereo matcher a processing time of 4.37 s was determined. Steps to reduce the processing time included parallelizing the image preparation of the two frames of the image pair. Further reduction in total processing time was achieved by processing multiple image pairs simultaneously and using storage formats without compression. A total processing time of 0.8 s could be achieved by outsourcing the stereo matching to the graphics card. However, the tested method did not achieve the desired resolutions in depth as well as in the image plane. This was made possible by using semi-global matchers, which are up to 10 times slower but significantly more accurate, and which were therefore used for further investigations of the forest path profile.
https://doi.org/10.22032/dbt.58859
Intelligent classification and data augmentation for high accuracy AI applications for quality assurance of mineral aggregates. - In: Engineering for a changing world, (2023), 2.2.049, S. 1-18
In this work, a method for automatic analysis of natural aggregates using hyperspectral imaging and high-resolution RGB imaging combined with AI algorithms consisting of an intelligent deep-learning-based recognition routine in form of hybrid cascaded recognition routine, and a necessary demonstration setup are demonstrated. Mineral aggregates are an essential raw material for the production of concrete. Petrographic analysis represents an elementary quality assurance measure for the production of high-quality concrete. Petrography is still a manual examination by specially trained experts, and the difficulty of the task lies in a large intra-class variability combined with low inter-class variability. In order to be able to increase the recognition performance, innovative new classification approaches have to be developed. As a solution, this paper presents an innovative cascaded deep-learning-based classification and uses a deep-learning-based data augmentation method to synthetically generate images to optimize the results.
https://doi.org/10.22032/dbt.58855
Mobile 3D sensor for documenting maintenance processes of large complex structures. - In: Engineering for a changing world, (2023), 2.2.035, S. 1-10
With the new handheld goSCOUT3D sensor system, the entire surface of complex industrial machinery spanning several meters can be captured three-dimensionally within a matter of minutes. In addition, a comprehensive photo collection is registered and precisely assigned to the corresponding 3D object points in one hybrid 2D/3D model. At the basis of the robust 3D digitization are the measuring principles of photogrammetric reconstruction using a high-resolution color camera and simultaneous localization and imaging using a tracking unit. Following image acquisition, the process leading to generation of the complete hybrid model is fully automated. Under continuous movement of the sensor head, up to six images per second and a total of up to several thousand images can be recorded. Those images are then aligned in 3D space and used to reconstruct the 3D model. Results regarding accuracy measurements are presented as well as application examples of digitized technical machinery under maintenance and inspection.
https://doi.org/10.22032/dbt.58857
Multimodal 3D sensor for object recognition in hospital settings. - In: Applied Optical Metrology V, (2023), 126720H, S. 126720H-1-126720H-8
In hospitals but also in other public facilities, it is essential to minimize the risk of contagion from infected persons. One of the key aspects is therefore to avoid contact infections caused by touching contaminated surfaces. While the current practice of wipe disinfection carried out by cleaning staff is expedient, it makes objective documentation difficult, can lead to surface damage by sanitizer overdosage, and can even put people at risk due to the released vapors. Consequently, it would be beneficial to implement technical solutions for both efficient and gentle disinfection of surfaces, e.g., a mobile platform with a sanitization module attached to a robotic arm. For a targeted cleaning and disinfection, which is tailored to specific objects and materials, such a system requires sensor technology for analyzing the environment. With this purpose in mind, we have developed a multimodal 3D sensor for detecting objects that can typically be found in a hospital environment. We started by examining specific materials using a spectrometer as well as cameras of various spectral ranges. Based on the results, we developed a sensor that can provide multimodal surface data with high spatial and temporal resolution. In experiments, we investigated how the generated data stream can be utilized for the targeted identification and treatment of typical hospital objects.
https://doi.org/10.1117/12.2675748
Multimodal RGB-D-T-scanning for interactive robot teaching. - In: Automated Visual Inspection and Machine Vision V, (2023), 126230H, S. 126230H-1-126230H-15
In this paper, we introduce an interactive multimodal vision-based robot teaching method. Here, a multimodal 3D image (color (RGB), thermal (T) and point cloud (3D)) was used to capture the temperature, texture and geometry information required to analyze human action. By our method, we only need to move our finger on an object surface, and then the heat traces left by the finger on the object surface will be recorded by the multimodal 3D sensor. By analyzing the multimodal point cloud dynamically, the accurate finger trace on the object is recognized. A robot trajectory is computed using this finger trace.
https://doi.org/10.1117/12.2673406
Oxygen saturation estimation from near-infrared multispectral video data using 3D convolutional residual networks. - In: Multimodal Sensing and Artificial Intelligence: Technologies and Applications III, (2023), 26210O, S. 26210O-1-26210O-15
Non-contact methods can expand the application scenarios of blood oxygen measurement with better hygiene and comfort, but the traditional non-contact methods are usually less accurate. In this study a novel non-contact approach for measuring peripheral oxygen saturation (SpO2) using deep learning and near-infrared multispectral videos is proposed. After a series of data processing including shading correction, global detrending and spectral channel normalization to reduce the influences from illumination non-uniformity, ambient light, and skin tone, the preprocessed video data are split into half-second clips (30 frames) as input of the 3D convolutional residual network. In the experiment, multispectral videos in 25 channels of hand palms from 7 participants were captured. The experimental results show that the proposed approach can accurately estimate SpO2 from near-infrared multispectral videos, which demonstrates the agreement with commercial pulse oximeter. The study also evaluated the performance of the approach with different combinations of near-infrared channels.
https://doi.org/10.1117/12.2673109