Multimodal 3D sensor for object recognition in hospital settings. - In: Applied Optical Metrology V, (2023), 126720H, S. 126720H-1-126720H-8
In hospitals but also in other public facilities, it is essential to minimize the risk of contagion from infected persons. One of the key aspects is therefore to avoid contact infections caused by touching contaminated surfaces. While the current practice of wipe disinfection carried out by cleaning staff is expedient, it makes objective documentation difficult, can lead to surface damage by sanitizer overdosage, and can even put people at risk due to the released vapors. Consequently, it would be beneficial to implement technical solutions for both efficient and gentle disinfection of surfaces, e.g., a mobile platform with a sanitization module attached to a robotic arm. For a targeted cleaning and disinfection, which is tailored to specific objects and materials, such a system requires sensor technology for analyzing the environment. With this purpose in mind, we have developed a multimodal 3D sensor for detecting objects that can typically be found in a hospital environment. We started by examining specific materials using a spectrometer as well as cameras of various spectral ranges. Based on the results, we developed a sensor that can provide multimodal surface data with high spatial and temporal resolution. In experiments, we investigated how the generated data stream can be utilized for the targeted identification and treatment of typical hospital objects.
Multimodal RGB-D-T-scanning for interactive robot teaching. - In: Automated Visual Inspection and Machine Vision V, (2023), 126230H, S. 126230H-1-126230H-15
In this paper, we introduce an interactive multimodal vision-based robot teaching method. Here, a multimodal 3D image (color (RGB), thermal (T) and point cloud (3D)) was used to capture the temperature, texture and geometry information required to analyze human action. By our method, we only need to move our finger on an object surface, and then the heat traces left by the finger on the object surface will be recorded by the multimodal 3D sensor. By analyzing the multimodal point cloud dynamically, the accurate finger trace on the object is recognized. A robot trajectory is computed using this finger trace.
Oxygen saturation estimation from near-infrared multispectral video data using 3D convolutional residual networks. - In: Multimodal Sensing and Artificial Intelligence: Technologies and Applications III, (2023), 26210O, S. 26210O-1-26210O-15
Non-contact methods can expand the application scenarios of blood oxygen measurement with better hygiene and comfort, but the traditional non-contact methods are usually less accurate. In this study a novel non-contact approach for measuring peripheral oxygen saturation (SpO2) using deep learning and near-infrared multispectral videos is proposed. After a series of data processing including shading correction, global detrending and spectral channel normalization to reduce the influences from illumination non-uniformity, ambient light, and skin tone, the preprocessed video data are split into half-second clips (30 frames) as input of the 3D convolutional residual network. In the experiment, multispectral videos in 25 channels of hand palms from 7 participants were captured. The experimental results show that the proposed approach can accurately estimate SpO2 from near-infrared multispectral videos, which demonstrates the agreement with commercial pulse oximeter. The study also evaluated the performance of the approach with different combinations of near-infrared channels.
Investigation on automated visual SMD-PCB inspection based on multimodal one-class novelty detection. - In: Multimodal Sensing and Artificial Intelligence: Technologies and Applications III, (2023), 1262110, S. 1262110-1-1262110-11
In electronics manufacturing, the inspection of defects of electrical components on printed circuit boards (SMD-PCB) is an import part of the production chain. This process is normally implemented by automatic optical inspection (AOI) systems based on classical computer vision and multimodal imaging. Despite the highly developed image processing, misclassifications can occur due to the different, variable appearance of objects and defects and constantly emerging defect types, which can only be avoided by constant manual supervision and adaption. Therefore, a lot of manpower is needed to do this or to perform a subjective follow-up. In this paper, we present a new method using the principle of multimodal deep learning-based one-class novelty-detection to support AOIs and operators to detect defects more accurate or to determine whether something needs to be changed. By combining with a given AOI classification a powerful adaptive AOI system can be realized. To evaluate the performance of the multimodal novelty-detector, we conducted experiments with SMD-PCB-components imaged in texture and geometric modalities. Based on the idea of one-class-detection only normal data is needed to form training sets. Annotated defect data which is normally only insufficiently available, is only used in the tests. We report about some experiments in accordance with the consistence of data categories to investigate the applicability of this approach in different scenarios. Hereby we compared different state-of-the-art one-class novelty detection techniques using image data of different modalities. Besides the influence of different data fusion methods are discussed to find a good way to use this data and to show the benefits using multimodal data. Our experiments show an outstanding performance of defect detection using multimodal data based on our approach. Our best value of the widely known AUROC reaches more than 0.99 with real test data.
Thermal single-shot 3D shape measurement of transparent objects: optimization of the projected statistical LWIR pattern. - In: Optical Measurement Systems for Industrial Inspection XIII, (2023), 126180H, S. 126180H-1-126180H-10
Fast and non-contacting 3D shape measurements of objects for quality assurance, human machine interaction, or robot handling, e.g., in the industrial sector, have become well established. Recently, we have successfully combined thermography and triangulation to tackle the challenge of measuring the 3D shape of uncooperative materials, i.e., materials with optical properties such as being glossy, transparent, absorbent, or translucent. Therefore, we have developed the principle of thermal 3D measurements, a two-step process consisting of (1) the projection and absorption of projection patterns in the thermal infrared and (2) the stereo recording of heat patterns re-emitted by the object surface. We match image points by evaluating the temporal normalized cross correlation between pixels in both camera image stacks. In order to measure dynamic scenes, the previously achieved measurement times of a few seconds must be reduced by at least one order of magnitude to the range < 0.1 s. For this purpose, we apply established single-shot methods from the visible spectral range to our thermal 3D approach. Instead of temporal sequences of multi-fringe patterns or scanning single fringes, we now project statistical point patterns and record only one thermal stereo image pair. In this paper, we theoretically investigate our approach by using a simulation model for thermal point pattern generation on static measurement objects. We analyze the temporal and spatial behavior of the heat patterns taking the material parameters into account. Finally, we show a first thermal single-shot 3D measurement.
Design and implementation of a tunable crystal-heater for spectral variation of entangled non-degenerate photon pairs generated by spontaneous parametric down-conversion. - In: Photonics for Quantum 2023, (2023), 126330I, S. 126330I-1-126330I-11
Efficient entangled photon pair sources are the main component for several applications based on quantum imaging. Specifically for ghost imaging, different wavelengths of signal (imaging photons) and idler (interaction with the object) photons are desired. An efficient and narrowband generation of entangled photons exploiting spontaneous parametric down-conversion using periodically poled (pp) nonlinear crystals is therefore a fundamental preliminary requirement to achieve (the process of) ghost imaging. This work presents the design and implementation of a precise and efficient crystalheater as a variable photon pair source and compares the achieved experimental values of the SPDC-wavelengths with theoretical calculations. A periodically poled nonlinear crystal from potassium titanyl phosphate (ppKTP) can generate various non-degenerate wavelengths from a pump radiation of 405 nm by temperature changes and satisfaction of energy conservation and quasi-phase-matching conditions. For this purpose, the crystal is securely housed in a custom-built mechanical mount. A computation and adjustment of various control parameters, as well as a precise determination of the current temperature via two temperature sensors allow the heater to set the target temperature with an accuracy of 0.1 ˚C±0.015 ˚C. A method for the theoretical determination of the temperature-dependent shift of the nondegenerate wavelengths, provides a foundation from which experimental verification of achievable wavelengths and intensities can be compared. By experimental verification, the efficiency and functionality of the photon pair source and SPDC-process is verified. These presented investigations and the design of the crystal-heater provide the basis for a precise and effective photon pair source, for subsequent studies in the field of ghost imaging.
OHO: a multi-modal, multi-purpose dataset for human-robot object hand-over. - In: Sensors, ISSN 1424-8220, Bd. 23 (2023), 18, 7807, S. 1-13
In the context of collaborative robotics, handing over hand-held objects to a robot is a safety-critical task. Therefore, a robust distinction between human hands and presented objects in image data is essential to avoid contact with robotic grippers. To be able to develop machine learning methods for solving this problem, we created the OHO (Object Hand-Over) dataset of tools and other everyday objects being held by human hands. Our dataset consists of color, depth, and thermal images with the addition of pose and shape information about the objects in a real-world scenario. Although the focus of this paper is on instance segmentation, our dataset also enables training for different tasks such as 3D pose estimation or shape estimation of objects. For the instance segmentation task, we present a pipeline for automated label generation in point clouds, as well as image data. Through baseline experiments, we show that these labels are suitable for training an instance segmentation to distinguish hands from objects on a per-pixel basis. Moreover, we present qualitative results for applying our trained model in a real-world application.
Triangle-Mesh-Rasterization-Projection (TMRP): an algorithm to project a point cloud onto a consistent, dense and accurate 2D raster image. - In: Sensors, ISSN 1424-8220, Bd. 23 (2023), 16, 7030, S. 1-28
The projection of a point cloud onto a 2D camera image is relevant in the case of various image analysis and enhancement tasks, e.g., (i) in multimodal image processing for data fusion, (ii) in robotic applications and in scene analysis, and (iii) for deep neural networks to generate real datasets with ground truth. The challenges of the current single-shot projection methods, such as simple state-of-the-art projection, conventional, polygon, and deep learning-based upsampling methods or closed source SDK functions of low-cost depth cameras, have been identified. We developed a new way to project point clouds onto a dense, accurate 2D raster image, called Triangle-Mesh-Rasterization-Projection (TMRP). The only gaps that the 2D image still contains with our method are valid gaps that result from the physical limits of the capturing cameras. Dense accuracy is achieved by simultaneously using the 2D neighborhood information (rx,ry) of the 3D coordinates in addition to the points P(X,Y,V). In this way, a fast triangulation interpolation can be performed. The interpolation weights are determined using sub-triangles. Compared to single-shot methods, our algorithm is able to solve the following challenges. This means that: (1) no false gaps or false neighborhoods are generated, (2) the density is XYZ independent, and (3) ambiguities are eliminated. Our TMRP method is also open source, freely available on GitHub, and can be applied to almost any sensor or modality. We also demonstrate the usefulness of our method with four use cases by using the KITTI-2012 dataset or sensors with different modalities. Our goal is to improve recognition tasks and processing optimization in the perception of transparent objects for robotic manufacturing processes.
Productive teaming under uncertainty : when a human and a machine classify objects together. - In: 2023 IEEE International Conference on Advanced Robotics and Its Social Impacts (ARSO), (2023), S. 9-14
Konferenz: IEEE International Conference on Advanced Robotics and Its Social Impacts, ARSO, Berlin, Germany, 05-07 June 2023
Komponenten und Methoden für die multimodale Gefahrenanalyse in öffentlichen Räumen. - In: 3D-NordOst 2022, (2023), S. 37-46