Publications without graduation papers

In this paper, we introduce an interactive multimodal vision-based robot teaching method. Here, a multimodal 3D image (color (RGB), thermal (T) and point cloud (3D)) was used to capture the temperature, texture and geometry information required to analyze human action. By our method, we only need to move our finger on an object surface, and then the heat traces left by the finger on the object surface will be recorded by the multimodal 3D sensor. By analyzing the multimodal point cloud dynamically, the accurate finger trace on the object is recognized. A robot trajectory is computed using this finger trace.

In electronics manufacturing, the inspection of defects of electrical components on printed circuit boards (SMD-PCB) is an import part of the production chain. This process is normally implemented by automatic optical inspection (AOI) systems based on classical computer vision and multimodal imaging. Despite the highly developed image processing, misclassifications can occur due to the different, variable appearance of objects and defects and constantly emerging defect types, which can only be avoided by constant manual supervision and adaption. Therefore, a lot of manpower is needed to do this or to perform a subjective follow-up. In this paper, we present a new method using the principle of multimodal deep learning-based one-class novelty-detection to support AOIs and operators to detect defects more accurate or to determine whether something needs to be changed. By combining with a given AOI classification a powerful adaptive AOI system can be realized. To evaluate the performance of the multimodal novelty-detector, we conducted experiments with SMD-PCB-components imaged in texture and geometric modalities. Based on the idea of one-class-detection only normal data is needed to form training sets. Annotated defect data which is normally only insufficiently available, is only used in the tests. We report about some experiments in accordance with the consistence of data categories to investigate the applicability of this approach in different scenarios. Hereby we compared different state-of-the-art one-class novelty detection techniques using image data of different modalities. Besides the influence of different data fusion methods are discussed to find a good way to use this data and to show the benefits using multimodal data. Our experiments show an outstanding performance of defect detection using multimodal data based on our approach. Our best value of the widely known AUROC reaches more than 0.99 with real test data.

Efficient entangled photon pair sources are the main component for several applications based on quantum imaging. Specifically for ghost imaging, different wavelengths of signal (imaging photons) and idler (interaction with the object) photons are desired. An efficient and narrowband generation of entangled photons exploiting spontaneous parametric down-conversion using periodically poled (pp) nonlinear crystals is therefore a fundamental preliminary requirement to achieve (the process of) ghost imaging. This work presents the design and implementation of a precise and efficient crystalheater as a variable photon pair source and compares the achieved experimental values of the SPDC-wavelengths with theoretical calculations. A periodically poled nonlinear crystal from potassium titanyl phosphate (ppKTP) can generate various non-degenerate wavelengths from a pump radiation of 405 nm by temperature changes and satisfaction of energy conservation and quasi-phase-matching conditions. For this purpose, the crystal is securely housed in a custom-built mechanical mount. A computation and adjustment of various control parameters, as well as a precise determination of the current temperature via two temperature sensors allow the heater to set the target temperature with an accuracy of 0.1 ˚C±0.015 ˚C. A method for the theoretical determination of the temperature-dependent shift of the nondegenerate wavelengths, provides a foundation from which experimental verification of achievable wavelengths and intensities can be compared. By experimental verification, the efficiency and functionality of the photon pair source and SPDC-process is verified. These presented investigations and the design of the crystal-heater provide the basis for a precise and effective photon pair source, for subsequent studies in the field of ghost imaging.

The projection of a point cloud onto a 2D camera image is relevant in the case of various image analysis and enhancement tasks, e.g., (i) in multimodal image processing for data fusion, (ii) in robotic applications and in scene analysis, and (iii) for deep neural networks to generate real datasets with ground truth. The challenges of the current single-shot projection methods, such as simple state-of-the-art projection, conventional, polygon, and deep learning-based upsampling methods or closed source SDK functions of low-cost depth cameras, have been identified. We developed a new way to project point clouds onto a dense, accurate 2D raster image, called Triangle-Mesh-Rasterization-Projection (TMRP). The only gaps that the 2D image still contains with our method are valid gaps that result from the physical limits of the capturing cameras. Dense accuracy is achieved by simultaneously using the 2D neighborhood information (rx,ry) of the 3D coordinates in addition to the points P(X,Y,V). In this way, a fast triangulation interpolation can be performed. The interpolation weights are determined using sub-triangles. Compared to single-shot methods, our algorithm is able to solve the following challenges. This means that: (1) no false gaps or false neighborhoods are generated, (2) the density is XYZ independent, and (3) ambiguities are eliminated. Our TMRP method is also open source, freely available on GitHub, and can be applied to almost any sensor or modality. We also demonstrate the usefulness of our method with four use cases by using the KITTI-2012 dataset or sensors with different modalities. Our goal is to improve recognition tasks and processing optimization in the perception of transparent objects for robotic manufacturing processes.