FPGA-based multi-view stereo system with flexible measurement setup. - In: Measurement: sensors, ISSN 2665-9174, Bd. 24 (2022), 100425, S. 1-9
In recent years, stereoscopic image processing algorithms have gained importance for a variety of applications. To capture larger measurement volumes, multiple stereo systems are combined into a multi-view stereo (MVS) system. To reduce the amount of data and the data rate, calculation steps close to the sensors are outsourced to Field Programmable Gate Arrays (FPGAs) as upstream computing units. The calculation steps include lens distortion correction, rectification and stereo matching. In this paper a FPGA-based MVS system with flexible camera arrangement and partly overlapping field of view is presented. The system consists of four FPGA-based passive stereoscopic systems (Xilinx Zynq-7000 7020 SoC, EV76C570 CMOS sensor) and a downstream processing unit (Zynq Ultrascale ZU9EG SoC). This synchronizes the sensor near processing modules and receives the disparity maps with corresponding left camera image via HDMI. The subsequent computing unit calculates a coherent 3D point cloud. Our developed FPGA-based 3D measurement system captures a large measurement volume at 24 fps by combining a multiple view with eight cameras (using Semi-Global Matching for an image size of 640 px × 460 px, up to 256 px disparity range and with aggregated costs over 4 directions). The capabilities and limitation of the system are shown by an application example with optical non-cooperative surface.
Investigations on the potential application of machine vision lenses for depth measurement by exploiting chromatic aberrations. - In: Measurement: sensors, ISSN 2665-9174, Bd. 23 (2022), 100410, S. 1-9
Chromatic (spectral) aberrations are image imperfections that are disadvantageous for standard image processing tasks and are typically compensated through the application of different types of glass during lens design. The longitudinal chromatic aberration causes a relative unsharpness over different spectral channels. Since this error is corrected in most multi-chromatic lenses, this paper investigates to which extent the shift of the focal planes in a standard lens can be used specifically for image processing applications. Theoretical investigations of the longitudinal chromatic aberration are carried out. Based on this, conditions and a method to generate a 3D depth reconstruction out of different spectral channels are presented.
High-resolution 3D shape measurement with extended depth of field using fast chromatic focus stacking. - In: Optics express, ISSN 1094-4087, Bd. 30 (2022), 13, S. 22590-22607
Close-range 3D sensors based on the structured light principle have a constrained measuring range due to their depth of field (DOF). Focus stacking is a method to extend the DOF. The additional time to change the focus is a drawback in high-speed measurements. In our research, the method of chromatic focus stacking was applied to a high-speed 3D sensor with 180 fps frame rate. The extended DOF was evaluated by the distance-dependent 3D resolution derived from the 3D-MTF of a tilted edge. The conventional DOF of 14 mm was extended to 21 mm by stacking two foci at 455 and 520 nm wavelength. The 3D sensor allowed shape measurements with extended DOF within 44 ms.
Automatic detection and prediction of discontinuities in laser beam butt welding utilizing deep learning. - In: Journal of advanced joining processes, ISSN 2666-3309, Bd. 6 (2022), 100119, S. 1-11
Laser beam butt welding of thin sheets of high-alloy steel can be really challenging due to the formation of joint gaps, affecting weld seam quality. Industrial approaches rely on massive clamping systems to limit joint gap formation. However, those systems have to be adapted for each individually component geometry, making them very cost-intensive and leading to a limited flexibility. In contrast, jigless welding can be a high flexible alternative to substitute conventionally used clamping systems. Based on the collaboration of different actuators, motions systems or robots, the approach allows an almost free workpiece positioning. As a result, jigless welding gives the possibility for influencing the formation of the joint gap by realizing an active position control. However, the realization of an active position control requires an early and reliable error prediction to counteract the formation of joint gaps during laser beam welding. This paper proposes different approaches to predict the formation of joint gaps and gap induced weld discontinuities in terms of lack of fusion based on optical and tactile sensor data. Our approach achieves 97.4 % accuracy for video-based weld discontinuity detection and a mean absolute error of 0.02 mm to predict the formation of joint gaps based on tactile length measurements by using inductive probes.
A new sensor system for accurate 3D surface measurements and modeling of underwater ojects. - In: Applied Sciences, ISSN 2076-3417, Bd. 12 (2022), 9, 4139, S. 1-15
A new underwater 3D scanning device based on structured illumination and designed for continuous capture of object data in motion for deep sea inspection applications is introduced. The sensor permanently captures 3D data of the inspected surface and generates a 3D surface model in real time. Sensor velocities up to 0.7 m/s are directly compensated while capturing camera images for the 3D reconstruction pipeline. The accuracy results of static measurements of special specimens in a water basin with clear water show the high accuracy potential of the scanner in the sub-millimeter range. Measurement examples with a moving sensor show the significance of the proposed motion compensation and the ability to generate a 3D model by merging individual scans. Future application tests in offshore environments will show the practical potential of the sensor for the desired inspection tasks.
Overview of the state of the art in the digitization of drivable forestry roads. - In: Proceedings of SPIE, Bd. 12091 (2022), S. 120910A-1-120910A-10
Mobile mapping becomes a more and more important and interesting field of sensing technologies and their application scenarios. Various applications range from airborne sensing of specific environments and characteristics to ground-based applications such as multimodal 3-dimensional registration of environments in the infrastructure sector or for assistance systems. In the specific case of infrastructure systems, known fields of application range from the detection of the surface condition of roads to the digitization of entire railroad lines, including their clearance diagrams. From the technical point of view, it also combines a wide variety of sensory approaches for sensing relevant features. For example, known systems use both LiDAR and GNSS and image processing-based subsystems. This work summarizes the state-of-the-art mobile mapping technologies in the framework of road detection and digitization concerning the application of georeferenced condition monitoring. In the first part, the relevant historical development will be briefly reviewed and compared regarding technological progress furthermore, various sensing systems will be compared regarding their applications, applicability and limitations. The aim is to clearly identify shortcomings regarding the application case of road detection in the forestry sector and thus to lay the foundation for subsequent research and development work for multimodal sensing systems. It is also the starting point for upcoming work for a multimodal sensing system that is able to digitalize and characterize the structure of forestry trails. The data obtained in this way will later be used for a planning tool that will derive measures for the maintenance and repair these forest roads.
Investigation on surface inspection using polarizing image sensors. - In: Proceedings of SPIE, Bd. 12091 (2022), S. 120910F-1-120910F-13
Surface inspection in industrial automated processes is very often challenging. Especially the detection of transparent liquid materials such as water represent a major challenge for standard imaging systems. One approach to overcome the limitation of these imaging systems lies in the exploitation of the polarization effect. This effect surely can only be applied if the contaminants have polarizing features but can help to use invisible characteristics of light for quality inspection tasks. In this work investigations on surfaces which are contaminated with water will be presented. Therefore, an imaging system using an RGB dome light illumination was set up in combination with a four-channel polarizing camera. The dome light, which is equipped with three different LED wavelengths, will be mixed so that the illumination which hits the sample is completely unpolarized. So, any effect on the surface which leads to a polarizing effect can be observed. The system delivers a four-channel image with different polarization angles that have to be processed. Therefore, an algorithm realizes a demosaicing which separates the four different polarized pixels into individual images. Based on this, the stokes equation which allows the calculation of the degree of polarization and the angle of polarization has to be processed for the image presentation. To achieve a better visualization of the degree of polarization an HSV-transformation based on the polarization parameters was also realized.
Multispectral imaging system for short wave infrared applications. - In: Proceedings of SPIE, Bd. 12094 (2022), S. 120940Z-1-120940Z-14
A lot of applications as well as in the laboratory range and industrial range need a short-wave infrared imaging system. Especially multispectral imaging in that wavelength region are often enable new applications for quality assurance and monitoring of industrial processes. Due to the cost of the SWIR image sensors a multispectral imaging system should be flexible and adjustable to realize a maximum of applications. Normally push broom devices will fulfill these requirements. Disadvantages are that the sample must be moved, the spectral crosstalk and the blurring between the different channels can disturb the processing, and the correctness of spatial resolution along the scanning direction lead to uncertainties for dimensional measurements. For that reasons a twelve-channel filter-based imaging system was designed. A motor driven filter-wheel with a high precision drive will ensure that the filters will be placed in front of a SWIR Image sensor very precisely. To compensate the spectral aberrations along the optical axis a second drive positioning the image sensor into the focus plane. This enable sharp images in all spectral channels as well as high SNR (depends also on the capability of the SWIR Sensor). Furthermore, this implementation allows also to focus without changing the adjustment of the objective lens. A special designed dome light using halogen bulbs delivers very homogeneous radiation on the sample table. For the image readout, the control of the drives as well as the image presentation a software with a graphical user interface was developed. To export the image stacks in 3rd party software the imaging software saves the multispectral images in the envi-format.
Human-robot interaction booth with shape-from-silhouette-based real-time proximity sensor. - In: Proceedings of SPIE, Bd. 12098 (2022), S. 120980B-1-120980B-9
Industrial robots have been an essential part of production facilities for many years. They allow fast and precise positioning of even the largest loads with very high repeatability. However, there are still many processes which are superiorly or more economically executed by humans. If a component requires several work steps, some of which are better suited to a robot and some to a human worker, cooperation between humans and robots would be beneficial. Due to the enormous power and speed of industrial robots, this poses a considerable risk to the worker. Therefore, tasks to be performed by humans and robots are usually completely decoupled in terms of space or time. We suggest an approach, which allows a human worker to interact safely with a fast industrial robot. We achieve this by constantly monitoring the position of both robot and human and adjusting the robot's velocity according to its proximity to the worker. We present an interaction booth, which can be entered by a robot arm from the back and a worker from the front such that they can both access the machinery within. A multicamera sensor, which is based on the shape-from-silhouette principle, constantly observes the booth to monitor its occupancy. We demonstrate that within 50 ms, our sensor can (1) detect a change in occupancy in the booth, (2) classify sub-volumes as "robot", "human", or "other object", (3) calculate the distance between human and robot, and (4) output this information to the robot controller. The working speed of the robot is then adjusted according to its distance to the worker.
Optimisation of a stereo image analysis by densify the disparity map based on a deep learning stereo matching framework. - In: Proceedings of SPIE, Bd. 12098 (2022), S. 120980D-1-120980D-16
Stereo vision is used in many application areas, such as robot-assisted manufacturing processes. Recently, many different efficient stereo matching algorithms based on deep learning have been developed to solve the limitations of traditional correspondence point analysis, among others. The challenges include texture-poor objects or non-cooperative objects. One of these end-to-end learning algorithms is the Adaptive Aggregation Network (AANet/AANet+), which is divided into five steps: feature extraction, cost volume construction, cost aggregation, disparity computation and disparity refinement. By combining different components, it is easy to create an individual stereo matching model. Our goal is to develop efficient learning methods for robot-assisted manufacturing processes for cross-domain data streams. The aim is to improve recognition tasks and process optimisation. To achieve this, we have investigated the AANet+ in terms of usability and efficiency on our own test-dataset with different measurement setups (passive stereo system). Input of the AANet+ are rectified stereo pairs of the test-dataset and a pre-trained model. Instead of generating our own training dataset, we used two pre-trained models based on the KITTI-2015 and SceneFlow datasets. Our research has shown that the pretrained model based on the Scene Flow dataset predicts disparities with better object delimination. Due to the Out-of-Distribution inputs, only reliable disparity predictions of the AANet are possible for test data sets with parallel measurement setup. We compared the results with two traditional stereo matching algorithms (SemiGlobal block matching and DAISY). Compared to the traditionally computed disparity maps, the AANet+ is able to robustly detect texture-poor objects and optically non-cooperative objects.