Fusion of multimodal imaging and 3D digitization using photogrammetry. - In: Sensors, ISSN 1424-8220, Bd. 24 (2024), 7, 2290, S. 1-20
Multimodal sensors capture and integrate diverse characteristics of a scene to maximize information gain. In optics, this may involve capturing intensity in specific spectra or polarization states to determine factors such as material properties or an individual’s health conditions. Combining multimodal camera data with shape data from 3D sensors is a challenging issue. Multimodal cameras, e.g., hyperspectral cameras, or cameras outside the visible light spectrum, e.g., thermal cameras, lack strongly in terms of resolution and image quality compared with state-of-the-art photo cameras. In this article, a new method is demonstrated to superimpose multimodal image data onto a 3D model created by multi-view photogrammetry. While a high-resolution photo camera captures a set of images from varying view angles to reconstruct a detailed 3D model of the scene, low-resolution multimodal camera(s) simultaneously record the scene. All cameras are pre-calibrated and rigidly mounted on a rig, i.e., their imaging properties and relative positions are known. The method was realized in a laboratory setup consisting of a professional photo camera, a thermal camera, and a 12-channel multispectral camera. In our experiments, an accuracy better than one pixel was achieved for the data fusion using multimodal superimposition. Finally, application examples of multimodal 3D digitization are demonstrated, and further steps to system realization are discussed.
https://doi.org/10.3390/s24072290
Data fusion of RGB and depth data with image enhancement. - In: Journal of imaging, ISSN 2313-433X, Bd. 10 (2024), 3, 73, S. 1-17
Since 3D sensors became popular, imaged depth data are easier to obtain in the consumer sector. In applications such as defect localization on industrial objects or mass/volume estimation, precise depth data is important and, thus, benefits from the usage of multiple information sources. However, a combination of RGB images and depth images can not only improve our understanding of objects, capacitating one to gain more information about objects but also enhance data quality. Combining different camera systems using data fusion can enable higher quality data since disadvantages can be compensated. Data fusion itself consists of data preparation and data registration. A challenge in data fusion is the different resolutions of sensors. Therefore, up- and downsampling algorithms are needed. This paper compares multiple up- and downsampling methods, such as different direct interpolation methods, joint bilateral upsampling (JBU), and Markov random fields (MRFs), in terms of their potential to create RGB-D images and improve the quality of depth information. In contrast to the literature in which imaging systems are adjusted to acquire the data of the same section simultaneously, the laboratory setup in this study was based on conveyor-based optical sorting processes, and therefore, the data were acquired at different time periods and different spatial locations. Data assignment and data cropping were necessary. In order to evaluate the results, root mean square error (RMSE), signal-to-noise ratio (SNR), correlation (CORR), universal quality index (UQI), and the contour offset are monitored. With JBU outperforming the other upsampling methods, achieving a meanRMSE = 25.22, mean SNR = 32.80, mean CORR = 0.99, and mean UQI = 0.97.
https://doi.org/10.3390/jimaging10030073
Recurrent autoencoder for weld discontinuity prediction. - In: Journal of advanced joining processes, ISSN 2666-3309, Bd. 9 (2024), 100203, S. 1-12
Laser beam butt welding is often the technique of choice for a wide range of industrial tasks. To achieve high quality welds, manufacturers often rely on heavy and expensive clamping systems to limit the sheet movement during the welding process, which can affect quality. Jiggless welding offers a cost-effective and highly flexible alternative to common clamping systems. In laser butt welding, the process-induced joint gap has to be monitored in order to counteract the effect by means of an active position control of the sheet metal. Various studies have shown that sheet metal displacement can be detected using inductive probes, allowing the prediction of weld quality by ML-based data analysis. The probes are dependent on the sheet metal geometry and are limited in their applicability to complex geometric structures. Camera systems such as long-wave infrared (LWIR) cameras can instead be mounted directly behind the laser to overcome a geometry dependent limitation of the jiggles system. In this study we will propose a deep learning approach that utilizes LWIR camera recordings to predict the remaining welding process to enable an early detection of weld interruptions. Our approach reaches 93.33% accuracy for time-wise prediction of the point of failure during the weld.
https://doi.org/10.1016/j.jajp.2024.100203
Synthetic training data in AI-driven quality inspection: the significance of camera, lighting, and noise parameters. - In: Sensors, ISSN 1424-8220, Bd. 24 (2024), 2, 649, S. 1-18
Industrial-quality inspections, particularly those leveraging AI, require significant amounts of training data. In fields like injection molding, producing a multitude of defective parts for such data poses environmental and financial challenges. Synthetic training data emerge as a potential solution to address these concerns. Although the creation of realistic synthetic 2D images from 3D models of injection-molded parts involves numerous rendering parameters, the current literature on the generation and application of synthetic data in industrial-quality inspection scarcely addresses the impact of these parameters on AI efficacy. In this study, we delve into some of these key parameters, such as camera position, lighting, and computational noise, to gauge their effect on AI performance. By utilizing Blender software, we procedurally introduced the “flash” defect on a 3D model sourced from a CAD file of an injection-molded part. Subsequently, with Blender’s Cycles rendering engine, we produced datasets for each parameter variation. These datasets were then used to train a pre-trained EfficientNet-V2 for the binary classification of the “flash” defect. Our results indicate that while noise is less critical, using a range of noise levels in training can benefit model adaptability and efficiency. Variability in camera positioning and lighting conditions was found to be more significant, enhancing model performance even when real-world conditions mirror the controlled synthetic environment. These findings suggest that incorporating diverse lighting and camera dynamics is beneficial for AI applications, regardless of the consistency in real-world operational settings.
https://doi.org/10.3390/s24020649
Qualitätsmanagement-Grundlagen : Aufbau und Zertifizierung von Managementsystemen, Metrologie, Messtechnik
5., vollständig überarbeitete Auflage. - München : Hanser, 2024. - 1 Online-Ressource (XV, 368 Seiten). - (Hanser eLibrary) ISBN 978-3-446-47695-0
Umfassendes praxisorientiertes Lehr- und Arbeitsbuch Dieses Lehr- und Arbeitsbuch vermittelt das Grundwissen zum Qualitätsmanagement (QM) und stellt Zusammenhänge zu anderen Wissensgebieten, insbesondere zur Messtechnik und Metrologie her. Kapitel zu Normen für das QM, Qualitätsregelkreisen, Struktur und Aufbau integrierter QM-Systeme, Prozessmanagement, staatlich-metrologischer Infrastruktur und zur Einführung und Zertifizierung von QM-Systemen runden die Darstellungen ab. - Berücksichtigt die aktuelle Normenfamilie ISO 9000 ff. - Anleitung zum Aufbau und zur Pflege von QM-Dokumentationen - Umsetzungsorientiert und kompakt - Sowohl im Studium als auch in der Praxis einsetzbar - Zum Download: Umfangreiches Paket mit praktischen Arbeitshilfen
https://doi.org/10.3139/9783446476950
Data augmentation for solving industrial recognition tasks with underrepresented defect classes. - In: Acta IMEKO, ISSN 2221-870X, Bd. 12 (2023), 4, S. 1-5
This paper discusses neural network-based data augmentation to increase the performance of neural networks in classification of datasets with underrepresented defect classes. The performance of deep neural networks suffers from an inhomogeneous class distribution in recognition tasks. In particular, applications of deep neural networks to solve quality assurance tasks in industrial production suffer from such unbalanced class distributions. In order to train deep learning networks, a large amount of data is needed to avoid overfitting and to give the network a good generalisation ability. Therefore, a large amount of defect class objects is needed. However, when it comes to producing defect classes, obtaining a dataset for training can be costly. To reduce this costs, artificial intelligence in the form of Generative Adversarial Networks (GANs) can be used to generate images without producing real objects of defect classes. This allows a cost-effective solution for any kind of underrepresented classes. However, the focus of this work is on defect classes. In this paper a comparison of GANs for data augmentation with classical data augmentation methods for simulating images of defect classes in an industrial context is presented. The results show the positive effect of both, classical and GAN-based data augmentation. By applying both methods parallel the best results for defect-class recognition tasks of datasets with underrepresented classes can be achieved.
https://doi.org/10.21014/actaimeko.v12i4.1320
Artificial intelligence methods for in-process high-speed image analysis in laser beam welding of hairpins. - In: NOLAMP- Nordic Laser Materials Processing Conference (19TH-NOLAMP-2023), 22/08/2023-24/08/2023, Turku, Finland, (2023), 012007, S. 1-13
In the production of modern electric drives for battery electric vehicles, hairpin technology is used to increase the copper fill factor in the stator of a permanently excited synchronous machine. A central process in the production of these stators is the contacting of the hairpin ends by means of laser beam welding. This welding process is characterized by geometric and process-related deviations from previous process steps, which influence the result of the welded joint. It is desirable to find an in-process method for monitoring. As part of the process monitoring of welded joints, high-speed camera images are often used to detect weld spatter. These can be detected by a program based on a static algorithm. For this reason, a feasibility analysis is performed regarding the application of AI for the detection of spatters, in which the methods of semantic segmentation and single-image classification prove to be useful. In a preliminary experiment, three base networks for each of the two methods are evaluated with respect to the best training results. The single-image classification method will then be extended by a subsequent static algorithm, so that a hybrid use of AI and static algorithm will be investigated. The evaluation and final comparison of all evaluation methods is performed using data from a welding experiment. It turns out that the hybrid approach of single-image classification and static algorithm has numerous advantages in the detection of spatter compared to semantic segmentation and the static algorithm.
https://doi.org/10.1088/1757-899X/1296/1/012007
Contactless optical detection of nocturnal respiratory events. - In: Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, VISIGRAPP, (2023), S. 336-344
Obstructive sleep apnea (OSA) is a common sleep-related breathing disorder characterized by the collapse of the upper airway and associated with various diseases. For clinical diagnosis, a patient’s sleep is recorded during the night via polysomnography (PSG) and evaluated the next day regarding nocturnal respiratory events. The most prevalent events include obstructive apneas and hypopneas. In this paper, we introduce a fully automatic contactless optical method for the detection of nocturnal respiratory events. The goal of this study is to demonstrate how nocturnal respiratory events, such as apneas and hypopneas, can be autonomously detected through the analysis of multi-spectral image data. This represents the first step towards a fully automatic and contactless diagnosis of OSA. We conducted a trial patient study in a sleep laboratory and evaluated our results in comparison with PSG, the gold standard in sleep diagnostics. In a study sample with three patients, 24 hours of recor ded video materials and 245 respiratory events, we have achieved a classification accuracy of 82 % with a random forest classifier.
https://doi.org/10.5220/0011694400003417
TranSpec3D: a novel measurement principle to generate a non-synthetic data set of transparent and specular surfaces without object preparation. - In: Sensors, ISSN 1424-8220, Bd. 23 (2023), 20, 8567, S. 1-24
Estimating depth from images is a common technique in 3D perception. However, dealing with non-Lambertian materials, e.g., transparent or specular, is still nowadays an open challenge. However, to overcome this challenge with deep stereo matching networks or monocular depth estimation, data sets with non-Lambertian objects are mandatory. Currently, only few real-world data sets are available. This is due to the high effort and time-consuming process of generating these data sets with ground truth. Currently, transparent objects must be prepared, e.g., painted or powdered, or an opaque twin of the non-Lambertian object is needed. This makes data acquisition very time consuming and elaborate. We present a new measurement principle for how to generate a real data set of transparent and specular surfaces without object preparation techniques, which greatly reduces the effort and time required for data collection. For this purpose, we use a thermal 3D sensor as a reference system, which allows the 3D detection of transparent and reflective surfaces without object preparation. In addition, we publish the first-ever real stereo data set, called TranSpec3D, where ground truth disparities without object preparation were generated using this measurement principle. The data set contains 110 objects and consists of 148 scenes, each taken in different lighting environments, which increases the size of the data set and creates different reflections on the surface. We also show the advantages and disadvantages of our measurement principle and data set compared to the Booster data set (generated with object preparation), as well as the current limitations of our novel method.
https://doi.org/10.3390/s23208567
Intelligente Qualitätssicherung im industriellen Produktionsprozess unter Verwendung von KI-Algorithmen. - In: Nachhaltiges Qualitätsdatenmanagement, (2023), S. 120-138
In diesem Beitrag werden intelligente Qualitätssicherungslösungen für die automatisierte Erkennung verschiedener Fehlerklassen im industriellen Fertigungsprozess auf Basis optischer Bilderfassung, intelligenter digitaler Bildverarbeitung sowie Verfahren der Künstlichen Intelligenz vorgestellt. Hierbei werden schnelle automatisierte QS-Lösungen, sowohl für den Kunststoffspritzguss von Bauteilen im Automobilbau auf der Basis Robotik-assistierter Farbbildaufnahmen, als auch für die Metalloberflächenanalyse im Fräsbearbeitungsprozess auf der Basis von Farbbildern, aufgezeigt. Für beide QS-Lösungen ist die Realisierung einer angepassten Bildverarbeitungs- und Mustererkennungskette sowie angepasster leistungsfähiger Algorithmen der Künstlichen Intelligenz (KI) mit hoher Generalisierungs- und Abstraktionsfähigkeit auf Basis gewonnener Bildinformationen essentiell. Die wesentlichen Voraussetzungen und Schritte zur Lösung der beiden QS-Aufgaben werden vorgestellt und die verschiedenen Aspekte einer angepassten Bilderfassung, eines KI-Klassifikationsroutinen-Designs sowie der Validierung der Klassifikationsleistung und mögliches Optimierungspotential durch eine ressourcenschonende und effiziente Datennutzung unter künstlicher Datenaugmentierung unterrepräsentierter Klassen beleuchtet. Eine mögliche Performance-Steigerung von vortrainierten Deep-Learning-Modellen wird sowohl für die synthetische Bilddatensatzerweiterung unter Verwendung von klassischen Bildverarbeitungsmethoden als auch für die innovative Bilddatensatzerweiterung unter Verwendung von Generative Adversarial Networks (GANs) für die beiden gewählten QS-Applikationen aufgezeigt.
https://doi.org/10.1007/978-3-658-40588-5_7