Publications

Communication technologies play an important role in maintaining the grandparent-grandchild (GP-GC) relationship. Based on Media Richness Theory, this study investigates the frequency of use (RQ1) and perceived quality (RQ2) of established media as well as the potential use of selected innovative media (RQ3) in GP-GC relationships with a particular focus on digital media. A cross-sectional online survey and vignette experiment were conducted in February 2021 among N = 286 university students in Germany (mean age 23 years, 57% female) who reported on the direct and mediated communication with their grandparents. In addition to face-to-face interactions, non-digital and digital established media (such as telephone, texting, video conferencing) and innovative digital media, namely augmented reality (AR)-based and social robot-based communication technologies, were covered. Face-to-face and phone communication occurred most frequently in GP-GC relationships: 85% of participants reported them taking place at least a few times per year (RQ1). Non-digital established media were associated with higher perceived communication quality than digital established media (RQ2). Innovative digital media received less favorable quality evaluations than established media. Participants expressed doubts regarding the technology competence of their grandparents, but still met innovative media with high expectations regarding improved communication quality (RQ3). Richer media, such as video conferencing or AR, do not automatically lead to better perceived communication quality, while leaner media, such as letters or text messages, can provide rich communication experiences. More research is needed to fully understand and systematically improve the utility, usability, and joy of use of different digital communication technologies employed in GP-GC relationships.

Previous studies have explored the perception of various types of embodied avatars in immersive environments. However, the impact of eye and face tracking with personalized avatars is yet to be explored. In this paper, we investigate the impact of eye and face tracking on embodiment, enfacement, and the uncanny valley with four types of avatars using a VR-based mirroring task. We conducted a study (N=12) and created self-avatars with two rendering styles: a cartoon avatar (created in an avatar generator using a picture of the user’s face) and a photorealistic scanned avatar (created using a 3D scanner), each with and without eye and face tracking and respective adaptation of the mirror image. Our results indicate that adding eye and face tracking can be beneficial for certain enfacement scales (belonged), and we confirm that compared to a cartoon avatar, a scanned realistic avatar results in higher body ownership and increased enfacement (own face, belonging, mirror) - regardless of eye and face tracking. We critically discuss our experiences and outline the limitations of the applied hardware and software with respect to the provided level of control and the applicability for complex tasks such as displaying emotions. We synthesize these findings into a discussion about potential improvements for facial animation in VR and highlight the need for a better level of control, the integration of additional sensing and processing technologies, and an objective metric for comparing facial animation systems.

Previous research has attempted to understand the influence of virtual reality (VR) on human perception of time, but neither a comprehensive understanding nor conclusive results have been achieved. To extend and continue research on this topic, we closely replicated a previous study and included new elements in a constructive replication study. To do this, we replicate the original setup and investigate the influence of workload and sun speed on time production. Contrary to previous findings, we did not find significant differences in virtual sun movements on time judgments. However, consistent with the original study, time perception in VR was affected by cognitive workload. In addition, we found that immersion in the virtual environment influenced time perception after VR. The contrasting results highlight the need for further research into the factors contributing to time perception in and after VR to fully explain the phenomenon of altered perception.

Realistic lighting approaches typically rely on physically-based rendering which in turn often makes use of image-based lighting. Enabling these techniques in augmented reality on mobile devices requires unique approaches to estimate light probes, given the limited camera and sensor data available. In this paper, we evaluate different time-dependent and time-independent techniques for light probe estimation in augmented reality applications that try to predict the environment lighting using single images or video streams in combination with inpainting techniques. We simulate real-world applications using an evaluation framework where a simulated mobile device captures camera streams from different scenarios following a pre-defined path. The resulting camera streams are fed to a total of six estimation techniques in order to create light probes which are then used to render virtual objects while applying various materials. By comparing the rendered images as well as the light probe estimations, we perform a quantitative evaluation. We show how approaches that are able to process continuous video streams can provide more plausible results in cases where sufficient camera movement is present. Additionally, we investigate the visual impression of different types of materials showing that rough surfaces with distinct colors are less likely to produce divergent estimation results.

The combination of smartphone Augmented Reality (AR) and Virtual Reality (VR) makes it possible for on-site and remote users to simultaneously explore a physical space and its digital twin through an asymmetric Collaborative Virtual Environment (CVE). In this paper, we investigate two spatial awareness visualizations to enable joint exploration of a space for dyads consisting of a smartphone AR user and a head-mounted display VR user. Our study revealed that both, a mini-map-based method and an egocentric compass method with a path visualization, enabled the on-site visitors to locate and follow a virtual companion reliably and quickly. Furthermore, the embodiment of the AR user by an inverse kinematics avatar allowed the use of natural gestures such as pointing and waving which was preferred over text messages by the participants of our study. In an expert review in a museum and its digital twin we observed an overall high social presence for on-site AR and remote VR visitors and found that the visualizations and the avatar embodiment successfully facilitated their communication and collaboration.

Eine effektive und effiziente universitäre Programmierausbildung erfordert zunehmend den Einsatz automatisierter Bewertungssysteme. Im Rahmen des Projekts examING erprobt das Teilprojekt AutoPING den Einsatz des quelloffenen Autograding-Systems CodeOcean für übergreifende Lehrangebote und Prüfungen an der TU Ilmenau mit dem Ziel, selbstgesteuertes und kompetenzorientiertes Lernen zu ermöglichen und zu fördern. Der Beitrag gibt einen Überblick über erste Projekterfahrungen bei der Adaption didaktischer Szenarien in der Programmierausbildung hin zu testgetriebener Softwareentwicklung sowie der Generierung von Feedback. Es werden wesentliche Erkenntnisse aus Sicht der Studierenden und Lehrenden erörtert, Herausforderungen und Lösungsansätze zur Integration und Erweiterung von CodeOcean für neue Anwendungsfelder diskutiert sowie zukünftige Perspektiven eröffnet.

Augmented Reality (AR) and Virtual Reality (VR) are pushing from the labs towards consumers, especially with social applications. These applications require visual representations of humans and intelligent entities. However, displaying and animating photo-realistic models comes with a high technical cost while low-fidelity representations may evoke eeriness and overall could degrade an experience. Thus, it is important to carefully select what kind of avatar to display. This article investigates the effects of rendering style and visible body parts in AR and VR by adopting a systematic literature review. We analyzed 72 papers that compare various avatar representations. Our analysis includes an outline of the research published between 2015 and 2022 on the topic of avatars and agents in AR and VR displayed using head-mounted displays, covering aspects like visible body parts (e.g., hands only, hands and head, full-body) and rendering style (e.g., abstract, cartoon, realistic); an overview of collected objective and subjective measures (e.g., task performance, presence, user experience, body ownership); and a classification of tasks where avatars and agents were used into task domains (physical activity, hand interaction, communication, game-like scenarios, and education/training). We discuss and synthesize our results within the context of today's AR and VR ecosystem, provide guidelines for practitioners, and finally identify and present promising research opportunities to encourage future research of avatars and agents in AR/VR environments.

What is Virtual Reality (VR)? What is Augmented Reality (AR)? What is the purpose of VR/AR? What are the basic concepts? What are the hard- and software components of VR/AR systems? How has VR/AR developed historically? The first chapter examines these questions and provides an introduction to this textbook. This chapter is fundamental for the whole book. All subsequent chapters build on it and do not depend directly on one another. Therefore, these chapters can be worked through selectively and in a sequence that suits the individual interests and needs of the readers. Corresponding tips on how this book can be used efficiently by different target groups (students, teachers, users, technology enthusiasts) are provided at the end of the chapter, as well as a summary, questions for reviewing what has been learned, recommendations for further reading, and the references used in the chapter.

This chapter discusses output devices and technologies for Virtual Reality (VR) and Augmented Reality (AR). The goal of using output devices is to enable the user to dive into the virtual world or to perceive the augmented world. Devices for visual output play a crucial role here, they are of central importance for the use of VR and AR. First and foremost, Head-Mounted Displays (HMD) must be mentioned, the different types of which are discussed in detail here. However, VR also uses different forms of stationary displays, which are another major topic of this chapter. Finally, output devices for other senses are reviewed, namely acoustic and haptic outputs.

This chapter deals with the authoring of VR and AR applications. The focus here is on the use of authoring tools in the form of software development kits (SDKs) or game engines. First, the actual authoring process will be briefly discussed before selected authoring tools for VR and AR are reviewed. Subsequently, the authoring process and the use of the tools will be illustrated through typical case studies. The other chapters of this book deal with the fundamentals and methodologies of VR and AR. These are generally applicable over a longer period. In contrast to this, this chapter looks at some very specific authoring tools and the authoring process based on them, which can inevitably only represent a snapshot in time. Features, releases and availability of these tools can change at short notice, so that individual sections may no longer be up to date when this book is in press. To take this aspect into account, the case studies listed here are stored in an online repository, where they are regularly updated to reflect the latest versions of the authoring tools and runtime environments.

In recent years, various depth sensors that are small enough to be used with mobile hardware have been introduced. They provide important information for use cases like 3D reconstruction or in the context of augmented reality where tracking and camera data alone would be insufficient. However, depth sensors may not always be available due to hardware limitations or when simulating augmented reality applications for prototyping purposes. In these cases, different approaches like stereo matching or depth estimation using neural networks may provide a viable alternative. In this paper, we therefore explore the imitation of depth sensors using deep neural networks. For this, we use a state-of-the-art network for depth estimation and adapt it in order to mimic a Structure Sensor as well as an iPad LiDAR sensor. We evaluate the network which was pre-trained on NYU V2 directly as well as several variations where transfer learning is applied in order to adapt the network to different depth sensors while using various data preprocessing and augmentation techniques. We show that a transfer learning approach together with appropriate data processing can enable an accurate modeling of the respective depth sensors.

Recent developments in augmented reality for TV productions encouraged broadcasters to enhance interaction with virtual content for moderators. However, traditional interaction methods are considered distracting and not intuitive. To overcome these issues, we performed a gesture elicitation study with a follow-up evaluation. For this, we considered TV moderators as primary users of the gestures as well as viewers as recipients. The elicited gesture set consists of five gestures for two types of camera shots (long shot and close shot). Findings of the evaluation study indicate that the derived set of gestures requires low physical and concentration effort from moderators. Also, both moderators and viewers found them appropriate to be used in TV with respect to understandability, distraction, likeability, and appropriateness. Using these gestures would allow moderators to control AR content in TV and tell stories in a modern and more expressive way.

When operating a conditionally automated vehicle, humans occasionally have to take over control. If the driver is out of the loop, a certain amount of time is necessary to gain situation awareness. This work evaluates the potential of stereoscopic 3D (S3D) dashboards for presenting smart S3D take-over-requests (TORs) to support situation assessment. In a driving simulator study with a 4 × 2 between-within design, we presented 3 smart TORs showing the current traffic situation and a baseline TOR in 2D and S3D to 52 participants doing the n-back task. We further investigate if non-standard locations affect the results. Take-over performance indicates that participants looked at and processed the TORs' visual information and by that, could perform more safe take-overs. S3D warnings in general, as well as warnings appearing at the participants’ focus of attention and warnings at the instrument cluster, performed best. We conclude that visual warnings, presented on an S3D dashboard, can be a valid option to support take-over while not increasing workload. We further discuss participants’ gaze behavior in the context of visual warnings for automotive user interfaces.

Smart home devices often blend in seamlessly into the environment and operate ubiquitously, providing almost no contextual information such as data collection or activated sensors. Augmented Reality (AR) could, for example in form of head-mounted-displays (HMD), offer users a non-intrusive way to query the devices and to get privacy-related information. This pilot study explored how privacy icons, displayed by an AR-HMD and co-located with smart home devices, affect users' privacy awareness. In a qualitative within-subject study, 16 participants experienced first a setup without AR privacy information and then one with such information. Participants' answers indicate a high potential and excitement towards such a setup. Among others, they stated changed privacy awareness after experiencing AR privacy icons: While the icons prioritized privacy-related information for users and educated them to promote conscious decision-making, they sometimes also reinforced existing negative attitudes towards smart home devices. Further, the display of icons in AR has the potential to inspire trust towards manufacturers and providers, potentially leading to a false sense of security. Next to the discussion of participants' answers, we outline the implications of our findings and provide recommendations for future research including trust and learning effects.

Die Visualisierung virtueller Objekte in der Realität erfolgt bei vielen Anwendungen oftmals durch eine vereinfachte Darstellung ohne Bezug zur umliegenden Umgebung. Dabei ist die nahtlose Verschmelzung der virtuellen und realen Umgebung in vielen Bereichen ein wesentlicher Faktor, der insbesondere bei der Beleuchtungsberechnung in gemischten Realitäten von großer Bedeutung ist. Aktuelle Ansätze legen den Fokus auf Approximationen, welche eine Berechnung der diffusen Beleuchtung ermöglichen, wobei die Darstellung glänzender Beleuchtungseigenschaften vernachlässigt wird. Das Ziel dieser Arbeit ist eine Visualisierung von spiegelnden Oberflächen in erweiterten Realitäten zu ermöglichen. Um dieses Ziel zu erreichen werden verschiedene Verfahren aufgezeigt, die eine hochwertige Darstellung virtueller Objekte in Echtzeit ermöglichen, wobei der Fokus auf der Verwendung üblicher Hardware wie Kameras, Sensoren in mobilen Endgeräten und teilweise Tiefensensoren liegt. Das erste Verfahren verwendet das aktuelle Kamerabild zur Rekonstruktion einer 360-Grad Beleuchtung durch die Transformation des Bildes auf eine Würfelkarte. Anschließend wird dieser Ansatz durch ein Stitching erweitert, bei dem das aktuelle Bild zusammen mit der Position und Rotation der Kamera genutzt wird, um eine Umgebungsbeleuchtung zu erstellen. Dabei von der Kamera nicht erfasste Bereiche werden durch ein semantisches Inpainting, basierend auf einem neuronalen Netz, aufgefüllt. Hierdurch kann das komplette Umgebungslicht approximiert werden, welches die Darstellung detaillierter Reflexionen ermöglicht. Die Ergebnisse der Ansätze zeigen neuartige Möglichkeiten, geometrische Formen mit glänzenden Oberflächen in die reale Umgebung einzubetten und bietet im Vergleich zu bestehenden Methoden einen höheren Detailgrad in den Reflexionen. Alle Ansätze sind für eine Verwendung auf mobilen Endgeräten konzipiert, wodurch neue Möglichkeiten für verschiedene Einsatz- und Anwendungsbereiche der erweiterten Realität existieren.

State-of-the-art methods for diminished reality propagate pixel information from a keyframe to subsequent frames for real-time inpainting. However, these approaches produce artifacts, if the scene geometry is not sufficiently planar. In this article, we present InpaintFusion, a new real-time method that extends inpainting to non-planar scenes by considering both color and depth information in the inpainting process. We use an RGB-D sensor for simultaneous localization and mapping, in order to both track the camera and obtain a surfel map in addition to RGB images. We use the RGB-D information in a cost function for both the color and the geometric appearance to derive a global optimization for simultaneous inpainting of color and depth. The inpainted depth is merged in a global map by depth fusion. For the final rendering, we project the map model into image space, where we can use it for effects such as relighting and stereo rendering of otherwise hidden structures. We demonstrate the capabilities of our method by comparing it to inpainting results with methods using planar geometric proxies.

In der Natur ist jede Interaktion des Lichts mit Materie in einen globalen Kontext eingebunden, weswegen alle natürlichen Beleuchtungsphänomene in unserer Umwelt das Resultat globaler Beleuchtung sind. Diese basiert auf der Anwendung mannigfaltiger Absorptions-, Reflexions- und Brechungsgesetze, deren Simulation so komplex ist, dass interaktive Anwendungen diese nicht in wenigen Millisekunden berechnen können. Deshalb wurde bisher in vielen interaktiven Systemen auf die Abbildung von solchen globalen Beleuchtungsphänomenen verzichtet, was jedoch zu einer unrealistischen und synthetisch-wirkenden Darstellung führte. Diese unrealistische Darstellung ist besonders für die Anwendungsfelder Virtual Reality und Augmented Reality, bei denen der Nutzer eine möglichst realitätsnahe Simulation erfahren soll, ein gewichtiger Nachteil. In dieser Arbeit wird das LightSkin-Verfahren vorgestellt, das es erlaubt, globale Beleuchtungsphänomene in einer Echtzeitanwendung darzustellen. Das Verfahren wurde speziell für die Anwendungsfelder Virtual Reality und Augmented Reality entwickelt und erfüllt spezifische Anforderungen, die diese an eine Echtzeitanwendung stellen. Bei dem Verfahren wird das indirekte Licht durch eine geringe Anzahl von Punktlichtquellen (Proxy-Lichtquellen) repräsentiert, die für eine lose Menge von Oberflächenpunkten (Caches) berechnet und anschließend über die komplette sichtbare Szene interpoliert werden. Diese neue Form der Repräsentation der indirekten Beleuchtung erlaubt eine effiziente Berechnung von diffusen und glänzenden indirekten Reflexionen, die Abbildung von weichen Schatten und die Simulation von Multiple-Subsurface-Scattering-Effekten in Echtzeit für komplexe und voll dynamische Szenen. Ferner wird gezeigt, wie das Verfahren modifiziert werden kann, um globale Lichtwechselwirkungen zwischen realen und virtuellen Objekten in einer Augmented-Reality-Anwendung zu simulieren. Im Gegensatz zu den meisten existierenden Echtzeitverfahren zur Simulation von globalen Beleuchtungseffekten benötigt der hier vorgestellte Ansatz keine aufwändigen zusätzlichen Berechnungen bei Animationen und erzeugt darüber hinaus für diese keine visuellen Artefakte. Diese Arbeit enthält alle Informationen, die zum Verständnis, zur Implementierung und zur Evaluation des LightSkin-Verfahrens benötigt werden und gibt darüber hinaus einen umfassenden Überblick über das Forschungsfeld.

Diminished Reality ist eine neue faszinierende Technologie, die es ermöglicht, reale Inhalte aus Live-Kamerabildern zu entfernen. Die Live-Videomanipulation entfernt reale Objekte und erzeugt in Echtzeit einen kohärenten Video-Stream. Dabei können die Betrachter keine Manipulation feststellen. Derzeit existierende Ansätze sind auf sich bewegende Objekte und statische oder weitestgehend statische Kameras beschränkt und erlauben keine Manipulation des Videoinhalts in Echtzeit. In dieser Arbeit wird ein innovativer Ansatz vorgestellt, der das Entfernen von Objekten bei beliebigen Kamerabewegungen ermöglicht. Dabei gilt es vorrangig, zwei Herausforderungen zu lösen. Zum einen wird eine Bild-Inpainting-Methode benötigt, die innerhalb weniger Millisekunden auf jedes Kamerabild angewandt werden kann und qualitativ hochwertige Ergebnisse liefert. Zum anderen muss ohne die Verwendung vergangener oder zukünftiger Kamerabilder eine Bild-zu-Bild-Kohärenz erzeugt werden. In dieser Arbeit wird eine neuartige und leistungsfähige Selektionsstrategie vorgestellt, die auf sogenannten Fingerabdrücken basiert, damit unerwünschte Inhalte selbst auf heterogenen Hintergründen bestimmt werden können. Der Inpainting-Ansatz wurde ausgehend von bereits bekannten randomisierten Ansätzen und Verfahren mit mehreren Bildebenen entwickelt. Qualitativ hochwertige Ergebnisse können in Echtzeit durch die Anwendung einer einzigartigen Initialisierungsstrategie und einer neuartigen Kostenfunktion, die die Kohärenzabweichung miniert, erzeugt werden. Die Kostenfunktion kombiniert räumliche und erscheinungsbasierende Kosten. Der auf Homographie basierte Ansatz erzeugt eine Bild-zu-Bild-Kohärenz beim Entfernen von Objekten, vor überwiegend ebenen Hintergründen und unterstützt Rotationsbewegungen der Kamera um das zu entfernende Objekt. Obwohl der Ansatz etwa zwei Zehnerpotenzen schneller ist als aktuelle, vergleichbare Inpainting-Ansätze, zeigt sich, dass er für bekannte Testbilder durchgängig qualitativ gleichwertige oder bessere Ergebnisse erzeugt. Ein erster Test mit Nutzern zeigt, dass Videomanipulationen, die mit diesem Ansatz durchgeführt werden, für Testpersonen kaum zu erkennen sind, sogar wenn diese sich der Manipulationsmöglichkeiten bewusst sind. Basierend auf diesen Ergebnissen eröffnet der in der Arbeit vorgestellte Ansatz eine Fülle neuer Möglichkeiten für interaktive und echtzeitfähige Manipulationen, die vor allem in TV- und Filmproduktionen sowie im Bereich der Werbung eingesetzt werden können.

"This book discusses how collaboration between humans and intelligent systems can potentially improve access to and processing of complex information, contributing to the synthesis, storage and integration of knowledge"--Provided by publisher

Previous paradigms for presence research were primarily established in the context of VR. The objective of this paper is to introduce a new agenda for research on presence suitable for the domain of MR. While established assumptions and methods of presence research from VR are applicable to Mr experiences, we argue that they are not necessarily meaningful or informative. Specifically, a shift of attention is needed away from psycho-physiological studies coming from a laboratory experiment tradition, toward an ecological-cultural approach that is applicable in real world situations and relies on ethnographic rather than fully controlled methods. We give a series of examples taken from the work on the European integrated research project IPcity, and discuss the implications of our findings.

The current state of affair in learning systems in general and in e-learning in particular suffers from a lack of an explicit and adaptable didactic design. Students complain about the insufficient adaptability of e-learning to the learners' needs. Learning content and services need to reach their audience properly. That is, according to their different prerequisites, needs, and different learning conditions. After a short introduction to the storyboard concept, which is a way to address these concerns, we present an example of using storyboards for the didactic design of a university course on Intelligent Systems. In particular, we show the way to express didactic variants and didactic intentions within storyboards. Finally, we sketch ideas to for a machine supported knowledge processing, knowledge evaluation, knowledge refinement, and knowledge engineering with storyboards.

Currently, e-learning systems as well as learning environments are still suffering from a lack of explicit and adaptable didactic design - an issue of quality management. Students complain about insufficient adaptability of e-learning offers to the learners’ needs. This is especially the case in university teaching. University professors are not necessarily experts in the didactics of teaching their subject. As a way out of this dilemma the authors utilize their formerly developed storyboard concept. Storyboards are an explicit representation of a didactic design, which enjoys simplicity, clarity, visual appearance, and the chance for standardized development. Everybody may easily become a storyboard author - a step toward pedagogical engineering. By representing didactics explicitly, learners are supported to find their individually appropriate way. Besides this processing issue, storyboards have the potential to identify successful didactic patterns. This is, in fact, the vision of knowledge discovery in didactics. In the authors' experience, embedding topical content into game adventures appears to be a promising didactic mean when properly linked in a related didactic pattern. The success of the authors' approach is demonstrated by an application to one of the authors' real life courses.

Die Personas-Methode wird im Bereich der Website- und Softwareentwicklung zur Evaluation, aber auch zum Design komplexer Systeme zunehmend eingesetzt. Welche Potenziale und welche Probleme birgt die Anwendung des Verfahrens? Wie können Personas in den Entwicklungsprozess integriert werden? Auf diese und weiterführende Fragen gibt der UPA Workshop im Rahmen der Mensch & Computer 2005 Antworten. Praxisbeispiele und Erfahrungsberichte sollen den Teilnehmern Hintergrundinformationen liefern und zur Diskussion motivieren.