Multimodal Technologies and Interaction

Throughout the last years, Intelligent Virtual Assistants (IVAs), such as Alexa and Siri, have increasingly gained in popularity. Yet, privacy advocates raise great concerns regarding the amount and type of data these systems collect and consequently process. Among many other things, it is technology trust which seems to be of high significance here, particularly when it comes to the adoption of IVAs, for they usually provide little transparency as to how they function and use personal and potentially sensitive data. While technology trust is influenced by many different socio-technical parameters, this article focuses on human personality and its connection to respective trust perceptions, which in turn may further impact the actual adoption of IVA products. To this end, we report on the results of an online survey (n=367). Findings show that on a scale from 0 to 100%, people trust IVAs 51.59% on average. Furthermore, the data point to a significant positive correlation between people’s propensity to trust in general technology and their trust in IVAs. Yet, they also show that those who exhibit a higher propensity to trust in technology tend to also have a higher affinity for technology interaction and are consequently more likely to adopt IVAs.

Lisa Schadelbauer

Stephan Schlögl

Aleksander Groth

Linking Personality and Trust in Intelligent Virtual Assistants

智能虚拟助手中个性与信任的关联

Recognizing places of interest (POIs) can be challenging for humans, especially in foreign environments. In this study, we leverage smartphone sensors (i.e., camera, GPS) and deep learning algorithms to propose an intelligent solution to recognize POIs in an urban environment. Recent studies have approached landmark recognition as an image retrieval problem. However, visual similarity alone is not robust against challenging conditions such as extreme appearance variance and perceptual aliasing in urban environments. To this end, we propose to fuse visual, textual, and positioning information. Our contributions are as follows. Firstly, we propose VPR through text reading pipeline (VPRText) that uses off-the-shelf text spotting algorithms for word spotting followed by layout analysis and text similarity search modules. Secondly, we propose a hierarchical architecture that combines VPRText and image retrieval. Thirdly, we perform a comprehensive empirical study on the applicability of state-of-the-art text spotting methods for the VPR task. Additionally, we introduce a challenging purpose-built urban dataset for VPR evaluation. The proposed VPR architecture achieves a superior performance overall, especially in challenging conditions (i.e., perceptually aliased and illuminated environments).

Dulmini Hettiarachchi

Ye Tian

Han Yu

Shunsuke Kamijo

Text Spotting towards Perceptually Aliased Urban Place Recognition

Chemical phenomena are only observable on a macroscopic level, whereas they are explained by entities on a non-visible level. Students often demonstrate limited ability to link these different levels. Augmented reality (AR) offers the possibility to increase contiguity by embedding virtual models into hands-on experiments. Therefore, this paper presents a pre- and post-test study investigating how learning and cognitive load are influenced by AR during hands-on experiments. Three comparison groups (AR, animation and filmstrip), with a total of N = 104 German secondary school students, conducted and explained two hands-on experiments. Whereas the AR group was allowed to use an AR app showing virtual models of the processes on the submicroscopic level during the experiments, the two other groups were provided with the same dynamic or static models after experimenting. Results indicate no significant learning gain for the AR group in contrast to the two other groups. The perceived intrinsic cognitive load was higher for the AR group in both experiments as well as the extraneous load in the second experiment. It can be concluded that AR could not unleash its theoretically derived potential in the present study.

Hendrik Peeters

Sebastian Habig

Sabine Fechner

Does Augmented Reality Help to Understand Chemical Phenomena during Hands-On Experiments?–Implications for Cognitive Load and Learning

增强现实是否有助于理解动手实验中的化学现象？——对认知负荷和学习的影响

Trust plays an essential role in all human relationships. However, measuring trust remains a challenge for researchers exploring psychophysiological signals. Therefore, this article aims to systematically map the approaches used in studies assessing trust with psychophysiological signals. In particular, we examine the numbers and frequency of combined psychophysiological signals, the primary outcomes of previous studies, and the types and most commonly used data analysis techniques for analyzing psychophysiological data to infer a trust state. For this purpose, we employ a systematic mapping review method, through which we analyze 51 carefully selected articles (studies focused on trust using psychophysiology). Two significant findings are as follows: (1) Psychophysiological signals from EEG(electroencephalogram) and ECG(electrocardiogram) for monitoring peripheral and central nervous systems are the most frequently used to measure trust, while audio and EOG(electro-oculography) psychophysiological signals are the least commonly used. Moreover, the maximum number of psychophysiological signals ever combined so far is three (2). Most of which are peripheral nervous system monitoring psychophysiological signals that are low in spatial resolution. (3) Regarding outcomes: there is only one tool proposed for assessing trust in an interpersonal context, excluding trust in a technology context. Moreover, there are no stable and accurate ensemble models that have been developed to assess trust; all prior attempts led to unstable but fairly accurate models or did not satisfy the conditions for combining several algorithms (ensemble). In conclusion, the extent to which trust can be assessed using psychophysiological measures during user interactions (real-time) remains unknown, as there several issues, such as the lack of a stable and accurate ensemble trust classifier model, among others, that require urgent research attention. Although this topic is relatively new, much work has been done. However, more remains to be done to provide clarity on this topic.

Ighoyota Ben. Ajenaghughrure

Sonia Da Costa Sousa

David Lamas

Measuring Trust with Psychophysiological Signals: A Systematic Mapping Study of Approaches Used

使用心理生理信号测量信任：采用方法的系统映射研究

Bi-directional gaze-based communication offers an intuitive and natural way for users to interact with systems. This approach utilizes the user’s gaze not only to communicate intent but also to obtain feedback, which promotes mutual understanding and trust between the user and the system. In this review, we explore the state of the art in gaze-based communication, focusing on both directions: From user to system and from system to user. First, we examine how eye-tracking data is processed and utilized for communication from the user to the system. This includes a range of techniques for gaze-based interaction and the critical role of intent prediction, which enhances the system’s ability to anticipate the user’s needs. Next, we analyze the reverse pathway—how systems provide feedback to users via various channels, highlighting their advantages and limitations. Finally, we discuss the potential integration of these two communication streams, paving the way for more intuitive and efficient gaze-based interaction models, especially in the context of Artificial Intelligence. Our overview emphasizes the future prospects for combining these approaches to create seamless, trust-building communication between users and systems. Ensuring that these systems are designed with a focus on usability and accessibility will be critical to making them effective communication tools for a wide range of users.

Björn Rene Severitt

Nora Castner

Siegfried Wahl

Bi-Directional Gaze-Based Communication: A Review

基于双向凝视的交流：综述

Collaborative decision-making increasingly involves wall-sized displays (WSDs), allowing teams to view, analyse and discuss large amounts of data. To enhance workspace awareness for mixed-presence meetings, previous work proposes digital cues to share gestures, gaze, or entire postures. While several isolated cues were proposed and demonstrated useful in different workspaces, it is unknown whether results from previous studies can be transferred to a mixed-presence WSD context and to what extend such cues can be used in a combined way. In this paper, we report on the results from a user study with 24 participants (six groups of four participants), testing a mixed-presence collaboration scenario on two different setups of connected WSDs: audio-video link only vs. full setup with seven complementary cues. Our results show that the version with cues enhances workspace awareness, user experience, team orientation and coordination, and leads teams to take more correct decisions.

Valérie Maquil

Adrien Coppens

Lou Schwartz

Dimitra Anastasiou

Mixed-Presence Collaboration with Wall-Sized Displays: Empirical Findings on the Benefits of Awareness Cues

大型显示墙混合在场协作：关于意识线索益处的实证发现

Despite the increasing sophistication of autonomous vehicles (AVs) and promises of increased safety, accidents will occur. These will corrode public trust and negatively impact user acceptance, adoption and continued use. It is imperative to explore methods that can potentially reduce this impact. The aim of the current paper is to investigate the efficacy of informational assistants (IAs) varying by anthropomorphism (humanoid robot vs. no robot) and dialogue style (conversational vs. informational) on trust in and blame on a highly autonomous vehicle in the event of an accident. The accident scenario involved a pedestrian violating the Highway Code by stepping out in front of a parked bus and the AV not being able to stop in time during an overtake manoeuvre. The humanoid (Nao) robot IA did not improve trust (across three measures) or reduce blame on the AV in Experiment 1, although communicated intentions and actions were perceived by some as being assertive and risky. Reducing assertiveness in Experiment 2 resulted in higher trust (on one measure) in the robot condition, especially with the conversational dialogue style. However, there were again no effects on blame. In Experiment 3, participants had multiple experiences of the AV negotiating parked buses without negative outcomes. Trust significantly increased across each event, although it plummeted following the accident with no differences due to anthropomorphism or dialogue style. The perceived capabilities of the AV and IA before the critical accident event may have had a counterintuitive effect. Overall, evidence was found for a few benefits and many pitfalls of anthropomorphising an AV with a humanoid robot IA in the event of an accident situation.

Christopher D. Wallbridge

Qiyuan Zhang

Victoria E. K. Marcinkiewicz

Louise Bowen

Theodor Kozlowski

Dylan M. Jones

Phillip L. Morgan

“Warning!” Benefits and Pitfalls of Anthropomorphising Autonomous Vehicle Informational Assistants in the Case of an Accident

“警告！” 在事故案例中拟人化自动驾驶车辆信息助手的好处与陷阱

When discussing the Education 4.0 concept and the role of technology-based learning systems along with creativity, it is interesting to explore how these are reflected as educational innovations in the case of design education. This study aims to provide an overview of interactive technologies used in product design education and examine their integration into the learning process. A literature search was conducted, analyzing scientific papers to review relevant articles. The findings highlight several categories of technologies utilized in design education, including virtual and augmented reality, robotics, interactive embedded systems, immersive technologies, and computational intelligence systems. These technologies are primarily integrated as supportive tools throughout different stages of the design process within learning environments. This study suggests that integrating such technologies alongside pedagogical methods positively impacts education, offering numerous opportunities for further research and innovation. In conclusion, this review contributes to ongoing research in technological advancements and innovations in design education, offering insights into the diverse applications of interactive technologies in enhancing learning environments.

Ioanna Nazlidou

Nikolaos Efkolidis

Konstantinos Kakoulis

Panagiotis Kyratsis

Innovative and Interactive Technologies in Creative Product Design Education: A Review

创意产品设计教育中的创新与互动技术：综述

This study aims to explore the evolution of the literature on the sociological implications of integrating artificial intelligence (AI) as an educational tool, particularly its influence on digital inequalities. While AI technologies, such as AI-based language models, have begun transforming educational practices by personalizing learning, fostering student autonomy, and supporting educators, concerns remain regarding access disparities, ethical implications, and the potential reinforcement of existing social inequalities. To address these issues, a bibliometric analysis employing science mapping was conducted on 1515 studies sourced from the Web of Science Core Collection. This analysis traces the thematic evolution of social science perspectives on AI’s role in education and its relationship with digital inequalities. The results indicate a growing academic interest in AI in education, with a notable progression from understanding its basic impact to exploring complex themes such as vulnerability, disability, bias, and community. The studies show that AI’s application has expanded from isolated research on specific populations to broader discussions on inclusivity, equity, and the impact of AI on governance, policy, and community. However, the findings also reveal a significant gap in sociological perspectives, particularly regarding issues like digital illiteracy and socio-economic access disparities. Although AI holds promise for promoting more inclusive education, further research is essential to address these sociological concerns and to guide the ethical, equitable implementation of AI as its influence on governance, policy, and community impact continues to grow.

Isotta Mac Fadden

Elena-María García-Alonso

Eloy López Meneses

Science Mapping of AI as an Educational Tool Exploring Digital Inequalities: A Sociological Perspective

教育工具人工智能的科学图谱：探索数字不平等的社会学视角

Abbreviation	MTI
Journal Impact	2.45
Quartiles(Global)	COMPUTER SCIENCE, INFORMATION SYSTEMS(Q3)

ISSN	2414-4088
h-index	32

Publication Information	Publisher: Multidisciplinary Digital Publishing Institute (MDPI)，Publishing cycle: ，Journal Type: journal，Open Access Journals: Yes
Basic data	Year of publication: 2017，Proportion of original research papers: 96.40%，Self Citation Rate:4.20%， Gold OA Rate: 100.00%
Journal Fees	$Details
Average review cycle	网友分享经验：4 Weeks

Multimodal Technologies and Interaction

Journal Citation Format

A journal article with 1 author

A journal article with 2 authors

A journal article with 3 authors

A journal article with 5 or more authors

Books Citation Format

Thesis Citation Format

Web sites Citation Format

Patent Citation Format

Staying up late manually editing references? ivySCI automatically matches journals and helps you generate references with a single click.

Share Submission Experience

Linking Personality and Trust in Intelligent Virtual Assistants

Text Spotting towards Perceptually Aliased Urban Place Recognition

Does Augmented Reality Help to Understand Chemical Phenomena during Hands-On Experiments?–Implications for Cognitive Load and Learning

Measuring Trust with Psychophysiological Signals: A Systematic Mapping Study of Approaches Used

Bi-Directional Gaze-Based Communication: A Review

Mixed-Presence Collaboration with Wall-Sized Displays: Empirical Findings on the Benefits of Awareness Cues

“Warning!” Benefits and Pitfalls of Anthropomorphising Autonomous Vehicle Informational Assistants in the Case of an Accident

Bi-Directional Gaze-Based Communication: A Review

Innovative and Interactive Technologies in Creative Product Design Education: A Review

Science Mapping of AI as an Educational Tool Exploring Digital Inequalities: A Sociological Perspective