[et_pb_section fb_built=”1″ custom_padding_last_edited=”off|desktop” admin_label=”Hero” _builder_version=”4.24.2″ background_enable_color=”off” custom_margin=”|||” custom_padding=”50px||0px|||” custom_padding_tablet=”130px||130px|” global_colors_info=”{}”][et_pb_row _builder_version=”4.27.4″ _module_preset=”default” width=”100%” max_width=”1280px” custom_padding=”|15px|3px|15px|false|false” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.24.2″ _module_preset=”default” global_colors_info=”{}”][et_pb_text _builder_version=”4.24.2″ _module_preset=”default” header_font=”|||on|||||” custom_margin=”||1px|||” global_colors_info=”{}”]<\/p>\n
[\/et_pb_text][et_pb_divider color=”#0c71c3″ divider_weight=”3px” _builder_version=”4.27.4″ _module_preset=”default” width=”50%” module_alignment=”left” global_colors_info=”{}”][\/et_pb_divider][\/et_pb_column][\/et_pb_row][et_pb_row _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_team_member name=”Professor Jo\u00e3o Magalh\u00e3es” position=”Full Professor at the Computer Science Dep. at Universidade NOVA de Lisboa and national co-Director of the CMU Portugal partnership.” image_url=”https:\/\/iberspeech.tech\/2026\/wp-content\/uploads\/2026\/04\/Joao-Magalhaes-profile.jpeg” _builder_version=”4.27.4″ _module_preset=”default” header_font_size=”28px” position_font_size=”18px” custom_margin=”0px||||false|false” hover_enabled=”0″ global_colors_info=”{}” sticky_enabled=”0″]<\/p>\n
<\/span><\/p>\n Jo\u00e3o Magalh\u00e3es holds a Ph.D. degree (2008) from Imperial College London, UK. His research aims to move vision and language AI closer to the way humans understand it and communicate. He has made scientific contributions to the fields of multimedia search and summarization, multimodal conversational AI, data mining and multimodal information representation. He is currently coordinating the creation of the sovereign LLM AMALIA, and, in the past, has coordinated and participated in several research projects (national, EU-FP7 and H2020) where he pursues robust and generalizable methods in different domains. He is regularly involved in review panels, organization of international conferences and program committees. His work and the work of his group has been awarded, or nominated for, several honours and distinctions, most notably the 1st prize in the Amazon Alexa Taskbot Challenge 2022. He was the General Chair of ECIR 2020 and ACM Multimedia 2022, Honorary Chair for ACM Multimedia Asia 2021 and will be the PC chair of ACM Multimedia 2026.<\/span><\/p>\n [\/et_pb_team_member][\/et_pb_column][\/et_pb_row][et_pb_row _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”][et_pb_toggle title=”Title of the talk: %22Multimodal Conversational Assistance of Complex Manual Tasks%22″ open=”on” toggle_icon=”C||divi||400″ _builder_version=”4.27.4″ _module_preset=”default” title_level=”h3″ title_font=”|||||on|||” body_font_size=”15px” custom_margin=”||30px|||” hover_enabled=”0″ global_colors_info=”{}” sticky_enabled=”0″]<\/p>\n Conversational agents have become an integral part of our daily routines, aiding humans in various tasks. Helping users in real-world manual tasks is a complex and challenging paradigm, where it is necessary to leverage multiple information sources, provide several multimodal stimuli, and be able to correctly ground the conversation in a helpful and robust manner. In this talk I will describe TWIZ, a conversational AI assistant that is helpful, multimodal, knowledgeable, and engaging, and designed to guide users towards the successful completion of complex manual tasks. To achieve this, we focused our efforts on three main research questions: (1) Humanly-Shaped Conversations, by providing information in a knowledgeable way; (2) Multimodal Stimulus, making use of various modalities including voice, images, and videos; and (3) Zero-shot Conversational Flows, to improve the robustness of the interaction to unseen scenarios. TWIZ is an assistant capable of supporting a wide range of unseen tasks — it leverages Generative AI methods to deliver several innovative features such as creative cooking, video navigation through voice, and the robust PlanLLM, a Large Language Model trained for dialoguing about complex manual tasks.<\/span><\/p>\n [\/et_pb_toggle][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section fb_built=”1″ _builder_version=”4.16″ _module_preset=”%22default%22″ custom_padding=”%220px||2px|||%22″ global_colors_info=”%22{}%22″][et_pb_row _builder_version=”4.27.4″ _module_preset=”default”][et_pb_column _builder_version=”4.27.4″ _module_preset=”default” type=”4_4″][et_pb_team_member name=”Prof. Dr. Elisabeth Andr\u00e9” position=”Full Professor of Computer Science, Chair of Human-Centered Artificial Intelligence, Faculty of Applied Informatics, Augsburg University” image_url=”https:\/\/iberspeech.tech\/2026\/wp-content\/uploads\/2026\/05\/elisabeth-small.png” _builder_version=”4.27.4″ _module_preset=”default” header_font_size=”28px” position_font_size=”18px” custom_margin=”0px||||false|false” hover_enabled=”0″ global_colors_info=”{}” sticky_enabled=”0″ title_text=”elisabeth-small”]<\/p>\n <\/span><\/p>\n Elisabeth Andr\u00e9 is a Full Professor of Computer Science and the Founding Chair of Human-Centered Artificial Intelligence at Augsburg University, Germany. A global leader in multimodal human-machine interaction and social signal processing, her work has been foundational in enabling machines to perceive and respond to human speech, gestures, and emotions in a natural, socially intelligent manner. Her research is dedicated to the development of “believable” virtual agents and social robots capable of sophisticated, human-like dialogue. For her pioneering contributions to artificial emotional intelligence, she was awarded the Gottfried Wilhelm Leibniz Prize by the German Research Foundation (DFG). As Germany\u2019s most prestigious research honor, the prize recognizes her trailblazing role in bridging the gap between Artificial Intelligence and Human-Computer Interaction to create technology that is more intuitive and empathetic. Beyond her technical research, Professor Andr\u00e9 is recognized as one of the most influential voices in the field. In 2024, Manager Magazin named her one of the 15 most important women in AI in Germany, and in 2019, she was honored by the National Society for Informatics (GI) as one of the ten most influential figures in the history of German AI. Her long-standing impact on the community was further recognized with the ICMI Sustained Accomplishment Award and the 2025 AI Visionary TIGER Award. Professor Andr\u00e9 is a member of the German Academy of Sciences (Leopoldina), the CHI Academy, and a EurAI Fellow. She currently co-leads the “Work\/Qualification and Human-Machine Interaction” group for Germany’s National Platform for Artificial Intelligence.<\/span><\/p>\n [\/et_pb_team_member][et_pb_toggle title=”Title of the talk: %22From Speech to Multimodal Interaction: Guardrails for Socially-Interactive AI%22″ open=”on” toggle_icon=”C||divi||400″ _builder_version=”4.27.4″ _module_preset=”default” title_level=”h3″ title_font=”|||||on|||” body_font_size=”15px” custom_margin=”||30px|||” hover_enabled=”0″ global_colors_info=”{}” sticky_enabled=”0″]<\/p>\n When speech moves from transcription to multimodal interaction, its requirements change fundamentally. In such settings, systems must operate under ambiguity and uncertainty while accounting for social context and application-specific constraints. This talk focuses on the design of guardrails that go beyond generic content filtering. I will present approaches for risk identification, evaluation, and mitigation, including mechanisms for uncertainty handling, policy compliance, and improving the reliability of LLM-based systems in multimodal interaction. These challenges and solutions are illustrated through three complementary perspectives from interdisciplinary projects: social coaching through role-play with robots (CAIDA) and virtual agents for children (CONFIDENCE), where particular care must be taken to avoid stigmatization of marginalized groups and psychologically harmful interaction scenarios; language-enabled robotics in healthcare (REGINA), where systems must ensure confidentiality, correctness, and certifiability while remaining accessible through natural language and low-code\/no-code interfaces; and CAR-bench, a benchmark for evaluating consistency, uncertainty awareness, and capability awareness in multi-turn, tool-using LLM agents for in-car assistant scenarios.<\/span><\/p>\n [\/et_pb_toggle][\/et_pb_column][\/et_pb_row][\/et_pb_section]<\/p>\n","protected":false},"excerpt":{"rendered":" KEYNOTE SPEAKERS Jo\u00e3o Magalh\u00e3es holds a Ph.D. degree (2008) from Imperial College London, UK. His research aims to move vision and language AI closer to the way humans understand it and communicate. He has made scientific contributions to the fields of multimedia search and summarization, multimodal conversational AI, data mining and multimodal information representation. He […]<\/p>\n","protected":false},"author":4,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_et_pb_use_builder":"on","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":"","_links_to":"","_links_to_target":""},"class_list":["post-778","page","type-page","status-publish","hentry"],"yoast_head":"\nAbstract<\/span><\/h4>\n
Abstract<\/span><\/h4>\n