UC Davis

Having an intelligent assistant that can communicate with humans to serve their needs is a fundamental challenge in Artificial Intelligence (AI) research. Recently, owing to the development of deep learning techniques and the large-scale datasets, we have witnessed a great advancement in dialogue systems. Nowadays, conversational agents have been deployed in millions of smart devices such as Alexa, Google home assistant, and Smartphones (e.g. Siri) to serve as personal assistants or chat companions for human users. Although tremendous success has been achieved, there are still major limitations. The majority of current dialogue systems can only process and communicate with language context, which limits their application to conversational tasks that require situational understanding such as language-guided visual navigation or fashion shopping assistant. Additionally, while there are more than 6500 different languages used in our world, the dialogue systems are mainly studied on English. In order to broaden the access of such AI techniques to non-English speakers, it is essential to build conversational AI agents that can communicate in multiple languages. To address these limitations, we aim to build multi-lingual multi-modal dialogue systems that learn to process context from multi-modal signals (vision and language) and communicate in various languages via interacting with real users. In this dissertation, we introduce our effort to approach this goal in two different research directions:1. Ground Vision and Action: we build multi-modal dialogue systems that can ground conversations in a visual environment and adopt optimal actions to improve task success. we also collect a new benchmark that helps the dialogue system to learn cross-modal grounding via simultaneously handling vision generation from textual context and text generation from visual context in a unified conversational task. 2. Cross-lingual Cross-modal Representation Learning: To enable dialogue systems to become multi-lingual speakers, we conduct research to align the vision and various languages in a learned semantic space. Specifically, we research multi-modal machine translation and cross-lingual cross-modal pre-training techniques to learn joint representations across languages and modalities. we have also introduced how to learn robust universal cross-modal representation without parallel image-text pairs. Below we give an overview of our past research as the initial exploration to build multi-lingual multi-modal dialogue systems via interactive learning.