Activities

Group3: Human level knowledge and concept acquisition

Latest Activity

Concept learning and multimodal recognition (Tatsuya HARADA)

To freely manipulate cybernetic avatars (CAs) in an environment where CAs coexist with people, CAs must have the ability to understand people's daily living environment, recognize objects, and understand common sense. Therefore, in this project, we will realize the real-world recognition functions of CAs and develop methods to acquire the knowledge and concepts essential for these functions. To achieve these advanced recognition functions, we will invent techniques to fuse information from the natural environment through vision and audio with semantics information described in natural language. We also aim to build methods for acquiring knowledge and concepts by fusing various modalities, utilizing a large amount of textual information on the Internet and other sources, and realizing communication technologies between humans and CAs. Furthermore, we aim to boost the ability of recognition using the acquired knowledge and concepts.

Lab website: https://www.mi.t.u-tokyo.ac.jp/
Corpus development for semantic understanding (Yusuke KUROSE)

Cybernetic avatars (CAs) are expected to have a variety of applications, and this project targets medical applications of CAs. Doctors make diagnoses through conversations obtained during counseling and medical interviews. However, counseling and medical interviews are time-consuming, and it would greatly reduce the burden on doctors if it were possible to perform them through CA. Therefore, this project aims to develop a system in which knowledge and concepts through CA can be used at a practical level to support doctors, and to develop a system for semantic understanding using multiple modalities of information, including visual information such as the patient's facial expression and behavior, audio information of the patient during medical interviews, and natural language information such as the content of conversations. In addition, the current deep learning requires a large amount of data, but it is often difficult to use such data in the medical field, and it is even more difficult to assign labels to all such data as supervised information due to the requirement of highly specialized knowledge of medical data. Therefore, in this project, we will also work on the development of an efficient learning method from a small amount of data or a small amount of annotated data, even in situations where such a large amount of data cannot be obtained.

Continuous Learning and Memory Mechanism (Lin Gu)

While the actual environment is constantly changing, the relation between user and cybernetic agent (CA) is also changing. To reflect the changing knowledge and concept that is necessary for CA to understand user’s real motivation, continuous learning is a necessary step. To achieve this step, we have conducted research on foundation of short-term and long-term memory as well as direct support of Nature Language Processing (NLP) and medical application. Now we are designing the artificial neuron representation, which holds the human-like perceptual constancy. We are also conducting research on knowledge graph based long-term memory. By continuously learning from the changing environment, the CA is expected to continuously improve its performance.

Research on time series prediction and estimation of the causality measure (Yusuke MUKUTA)

The function of causal inference is essential for CAs to understand events in the environment and to acquire knowledge and concepts about the environment while clarifying the relationship between the environment and CAs. In this R&D project, the function of causal inference between objects and events based on observations in the environment is realized so that CAs can share knowledge and concepts with humans who are active in the same environment. For this purpose, we propose a mathematical method for time series analysis to effectively extract information from miscellaneous series of observations in the environment, and an action planning method for CA to efficiently search in the environment and obtain information about the environment. Furthermore, we will realize a function to predict the actual future based on the inferred relationships, so that CA can work efficiently based on the predicted future. To date, we have proposed models for efficient time-series forecasting even with small sample sizes by utilizing symmetry, Bayesian estimation, and generative modeling techniques. In the future, we will build even higher-performance forecasting models as well as search methods to efficiently gather causal information from the real world.

Natural language processing (Jun SUZUKI)

To freely operate CA and to communicate with users autonomously via natural language, it is necessary for CA to equip advanced language understanding capabilities. To achieve this demand, this research project tackles to realize a natural language understanding technology suited for the CA use environment, and to realize a comprehensive real-world understanding technology that combines this with visual information. This project also tackles multilingualization of CA to extend the usability of CAs and to overcome transcends language barriers.