Course Overview
Thanks to rapid advances in digital sensors, large-scale data collection, and foundation models, we can now generate, process, and understand not only images and videos, but also text, audio, and other modalities at unprecedented scale. As a result, massive multimodal datasets are continuously created across the web, industry, and personal devices. For example, modern platforms host billions of multimodal items, with new content being generated and updated in real time.
These large-scale multimodal databases pose fundamental technical challenges, including representation learning, retrieval, reasoning, and efficient interaction between perception and language. Beyond classical image search, modern systems must retrieve and reason over heterogeneous data, align vision with language, and support downstream decision-making and generation.
In this course, we study scalable techniques for deep multimodal retrieval and their extension toward agentic AI systems. Topics include multimodal foundation models, advanced retrieval and Retrieval-Augmented Generation (RAG), and language models acting as agents that can plan, reason, and interact with retrieved knowledge. The course emphasizes both core algorithms and emerging applications built on web-scale multimodal data.
In summary, what you will gain by the end of the course:
- A broad understanding of multimodal retrieval across vision, language, and other modalities
- In-depth knowledge of modern retrieval and RAG techniques for web-scale data
- An understanding of how language models function as agents that leverage retrieval for reasoning and decision-making
- Exposure to emerging applications built on large-scale multimodal and agentic systems
What you will do:
- Choose and present a few papers from recent conferences.
- Final project: come up with your own idea related to the topic, (optionally) implement it to improve the state-of-the-art techniques.
- Mid-term exam: reviewing basic multi-model search and RAG methods.