Goal
Gaining an understanding of multimedia retrieval techniques, tools, systems and applications with an emphasis on image, video and audio.

Prerequisites
Basic knowledge of digital image or signal processing and machine learning.

Location
University of Amsterdam, Science park 904, 1098 XH Amsterdam

Course contents
With all media becoming digital, the need arises to be able to retrieve these media objects with the same ease as we are used to when accessing text or factual data. Multimedia retrieval tools for your private photo collection obtained with your new digital photo camera and uploaded onto Flickr, your personalized video broadcasts, news archives, forensic image and video collections, and all multimedia objects accessible through the web are urged for. Although it might seem that multimedia retrieval is the trivial extension of text retrieval it is in fact far more difficult. Most of the data is of sensory origin (image, sound, video) and hence techniques from digital signal processing and computer vision are required to extract relevant metadata. Such techniques in general yield syntactic features and do not relate directly to the user's perception of the data, a problem known as the semantic gap.
This course aims at understanding the characteristics of multimedia data and multimedia user queries and the impact it has on the design and development of multimedia retrieval systems. Techniques and tools will be presented ranging from low-level data analysis and automatic semantic indexing to complete interactive systems.

Lecturers
Prof. dr. Theo Gevers (University of Amsterdam)
Dr. Alan Hanjalic (Delft University of Technology)
Dr. Cees Snoek (University of Amsterdam)
Dr. Anja Volk (University Utrecht)
Dr. Marcel Worring (University of Amsterdam)

Contact
Marcel Worring
m.worring(at)uva.nl
Phone: +31 20 525 7521

Examination
A number of exercises will be presented during the course, and software tools for doing multimedia retrieval experiments will be provided. Participants can work on these problems during the lab courses. On the basis of this a report has to be written which has to be submitted within 2 weeks after the course. Evaluation will take place within 4 weeks after the course.

The report is composed of two parts

• Part 1: These are the answers to the 5 lab courses as they have to be performed each afternoon. The format and scope of these are different and are dependent on the lecturer.
• part 2: You should write a four page IEEE/ACM conference style paper (including references) in which you describe your own research (max 1 page) and then elaborate on how the techniques in the course could be helpful in your current work or how they could extend your work into new avenues. We expect you to be creative here. So don't consider only the techniques which are directly applicable, but try to extend your scope. So if e.g. you are working in video retrieval don't focus on the techniques of the second day, but look at audio retrieval techniques, social media and multimedia analytics.

All documents can be send to Marcel Worring (in .pdf format). Please have separate documents for each practical course for which documents have to be delivered and for the document describing your own work.

Multimedia Retrieval course Program

The overall schedule each day is approximately as follows:

• 9.15 start of the lecture
• 10.45 coffee break
• 12.30 lunch
• 13.30 practical course
• 15.00 tea break
• 16.30 finish

Day 1

• Marcel Worring: Introduction (Room D1.112)

  In this introductory lecture an overview will be given of the course and the multimedia retrieval field in general.

• Theo Gevers: Image Retrieval (Room D1.112) ( sheets )

  This lecture and practical work will focus on retrieving images, where a key component is formed by the notion of invariance. Being able to find back an image of a scene or object recorded under different circumstances critically depends on this notion of variance and different color spaces with varying invariance properties. The key challenge is to develop features which are invariant under the varying conditions, but which still have sufficient discriminatory power.

• Practical work on image retrieval (Room G0.10) ( assignment )

Day 2

• Cees Snoek: Video Retrieval (Room A1.14) ( sheets )

  Where the lecture on Image Retrieval has its focus on static low-level feature representations, we move our attention to video search using high-level concept detectors and dynamic events, as well as large-scale benchmark evaluations.

• Practical work on video retrieval (Room A1.20-1.22) ( assignment )

Day 3

• Alan Hanjalic and Cees Snoek: Social Media Retrieval (Room A1.14) ( sheets 1 sheets 2 )

  Images and video play an important in social media sites, and huge collections of these are available online. To search for social media requires true multimedia techniques as visual material cannot be considered without the associated tags or user information. Next to being interest material to search through, social media can also be leveraged to learn semantic detectors for visual material without actually having to label data for training. Both these aspects of social media will be considered.

• Practical work on social media retrieval (Room A1.16b) ( assignment )

Day 4

• Anja Volk: Music Retrieval (Room G0.10) ( sheets )

  Retrieving music requires not only to index the data with low-level representation, but also requires an understanding of how music is structured. Various techniques for music retrieval will be discussed here.

• Practical work on music retrieval (Room D1.111) ( assignment )

Day 5

• Marcel Worring: Multimedia Analytics (Room D1.115) ( sheets )

  When datasets become large and complex neither fully automatic nor fully manual approaches are suited. A combination of human and machine is needed which is best achieved by interactive visualizations which based on the results of automatic analysis. In this lecture we will consider the integration of Multimedia Analysis and Visual Analytics and see how it can help in building effective solutions for multimedia retrieval.

• Practical work on Multimedia Analytics (Room G.023) ( assignment )

• Marcel Worring: Closing with drinks