Multimedia Sensing Laboratory

With increasing integration of computing, communication, networking and the world-wide web, multimedia information has become pervasive and permeates many aspects of our lives. Increasingly, we find information coming from multiple sources and in different media forms, such as text, image, audio, video and novel sensory forms. Managing live data (be it symbolic text-feed or signal sensor-feed) is particularly challenging in this environment. For example, the ability to analyse and fuse information from different sources and in multiple media types has become critically important in many information processing tasks.

The multimedia sensing laboratory is organized such that it comprises researchers who look at handling of correlated multimedia information (live as well as archived) with a rigorous information-centric approach. Overall, the group conducts research related to the processing, interaction, storage and retrieval of multimedia sensor information. The group will focus on fundamental research with twin emphasis on theoretical and systems research with a long-term aim of creating real-world impact.

This project aims to develop the theoretical foundation, algorithms, architectures and prototypes for active sensing for a wide variety of applications such as surveillance, monitoring and the web. It requires the development of innovative techniques in active collaborative sensing involving: multiple sensors in multiple media; mobile sensors; devices; and biological systems. The idea here is to harness the power of diverse sensors in an orchestrated manner to optimally perform the given tasks. The areas of focus will include multimedia monitoring, mobile sensing, active devices and biologically plausible modeling.

Humans tend to organize their lives around events. Therefore, devices to capture events easily for archiving and sharing them are essential. Such devices will utilize different sensors to capture specific types of events and related experiential information effortlessly for use by people who are not technologists. Technology to capture specific type of events from multimodal information will then need to be developed. This is a complex area. The success will depend on developing specific event detection techniques by using all appropriate sensors and human input as much as required in early stages.

The focus will not be on fully automatic techniques from specific medium. This project will focus on specific event detection using multi-modal information with human assistance, as maybe suitable in specific contexts.

There is a need to building infrastructure for handling multimedia sensing data which can allow for seamless access of live as well as archived data. Since this is a dynamic and evolving infrastructure, the framework should have an internet-style plug and play philosophy. For example, each sensor or other data source is registered with the information source registry (ISR). This gives the location and viewpoint for the sensors and characteristics of other information sources.

When a new sensor is added to the system, it is entered in the registry and its key parameters for the transformation are also entered. Each sensor captures some physical attributes of the environment being observed. The nature of sensor data and its mapping to the environment should be part of the information source transformer that is available to the system. All data from each sensor is stored in the media storage. Though each sensor stores its data in terms of the sensor coordinate system in the storage, each sensor's relative offset with respect to the physical environment are available and stored in the ISR.

All assimilated information from different sensors is in an application independent manner - it should be as objective as possible. Application oriented views and functions could be defined on top of the objective system. These views could be computed and separate databases created for each application view. Thus, using the same data, one could have a surveillance view, or be enjoying an art gallery, or a video ethnography view.

All these views could be computed from the objective database using different event models and related attributes. The ultimate aim is to build flexible yet scalable middleware to handle multimdia sensing information which can be seamlessly accessed by the applications built on top of the middleware.

Keyword based navigation on the current WWW becomes very tedious in practice. There is a need to develop completely new approaches to combine navigation and search environments for finding appropriate events (which are captured using multimedia sensors).

Events are much better captured and represented using photos and video than text and hence will require novel presentation environment. Such presentation environment will require a rich combination of ideas from visual arts and human-computer interaction to present event experiences to people. When there are a huge number of events - past, current, and future - how do we find events of interest? All events should be somehow aggregated and an environment developed in which events of interests could be discovered. For accessing all these events, powerful and efficient indexing approaches must be developed.

Events may not be indexable using techniques employed in current search engines and relational databases. We may require a combination of multi-dimensional and inverted file approaches. It would be interesting to explore these ideas in a particular domain like education.