Project: NeTra-V -  Toward an Object-based Video Representation

People

Objective

We present here a prototype video analysis and retrieval system, called NeTra-V, that is being developed to build an object-based video representation for functionalities such as search and retrieval of video objects. A region-based content description scheme using low-level visual descriptors is proposed. In order to obtain regions for local feature extraction, a new spatio-temporal segmentation and region tracking scheme is employed. The segmentation algorithm uses all three visual features, color, texture, and motion in the video data. A group processing scheme similar to the one in the MPEG-2 standard is used to ensure the robustness of the segmentation. The proposed approach can handle complex scenes with large motion. After segmentation, regions are tracked through the video sequence using extracted local features. The results of tracking are sequences of coherent regions, called "sub-objects". Sub-objects are the fundamental elements in our low-level content description scheme, which can be used to obtain meaningful physical objects in a high-level content description scheme.

Examples

Publications

These materials are presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each authors copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

1. Y. Deng, D. Mukherjee, and B. S. Manjunath, "Netra-V: Towards an object based video representation," Proc. SPIE: Storage and Retrieval for Image and Video Databases, vol. 3312, pp. 202-213, San Jose, CA, Jan 1998.[abstract]

2. Y. Deng and B. S. Manjunath, "Spatio-temporal relationships and video object extraction," Proc. 32nd Asilomar Conference on Signals, Systems and Computers, Asilomar, California, Nov 1998. [abstract]