Framework and Algorithms for Multi-view Video Synopsis

Abstract

Summary or synopsis production of videos, shot with a single static camera, has been well studied in last one decade. It has numerous applications, especially in surveillance business. With the advent of multi-camera networks (MCN), newer challenges have surfaced before the research community and synopsis generation is one of them. It may be noted that, in MCN, a scene is recorded from multiple angles i.e. the network of cameras records multi-view videos. Adaptation of single video synopsis generation methods to each view of MCN would not only lead to redundancy but also make the comprehension of synopses cumbersome. Besides, the background of each camera view is different making it difficult to bring all views under a single view. Furthermore, the coherence among the multiple views is another issue that demands special attention to generate a single synopsis.

In this doctoral research, the focus is made on developing a framework that generates a single synopsis of multi-view videos. Alongside the framework, various methods are proposed that help in synopsis generation. The methods are grouped in three categories; pre-processing, synopsis generation, and post-processing. Some of the methods in pre-processing are adapted from existing literature while the rest are proposed.

The framework uses the top view of the surveillance site as the common ground plane, wherein objects detected from different views are mapped through homographic technique. The mapped object locations are clustered, spatially followed by temporally, adapting density based clustering algorithm to form the track of each object. An action recognition module is also used in the framework to recognize the objects’ action and prioritize them so that the objects performing important actions can be included in the synopsis leading omission of trivial content and reduction in synopsis length. Two more methods are suggested in the framework; interaction detection method makes the generated synopsis more rich in information, and collision detection method helps in excluding colliding tracks in the generated synopsis.

The generation of multi-view video synopsis is modelled as a scheduling problem. Two sets of solutions are suggested in the research; deterministic and non-deterministic. Under the deterministic category, four approaches are proposed. A table driven method has been proposed to schedule object tracks with zero collision by carefully selecting objects. A contradictory graph coloring based approach has been proposed that allows a small number of collision in the synopsis to reduce its length. A greedy based object scheduling method has also been proposed for scheduling more number of objects per schedule. Lastly, a dynamic programming based scheduling algorithm is proposed that considers both the number of collisions and action performed by the object to generate a synopsis. Under the non-deterministic head, the synopsis generation is modelled as a multi-objective optimization problem that takes into account components like synopsis length, number of collisions, actions, and interactions performed by the objects. Simulated Annealing (SA) and Genetic Algorithm (GA) are used to optimize the cost function.

A fuzzy based post-processing method is also proposed that further reduces the synopsis length by computing the visibility scores of each object track. The visualization of the generated synopsis is achieved by presenting the objects on top of the common ground plane. The proposed framework reveals its efficacy when tested on different datasets and compared with the state-of-the-art.