Object Tracking in Audio-Visual Scene

Abstract

The thesis aims at tracking of an object using visual and audio information simultaneously. Presently such kind of systems have limited use in traffic monitoring, security surveillance systems, spying and espionage etc. After substantial literature review it was found that Audio-Visual fusion is one of the techniques yet to be explored for object tracking due to the difficulty involved in synchronizing both the modalities viz audio and video. The advantage of such a system is that it is more robust in challenging environments. The aim is to overcome the shortcomings that are found in standalone audio and video techniques. In environments where low lighting conditions are found, the video techniques fail to track the object since the installed cameras don’t receive the proper images but the audio tracking assists in such cases. Similarly, we might have situations in which there is a lot of background noise, room reverberations or any combination of the above factors. In such a situation, audio tracking fails but video tracking works robustly. In this thesis, a colour based tracking approach using particle filter has been implemented by entering the gray scale indices of the red, green and blue planes of the object’s image into the system as the input. Then an audio source localization method based on Simplex Optimization is implemented. This approach is based on capturing the sound signals from a scene by a set of microphones and then locating the source location by using time delay estimation techniques and simplex optimization. Then an attempt to fuse the information obtained from both modes has been studied.