Experimental Study on Finding Audio-visual Association by Maximizing Mutual Information

Abstract

In this thesis, the fusion of audio and visual signal are done by considering one camera and one microphone. Although there are many approaches for audiovisual fusion, but still it is an open problem. Here, localizing and recognizing the speaker in a video by maximizing the mutual information occuring from audiovisual scene. Experiment is done using a method in which the arrival of audio and visual signals from a unique source are detected using a signal-level fusion technique. The information theoretic measure of audio-visual correspondence using probabilistic multi-modal generation statistical implicit model given in [1] is used here. In the used method non-parametric statistical density is able to give better result for characterizing the mutual information between signals from different domains by following some constraint for which entropy is maximized. By maximizing the above mutual information between different pairs of signals, it is possible to identify which person is speaking a given utterance, and we can verify whether the audio is associated with that visual seen or not. In doing so, no assumption is made about user appearance or speech of person. Further more this method doesn’t required any training of corpuses. In this thesis experimental results are demonstrated using CUAVE database.