Soft biometrics based person retrieval from unconstrained surveillance video

Abstract

In today’s world of security and surveillance, the prime objective is to search and locate the person of interest from unconstrained surveillance videos using a tex tual description. On the availability of such a description, the person of interest is retrieved by observing countless hours of video footage. Moreover, only a few per sons can continuously monitor surveillance videos on several display screens and multiple feeds. Such manual efforts take a considerable amount of time, and are also very ineffectual. In such situations, the capability to translate a textual description into input that is relevant to computer vision would be of great assistance. Image-based person retrieval methods are popular. They retrieve by querying the system with the image of the target person. The query image is from one cam era, and the retrieval methods re-identify the same person in the video footage of a different camera. Such person retrieval techniques fail when a query image is un available. In such scenarios, a textual description is the prime entity for retrieval, e.g., a tall female with a grey short sleeve t-shirt and white jeans with a handbag. Such a description has personal attributes like height, gender, clothing colour, and clothing type. These attributes are known as soft biometrics. This dissertation develops approaches for retrieving persons from uncon strained surveillance videos using a soft biometrics-based textual description. The initial approach leverages the advantage of semantic segmentation for precise feet and head point extraction for better height estimation. Person retrieval uses only three attributes, i.e., height, gender, and torso colour, where gender and colour mod els are trained using AlexNet. The initial approach is improved by introducing adaptive torso patch extrac tion for torso type, bounding box regression, better data augmentation strategy for colour and gender model, and DenseNet-169 based improved classification models. Person retrieval is done using height, torso type, torso colour-I, torso colour-II, and gender. Further improvement is made by utilising the contribution of each attribute, and a ranking strategy is introduced to retrieve the target person. This strategy per forms well for a person with partial occlusion and results in a higher degree of per formance. The resulting approach outperforms previous approaches and achieves state-of-the-art performance. The introduction of a single model for attribute recognition enables a solution towards an end-to-end person retrieval system. It considers the correlation among the attributes. Better dataset preparation and attention mechanism with focal loss proved to be best for attribute recognition. The resulting approach once again out performs existing techniques by utilising only five attributes. Findings from this dissertation contribute to achieving the best performance in-person retrieval from unconstrained surveillance video using soft biometrics