Sindhu Hegde

IIIT Hyderabad, CVIT(Centre for Visual Information Technolony), Graduate Student

Followers

Following

Public Views

Interests

Uploads

Papers by Sindhu Hegde

Towards Accurate Lip-to-Speech Synthesis in-the-Wild

Visual Speech Enhancement Without A Real Visual Stream

arXiv (Cornell University), Dec 20, 2020

In this work, we rethink the task of speech enhancement in unconstrained real-world environments.... more In this work, we rethink the task of speech enhancement in unconstrained real-world environments. Current state-of-the-art methods use only the audio stream and are limited in their performance in a wide range of real-world noises. Recent works using lip movements as additional cues improve the quality of generated speech over "audio-only" methods. But, these methods cannot be used for several applications where the visual stream is unreliable or completely absent. We propose a new paradigm for speech enhancement by exploiting recent breakthroughs in speech-driven lip synthesis. Using one such model as a teacher network, we train a robust student network to produce accurate lip movements that mask away the noise, thus acting as a "visual noise filter". The intelligibility of the speech enhanced by our pseudo-lip approach is comparable (< 3% difference) to the case of using real lips. This implies that we can exploit the advantages of using lip movements even in the absence of a real video stream. We rigorously evaluate our model using quantitative metrics as well as human evaluations. Additional ablation studies and a demo video on our website containing qualitative comparisons and results clearly illustrate the effectiveness of our approach. We provide a demo video which clearly illustrates the effectiveness of our proposed approach on our website: http://cvit.iiit.ac.in/research/projects/cvit-projects/ visual-speech-enhancement-without-a-real-visual-stream. The code and models are also released for future research: https://github.com/Sindhu-Hegde/ pseudo-visual-speech-denoising.

Download

GestSync: Determining who is speaking without a talking head

arXiv (Cornell University), Oct 7, 2023

In this paper we introduce a new synchronisation task, Gesture-Sync: determining if a person's ge... more In this paper we introduce a new synchronisation task, Gesture-Sync: determining if a person's gestures are correlated with their speech or not. In comparison to Lip-Sync, Gesture-Sync is far more challenging as there is a far looser relationship between the voice and body movement than there is between voice and lip motion. We introduce a dual-encoder model for this task, and compare a number of input representations including RGB frames, keypoint images, and keypoint vectors, assessing their performance and advantages. We show that the model can be trained using self-supervised learning alone, and evaluate its performance on the LRS3 dataset. Finally, we demonstrate applications of Gesture-Sync for audiovisual synchronisation, and in determining who is the speaker in a crowd, without seeing their faces. The code, datasets and pre-trained models can be found at: https://www.robots.ox.ac.uk/~vgg/research/gestsync. Figure 1: Who is speaking in these scenes? Our model, dubbed GestSync learns to identify whether a person's gestures and speech are "in-sync". The learned embeddings from our model are used to determine "who is speaking" in the crowd, without looking at their faces. Please refer to the demo video for examples.

format_quoteIntroduced Gesture-Sync task shows promising results on LRS3 dataset, demonstrating the model's effectiveness in gesture-driven audio-visual synchronization.format_quote

Download

Extreme-scale Talking-Face Video Upsampling with Audio-Visual Priors

Proceedings of the 30th ACM International Conference on Multimedia

format_quoteWe successfully upscale 8×8 videos to 256×256 through a novel audio-visual upsampling network, leveraging audio priors for detail recovery.format_quote

Download

Towards Automatic Speech to Sign Language Generation

arXiv: Computer Vision and Pattern Recognition, Jun 24, 2021

We aim to solve the highly challenging task of generating continuous sign language videos solely ... more We aim to solve the highly challenging task of generating continuous sign language videos solely from speech segments for the first time. Recent efforts in this space have focused on generating such videos from human-annotated text transcripts without considering other modalities. However, replacing speech with sign language proves to be a practical solution while communicating with people suffering from hearing loss. Therefore, we eliminate the need of using text as input and design techniques that work for more natural, continuous, freely uttered speech covering an extensive vocabulary. Since the current datasets are inadequate for generating sign language directly from speech, we collect and release the first Indian sign language dataset comprising speech-level annotations, text transcripts, and the corresponding sign-language videos. Next, we propose a multi-tasking transformer network trained to generate signer's poses from speech segments. With speech-to-text as an auxiliary task and an additional cross-modal discriminator, our model learns to generate continuous sign pose sequences in an end-to-end manner. Extensive experiments and comparisons with other baselines demonstrate the effectiveness of our approach. We also conduct additional ablation studies to analyze the effect of different modules of our network. A demo video containing several results is attached to the supplementary material.

Download

Fake Review Detection Using Hybrid Ensemble Learning

Lecture Notes in Electrical Engineering, 2021

Opinion spam on online restaurant review sites are a major problem as the reviews influence the u... more Opinion spam on online restaurant review sites are a major problem as the reviews influence the users’ choice to visit or not to a restaurant. In this paper, we address the problem of detecting genuine and fake reviews in restaurant online reviews. We propose a fake review detection technique comprising data preprocessing, detection and ensemble learning that learns the reviews and their features to filter out the fake reviews. Initially, we preprocess to obtain the refined reviews and employ two independent classifiers using deep machine learning and feature-based machine learning techniques for detection. These classifiers tackle the problem in two aspects, i.e., the deep machine learning model learns the word distributions and the feature-based machine learning model extracts the relevant features from the reviews. Finally, a hybrid ensemble model from the two classifiers are built to detect the genuine and fake reviews. The experimental analysis of the proposed approach on Yelp datasets outperforms the existing state-of-the-art methods.

Audio-Visual Speech Super-Resolution

In this paper, we present an audio-visual model to perform speech super-resolution at large scale... more In this paper, we present an audio-visual model to perform speech super-resolution at large scale-factors (8× and 16×). Previous works attempted to solve this problem using only the audio modality as input and thus were limited to low scale-factors of 2× and 4×. In contrast, we propose to incorporate both visual and auditory signals to superresolve speech of sampling rates as low as 1kHz. In such challenging situations, the visual features assist in learning the content and improves the quality of the generated speech. Further, we demonstrate the applicability of our approach to arbitrary speech signals where the visual stream is not accessible. Our “pseudo-visual network” precisely synthesizes the visual stream solely from the low-resolution speech input. Extensive experiments and the demo video illustrate our method’s remarkable results and benefits over state-of-the-art audio-only speech super-resolution approaches. Figure 1: We present an audio-visual model for super-resolving v...

format_quoteIntroduced a pseudo-visual model generating visual streams without real-world faces, enabling applicability when visual data is unavailable.format_quote

Download

Evaluation of Point Cloud Categorization for Rigid and Non-Rigid 3D Objects

Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing, 2018

In this paper, we address the problem of 3D object categorization for point cloud data. With the ... more In this paper, we address the problem of 3D object categorization for point cloud data. With the availability of inexpensive scanning devices and powerful computational resources, there is a rapid growth of point cloud data. This necessitates efficient classification techniques which form the basis for analysis and processing of 3D point cloud data. In order to address the classification problem, we propose a 3D object categorization framework for both rigid and non-rigid objects. Initially, the proposed approach extracts the feature descriptors using improved wave kernel signature by approximating Laplace-Beltrami operator on point cloud data for non-rigid objects. For rigid objects, our approach uses the geometric features, namely, metric tensor and Christoffel symbols by modifying the geodesic distance computation. These feature descriptors are then represented using bag-of-features and improved Fisher vector encoding techniques. Finally, the support vector machine classifies the 3D objects into predefined set of classes. We also provide an exhaustive performance evaluation of the proposed 3D object categorization framework on state-of-the-art datasets, namely, SHREC'10, SHREC'11, SHREC'12, SHREC'15 and Princeton Shape Benchmark. The evaluation results reveal that the proposed approach outperforms the existing object categorization methods for both rigid and non-rigid 3D objects.

An Evaluation of Feature Encoding Techniques for Non-Rigid and Rigid 3D Point Cloud Retrieval

In this paper, we address the 3D point cloud based retrieval problem for both nonrigid and rigid ... more In this paper, we address the 3D point cloud based retrieval problem for both nonrigid and rigid 3D data. As powerful computation resources and scanning devices have led to an exponential growth of 3D point cloud data, retrieving the relevant 3D objects from databases is a challenging task. The local descriptors provide only the abstract representations that do not enable the exploration of shape variability to solve the 3D object retrieval problem. Thus, it is not just the local descriptors but also the encoding of local signatures into global descriptors which is of crucial importance for enhancing the performance. To create a compact shape signature that constitutes the 3D object as a whole, various encoding techniques have been proposed in the literature. The most popular among them are bag-of-features [28], Fisher vector [20] and vector of locally aggregated descriptors [11]. However evaluating the different encoding techniques and analyzing the critical aspects to boost the pe...

Download

PIG-Net: Inception based deep learning architecture for 3D point cloud segmentation

Computers & Graphics, 2021

Point clouds, being the simple and compact representation of surface geometry of 3D objects, have... more Point clouds, being the simple and compact representation of surface geometry of 3D objects, have gained increasing popularity with the evolution of deep learning networks for classification and segmentation tasks. Unlike human, teaching the machine to analyze the segments of an object is a challenging task and quite essential in various machine vision applications. In this paper, we address the problem of segmentation and labelling of the 3D point clouds by proposing a inception based deep network architecture called PIG-Net, that effectively characterizes the local and global geometric details of the point clouds. In PIG-Net, the local features are extracted from the transformed input points using the proposed inception layers and then aligned by feature transform. These local features are aggregated using the global average pooling layer to obtain the global features. Finally, feed the concatenated local and global features to the convolution layers for segmenting the 3D point clouds. We perform an exhaustive experimental analysis of the PIG-Net architecture on two state-of-the-art datasets, namely, ShapeNet [1] and PartNet [2]. We evaluate the effectiveness of our network by performing ablation study.

Download

Sentiment based Food Classification for Restaurant Business

2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2018

In this paper, we address the problem of classifying the restaurant reviews into different meal c... more In this paper, we address the problem of classifying the restaurant reviews into different meal courses based on the review sentiments. Based on our survey, we observe that there are no works which focus on classifying the food items into different meal courses with top $\boldsymbol{n}$ food items under each course using restaurant reviews. Most of the works in literature address the issues like predicting the ratings of the restaurant or restaurant business strategies by considering various attributes of the restaurant. Thus, in this paper, we propose a sentiment based food classification framework consisting of two tasks, namely, sentiment classification and four-course meal classification. In sentiment classification, we classify the reviews into positive and negative categories based on the sentiments of the reviews. In four-course meal classification, we categorize the reviews into four courses, namely, soups and salads, appetizers, main course and desserts. We list the top $\boldsymbol{n}$ food items liked by most of the customers in each of these courses. In order to select the suitable classification technique for addressing the identified problem, we analyze the learning curves based on the bias-variance trade-off. We observe that support vector machines (SVM) classification technique outperforms other techniques. The performance analysis of the proposed framework is carried out on the standard Yelp dataset. The top 5 food items obtained in each of the categories are soups and salads: salad, corn, coffee, cocktail, tea; appetizers: burger, pizza, bread, cheese, frie; main course: steak, meal, chicken, meat, beef; and desserts: doughnut, cream, cake, chocolate, flan.

Restaurant setup business analysis using yelp dataset

2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2017

In this paper, we address the issues associated with setting-up of a new restaurant business. To ... more In this paper, we address the issues associated with setting-up of a new restaurant business. To strategize a new restaurant business, we propose a restaurant business framework which comprises of 3 most important tasks, namely, high frequency attributes, most crowded day and location of the restaurant. First, we identify the features/attributes of the restaurants in which the customers are most interested in and provide those facilities and services to increase the profit. Next, we identify the day of the week when the restaurants are heavily crowded so that the best recipes and offers are made available on those days. Finally, since location has a profound effect on the success of a restaurant business, we consider location to be the most important to know the nearby restaurants and their facilities before coming up with a new restaurant business. The performance analysis of the proposed framework was carried out on the standard Yelp dataset. Thus, we found credit card to be the most preferred attribute, the most crowded day to be Monday and Divey to be the most desired ambience among the customers. We also demonstrate how the new restaurant can be setup by identifying the nearest restaurants and the services.

Visual Speech Enhancement Without A Real Visual Stream

2021 IEEE Winter Conference on Applications of Computer Vision (WACV), 2021

Download

Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild

Proceedings of the 30th ACM International Conference on Multimedia

Figure 1: We address the problem of generating speech from silent lip videos for any speaker in t... more Figure 1: We address the problem of generating speech from silent lip videos for any speaker in the wild. Previous works train either on large amounts of data of isolated speakers or in laboratory settings with a limited vocabulary. Conversely, we can generate speech for the lip movements of arbitrary identities in any voice without additional speaker-specific fine-tuning. Our new VAE-GAN approach allows us to learn strong audiovisual associations despite the ambiguous nature of the task.

Download

Towards Automatic Speech to Sign Language Generation

Interspeech 2021

Sindhu Hegde

Uploads

Papers by Sindhu Hegde

Log In