Skip to main content

Nicolas Thome

Followers

13

Following

3

Public Views

Interests

Uploads

Papers by Nicolas Thome

SHARE: Regularization for Deep Learning

Regularization is a big issue for training deep neural networks. In this paper, we propose a new ... more Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHARE for SHAnnon DEcay. The originality of the approach is to define a prior based on conditional entropy, which explicitly decouples the learning of invariant representations in the regularizer and the learning of correlations between inputs and labels in the data fitting term. Our second contribution is to derive a stochastic version of the regularizer compatible with deep learning, resulting in a tractable training scheme. We empirically validate the efficiency of our approach to improve classification performances compared to common regularization schemes on several standard architectures.

M2CAI Workflow Challenge: Convolutional Neural Networks with Time Smoothing and Hidden Markov Model for Video Frames Classification

ArXiv, 2016

Our approach is among the three best to tackle the M2CAI Workflow challenge. The latter consists ... more Our approach is among the three best to tackle the M2CAI Workflow challenge. The latter consists in recognizing the operation phase for each frames of endoscopic videos. In this technical report, we compare several classification models and temporal smoothing methods. Our submitted solution is a fine tuned Residual Network-200 on 80% of the training set with temporal smoothing using simple temporal averaging of the predictions and a Hidden Markov Model modeling the sequence.

format_quoteAchieved a Jaccard index of 71.9 using a fine-tuned Residual Network-200 in M2CAI Workflow challenge.format_quote

A Deep Physical Model for Solar Irradiance Forecasting with Fisheye Images

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020

We present a new deep learning approach for short-term solar irradiance forecasting based on fish... more We present a new deep learning approach for short-term solar irradiance forecasting based on fisheye images. Our architecture, based on recent works on video prediction with partial differential equations, extracts spatio-temporal features modelling cloud motion to accurately anticipate future solar irradiance. Our method obtains state-of-theart results on video prediction and 5min-ahead irradiance forecasting against strong recent baselines, highlighting the benefits of incorporating physical knowledge in deep models for real-world physical process forecasting.

DualDis: Dual-Branch Disentangling with Adversarial Learning

In computer vision, disentangling techniques aim at improving latent representations of images by... more In computer vision, disentangling techniques aim at improving latent representations of images by modeling factors of variation. In this paper, we propose DualDis, a new auto-encoder-based framework that disentangles and linearizes class and attribute information. This is achieved thanks to a two-branch architecture forcing the separation of the two kinds of information, accompanied by a decoder for image reconstruction and generation. To effectively separate the information, we propose to use a combination of regular and adversarial classifiers to guide the two branches in specializing for class and attribute information respectively. We also investigate the possibility of using semi-supervised learning for an effective disentangling even using few labels. We leverage the linearization property of the latent spaces for semantic image editing and generation of new images. We validate our approach on CelebA, Yale-B and NORB by measuring the efficiency of information separation via cl...

Robust and Decomposable Average Precision for Image Retrieval

ArXiv, 2021

In image retrieval, standard evaluation metrics rely on score ranking, e.g. average precision (AP... more In image retrieval, standard evaluation metrics rely on score ranking, e.g. average precision (AP). In this paper, we introduce a method for robust and decomposable average precision (ROADMAP) addressing two major challenges for end-to-end training of deep neural networks with AP: non-differentiability and non-decomposability. Firstly, we propose a new differentiable approximation of the rank function, which provides an upper bound of the AP loss and ensures robust training. Secondly, we design a simple yet effective loss function to reduce the decomposability gap between the AP in the whole training set and its averaged batch approximation, for which we provide theoretical guarantees. Extensive experiments conducted on three image retrieval datasets show that ROADMAP outperforms several recent AP approximation methods and highlight the importance of our two contributions. Finally, using ROADMAP for training deep models yields very good performances, outperforming state-of-the-art r...

format_quoteIntroduced L SupAP offers a robust smooth approximation of AP, providing an essential upper bound for effective gradient-based optimization.format_quote

HybridNet: Classification and Reconstruction Cooperation for Semi-Supervised Learning

In this paper, we introduce a new model for leveraging unlabeled data to improve generalization p... more In this paper, we introduce a new model for leveraging unlabeled data to improve generalization performances of image classifiers: a two-branch encoder-decoder architecture called HybridNet. The first branch receives supervision signal and is dedicated to the extraction of invariant class-related representations. The second branch is fully unsupervised and dedicated to model information discarded by the first branch to reconstruct input data. To further support the expected behavior of our model, we propose an original training objective. It favors stability in the discriminative branch and complementarity between the learned representations in the two branches. HybridNet is able to outperform state-of-the-art results on CIFAR-10, SVHN and STL-10 in various semi-supervised settings. In addition, visualizations and ablation studies validate our contributions and the behavior of the model on both CIFAR-10 and STL-10 datasets.

MUTAN: Multimodal Tucker Fusion for Visual Question Answering

Bilinear models provide an appealing framework for mixing and merging information in Visual Quest... more Bilinear models provide an appealing framework for mixing and merging information in Visual Question Answering (VQA) tasks. They help to learn high level associations between question meaning and visual concepts in the image, but they suffer from huge dimensionality issues. We introduce MUTAN, a multimodal tensor-based Tucker decomposition to efficiently parametrize bilinear interactions between visual and textual representations. Additionally to the Tucker framework, we design a low-rank matrix-based decomposition to explicitly constrain the interaction rank. With MUTAN, we control the complexity of the merging scheme while keeping nice interpretable fusion relations. We show how our MUTAN model generalizes some of the latest VQA architectures, providing state-of-the-art results.

Addressing Failure Prediction by Learning Model Confidence

Assessing reliably the confidence of a deep neural network and predicting its failures is of prim... more Assessing reliably the confidence of a deep neural network and predicting its failures is of primary importance for the practical deployment of these models. In this paper, we propose a new target criterion for model confidence, corresponding to the True Class Probability (TCP). We show how using the TCP is more suited than relying on the classic Maximum Class Probability (MCP). We provide in addition theoretical guarantees for TCP in the context of failure prediction. Since the true class is by essence unknown at test time, we propose to learn TCP criterion on the training set, introducing a specific learning scheme adapted to this context. Extensive experiments are conducted for validating the relevance of the proposed approach. We study various network architectures, small and large scale datasets for image classification and semantic segmentation. We show that our approach consistently outperforms several strong methods, from MCP to Bayesian uncertainty, as well as recent approa...

Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings

Designing powerful tools that support cooking activities has rapidly gained popularity due to the... more Designing powerful tools that support cooking activities has rapidly gained popularity due to the massive amounts of available data, as well as recent advances in machine learning that are capable of analyzing them. In this paper, we propose a cross-modal retrieval model aligning visual and textual data (like pictures of dishes and their recipes) in a shared representation space. We describe an effective learning scheme, capable of tackling large-scale problems, and validate it on the Recipe1M dataset containing nearly 1 million picture-recipe pairs. We show the effectiveness of our approach regarding previous state-of-the-art models and present qualitative results over computational cooking use cases.

Multitask Classification and Segmentation for Cancer Diagnosis in Mammography

Annotation cost is a bottleneck for collecting massive data in mammography, especially for traini... more Annotation cost is a bottleneck for collecting massive data in mammography, especially for training deep neural networks. In this paper, we study the use of heterogeneous levels of annotation granularity to improve predictive performances. More precisely, we introduce a multi-task learning scheme for training convolutional neural network (ConvNets), which combines segmentation and classification, using image-level and pixel-level annotations. In this way, different objectives can be used to regularize training by sharing intermediate deep representations. Successful experiments are carried out on the Digital Database of Screening Mammography (DDSM) to validate the relevance of the proposed approach.

Jkernelmachines: A simple framework for kernel machines

Editor: JKernelMachines is a Java library for learning with kernels. It is primarily designed to ... more Editor: JKernelMachines is a Java library for learning with kernels. It is primarily designed to deal with custom kernels that are not easily found in standard libraries, such as kernels on structured data. These types of kernels are often used in computer vision or bioinfor-matics applications. We provide several kernels leading to state of the art classification performances in computer vision, as well as various kernels on sets. The main focus of the library is to be easily extended with new kernels. Standard SVM optimization algorithms are available, but also more sophisticated learning-based kernel combination methods such as Multiple Kernel Learning (MKL), and a recently published algorithm to learn powered products of similarities (Product Kernel Learning).

format_quoteJKernelMachines aims to simplify kernel use in specialized fields like computer vision and bioinformatics, emphasizing easy kernel development.format_quote

Deformable Part-based Fully Convolutional Network for Object Detection

Existing region-based object detectors are limited to regions with fixed box geometry to represen... more Existing region-based object detectors are limited to regions with fixed box geometry to represent objects, even if those are highly non-rectangular. In this paper we introduce DP-FCN, a deep model for object detection which explicitly adapts to shapes of objects with deformable parts. Without additional annotations, it learns to focus on discriminative elements and to align them, and simultaneously brings more invariance for classification and geometric information to refine localization. DP-FCN is composed of three main modules: a Fully Convolutional Network to efficiently maintain spatial resolution, a deformable part-based RoI pooling layer to optimize positions of parts and build invariance, and a deformation-aware localization module explicitly exploiting displacements of parts to improve accuracy of bounding box regression. We experimentally validate our model and show significant gains. DP-FCN achieves state-of-the-art performances of 83.1% and 80.9% on PASCAL VOC 2007 and 2...

Shade: Information-Based Regularization for Deep Learning

2018 25th IEEE International Conference on Image Processing (ICIP)

Regularization is a big issue for training deep neural networks. In this paper, we propose a new ... more Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define a prior based on conditional entropy, which explicitly decouples the learning of invariant representations in the regularizer and the learning of correlations between inputs and labels in the data fitting term. Our second contribution is to derive a stochastic version of the regularizer compatible with deep learning, resulting in a tractable training scheme. We empirically validate the efficiency of our approach to improve classification performances compared to common regularization schemes on several standard architectures.

Image Classification Using Object Detectors

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci-entific ... more HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et a ̀ la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Probabilistic Time Series Forecasting with Structured Shape and Temporal Diversity

Probabilistic forecasting consists in predicting a distribution of possible future outcomes. In t... more Probabilistic forecasting consists in predicting a distribution of possible future outcomes. In this paper, we address this problem for non-stationary time series, which is very challenging yet crucially important. We introduce the STRIPE model for representing structured diversity based on shape and time features, ensuring both probable predictions while being sharp and accurate. STRIPE is agnostic to the forecasting model, and we equip it with a diversification mechanism relying on determinantal point processes (DPP). We introduce two DPP kernels for modeling diverse trajectories in terms of shape and time, which are both differentiable and proved to be positive semi-definite. To have an explicit control on the diversity structure, we also design an iterative sampling mechanism to disentangle shape and time representations in the latent space. Experiments carried out on synthetic datasets show that STRIPE significantly outperforms baseline methods for representing diversity, while...

Unsupervised and supervised visual codes with restricted boltzmann machines,” ECCV

Abstract. Recently, the coding of local features (e.g. SIFT) for image categorization tasks has b... more Abstract. Recently, the coding of local features (e.g. SIFT) for image categorization tasks has been extensively studied. Incorporated within the Bag of Words (BoW) framework, these techniques optimize the pro-jection of local features into the visual codebook, leading to state-of-the-art performances in many benchmark datasets. In this work, we propose a novel visual codebook learning approach using the restricted Boltzmann machine (RBM) as our generative model. Our contribution is three-fold. Firstly, we steer the unsupervised RBM learning using a regularization scheme, which decomposes into a combined prior for the sparsity of each feature’s representation as well as the selectivity for each codeword. The codewords are then fine-tuned to be discriminative through the super-vised learning from top-down labels. Secondly, we evaluate the proposed method with the Caltech-101 and 15-Scenes datasets, either matching or outperforming state-of-the-art results. The codebooks are compact a...

M.: Biasing restricted Boltzmann machines to manipulate latent selectivity and sparsity

This paper proposes a modification to the restricted Boltzmann machine (RBM) learning algorithm t... more This paper proposes a modification to the restricted Boltzmann machine (RBM) learning algorithm to incorporate inductive biases. These latent activation biases are ideal solutions of the latent activity and may be designed either by modeling neural phenomenon or inductive principles of the task. In this paper, we design ac-tivation biases for sparseness and selectivity based on the activation distributions of biological neurons. With this model, one can manipulate the selectivity of indi-vidual hidden units and the sparsity of population codes. The biased RBM yields a filter bank of Gabor-like filters when trained on natural images, while model-ing handwritten digits results in filters with stroke-like features. We quantitatively verify that the latent representations assume the properties of the activation bi-ases. We further demonstrate that RBMs biased with selectivity and sparsity can significantly outperform standard RBMs for discriminative tasks. 1

format_quoteThe proposed activation biases for RBMs are inspired by biological neuron activity distributions, leading to better selectivity and sparsity.format_quote

Unsupervised and supervised visual codes with restricted Boltzmann machines

Abstract. Recently, the coding of local features (e.g. SIFT) for image categorization tasks has b... more Abstract. Recently, the coding of local features (e.g. SIFT) for image categorization tasks has been extensively studied. Incorporated within the Bag of Words (BoW) framework, these techniques optimize the projection of local features into the visual codebook, leading to state-of-theart performances in many benchmark datasets. In this work, we propose a novel visual codebook learning approach using the restricted Boltzmann machine (RBM) as our generative model. Our contribution is three-fold. Firstly, we steer the unsupervised RBM learning using a regularization scheme, which decomposes into a combined prior for the sparsity of each feature’s representation as well as the selectivity for each codeword. The codewords are then fine-tuned to be discriminative through the supervised learning from top-down labels. Secondly, we evaluate the proposed method with the Caltech-101 and 15-Scenes datasets, either matching or outperforming state-of-the-art results. The codebooks are compact and ...

Apprentissage Interactif pour la RI Contexte Plan

1 Sélection d’une image requête par l’utilisateur 2 Tri de la base par ordre décroissant de si... more

head detection and tracking using skeleton graph for people