James DiCarlo

Followers

Following

Public Views

Interests

Uploads

Papers by James DiCarlo

Functionally distinct sub-regions of the parahippocampal place area revealed by model-based neural control

Journal of Vision, Dec 5, 2022

Computational Models Recapitulate Key Signatures of Face, Body and Scene Processing in the FFA, EBA, and PPA

Journal of Vision

Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolution

arXiv (Cornell University), Oct 15, 2022

Neuroscience has long been an essential driver of progress in artificial intelligence (AI). We pr... more Neuroscience has long been an essential driver of progress in artificial intelligence (AI). We propose that to accelerate progress in AI, we must invest in fundamental research in NeuroAI. A core component of this is the embodied Turing test, which challenges AI animal models to interact with the sensorimotor world at skill levels akin to their living counterparts. The embodied Turing test shifts the focus from those capabilities like game playing and language that are especially well-developed or uniquely human to those capabilities-inherited from over 500 million years of evolution-that are shared with all animals. Building models that can pass the embodied Turing test will provide a roadmap for the next generation of AI. Over the coming decades, Artificial Intelligence (AI) will transform society and the world economy in ways that are as profound as the computer revolution of the last half century and likely at an even faster pace. This AI revolution presents tremendous opportunities to unleash human creativity and catalyze economic growth, relieving workers from performing the most dangerous and menial jobs. However, to reach this potential, we still require advances that will make AI more human-like in its capabilities. Historically, neuroscience has been a critical driver and source of inspiration for improvements in AI, particularly those that made AI more proficient in areas that humans and other animals excel at, such as vision, reward-based learning, interacting with the physical world, and language 1,2. It can still play this role. To accelerate progress in AI and realize its vast potential, we must invest in fundamental research in "NeuroAI." The seeds of the current AI revolution were planted decades ago, mainly by researchers attempting to understand how brains compute 3. Indeed, the earliest efforts to build an "artificial brain" led to the invention of the modern "von Neumann computer architecture," for which John von Neumann explicitly drew upon the very limited knowledge of the brain available to him in the 1940s 4,5. Later, the Nobel-prize winning work of David Hubel and Torsten Wiesel on visual processing circuits in the cat neocortex inspired the deep convolutional networks that have catalyzed the recent revolution in modern AI 6-8. Similarly, the development of reinforcement learning was directly inspired by insights into animal behavior and neural activity during learning 9-15. Now, decades later, applications of ANNs and RL are coming so quickly that many observers assume that the long-elusive goal of human-level intelligencesometimes referred to as "artificial general intelligence"-is within our grasp. However, in contrast to the optimism of those outside the field, many front-line AI researchers believe that major breakthroughs are needed before we can build artificial systems capable of doing all that a human, or even a much simpler animal like a mouse, can do. Although AI systems can easily defeat any human opponent in games such as chess 16 and Go 17 , they are not robust and often struggle when faced with novel situations. Moreover, we have yet to build effective systems that can walk to the shelf, take down the chess set, set up the pieces, and move them around during a game, although recent progress is encouraging 18. Similarly, no machine can build a nest, forage for berries, or care for young. Today's AI systems cannot compete with the sensorimotor capabilities of a four-year old child or even simple animals. Many basic capacities required to navigate new situationscapacities that animals have or acquire effortlessly-turn out to be deceptively challenging for AI, partly because AI systems lack even the basic abilities to interact with an unpredictable world. A growing number of AI researchers doubt that merely scaling up current approaches will overcome these limitations. Given the need to achieve more natural intelligence in AI, it is quite likely that new inspiration from naturally intelligent systems is needed 19. Historically, many key AI advances, such as convolutional ANNs and reinforcement learning, were inspired by neuroscience. Neuroscience continues to provide guidance-e.g., attention-based neural networks were loosely inspired by attention mechanisms in the brain 20-23-but this is often based on findings that are decades old. The fact that such cross-pollination between AI and neuroscience is far less common than in the past represents a missed opportunity. Over the last decades, through efforts such as the NIH BRAIN initiative and others, we have amassed an enormous amount of knowledge about the brain. The emerging field of NeuroAI, at the intersection of neuroscience and AI, is based on the premise that a better understanding of neural computation will reveal fundamental ingredients of intelligence and catalyze the next revolution in AI. This will eventually lead to artificial agents with capabilities that match those of humans. The NeuroAI program we advocate is driven by the recognition that AI historically owes much to neuroscience and the promise that AI will continue to learn from it-but only if there is a large enough community of researchers fluent in both domains. We believe the time is right for a large-scale effort to identify and understand the principles of biological intelligence and abstract those for application in computer and robotic systems.

Download

How Well Do Contrastive Learning Algorithms Model Human Real-time and Life-long Learning?

2022 Conference on Cognitive Computational Neuroscience

SanghaviJozwik2020

Documentation for neural data collected with BOLD5000 images.

SanghaviMurty2020

Documentation for neural data collected with images from (Rust & DiCarlo, 2012).

Single fiber endoscopy for deep brain imaging

Imaging and Applied Optics 2017 (3D, AIO, COSI, IS, MATH, pcAOP), 2017

Fast optical imaging of individual neurons in deep brain circuits remains a critical challenge in... more

Linking image-by-image population dynamics in the macaque inferior temporal cortex to core object recognition behavior

2018 Conference on Cognitive Computational Neuroscience, 2018

Task-Driven Convolutional Recurrent Models of the Visual System

Feed-forward convolutional neural networks (CNNs) are currently state-of-the-art for object class... more Feed-forward convolutional neural networks (CNNs) are currently state-of-the-art for object classification tasks such as ImageNet. Further, they are quantitatively accurate models of temporally-averaged responses of neurons in the primate brain's visual system. However, biological visual systems have two ubiquitous architectural features not shared with typical CNNs: local recurrence within cortical areas, and long-range feedback from downstream areas to upstream areas. Here we explored the role of recurrence in improving classification performance. We found that standard forms of recurrence (vanilla RNNs and LSTMs) do not perform well within deep CNNs on the ImageNet task. In contrast, novel cells that incorporated two structural features, bypassing and gating, were able to boost task accuracy substantially. We extended these design principles in an automated search over thousands of model architectures, which identified novel local recurrent cells and long-range feedback conne...

Download

– September 11 , 2006 Spiking Neural Circuits for Gaussian Tuning and the Maximum Operation

s of NIH-Conte Meeting – September 11, 2006 Spiking Neural Circuits for Gaussian Tuning and the M... more s of NIH-Conte Meeting – September 11, 2006 Spiking Neural Circuits for Gaussian Tuning and the Maximum Operation Jake Bouvrie and Ulf Knoblich Massachusetts Institute of Technology In an effort to realize an additional layer of biophysical plausibility in the model, we present preliminary spiking neural circuits which separately implement the maximum operation and an approximation to a normalized Gaussian tuning function. We investigate these circuits on two levels of complexity. First we simulate biophysically detailed networks of model neurons with realistic synapses and different network topologies in order to study their responses to different types of inputs and compare them to data from electrophysiological experiments. We then implement a simplified version of these circuits retaining the basic necessary mechanisms but decreasing their complexity in order to make them computationally less expensive. Our MAX design is based on a winner-take-all computation with integrate-and-...

format_quoteHighly selective neurons exhibit lower tolerance to object appearance changes; revealing a critical tradeoff in visual processing.format_quote

Download

Selective behavioral deficits from focal inactivation of primate inferior temporal (IT) cortex: a new quantitative constraint for models of core object recognition

Primate core visual object recognition is thought to rely on the ventral visual stream, a hierarc... more Primate core visual object recognition is thought to rely on the ventral visual stream, a hierarchy of cortical areas culminating in inferior temporal (IT) cortex. Previous work has shown that the IT population responses accurately predict primate object recognition behavior, suggesting that these IT codes underlie these behaviors. However, direct causal evidence for this decoding hypothesis has been equivocal at best, especially beyond the specific case of face-selective sub-regions of IT. Here, we tested the general causal role of IT in core object recognition by reversibly inactivating individual, millimeter-scale regions of IT via injection of muscimol while monkeys performed several binary object discrimination tasks, interleaved trial-by-trial. Our results show that inactivating different millimeter-scale sub-regions of primate IT resulted in different patterns of task deficits. These results provide new constraints for computational models of the ventral stream and this behav...

Download

The inferior temporal cortex is a potential cortical precursor of orthographic processing in untrained monkeys

Nature Communications, 2020

The ability to recognize written letter strings is foundational to human reading, but the underly... more The ability to recognize written letter strings is foundational to human reading, but the underlying neuronal mechanisms remain largely unknown. Recent behavioral research in baboons suggests that non-human primates may provide an opportunity to investigate this question. We recorded the activity of hundreds of neurons in V4 and the inferior temporal cortex (IT) while naïve macaque monkeys passively viewed images of letters, English words and non-word strings, and tested the capacity of those neuronal representations to support a battery of orthographic processing tasks. We found that simple linear read-outs of IT (but not V4) population responses achieved high performance on all tested tasks, even matching the performance and error patterns of baboons on word classification. These results show that the IT cortex of untrained primates can serve as a precursor of orthographic processing, suggesting that the acquisition of reading in humans relies on the recycling of a brain network e...

format_quoteAdded linear decoders on macaque IT representation successfully approximated orthographic processing in visual stimuli.format_quote

Download

Topographic Deep Artificial Neural Networks (TDANNs) predict face selectivity topography in primate inferior temporal (IT) cortex

2018 Conference on Cognitive Computational Neuroscience, 2018

Deep convolutional neural networks are biologically driven models that resemble the hierarchical ... more Deep convolutional neural networks are biologically driven models that resemble the hierarchical structure of primate visual cortex and are the current best predictors of the neural responses measured along the ventral stream. However, the networks lack topographic properties that are present in the visual cortex, such as orientation maps in primary visual cortex and categoryselective maps in inferior temporal (IT) cortex. In this work, the minimum wiring cost constraint was approximated as an additional learning rule in order to generate topographic maps of the networks. We found that our topographic deep artificial neural networks (ANNs) can reproduce the category selectivity maps of the primate IT cortex.

Download

How far can you get with a modern face recognition test set using only simple features?

2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009

In recent years, large databases of natural images have become increasingly popular in the evalua... more In recent years, large databases of natural images have become increasingly popular in the evaluation of face and object recognition algorithms. However, Pinto et al. previously illustrated an inherent danger in using such sets, showing that an extremely basic recognition system, built on a trivial feature set, was able to take advantage of low-level regularities in popular object [10] and face [11] recognition sets, performing on par with many state-of-the-art systems. Recently, several groups have raised the performance "bar" for these sets, using more advanced classification tools. However, it is difficult to know whether these improvements are due to progress towards solving the core computational problem, or are due to further improvements in the exploitation of low-level regularities. Here, we show that even modest optimization of the simple model introduced by Pinto et al. using modern multiple kernel learning (MKL) techniques once again yields "state-of-the-art" performance levels on a standard face recognition set ("Labeled Faces in the Wild" [7]). However, at the same time, even with the inclusion of MKL techniques, systems based on these simple features still fail on a synthetic face recognition test that includes more "realistic" view variation by design. These results underscore the importance of building test sets focussed on capturing the central computational challenges of real-world face recognition.

Download

Teacher Guided Architecture Search

2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019

Much of the recent improvement in neural networks for computer vision has resulted from discovery... more Much of the recent improvement in neural networks for computer vision has resulted from discovery of new networks architectures. Most prior work has used the performance of candidate models following limited training to automatically guide the search in a feasible way. Could further gains in computational efficiency be achieved by guiding the search via measurements of a high performing network with unknown detailed architecture (e.g. the primate visual system)? As one step toward this goal, we use representational similarity analysis to evaluate the similarity of internal activations of candidate networks with those of a (fixed, high performing) teacher network. We show that adopting this evaluation metric could produce up to an order of magnitude in search efficiency over performanceguided methods. Our approach finds a convolutional cell structure with similar performance as was previously found using other methods but at a total computational cost that is two orders of magnitude lower than Neural Architecture Search (NAS) and more than four times lower than progressive neural architecture search (PNAS). We further show that measurements from only ∼300 neurons from primate visual system provides enough signal to find a network with an Imagenet top-1 error that is significantly lower than that achieved by performance-guided architecture search alone. These results suggest that representational matching can be used to accelerate network architecture search in cases where one has access to some or all of the internal representations of a teacher network of interest, such as the brain's sensory processing networks.

Download

Natural and Artificial Intelligence: A brief introduction to the interplay between AI and neuroscience research

Neural Networks, 2021

Neuroscience and artificial intelligence (AI) share a long history of collaboration. Advances in ... more Neuroscience and artificial intelligence (AI) share a long history of collaboration. Advances in neuroscience, alongside huge leaps in computer processing power over the last few decades, have given rise to a new generation of in silico neural networks inspired by the architecture of the brain. These AI systems are now capable of many of the advanced perceptual and cognitive abilities of biological systems, including object recognition and decision making. Moreover, AI is now increasingly being employed as a tool for neuroscience research and is transforming our understanding of brain functions. In particular, deep learning has been used to model how convolutional layers and recurrent connections in the brain's cerebral cortex control important functions, including visual processing, memory, and motor control. Excitingly, the use of neuroscience-inspired AI also holds great promise for understanding how changes in brain networks result in psychopathologies, and could even be utilized in treatment regimes. Here we discuss recent advancements in four areas in which the relationship between neuroscience and AI has led to major advancements in the field; (1) AI models of working memory, (2) AI visual processing, (3) AI analysis of big neuroscience datasets, and (4) computational psychiatry.

format_quoteModern AI's reliance on backpropagation lacks clear biological evidence in synaptic connections, questioning the biological plausibility of current models.format_quote

Download

Next-generation models of recurrent computations in the ventral visual stream

Journal of Vision, 2021

Recurrent Connections in the Primate Ventral Visual Stream Mediate a Tradeoff Between Task Performance and Network Size During Core Object Recognition

The computational role of the abundant feedback connections in the ventral visual stream (VVS) is... more The computational role of the abundant feedback connections in the ventral visual stream (VVS) is unclear, enabling humans and non-human primates to effortlessly recognize objects across a multitude of viewing conditions. Prior studies have augmented feedforward convolutional neural networks (CNNs) with recurrent connections to study their role in visual processing; however, often these recurrent networks are optimized directly on neural data or the comparative metrics used are undefined for standard feedforward networks that lack these connections. In this work, we develop task-optimized convolutional recurrent (ConvRNN) network models that more correctly mimic the timing and gross neuroanatomy of the ventral pathway. Properly chosen intermediate-depth ConvRNN circuit architectures, which incorporate mechanisms of feedforward bypassing and recurrent gating, can achieve high performance on a core recognition task, comparable to that of much deeper feedforward networks. We then devel...

Download

Convolutional recurrent neural network models of dynamics in higher visual cortex

Journal of Vision, 2018

Neurons in the ventral visual pathway exhibit behaviorally relevant temporal dynamics during imag... more Neurons in the ventral visual pathway exhibit behaviorally relevant temporal dynamics during image viewing. However, the most accurate existing computational models of this system are feedforward hierarchical convolutional neural networks (HCNNs), which capture neurons’ time-averaged responses, but do not account well for their complex temporal trajectories. Here we show that HCNNs augmented with both local and global recurrent connections are quantitatively accurate models of dynamics in higher visual cortex. We began with a five-layer HCNN that achieved state-of-the-art predictions of temporally-averaged visual responses in macaque V4 and IT neurons. To model within-area dynamics, we replaced units in each layer with one of several local recurrent circuit motifs, including simple Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs), and Long Short-Term Memory (LSTM) units. We also included combinations of global feedback connections, in which outputs of later convolutional layers were added to inputs of earlier layers. Using backpropagation through time, these new parameters were optimized to predict V4 and IT neural response patterns. Finally, we tested these networks’ ability to predict responses on held-out images and neurons not used for model optimization. We found that the best network structure led to substantial improvements over the feedforward baseline, explaining close to 100% of the explainable variance in V4 neurons and above 75% in IT neurons on average across time points. This network made use of gated local recurrence, with LSTMs and GRUs proving superior to simple RNNs. Furthermore, the presence of specific global feedback connections in this network was critical for best predicting V4 neuron dynamics. In summary, we have developed a deep recurrent neural network architecture that accurately captures temporal dynamics in several ventral cortical areas, opening the door to more detailed computational study of the circuit structures underlying complex visual behaviors.

Feature-based encoding of face identity by single neurons in the human amygdala and hippocampus

Neurons in the human amygdala and hippocampus that are selective for the identity of specific peo... more Neurons in the human amygdala and hippocampus that are selective for the identity of specific people are classically thought to encode a person’s identity invariant to visual features (e.g., skin tone, eye shape). However, it remains largely unknown how visual information from higher visual cortical areas is translated into such a semantic representation of an individual person. Here, we show that some amygdala and hippocampal neurons are selective to multiple different unrelated face identities based on shared visual features. The encoded identities form clusters in the representation of a deep neural network trained to recognize faces. Contrary to prevailing views, these neurons thus represent an individual’s face with a visual feature-based code rather than one based on association with known concepts. Feature neurons encoded faces regardless of their identity, race, gender, familiarity, or pixel-level visual features; and the region of feature space to which feature neurons are ...

Download

James DiCarlo

Uploads

Papers by James DiCarlo

Log In