Understanding the Mechanisms of Visual Processing in the Macaque Brain

On November 6-7, 2021, the 10th anniversary celebration of the IDG/McGovern Institute for Brain Research at Peking University and the International Forum on Brain Science successfully took place. Renowned scholars from various institutions, including Harvard University, Stanford University, UC Berkeley, University College London, Max Planck Institute, University of Tsukuba, Beijing Normal University, and the Chinese Academy of Sciences, passionately shared their latest progress in cutting-edge brain science research from multiple perspectives, including molecular, cellular, circuitry, systems, cognitive, psychological, computational, and neuropsychiatric disorders. They frequently interacted with online and offline audiences, sparking innovative ideas through the collision of thoughts. We are now gradually releasing related academic notes to revisit the highlights and embark on new journeys!

This academic note is based on the academic report titled “Studying macaque visual cortex with a large set of natural images” presented by Researcher Tang Shiming from the IDG/McGovern Institute for Brain Research at Peking University.

How should we begin to understand the principles of brain cognition, break through artificial intelligence, or construct a physical system that can perceive and think like the brain? The most basic intelligence starts from the sensory system, and perceptual invariance corresponds to the fundamental elements of thought—concepts—thus forming the cornerstone for the brain to construct knowledge systems and produce intelligence, similar to the underlying operating systems and assembly languages that, while difficult to understand, are essential for comprehending brain cognition. Over 80% of the information input to the human brain comes from vision, and the internal operations of brain cognition are primarily based on visual concepts. Vision is the most important sensory channel for higher animals. Research on the neural mechanisms of visual cognition is crucial for understanding the mysteries of brain intelligence and constructing brain-inspired computing to achieve breakthroughs in artificial intelligence technology. However, due to technical limitations, the number of neurons that can be recorded and the duration of recording in current studies of visual cortex neurophysiology are limited. Therefore, researchers often adopt a “hypothesis-driven” research model, selecting a relatively small number of artificial or natural image stimuli to collect neural response data. Using such biased stimulus sets can easily lead to misunderstandings of the visual cortex. To address this issue, Tang Shiming’s team developed long-term stable two-photon imaging technology and large-field single-photon imaging technology for awake monkeys. Using these technologies, the laboratory achieved recordings of neural responses to tens of thousands of stimulus images, thereby transforming the research paradigm into a “big data-driven” approach to explore the neural mechanisms of visual information processing.

The core of the “big data-driven” research paradigm is to collect neural response big data and establish quantitative models that can explain the response big data. The response big data comes from the neural responses to a large number of ImageNet natural images. By using deep learning models to summarize the response patterns in neural big data, we obtained a neural model with high response prediction accuracy. With such a quantitative model, we can perform various tests in the computer, including feature visualization and strong activation image search, to better understand the encoding characteristics of neurons.

Under this research paradigm, the laboratory conducted studies on neural responses at two scales: single cells and large-field neuron clusters.

In the single-cell research aspect, by using an expanded stimulus set, Tang Shiming’s team discovered that cells previously defined as “orientation-sensitive neurons” in the primary visual cortex exhibited more complex selectivity for patterns than traditional cognition. Furthermore, using long-term stable awake monkey two-photon imaging technology, the laboratory recorded neural responses to over 40,000 natural images over a continuous week of measurements, constructing a big data set of macaque V1 neurons’ responses to natural images. Based on this data, a convolutional neural network (CNN) model with a prediction accuracy of 76% was obtained, and feature visualization of the model showed V1’s selectivity for local structures. To prove that the high selectivity of neural responses is not an artifact introduced by calcium imaging, the team further validated the ultra-sparse representation of V1, V4, and IT neurons to natural images using the Loose-patch method.

In the research of large-field neuron clusters, Tang Shiming’s team first developed stable large-field calcium imaging technology for awake monkeys. To address the issue of light bleaching in long-term recordings, the team adopted an intermittent exposure measurement scheme. This large-field imaging technology demonstrated good response reliability and long-term stability in tests.

With this technology, the laboratory collected neural responses of V4 neuron populations to 19,900 natural images during a 5-day recording period, obtaining a corresponding neural response big data set. During this data collection, daily tests of 100 detection images were repeated to monitor the consistency of responses over multiple days.

Based on the obtained neural response data, the team used the “knowledge distillation + model fine-tuning” transfer training method to fit a high-precision neural model. The neural model supports larger-scale tests that are not permitted in computer experiments. The team tested the cortex’s responses to 50,000 natural images on the model and used the model’s predicted responses for the top 9 images to represent the response selectivity across various areas of the cortex, thus obtaining the functional map of the V4 brain region. The functional map reflects the partitioned encoding of rich natural image features, such as color, orientation, curves, dot arrays, grids, and faces, in the V4 brain region.

To further verify the functional map derived from the model, researchers experimentally tested the cortex’s responses to various natural images, such as curves, faces, and purple objects. The cortical responses matched well with the V4 functional map.

Through self-developed long-term stable two-photon imaging technology and large-field calcium imaging technology for awake monkeys, Tang Shiming’s team obtained big data on neural responses from the cerebral cortex using natural scene stimuli, thereby supporting a data-driven research paradigm. In this study, the research team aimed to gain a more comprehensive understanding of visual neural information processing while providing a new database for brain-inspired artificial intelligence. Ultimately, Researcher Tang Shiming’s vision is to explore how intelligence operates in our brains. This journey is long, and we look forward to knowledgeable individuals joining Tang Shiming’s research team to work together and advance.