top of page
01
Natural Vision
Neural processing during active, unconstrained interaction with the visual world.
+
OVERVIEW
Natural vision requires the brain to extract meaningful information from a continuously changing world while actively deciding where to look. We study how visual information is represented and transformed during spontaneous, real-world behavior, where perception unfolds through ongoing interactions among sensory input, eye movements, behavior, and internal state. By recording directly from the human brain during natural interactions, we aim to uncover the neural computations that support visual perception outside the constraints of traditional laboratory tasks.
KEY QUESTIONS
How does the brain represent people, objects, expressions, and movements during unconstrained visual behavior?
How do eye movements and active information sampling shape neural representations during natural vision?
How are moment-to-moment visual features transformed into distributed representations across cortical networks?
APPROACH
We combine intracranial electrophysiology with wearable eye tracking, video and audio recordings, computer vision, and machine-learning methods to study visual processing during spontaneous behavior. Neural activity is synchronized with detailed measurements of what participants see and where they look, allowing us to model relationships between dynamic visual information and millisecond-scale brain activity. This multimodal framework makes it possible to study neural representations and information flow as they emerge during natural interactions with friends, family, clinicians, and the surrounding environment.
Figure: A participant in the UPMC Epilepsy Monitoring Unit implanted with iEEG electrodes, secured with bandaging, and fitted with SensoMotoric Instrument’s (SMI) ETG 2 Eye Tracking Glasses that have been modified with an ergonomic Velcro strap. b) An over-the-shoulder view of the participant and the visual scene during an interaction with a researcher. c) Front (top) and back (bottom) view of the SMI ETG 2 Eye Tracking Glasses with the egocentric video camera (green circle) and inward facing eye-tracking cameras (red ellipses). d) A snapshot of the participant’s view (top) through the SMI ETG 2 Eye Tracking Glasses corresponding to b), and their eye movement (bottom) captured by the inward facing eye-tracking cameras. Alreja et. Al
SELECTED PUBLICATIONS
Alreja, A., Ward, M. J., Ma, Q., Russ, B. E., Bickel, S., Van Wouwe, N. C., . . . Ghuman, A. S. (2023). A new paradigm for investigating real-world social behavior and its neural underpinnings. Behavior Research Methods, 55(5), 2333-2352. doi:10.3758/s13428-022-01882-9
Alreja A, Ward MJ, Ma Q, Russ BE, Bickel S, Van Wouwe NC, González-Martínez JA, Neimat JS, Abel TJ, Bagić A, Parker LS, Richardson RM, Schroeder CE, Morency LP, Ghuman AS. A new paradigm for investigating real-world social behavior and its neural underpinnings. Behav Res Methods. 2023 Aug;55(5):2333-2352. doi: 10.3758/s13428-022-01882-9. Epub 2022 Jul 25. PMID: 35877024; PMCID: PMC10841340.
02
Brain Dynamics
Across Days
The organization of human brain states across minutes, hours, and days.
+
OVERVIEW
Whether we are fatigued or energetic, anxious or happy, cognition and related neurocognitive processes fluctuate over minutes to hours. Yet much of what we know about human brain function comes from experiments spanning milliseconds to seconds, or longitudinal studies spanning months to years. We leverage the rare opportunity provided by continuous intracranial recordings to study brain network dynamics across minutes, hours, and days during natural behavior.
KEY QUESTIONS
How does the brain maintain stable neurocognitive states while remaining flexible enough to rapidly adapt as behavior changes?
What neural dynamics organize cognition and behavior across minutes, hours, and days?
How are long-timescale brain dynamics related to sleep, physiology, arousal, and ongoing behavior?
APPROACH
We combine continuous, multi-day iEEG recordings with rich behavioral, video, wearable, and physiological measurements to study brain dynamics during natural behavior. Using recurrent neural networks, dynamical-systems approaches, and network analyses, we characterize how neural activity evolves across long timescales and identify the processes that allow the brain to balance stability with rapid, flexible transitions between behavioral and neurocognitive states.
Figure: Neural dynamics undergoes bursts of high transitory speed when natural behavior shifts.
(A-left) Network activations plotted over the week for one participant where each color represents the activity of a different network (sum of activity normalized to one for visualization purposes) along with how quickly the brain changed network activations every five seconds. (A-right) The average time between windows with the top 10% of transition speed across all participants is shown in blue vs the expected time if windows of high speed occurred via homogenous Poisson process (λ=0.1). Wang et. al
SELECTED PUBLICATIONS
MB Wang, M G’Sell, RM Richardson, and AS Ghuman (In Press). A Week in the Life of the Human Brain: Stable States Punctuated by Chaotic-Like Transitions. Nature Communications.
03
Social & Affective Perception
Neural representations of faces, expressions, bodies, and socially meaningful information.
+
OVERVIEW
We use information from a person’s face, body, voice, and movements to infer their identity, actions, intentions, and emotional state. We study how the brain integrates these sources of information to support social and affective perception, with a particular focus on the dynamics and distributed neural representations underlying face and expression processing.
KEY QUESTIONS
How does the brain transform visual information about faces and bodies into representations of identity, expression, and social meaning?
How are different stages of social and affective information processing organized across time and across brain regions?
To what extent are identity, expression, body, voice, and other socially relevant signals processed independently or through shared neural representations?
APPROACH
We combine iEEG recordings from brain regions involved in social and affective processing with multivariate pattern analysis, computational modeling, and controlled behavioral experiments. These methods allow us to characterize when and where information about facial identity, expression, bodies, and other socially relevant features emerges, and to test how these representations interact across distributed neural systems.
Examples of body-to-face identity adaptation stimuli, including face morphs, gender specific bodies, and two adaptation trial sequences. (Ghuman et al., 2010)
SELECTED PUBLICATIONS
Li Y, Richardson RM, Ghuman AS. Posterior Fusiform and Midfusiform Contribute to Distinct Stages of Facial Expression Processing. Cereb Cortex. 2019 Jul 5;29(7):3209-3219. doi: 10.1093/cercor/bhy186. PMID: 30124788; PMCID: PMC6611455.
Schwartz et al., 2023 — Intracranial Electroencephalography and Deep Neural Networks Reveal Shared Substrates for Representations of Face Identity and Expressions
04
Visual Recognition
Dynamic neural computations underlying recognition of faces, words, and objects.
+
OVERVIEW
Our remarkable capacity to rapidly recognize faces, words, objects, and other meaningful visual information depends on distributed regions of temporal, occipital, and parietal cortex that are tuned to different visual categories. We study the dynamic information-processing stages these networks use to transform visual input into increasingly specific and behaviorally meaningful representations.
KEY QUESTIONS
How does neural activity evolve from coarse category information into more specific representations of visual identity and meaning?
How do distributed category-selective regions interact over time to support recognition of faces, words, and objects?
What features of neural population activity carry and organize visual information across different stages of processing?
APPROACH
We combine human intracranial electrophysiology with multivariate machine learning, advanced signal processing, and causal perturbation to characterize visual information processing with high spatial and temporal precision. By decoding how information represented in neural activity changes from moment to moment, and examining its temporal and spectral organization, we aim to determine how category-selective brain networks dynamically construct representations of faces, words, and objects.
Figure: Results from one electrode during our visual localizer experiment.
SELECTED PUBLICATIONS
Ghuman et al., 2014 Dynamic encoding of face information in the human fusiform gyrus
Nature Communications
Boring et al., 2021. Multiple Adjoining Word- and Face-Selective Regions in Ventral Temporal Cortex Exhibit Distinct Dynamic
05
Brain-Body
Interactions
Coupled brain, behavioral, and physiological dynamics across multiple timescales.
+
OVERVIEW
Brain function does not occur in isolation from the body. Neural activity continuously interacts with physiological processes—including cardiac, respiratory, metabolic, autonomic, and circadian rhythms—that fluctuate over seconds, minutes, hours, and days. We study how these reciprocal brain–body interactions shape brain network dynamics and excitability, and how their coordinated fluctuations can move the system between different cognitive, physiological, and pathological states.
KEY QUESTIONS
How do ongoing physiological rhythms interact with brain network activity and excitability across multiple timescales?
How do coupled changes in brain, behavior, and physiology give rise to transitions between neurocognitive states?
Can the dynamics of the brain–body system reveal when the brain is entering a state of increased vulnerability to pathological events such as seizures?
APPROACH
We are developing a multimodal framework that synchronizes continuous intracranial electrophysiology with wearable physiology, behavioral measurements, and molecular biomarkers across extended periods of natural behavior. We combine advanced signal processing with machine learning and dynamical-systems approaches to model interactions among these signals and identify the latent states that organize brain–body dynamics. These methods allow us to ask how physiological processes shape neural excitability and whether changes in the coupled system create windows in which the brain is more or less vulnerable to transitions into pathological states.
SELECTED PUBLICATIONS
New Grant- Coming Soon!
06
Methods for Neural Information Processing
Computational methods for decoding neural representations, communication, and dynamical structure.
+
OVERVIEW
How do brain areas communicate with one another? What information do they exchange, and what neural codes support that exchange? We develop computational methods to characterize the information represented within neural populations and the interactions among distributed brain regions. Our goal is to move beyond identifying where neural activity occurs to understand what information that activity carries, how it changes over time, and how it is transformed across brain networks.
KEY QUESTIONS
What information is encoded in patterns of neural population activity?
How is information exchanged and transformed between interacting brain regions?
How can machine learning and dynamical-systems approaches reveal computational structure that is not apparent from neural activity alone?
APPROACH
We combine multivariate machine learning, advanced signal processing, network analysis, and dynamical-systems methods to characterize neural representations and communication. These approaches include decoding the information contained in population activity, modeling the representational structure of interactions between brain regions, and learning low-dimensional dynamical structure from high-dimensional neural recordings. We develop these tools both to answer specific questions about human cognition and to provide more general frameworks for understanding information processing in distributed neural systems.
Figure: iEEG experiments and Multi-Connection Pattern Analysis results. (a) Location of the electrodes of interest. The blue dot corresponds to the location of the FFA contact while the red dot corresponds to the location of the OFA contacts. (b) MCPA applied between (1) the OFA and FFA channels, (2) the FFA channel and the control channel, (3) the OFA channel and the control channel. The mean d′ of pairwise face classification over all 2415 pair of faces across the 250–450 ms timewindow after stimulus onset is plotted. *p < 0.01, permutation test. (c) Averaged absolute loading weights in the functional connectivity model of MCPA for OFA and FFA across the frequency spectrum during the time window of 250–450 ms after stimulus onset. (chance level: d′ = 0, accuracy = 50%). Li et al
SELECTED PUBLICATIONS
Ghuman, McDaniel & Martin, 2011
A wavelet-based method for measuring the oscillatory dynamics of resting-state functional connectivity in MEG
NeuroImage
Li, Richardson & Ghuman, 2017
Multi-Connection Pattern Analysis: Decoding the representational content of neural communication
NeuroImage
Natural Vision recording session in the EMU
Recorded Experiment: Letter misperception with stimulation to the left mid-fusiform
All individuals featured in footage provided written consent to public presentation of these materials
bottom of page