Prithvijit Chattopadhyay

I am a Research Scientist at NVIDIA Research, focusing on Generative AI. I earned my Ph.D. in Computer Science in August 2024 at Georgia Tech, where I was advised by Prof. Judy Hoffman. During my Ph.D., I broadly worked on distribution shift problems in computer vision. My doctoral thesis (see here) was focused on utilizing synthetic data to train robust and reliable vision models. Specifically, I studied how we can use synthetic data for the following purposes:

• Measuring Robustness - Improving resistance to changing conditions requires exposing and understanding failure modes. Some of my work has focused on developing accessible benchmarks and datasets to better understand model behavior by comprehensively evaluating them under diverse distribution shifts and training conditions.
• Improving Generalization - Curating data that covers all anticipated test-time conditions may be infeasible. Some of my work has focused on developing robustness enhancing algorithms that can effectively leverage the nature of available training time data sources.
• Ensuring Reliability - While maintaining performance is important, practical deployment also requires ensuring reliable (confidence-calibrated) predictions under changing conditions. Some of my work has also focused on designing patches that ensure adaptation methods make calibrated and reliable predictions under distribution shifts.

[Past Life]

Past Life: Before this, I earned my Masters in Computer Science (awarded M.S. Research Award) in Spring 2019 from Georgia Tech, advised by Prof. Devi Parikh and Prof. Dhruv Batra, where I worked on a host of vision-language & ML problems – exploring human-AI teams to evaluate explanations, improving responses of & developing cooperative testing for conversational agents, zero-shot transfer, state abstraction in RL and scene-understanding. I earned my Bachelors in Electrical Engineering in 2016 from Delhi Technological University, India, where I worked on developing (now outdated) autonomous underwater vehicles (AUVs).

In past years, I have had the fortune to conduct research at PRIOR, Allen Institute of Artificial Intelligence (Summer 2022, Summer 2020) mentored by Ani Kembhavi & Roozbeh Mottaghi; Deep Learning Group, Microsoft Research Redmond (Summer 2018) mentored by Hamid Palangi; Robotics Research Lab, IIIT Hyderabad (Winter 2014) mentored by Dr. K. Madhava Krishna and Indian Association for the Cultivation of Science (IACS), Kolkata (Summer 2014) mentored by Dr. Soumitra Sengupta on a diverse set of topics - ranging from embodied AI, vision & language to physics.

I also actively participate in reviewing for top computer vision and machine learning conferences & workshops (have accumulated a few reviewer awards - CVPR 2023, CVPR 2022, CVPR 2021, ICLR 2022, MLRC 2021, ICML 2020, NeurIPS 2019, ICLR 2019, NeurIPS 2018 - in the process).

Accepted to ICCV 2023 Doctoral Consortium! Recognized as an outstanding reviewer for CVPR 2023! Recognized as an outstanding reviewer for CVPR 2022! Recognized as a highlighted reviewer for ICLR 2022! Recognized as an outstanding reviewer for CVPR 2021! Recognized as an outstanding reviewer for ML Reproducibility Challenge 2021! Recognized to be among the top 33 percent reviewers for ICML 2020! Likelihood Landscapes received the NVIDIA Best Runner Up paper award at AROW, ECCV 2020! Awarded the College of Computing's Rising Star Doctoral Student Research Award (formerly known as the CS7001 Research Award) at Georgia Tech! ... Recognized as one of the best reviewers for NeurIPS 2019! Recognized as an outstanding reviewer for ICLR 2019! Recognized to be among the top 30 percent highest scoring reviewers for NeurIPS 2018! Awarded the College of Computing's MS Research Award at Georgia Tech! Our team won VT Hacks 2017, a Major League Hacking Event, 2017! Our undergraduate team, DTU-AUV, qualified for the semi-finals at AUVSI Robosub 2014! Awarded Merit Scholarships from 2012-2014 for undergraduate academic performance! Selected for KVPY and INSPIRE Fellowships, 2012 for undergraduate studies in basic sciences! Placed among the top 1 percent students in the country in INPhO 2012! Selected for rigorous mathematical training camps conducted by mathematicians from Bhabha Atomic Research Center (BARC) and Indian Institute of Science (IISc) in 2012! Selected for CSIR Programme on Youth Leadership in Science, 2010! Read More

July 2024: Successfully defended my dissertation (slides here)! July 2024: SkyScenes accepted to ECCV 2024 (updated preprint coming soon). Dec 2023: We'll be presenting two main conference papers (Battle-of-the-Backbones, LANCE) at NeurIPS 2023. Aug 2023: I'll be presenting two main conference papers (Sim2Real Generalization, Low-Shot Robustness) and an extended abstract on calibration under domain shifts at ICCV 2023. Jan 2023: Invited talk on "Reliable Computer Vision" at Machine Perception, Google Research. May 2022: I'll be interning in the PRIOR Team at Allen Institute for Artificial Intelligence. May 2020: I'll be interning in the PRIOR Team at Allen Institute for Artificial Intelligence. Serving as a reviewer for CVPR 2018-23, ICRA 2021-22, ECCV 2018, NeurIPS 2018-23, ICLR 2019-22, ICML 2019-20, ACL 2019. ... May 2019: Presented our work on Discovery of Decision States through Intrinsic Control at the Task-Agnostic Reinforcement Learning (TARL) Workshop at ICLR 2019. May 2019: Completed my Masters in Computer Science (focus on Machine Learning) with a thesis centered on Evaluating Visual Conversational Agents in the Context of Human-AI Cooperative Games! Feb 2019: Our technical report describing EvalAI - an open source platform to evaluate and compare AI algorithms at scale - is out on ArXiv! Dec 2018: Presented our work interpretable zero-shot learning at the Continual Learning and Visually Grounded Interaction and Language (ViGIL) workshops at NeurIPS 2018. Aug 2018: Our paper titled 'Do explanation modalities make VQA models more predictable to a human?' was accepted in EMNLP, 2018! Jul 2018: Our paper titled 'Choose Your Neuron: Incorporating Domain Knowledge through Neuron-Importance' was accepted in ECCV, 2018! Apr 2018: I was awarded the College of Computing's MS Research Award at Georgia Tech! Feb 2018: I'll intern in the Deep Learning Group at Microsoft Research, Redmond in summer 2018. Aug 2017: Our paper titled 'Evaluating Visual Conversational Agents via Cooperative Human-AI Games' was accepted in HCOMP 2017 as an oral! Jul 2017: I will be joining Georgia Tech as a Masters of Computer Science student in Fall 2017. May 2017: I will be presenting 'Counting Everyday Objects in Everyday Scenes' at the LDV Vision Summit, 2017. Mar 2017: Our paper titled 'It Takes Two to Tango: Towards Theory of AI's Mind' is out on ArXiv! Feb 2017: Our paper titled 'Counting Everyday Objects in Everyday Scenes' was accepted in CVPR 2017 as a spotlight! Feb 2017: Our team built FilterAI - an image retrival engine - and won VT Hacks 2017, a Major League Hacking Event! Dec 2016: 'Counting Everyday Objects in Everyday Scenes' received an Amazon Academic Research Award, 2016! Read More

SkyScenes: A Synthetic Dataset for Aerial Scene Understanding Sahil Khose^*, Anisha Pal^*, Aayushi Agarwal^*, Deepanshi^*, Judy Hoffman, Prithvijit Chattopadhyay ECCV 2024 [PDF] [Data] [TL;DR] TL;DR, We introduce SkyScenes, a large-scale synthetic dataset of densely annotated aerial images captured from Unmanned Aerial Vehicle (UAV) perspectives. We carefully curate SkyScenes images from CARLA to comprehensively capture diversity across layout (urban and rural maps), weather conditions, times of day, pitch angles and altitudes with corresponding semantic, instance and depth annotations. Through our experiments using SkyScenes, we show that (1) Models trained on SkyScenes generalize well to different real-world scenarios, (2) augmenting training on real images with SkyScenes data can improve real-world performance, (3) controlled variations in SkyScenes can offer insights into how models respond to changes in viewpoint conditions, and (4) additionally incorporating other sensor modalities (depth) can improve aerial scene understanding.

AUGCAL: Improving Sim2Real Adaptation by Uncertainty Calibration on Augmented Synthetic Images Prithvijit Chattopadhyay, Bharat Goyal, Boglarka Ecsedi, Viraj Prabhu, Judy Hoffman ICLR 2024 Workshop on Uncertainty Quantification for Computer Vision, ICCV 2023 (Extended Abstract) [PDF] [TL;DR] TL;DR, Mispredictions made by Sim2Real adaptation methods on real data can often be attributed to “miscalibration” – often caused by overconfident predictions. We propose a simple patch, AugCal, to improve uncertainty calibration of existing Sim2Real adaptation methods. Given a base Sim2Real adaptation algorithm, at training time, AugCal involves replacing vanilla "Sim" images with strongly augmented views (Aug intervention) and additionally optimizing for a training time calibration loss on augmented "Sim" predictions (Cal intervention). Through our experiments, we empirically show the efficacy of AugCal across multiple adaptation methods, backbones, tasks and shifts.

We're Not Using Videos Effectively: An Updated Domain Adaptive Video Segmentation Baseline Simar Kareer, Vivek Vijaykumar, Harsh Maheshwari, Prithvijit Chattopadhyay, Judy Hoffman, Viraj Prabhu TMLR 2024 [PDF] [Code] [TL;DR] TL;DR, Domain Adaptive Semantic Segmentation (DAS) seeks to adapt a model trained on images from a labeled source domain to an unlabeled target domain. Unlike the traditional Image-DAS settings, a few Video-DAS works have sought to additionally leverage the temporal signal present in videos on a distinct set of benchmarks from Image-DAS, with minimal cross-benchmarking. We address this gap by conducting experiments that reveal that (1) even after carefully controlling for data and model architecture, state-of-the-art Image-DAS methods (HRDA and HRDA+MIC) outperform Video-DAS methods on established Video-DAS benchmarks (+14.5 mIoU on Viper→Cityscapes-Seq, +19.0 mIoU on Synthia-Seq→Cityscapes-Seq), and (2) naive combinations of Image-DAS and Video-DAS techniques only lead to marginal improvements across datasets.

Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks Micah Goldblum^*, Hossein Souri^*, Renkun Ni, Manli Shu, Viraj Uday Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Adrien Bardes, Mark Ibrahim, Judy Hoffman, Rama Chellappa, Andrew Gordon Wilson, Tom Goldstein NeurIPS Datasets and Benchmarks 2023 [PDF] [Code] [TL;DR] TL;DR, Most neural network based computer vision systems are built on a backbone, a pretrained or randomly initialized feature extractor. Several years ago, the default option was an ImageNet-trained convolutional neural network. However, the recent past has seen the emergence of countless backbones pretrained using various algorithms and datasets. We benchmark a diverse suite of pretrained models across a diverse set of computer vision tasks ranging from classification to object detection to OOD generalization and more. Our Battle of Backbones (BoB) sheds light on promising directions for the research community to advance computer vision by illuminating strengths and weakness of existing backbones through a comprehensive analysis conducted on 1500 training runs.

LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images Viraj Prabhu, Sriram Yenamandra, Prithvijit Chattopadhyay, Judy Hoffman NeurIPS 2023 [PDF] [code] [project page] [TL;DR] TL;DR, We propose an automated algorithm to stress-test a trained visual model by generating language-guided counterfactual test images (LANCE). Our method leverages recent progress in large language modeling and text-based image editing to augment an IID test set with a suite of diverse, realistic, and challenging test images without altering model weights.

Benchmarking Low-Shot Robustness to Natural Distribution Shifts Aaditya Singh, Kartik Sarangmath, Prithvijit Chattopadhyay, Judy Hoffman ICCV 2023 [PDF] [code] [TL;DR] TL;DR, Robustness to natural distribution shifts has seen remarkable progress thanks to recent pre-training strategies combined with better fine-tuning methods. However, such fine-tuning assumes access to large amounts of labelled data, and the extent to which the observations hold when the amount of training data is not as high remains unknown. We address this gap by performing the first in-depth study of robustness to various natural distribution shifts in different low-shot regimes: spanning datasets, architectures, pre-trained initializations, and state-of-the-art robustness interventions.

PASTA: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalization Prithvijit Chattopadhyay^*, Kartik Sarangmath^*, Vivek Vijaykumar, Judy Hoffman ICCV 2023 [PDF] [code] [TL;DR] TL;DR, PASTA is a simple and effective frequency domain augmentation strategy to improve out-of-the-box synthetic-to-real (syn-to-real) generalization performance. PASTA involves perturbing the amplitude spectra of the synthetic images in the Fourier domain in a structured manner to generate augmented views. For the tasks of semantic segmentation (GTAV→Real), object detection (Sim10K→Real), and object recognition (VisDA-C Syn→Real), across a total of 5 syn-to-real shifts, we find that PASTA either outperforms or is consistently competitive with more complex state-of-the-art methods while being complementary to other generalization approaches.

RobustNav: Towards Benchmarking Robustness in Embodied Navigation Prithvijit Chattopadhyay, Judy Hoffman, Roozbeh Mottaghi, Ani Kembhavi ICCV 2021 Oral presentation [PDF] [code] [project page] [video] [TL;DR] TL;DR, As an attempt towards assessing the robustness of embodied navigation agents, we propose RobustNav, a framework to quantify the performance of embodied navigation agents when exposed to a wide variety of visual – affecting RGB inputs – and dynamics – affecting transition dynamics – corruptions. We find that standard end-to-end RL policies significantly underperform (or fail) in the presence of visual or dynamics corruptions, warranting more research in this direction.

Likelihood Landscapes: A Unifying Principle Behind Many Adversarial Defenses Fu Lin, Rohit Mittapali, Prithvijit Chattopadhyay, Daniel Bolya, Judy Hoffman Adversarial Robustness in the Real World (AROW), ECCV 2020 NVIDIA Best Paper Runner Up [PDF] [video] [TL;DR] TL;DR, Convolutional Neural Networks (CNNs) have been shown to be vulnerable to adversarial examples, which are known to locate in subspaces close to where normal data lies but are not naturally occurring and have low probability. In this work, we investigate the potential effect defense techniques have on the geometry of the likelihood landscape - likelihood of the input images under the trained model. We first propose a way to visualize the likelihood landscape by leveraging an energy-based model interpretation of discriminative classifiers. Then we introduce a measure to quantify the flatness of the likelihood landscape. We observe that a subset of adversarial defense techniques results in a similar effect of flattening the likelihood landscape. We further explore directly regularizing towards a flat landscape for adversarial robustness.

Learning to Balance Specificity and Invariance for In and Out of Domain Generalization Prithvijit Chattopadhyay, Yogesh Balaji, Judy Hoffman ECCV 2020 Visual Learning with Limited Labels (LwLL), CVPR 2020 [PDF] [code] [video] [TL;DR] TL;DR, We introduce Domain-specific Masks for Generalization, a model for improving both in-domain and out-of-domain generalization performance. To produce a model which best generalizes to both seen and unseen domains, we propose learning domain specific masks (encouraged to learn a balance of domain-invariant and domain-specific features) enabling a model to benefit from the predictive power of specialized features while retaining the universal applicability of domain-invariant features. We demonstrate competitive performance compared to naive baselines and state-of-the-art methods on both PACS and DomainNet.

Improving Generative Visual Dialog by Answering Diverse Questions Vishvak Murahari, Prithvijit Chattopadhyay, Dhruv Batra, Devi Parikh, Abhishek Das EMNLP 2019 Visual Question Answering and Dialog Workshop, CVPR 2019 [PDF] [code] [TL;DR] TL;DR, While generative visual dialog models trained with self-talk based RL perform better at the associated downstream task, they suffer from repeated interactions -- resulting in saturation in improvements as the number of rounds increase. To counter this, we devise a simple auxiliary objective that incentivizes Q-Bot to ask diverse questions, thus reducing repetitions and in turn enabling A-Bot to explore a larger state space during RL i.e., be exposed to more visual concepts to talk about, and varied questions to answer.

IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL Nirbhay Modhe, Prithvijit Chattopadhyay, Mohit Sharma, Abhishek Das,
Devi Parikh, Dhruv Batra, Ramakrishna Vedantam IJCAI 2020 Workshop on Task Agnostic Reinforcement Learning (TARL), ICLR 2019 [PDF] [TL;DR] TL;DR, We propose a novel framework to identify subgoals useful for exploration in sequential decision making tasks under partial observability. We utilize the variational intrinsic control framework (Gregor et.al., 2016) which maximizes empowerment – the ability to reliably reach a diverse set of states and show how to identify sub-goals as states with high necessary option information through an information theoretic regularizer. Despite being discovered without explicit goal supervision, our subgoals provide better exploration and sample complexity on challenging grid-world navigation tasks compared to supervised counterparts in prior work.

EvalAI: Towards Better Evaluation Systems for AI Agents Deshraj Yadav, Rishabh Jain, Harsh Agrawal, Prithvijit Chattopadhyay, Taranjeet Singh,
Akash Jain, Shiv Baran Singh, Stefan Lee, Dhruv Batra Workshop on AI Systems, SOSP 2019 [PDF] [code] [TL;DR] TL;DR, We introduce EvalAI, an open source platform for evaluating and comparing machine learning (ML) and artificial intelligence algorithms (AI) at scale. EvalAI is built to provide a scalable solution to the research community to fulfill the critical need of evaluating machine learning models and agents acting in an environment against annotations or with a human-in-the-loop. This will help researchers, students, and data scientists to create, collaborate, and participate in AI challenges organized around the globe.

Choose Your Neuron: Incorporating Domain-Knowledge through Neuron-Importance Ramprasaath R. Selvaraju^*, Prithvijit Chattopadhyay^*, Mohamed Elhoseiny,
Tilak Sharma, Dhruv Batra, Devi Parikh, Stefan Lee ECCV, 2018 Continual Learning Workshop, NeurIPS 2018 Visually Grounded Interaction and Language (ViGIL) Workshop, NeurIPS 2018 [PDF] [code] [article] [TL;DR] TL;DR, We introduce a simple, efficient zero-shot learning approach -- NIWT -- based on the observation that individual neurons in CNNs have been shown to implicitly learn a dictionary of semantically meaningful concepts (simple textures and shapes to whole or partial objects). NIWT learns to map domain knowledge about "unseen" classes onto this dictionary of learned concepts and optimizes for network parameters that can effectively combine these concepts - essentially learning classifiers by discovering and composing learned semantic concepts in deep networks.

Do explanation modalities make VQA Models more predictable to a human? Arjun Chandrasekaran^*, Viraj Prabhu^*, Deshraj Yadav^*, Prithvijit Chattopadhyay^*, Devi Parikh EMNLP 2018 [PDF] [TL;DR] TL;DR, A rich line of research attempts to make deep neural networks more transparent by generating human-interpretable 'explanations' of their decision process, especially for interactive tasks like Visual Question Answering (VQA). In this work, we analyze if existing explanations indeed make a VQA model -- its responses as well as failures -- more predictable to a human.

Evaluating Visual Conversational Agents via Cooperative Human-AI Games Prithvijit Chattopadhyay^*, Deshraj Yadav^*, Viraj Prabhu, Arjun Chandrasekaran, Abhishek Das, Stefan Lee, Dhruv Batra, Devi Parikh HCOMP 2017 Oral presentation [PDF] [code] [TL;DR] TL;DR, We design a cooperative game - GuessWhich - to measure human-AI team performance in the specific context of the AI being a visual conversational agent. GuessWhich involves live interaction between the human and the AI and is designed to gauge the extent to which progress in isolated metrics for AI (& AI-AI teams) transfers to human-AI collaborative scenarios.

It Takes Two to Tango: Towards Theory of AI's Mind Arjun Chandrasekaranu^*, Deshraj Yadav^*, Prithvijit Chattopadhyay^*, Viraj Prabhu^*, Devi Parikh Chalearn Looking at People Workshop, CVPR 2017 [PDF] [code] [TL;DR] TL;DR, To effectively leverage the progress in Artificial Intelligence (AI) to make our lives more productive, it is important for humans and AI to work well together in a team. In this work, we argue that for human-AI teams to be effective, in addition to making AI more accurate and human-like, humans must also develop a theory of AI's mind (ToAIM) - get to know its strengths, weaknesses, beliefs, and quirks.

Counting Everyday Objects in Everyday Scenes Prithvijit Chattopadhyay^*, Ramakrishna Vedantam^*, Ramprasaath R. Selvaraju, Dhruv Batra, Devi Parikh CVPR 2017 Spotlight presentation [PDF] [code] [TL;DR] TL;DR, We study the numerosity of object classes in natural, everyday images and build dedicated models for counting designed to tackle the large variance in counts, appearances, and scales of objects found in natural scenes. We propose a contextual counting approach inspired by the phenomenon of subitizing - the ability of humans to make quick assessments of counts given a perceptual signal, for small count values.

Investigating Visual Dialog Models for Goal-Driven Self-Talk Prithvijit Chattopadhyay (advised by Devi Parikh) 2019 [Report PDF]

Exploring Weak-Supervision and Generative Models for Semantic Segmentation 2018 Prithvijit Chattopadhyay, Ramprasaath R. Selvaraju, Viraj Prabhu [Report PDF]

DTU AUV: Autonomous Underwater Vehicle Prithvijit Chattopadhyay (Acoustics & Control Systems Department) (co-authored with DTU AUV members) 2012-2016 [Report PDF]

Evaluating Visual Conversational Agents in the Context of Human-AI Cooperative Games Masters in Computer Science (specialization Machine Learning) 2017-2019 [PDF]

Reviewing: CVPR 2018-23, ICCV 2023, ICRA 2021-22, ECCV 2018, NeurIPS 2018-21,23, ICLR 2019-22, ICML 2019-20, ACL 2019, TPAMI