Download PDF
Aug 2026 – Present

Mohamed bin Zayed University of Artificial Intelligence · Kang Lab

Research Collaborator · Advisor: Prof. Jian Kang

  • Investigating decorative verification in reasoning LMs: whether self-checking language such as verification, backtracking, and reconsideration can occur without the causal computation needed to update model beliefs.
  • Studying which reasoning operations—including alternative exploration, contradiction detection, backtracking, and evidence integration—genuinely improve calibration versus merely signal epistemic caution.
Remote / Abu Dhabi, UAE
Jun 2024 – Aug 2024

University of California, Santa Cruz · AIEA Lab

Research Assistant · Advisor: Prof. Leilani H. Gilpin

  • Investigated trustworthy logical reasoning in LLMs by translating natural language outputs into Prolog-based logic constraints to mathematically verify consistency and detect hallucinations.
  • Designed a Modular RAG architecture to decouple knowledge retrieval from reasoning, reducing false positive hallucination detection by integrating external ground-truth validation.
Remote / Santa Cruz, CA
Aug 2023 – May 2024

Jaypee Institute of Information Technology

Undergraduate Researcher · Advisor: Prof. Sandeep Kumar Singh

  • Conducted initial experiments on parameter-efficient fine-tuning (Unsloth/QLoRA) which revealed limitations in complex logic repair, motivating a strategic pivot from training-based optimization to an inference-time reasoning framework that became the core contribution of the Repair of Thought publication.
Noida, India
2026

Confidently Wrong, Silently So: Auditing Undetectable Failures of a Deployed On-Device Language Model

Shashwat Pandey*, Satwik Pandey*, Suresh Raghu

Audited a deployed on-device language model for reliability and oversight, showing that confident-correct and confident-wrong outputs are surface-indistinguishable (AUROC 0.55); introduced a model-agnostic audit protocol and a black-box consistency wrapper that reduced confident confabulation from 75% to 3%.

(Preprint)
2026

Proper Scoring Rules for Agentic Uncertainty Quantification

Suresh Raghu*, Satwik Pandey*, Shashwat Pandey

arXiv:2605.24756 | Accepted (poster) at CTB & FAGEN Workshops @ ICML 2026

Introduced a family of strictly proper trajectory-level scoring rules for evaluating uncertainty in LM agents, with a censored-trace extension and negative results establishing that standard trajectory-level calibration metrics (ECE, Brier) are not strictly proper in the agentic setting.

(Preprint)
2026

SELFDOUBT: Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratio

Satwik Pandey*, Suresh Raghu*, Shashwat Pandey

arXiv:2604.06389 | Accepted (poster) at the FAGEN Workshop @ ICML 2026 | Code

Proposed an O(1) black-box uncertainty framework that extracts behavioral hedge/verify signals from reasoning traces, significantly outperforming Semantic Entropy on discrimination (p=0.001) at 10× lower cost; a zero-hedge gate achieves 96.1% precision across 7 models and 3 benchmarks.

(Preprint)
2026

Don't Blink: Evidence Collapse during Multimodal Reasoning

Suresh Raghu*, Satwik Pandey*

arXiv:2604.04207 | Accepted (poster) at AIW & SCI-FM Workshops @ COLM 2026 | Code

Identified a universal evidence collapse phenomenon in reasoning VLMs, observing visual attention drops up to 90.8% during generation and discovering a task-conditional failure regime where confident but visually disengaged predictions are hazardous on sustained visual reference tasks but benign on symbolic tasks.

(Preprint)
2026

Repair of Thought: Advancing Automated Program Repair through a Dual-Model Reasoning Framework

Satwik Pandey, et al.

Under review at the Journal of Systems and Software | Code

Introduced a function-level APR framework achieving a SOTA 83.1% plausible repair rate on Defects4J, with an automated verification pipeline combining AST alignment, control-flow symbolic analysis, and semantic checks.

(Under Review)

* Equal contribution

Aug 2021 – May 2025

Jaypee Institute of Information Technology

Bachelor of Technology in Computer Science · CGPA: 3.3 / 4.0

Noida, India
Focus
Natural Language Processing Large Language Models Uncertainty Quantification Reliability Interpretability Trustworthy AI AI Safety / Security
Dec 2024 – Present

VFS Global

AI Research Engineer

  • Investigated dynamic execution paths for multi-agent frameworks. Designed a complexity-based routing mechanism that differentiates between deterministic tasks handled by parsers and ambiguous instances routed to reasoning-capable VLMs, optimizing the trade-off between inference cost and reasoning accuracy.
  • Integrated a two-stage confidence pipeline: token-level logit probabilities aggregated via Shannon entropy serve as a lightweight first-pass filter for extraction uncertainty, with high-entropy predictions escalated to a reasoning VLM-as-a-Judge for cross-modal extraction verification.
  • Designed a trace-guided re-extraction mechanism for cascade-rejected fields that fail both entropy screening and judge validation. The judge's reasoning traces are converted into corrective context for a secondary extraction model, enabling targeted recovery from extraction errors such as character misreads, boundary confusion, and semantic misattribution.
  • Designed a multi-level verification framework for biometric photo validation, separating signal-level quality assessment from semantic-level attribute verification with fine-tuned VLMs. The staged design isolated distinct failure modes and improved the interpretability of rejection decisions.
New Delhi, India