profile photo

Nick Crispino

PhD Student in Natural Language Processing

📚 Second-year PhD Student
👨‍🏫 Advisor: Professor Chenguang Wang
💻GitHub 🐦Twitter 📚Scholar 📄CV • Research Open Source

Research

I'm broadly interested in work on foundation models, agentic AI, and LLM alignment & safety.

Currently, I am spending a lot of time improving MassGen, an open-source multi-agent scaling system for LLMs. Please reach out to us on Discord if you are interested in contributing to the project!

project image

VMDT: Decoding the Trustworthiness of Video Foundation Models


Yujin Potter*, Zhun Wang*, Nicholas Crispino*, Kyle Montgomery*, Alexander Xiong*, Ethan Y. Chang, Francesco Pinto, Yuqi Chen, Rahul Gupta, Morteza Ziyadi, Christos Christodoulopoulos, Bo Li, Chenguang Wang, Dawn Song
in Thirty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS 2025)., 2025
paper / code / dataset

VMDT is a benchmark for evaluating the trustworthiness of text-to-video (T2V) and video-to-text (V2T) models across five key dimensions: safety, hallucination, fairness, privacy, and adversarial robustness.
project image

LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess


Sai Kolasani, Maxim Saplin, Nicholas Crispino, Kyle Montgomery, Jared Quincy Davis, Matei Zaharia, Chi Wang, Chenguang Wang
in Workshop on Foundations of Reasoning in Language Models (FoRLM @ NeurIPS 2025), 2025
paper / code

LLM CHESS is an evaluation framework designed to probe the generalization of reasoning and instruction-following abilities in LLMs through extended agentic interaction in the domain of chess.
project image

SteeringSafety: A Systematic Safety Evaluation Framework of Representation Steering in LLMs


Vincent Siu*, Nicholas Crispino*, David Park, Nathan W. Henry, Zhun Wang, Yang Liu, Dawn Song, Chenguang Wang
in Workshop on Socially Responsible and Trustworthy Foundation Models (ResponsibleFM @ NeurIPS 2025), 2025
arxiv / code / dataset

SteeringSafety is a benchmark measuring the behavioral entanglement that occurs on out of distribution datasets when using current representation steering methods to improve performance on a single behavior, along with a modular code framework allowing for standardization of training-free steering methods.
project image

RepIt: Steering Language Models with Concept-Specific Refusal Vectors


Vincent Siu, Nathan W. Henry, Nicholas Crispino, Yang Liu, Dawn Song, Chenguang Wang
in Workshop on Socially Responsible and Trustworthy Foundation Models (ResponsibleFM @ NeurIPS 2025), 2025
arxiv / code

RepIt isolates concept-specific representations of refusal that can be used to suppress refusal on target concepts while preserving the baseline model’s refusal on other concepts using steering methods in interpretability.
project image

COSMIC: Generalized Refusal Identification in LLM Activations


Vincent Siu, Nicholas Crispino, Zihao Yu, Sam Pan, Zhun Wang, Yang Liu, Dawn Song, Chenguang Wang
in Findings of the Association for Computational Linguistics (ACL), 2025
arxiv / code

COSMIC improves the direction selection step of steering refusal in LLMs by choosing a direction maximizing the cosine similarity between paired harmless and harmful behavior, allowing for the more robust application of activation steering.
project image

MLAN: Language-Based Instruction Tuning Improves Zero-Shot Generalization of Multimodal Large Language Models


Jianhong Tu, Zhuohao Ni, Nicholas Crispino, Zihao Yu, Michael Bendersky, Beliz Gunel, Ruoxi Jia, Xin Liu, Lingjuan Lyu, Dawn Song, Chenguang Wang
in Proceedings of the 3rd Workshop on Towards Knowledgeable Foundation Models (KnowFM @ ACL), 2025
arxiv / code

MLAN proposes focusing on text-only instances in instruction tuning to improve instruction following in both the vision and language modalities in multimodal large language models at a lower cost than traditional vision-only or vision-based methods.
project image

Humanity’s Last Exam


Long Phan, ..., Wenjin Zhang, Nick Crispino, Chenguang Wang, Daofeng Li, Jiawei Shen, Kyle Montgomery, Hannah Szlyk, Ting Wang, Summer Yue, Alexandr Wang, Dan Hendrycks, many others.
in arXiv:2501.14249, 2025
arxiv / code / dataset

HLE is a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage
project image

Agent Instructs Large Language Models to be General Zero-Shot Reasoners


Nicholas Crispino, Kyle Montgomery, Fankun Zeng, Dawn Song, Chenguang Wang
in Forty-first International Conference on Machine Learning (ICML), 2024
arxiv / code

Zero-shot AgentInstruct uses an agent to generate dataset-specific instructions to improve the zero-shot performance of instruction-following large language models.



Open Source Projects

Contributing to open source projects that advance LLM capabilities and allow research ideas to be shared with the broader community.

MassGen logo

MassGen - Multi-Agent LLM Scaling Framework


Lead Contributor
GitHub

I am one of the lead contributors to MassGen, an open-source framework for scaling multi-agent LLM systems. My work focuses on designing and implementing new features that enhance the framework's capabilities for agent collaboration and scalability.





Design and source code from Jon Barron's website. Template from Leonid Keselman. Modifications from me and Claude Code ✨.