All Posts

BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems

Andy K. Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, et al.

We introduce BountyBench, a cybersecurity benchmark featuring 25 systems with complex, real-world codebases, and 40 bug bounties that cover 9 of the OWASP Top 10 Web Application Security Risks.

Stanford AI Lab Papers and Talks at CVPR 2025

Compiled by Ruhana Azam

All the great work from the Stanford AI Lab accepted at CVPR 2025, all in one place.

Demystifying Verbatim Memorization in Large Language Models

Jing Huang, Diyi Yang, Christopher Potts

How do LLMs memorize long sequences of texts verbatim? In this work, we show that verbatim memorization is intertwined with the LM’s general capabilities.

Stanford AI Lab Papers and Talks at NAACL 2025

Compiled by Nitya Thakkar

All the great work from the Stanford AI Lab accepted at NAACL, all in one place.

Stanford AI Lab Papers and Talks at ICLR 2025

Compiled by Megha Srivastava

All the great work from the Stanford AI Lab accepted at ICLR 2025, all in one place.

MENTAT: A Clinician-Annotated Benchmark for Complex Psychiatric Decision-Making

Max Lamparth and Declan Grabb

We developed a new expert design and annotated clinical decision-making dataset that also allows for nuanced accuracy and fairness evaluations with expert preferences, uncertainty, and soft labels.