Zeyuan Li is a PhD student in Information Technology at MIT Sloan.

Frontier LLM agents are increasingly being deployed to transact on behalf of separate principals, often using natural language rather than structured APIs. Much of the safety literature studies misaligned LLM behavior through adversarial-elicitation evaluations on single agents or stylized tasks. Its prevalence and structure in settings that combine long horizons, separate principals, real operational state, and inter-agent natural-language exchange remain insufficiently measured. We study 2,583 inter-agent emails from 20 one-year simulation runs of Vending-Bench Arena, a competitive vending environment spanning 13 frontier LLMs. We operationalize speech-act misalignment as emails containing false factual claims, manipulation, collusion, or threats, combining message content with ground-truth simulator state and logged reasoning traces to classify and validate such behavior. Under our primary classifier, 12.6% of emails are labeled misaligned; misalignment appears in all 20 runs and in 74.7% of individual agent-runs. Both the magnitude and the composition of this misalignment are preserved under repeated classification at different sampling temperatures and under full-pipeline replication with judges from two other frontier-model families. However, available reasoning summaries rarely confirm explicit misaligned intent. Misalignment also shows reciprocal and stress-conditioned patterns: receiving a misaligned email from a counterparty raises the odds of a misaligned reply by 1.65×, and low-inventory conditions raise them by 1.58×. Across tests of capability-asymmetric exploitation, we find no evidence that higher-capability models differentially exploit weaker counterparties, and model performance rank does not predict misalignment rates. Together, these results indicate that measurable, state-dependent misalignment can arise in competitive multi-agent environments without engineered elicitation, in patterns associated with operational scarcity and counterparty behavior rather than model capability alone.

Kunal Jha is a third-year Computer Science and Engineering Ph.D. student at the University of Washington, advised by Natasha Jaques and Max Kleiman-Weiner. Supported by the CSERF Fellowship, ARCS Scholarship, Sony Focused Research Award, and Cooperative AI Early Career Grant, his research focuses on multi-agent systems, social learning, and emergence, drawing from cognitive science, economics, and physics. He holds a B.A. in Computer Science and Philosophy from Dartmouth College and previously spent two years as a visiting researcher at MIT. Outside of research, he enjoys mixed martial arts, basketball, and drumming.

Large language models (LLMs) can now improve themselves by revising the instructions they follow, and LLM agents are increasingly orchestrated to work together on complex problems. However, self-improvement methods typically optimize one system at a time, and multi-agent frameworks often have every model work toward a shared goal. We ask a different question. When each agent pursues its own reward, can self-improving LLMs learn from one another well enough to improve the whole population? We call this capability recursive social improvement. We study populations that revise skill files and choose whether, when, and whom to copy from. Independent search, learning from peers, and acting all share one token budget. In controlled environments, established social-learning algorithms benefit from peers, but three LLMs do not. They earn less reward per token than solo learners, and explore too narrowly or run out of tokens before acting. We then let the models write and revise their own skills. Observing peers changes how they improve, helping one model find useful skills sooner and another spend less on private search. Neither, however, outperforms independent learners at the same cost. Skills are copied, revised, and passed on, so one discovery can seed further search. Yet these exchanges concentrate the population around fewer independent discoveries. Together, these results show that LLMs can make learning more efficient by copying from peers, but not yet more effective.

Dimitris Michailidis is a postdoctoral researcher at the University of Amsterdam, studying the effects of link-recommendation algorithms on social networks through simulations and online experiments, and working on generative social simulation with LLM-based agents.

Generative Social Simulation (GSS) leverages Large Language Models (LLMs) to create sophisticated personas that define agents with demographics, emotions, and political opinions. While prior work shows that personas compress outputs into stereotypical caricatures, there is no comprehensive study of how these caricatures evolve into collective artifacts when they interact. Using a Twitter-like GSS platform with personas derived from the American National Elections Study (ANES), we ablate three layers of partisan identity: affective polarization markers, party identification, and vote choice, while retaining demographics, ideological positioning and important issues. We find that collective outcomes depend heavily on how survey responses are translated into personas. Removing a few explicitly partisan statements moves the network from near-complete segregation to cross-party ties approaching those of real Twitter networks. We argue that matching one outcome is not evidence for behavioral validity and point to salience as a possible explanation. Surveys ask for a position on every question regardless of its importance to the respondent, and persona construction turns each answer into a direct behavioral instruction.

Tan Zhi Xuan is an Assistant Professor in the Department of Computer Science of the National University of Singapore. Xuan’s research focuses on scaling cooperative intelligence via the rational analysis and design of intelligent systems. Together with her research group, the Cooperative Systems & Intelligence (CoSI) lab, Xuan aims to enable the development of AI systems that can cooperate as proficiently as humans, and to design institutions and incentives that promote mutual benefit and flourishing in an increasingly automated future.

One plausible extrapolation of current AI trends is that we are headed towards pluralistic AI futures: scenarios where a plethora of AI agents, systems and services are deployed towards various ends by various actors. How can we influence this process so that it occurs not just safely — avoiding AI-powered conflict and anti-human equilibria — but also in the direction of mutual flourishing? In order to address this challenge, I argue that we need to scale cooperative intelligence, advancing the cooperative capacities and dispositions of the agents we create, while creating institutions and incentives that promote AI and human cooperation at scale. To develop these capacities, I will introduce recent research on rational cooperative AI — a paradigm for building coherent AI agents that are cooperative-by-design. Across domains like assistive planning and natural language contracting, rational cooperative AI avoids the jaggedness and unreliability of current generative AI. Taking a rational approach also allows us to build “self-governing” AI agents that learn to follow and enforce societal norms. I will close by discussing the deeper connections between reasoning, rationality, and cooperation, and how a cooperative theory of reasoning could ultimately lead us to more human-aligned reasoners.

Andrea Baronchelli is Professor of Complexity Science at City St George’s, University of London. He investigates how interacting agents, human and artificial, develop and inhabit shared realities. These are the cognitive, social, and economic structures through which groups coordinate, assign meaning, and create value. His approach combines large-scale data analysis, mathematical modelling, and controlled experiments with human participants and AI agents. Andrea’s work has been published in journals including Nature, Science, PNAS, and Nature Human Behaviour, and has been widely covered in the press, informing public debate and helping shape policy.

Social conventions are the foundation of social coordination, shaping how individuals come together to form a society. In this talk, I will present theoretical and experimental findings that demonstrate the spontaneous emergence of social norms in human groups, as well as the existence of tipping points in social convention. I will then focus on the case of populations of large language models (LLMs). As AI agents increasingly communicate using natural language, understanding how they develop conventions is crucial for interpreting and managing their collective behaviour. I will show that LLM populations can establish social conventions and highlight how collective biases can emerge even when individual agents appear unbiased. I will conclude by showing how group size affects the dynamics in a non-linear way, revealing model-dependent dynamical regimes. These findings establish group size as a key driver of multi-agent dynamics and highlight the need to consider population-level effects when deploying LLM-based systems at scale.

After 6 months of stimulating STAMINA talks at the intersection of social tech, social science, and AI safety, we are organizing our 1st informational meeting! We’ll give overview of emerging STAMINA activities (our fresh slack workspace, upcoming conference workshops, shared tasks etc.) and discuss what the STAMINA interested community members are looking for. Looking forward to seeing you there!

Elias Stengel-Eskin is an Assistant Professor of Computer Science at the University of Texas at Austin. His research spans natural language processing, computational linguistics, and artificial intelligence, and focuses on developing AI agents that can intelligently communicate and collaborate with people and each other. This includes work on multi-agent communication and collaboration, converting language to action, grounding language to vision, and handling uncertainty, ambiguity, and underspecification. Before joining UT Austin, he was a Postdoctoral Research Associate at the University of North Carolina, Chapel Hill. He received a Ph.D. in Computer Science in 2023 from Johns Hopkins University and a B.A. & Sc. in Cognitive Science from McGill University in 2018.

As we scale from individual agents to teams of agents, inter-agent communication will become increasingly important. In this talk, I will describe a general paradigm for teaching multi-agent communication skills through multi-model reinforcement learning, which I will illustrate via three key collaborative skills: expressing confidence in a calibrated way, responding robustly to positive and negative persuasion, and expressing reasoning faithfully. I will show how these problems can be framed in terms of speaker-listener games, and how this framing allows us to teach models collaborative skills, often using games simulated on smaller models to train larger models.

Yunze (Lorenzo) Xiao is a Master’s student at Carnegie Mellon University’s Language Technologies Institute, advised by Prof. Mona Diab. His research aims to develop large language models that move beyond surface-level fluency toward genuine human-like intelligence, spanning anthropomorphism as a controllable modeling dimension, persona consistency, long-horizon memory, affective simulation, and multi-agent systems, with applications in education and therapy. He has published at ACL, EMNLP, and LREC-COLING, including InCharacter and ToxiCloakCN, and previously conducted research at SUTD and QCRI on multilingual NLP, propaganda detection, and AI safety. He co-organized the NeurIPS 2025 PersonaLLM workshop.

Applications based on large language models (LLMs), such as multi-agent simulations, require population diversity among agents. We identify a pervasive failure mode we term Persona Collapse: agents each assigned a distinct profile nonetheless converge into a narrow behavioral mode, producing a homogeneous simulated population. To quantify persona collapse, we propose a framework that measures how much of the persona space a population occupies (Coverage), how evenly agents spread across it (Uniformity), and how rich the resulting behavioral patterns are (Complexity). Evaluating ten LLMs on personality simulation (BFI-44), moral reasoning, and self-introduction, we observe persona collapse along two axes: (1) Dimensions: a model can appear diverse on one axis yet structurally degenerate on another, and (2) Domains: the same model may collapse the most in personality yet be the most diverse in moral reasoning. Furthermore, item-level diagnostics reveal that behavioral variation tracks coarse demographic stereotypes rather than the fine-grained individual differences specified in each persona. Counter-intuitively, the models achieving the highest per-persona fidelity consistently produce the most stereotyped populations. We release our toolkit and data to support population-level evaluation of LLMs.

Marwa Abdulhai is a PhD candidate at UC Berkeley advised by Sergey Levine. Her research focuses on enabling AI agents to better understand people and their interactions to build both safe and more AI capable systems. This includes improving the performance of existing large language models (LLMs) for multi-turn dialogue interactions, understanding how to protect against deception in AI systems, and exploring how AI can serve as a useful tool for social science research. Her research has been supported by the Quad Fellowship, AI Policy Hub, Open AI Research, and Cooperative AI PhD Fellowship.

Large Language Models (LLMs) are increasingly used to simulate human users in interactive settings such as therapy, education, and social role-play. While these simulations enable scalable training and evaluation of AI agents, off-the-shelf LLMs often drift from their assigned personas, contradict earlier statements, or abandon role-appropriate behavior. We introduce a unified framework for evaluating and improving consistency in LLM-generated dialogue with multi-turn RL, reducing inconsistency by over 55%, resulting in more coherent and trustworthy simulated users.

Ronen is a researcher and entrepreneur working on collective intelligence systems to help us think better, together. Ronen recently completed an Open Science fellowship at the Astera Institute, where he co-founded Cosmik, a mission driven R&D lab working on new kinds of social networks for collective sensemaking. Ronen also completed a PhD in computer science, with a focus on cognitive-inspired AI models for natural language comprehension. Ronen’s current research interests center around cooperative human-AI systems, institutional design for collective intelligence, and the role of epistemic environments in shaping human and machine intelligence.

What would social media look like if it were designed for sensemaking rather than engagement? We’re exploring this question with Semble, a platform where researchers curate shareable collections, create knowledge trails that others can build on, and discover relevant work through their network’s collective attention. Built on the AT Protocol, the open social networking protocol behind Bluesky, Semble offers researchers data portability and an open API designed for extension. We’ll discuss how Semble enables new kinds of research tooling, from living semantic citation graphs to collaborative review and annotation. We’ll also share how ATProto’s open data layer creates unique opportunities for studying and designing epistemic infrastructure — from observing how knowledge trails form across a network to experimenting with platform affordances that support collective sensemaking.

Zhijing Jin (she/her) is an Assistant Professor at the University of Toronto and Research Scientist at the Max Planck Institute. She serves as a CIFAR AI Chair, an ELLIS advisor, and a faculty member at the Vector Institute, and the Schwartz Reisman Institute. She co-chairs the ACL Ethics Committee, and the ACL Year-Round Mentorship. Her research focuses on Causal Reasoning with LLMs, and AI Safety in Multi-Agent LLMs. She has published over 80 papers and has received the ELLIS PhD Award, three Rising Star awards, and two Best Paper awards at NeurIPS 2024 Workshops.

While progress has been made in evaluating single-agent LLMs for persona modeling, the behavior of these models within multi-agent groups remains underexplored. This presentation outlines a research series dedicated to closing this gap by testing LLM cooperation through autonomous social simulations. Specifically, we ask: what happens when personas are tasked to interact and cooperate?

To answer this, we introduce a suite of simulation environments (GovSim, MoralSim, and SanctSim) designed to stress-test persona interaction. These environments simulate high-stakes scenarios, such as the tragedy of the commons and ethical trade-offs, allowing us to investigate whether simulated societies can autonomously negotiate social order and how personas with differing ethical constraints navigate social dilemmas.

Our findings highlight implications for persona modeling. We show that agents exhibit a functional “theory of mind,” capable of inferring the identities of their interlocutors and strategically adapting their behavior, sometimes exploiting specific model vulnerabilities. Furthermore, we discuss a counterintuitive phenomenon where advanced reasoning capabilities lead to exploitative behaviors that humans typically avoid, highlighting a significant misalignment between agent optimization and human social norms.

Marc-Antoine has worked in computational linguistics, knowledge representation, and more recently has focused on tools for augmented collective intelligence. He’s especially interested in how to represent emergent and disputed knowledge.

https://conversence.com

https://hyperknowledge.org

We present a model of the formation of a knowledge commons for democratic, collective decision-making in society, and explain how generative AI disrupts the formation of this knowledge commons. We will also present ways in which to reinforce the collective processes around a knowledge commons, including the possible contributions of hybrid AI.

This is based on a paper that was presented at the IJCAI’25 democracy and AI workshop.

Ryan is a Computer Scientist and Machine Learning researcher with a background in reinforcement learning and foundation models. He has worked as a Research Engineer over the past decade at Google Deepmind and he is also a PhD Student at the University of Toronto advised by Zhijing Jin. At GDM he works in the Concordia group led by Joel Leibo. At a high level his current research focus is on multi-agent systems, LLMs, and social learning. In this context he is interested in memory mechanisms, agent theory of mind, collective decision making, and simulating political systems.

Governing common-pool resources requires agents to develop enduring strategies through cooperation and self-governance to avoid collective failure. While foundation models have shown potential for cooperation in these settings, existing multi-agent research provides little insight into whether structured leadership and election mechanisms can improve collective decision making. The lack of such a critical organizational feature ubiquitous in human society presents a significant shortcoming of the current methods. In this work we aim to directly address whether leadership and elections can support improved social welfare and cooperation through multi-agent simulation with LLMs. We present a new framework that simulates leadership through elected personas and candidate-driven agendas and carry out an empirical study of LLMs under controlled governance conditions. Our experiments demonstrate that structured leadership can improve social welfare scores by 55.4% and survival time by 128.6% across a range of high performing LLMs. Through the construction of an agent social graph we compute centrality metrics to assess the social influence of leader personas and also analyze rhetorical and cooperative tendencies revealed through a sentiment analysis on leader utterances. This work lays the foundation for developing prosocial, self-governing multi-agent systems capable of navigating complex resource dilemmas.

Martin Weiss is Co-Founder of Tiptree Systems, a startup building AI agents that help ML researchers find, create, and share knowledge more efficiently. Tiptree is deployed to researchers across many top-tier institutes including Mila, ELLIS, MIT, and many more. Martin holds a PhD in AI from Mila, where he studied under Hugo Larochelle and Chris Pal. Before his PhD, he was an early employee at YesGraph, a social graph startup acquired by Lyft.

This talk examines three converging crises. First, the decoupling of control from comprehension — we can increasingly predict and manipulate systems without understanding why they work. Second, the collapse of the generator-verifier gap — AI makes it trivial to produce the aesthetics of deep thought. This makes peer review more difficult because we can no longer rely on easy-to-verify signals of work quality. Third, the credit assignment gap — our academic reward systems optimize for publication metrics, not the increase in understanding that a new paper produces.

Gian Marco Orlando is currently a PhD Student at the University of Naples Federico II. He earned his Master’s Degree in Computer Engineering from the University of Naples Federico II, graduating with honors. His thesis highlights his expertise in the intersection of Artificial Intelligence and Social Network Analysis. His research interests lie in Social Network Analysis, Agent-Based Modeling and Big Data Analytics.

Jinyi is a second-year CS PhD student co-advised by Dr. Emilio Ferrara and Dr. Luca Luceri. Her research lies in the intersection of computer science and social science, recently focusing on large-scale agentic simulations of human behavior, measuring and modeling collective dynamics in social networks, and empirical studies on AI and the future of work.

Mahdi Saeedi is currently exploring large-scale LLM simulations because he is fascinated by understanding how these models actually work under the hood. His background spans both the theoretical foundations and hands-on engineering of generative AI systems, which has been great preparation for this research. He have also spent time thinking about how people interact with AI through thoughtful interface design, since he believes making these tools intuitive and accessible is just as important as the underlying technology.

Generative agents are rapidly advancing in sophistication, raising urgent questions about how they might coordinate when deployed in online ecosystems. This is particularly consequential in information operations (IOs), influence campaigns that aim to manipulate public opinion on social media. While traditional IOs have been orchestrated by human operators and relied on manually crafted tactics, agentic AI promises to make campaigns more automated, adaptive, and difficult to detect. This work presents the first systematic study of emergent coordination among generative agents in simulated IO campaigns. Using generative agent-based modeling, we instantiate IO and organic agents in a simulated environment and evaluate coordination across operational regimes, from simple goal alignment to team knowledge and collective decision-making. As operational regimes become more structured, IO networks become denser and more clustered, interactions more reciprocal and positive, narratives more homogeneous, amplification more synchronized, and hashtag adoption faster and more sustained. Remarkably, simply revealing to agents which other agents share their goals can produce coordination levels nearly equivalent to those achieved through explicit deliberation and collective voting. Overall, we show that generative agents, even without human guidance, can reproduce coordination strategies characteristic of real-world IOs, underscoring the societal risks posed by increasingly automated, self-organizing IOs.