ArXiv: PAC-BENCH — What Happens When AI Agents Must Keep Secrets While Collaborating?

In the real world, AI agents from different organizations will need to collaborate, but they cannot freely share all data. PAC-BENCH is the first benchmark that systematically tests what happens when AI agents must collaborate while respecting privacy.

A problem nobody was measuring

Imagine a scenario: a hospital’s AI agent and a pharmaceutical company’s AI agent need to jointly analyze drug efficacy, but they must not share patients’ personal data. How do AI agents handle such situations? Until now, nobody had systematically measured this.

Three types of errors

The researchers identified three serious error patterns:

• Premature privacy disclosure — agents reveal confidential information in the early stages of collaboration that they should not be sharing
• Overly conservative abstraction — agents conceal data so aggressively that their responses become useless for collaboration
• Privacy-induced hallucinations — the most interesting finding: when agents cannot access real data due to restrictions, they start fabricating data that “sounds reasonable”

Why is this critical for the future?

As AI agents are increasingly used in business environments — from healthcare to finance — the question of privacy in collaboration becomes an unresolved challenge. PAC-BENCH results show that privacy significantly degrades collaboration quality and that outcomes become highly dependent on which agent initiates the communication.

This warns the industry that simply “adding privacy” to multi-agent systems is not a trivial task — it requires fundamentally different design approaches.