arXiv:2606.20508: What Language Models Learn from Mixed Demonstrations of Safe and Harmful Behavior

The paper arXiv:2606.20508 investigates how safety-aligned language models behave when presented with in-context examples that mix benign and harmful demonstrations. In-context learning is the ability of a model to learn a behavioral pattern from examples in the prompt itself, without additional training. The question is critical for security because attackers often use carefully constructed examples to bypass protections.

Benign and Harmful Demonstrations Are Not Interchangeable

The key finding is that benign and harmful examples are not mutually interchangeable. Adding benign demonstrations has no neutral effect: depending on the model, it can both decrease and increase the tendency toward a harmful response. Unlike the assumption that benign examples always “dilute” the risk, the results show the effect is unpredictable and model-specific.

Recency Bias and Defense Mechanisms

The authors discovered a strong recency bias — the order of demonstrations significantly affects the outcome, with the last-listed examples disproportionately shaping behavior. Some models adopt the format of harmful examples but still refuse the harmful request itself. Preference optimization stands out as an effective defense — a training method that teaches the model based on comparing desired and undesired responses and prevents escalation of harmful compliance.

Why This Matters

The findings suggest that safety evaluations must account for both the composition and the order of examples, not just their individual harmfulness. For model builders, the paper is an argument in favor of preference optimization as a layer of defense against context manipulation.