arXiv:2607.12463: function-aware FIM mid-training boosts coding agents up to +5.4 on SWE-Bench

What are FIM and mid-training?

FIM (Fill-in-the-Middle) is a training technique where the model fills in a code snippet between a given prefix and suffix — rather than just generating a continuation, it must understand the context that comes after. Mid-training is the phase between pre-training (on large corpora) and fine-tuning (on specific tasks): its goal is to shape the model’s foundations for a particular domain without destroying general capability. Yubo Wang and seven co-authors on arXiv combine these two techniques into a new method specifically for coding agents.

Function-aware FIM: the role of the dependency graph

Standard FIM masks random code snippets. The novelty of this research is the function-aware approach: masked regions are selected deliberately according to the program dependency graph that models dependencies between functions in the code. This means the model learns to fill in precisely those locations where understanding inter-function dependencies is critical — which corresponds to real-world agentic programming tasks. The training corpus spans 2.6 billion tokens from 968 GitHub repositories, providing a statistically representative sample of real codebases.

Results on SWE-Bench: concrete jumps

The method was validated on two standard benchmarks for agentic programming:

• SWE-Bench-Verified: improvement of +2.8 to +3.2 points on Qwen2.5-Coder (7B and 14B variants) and Qwen3-8B
• SWE-Bench-Lite: improvement of +3.7 to +5.4 points

Compare with typical fine-tuning improvements on specific tasks, which rarely exceed 2-3 points without regression on general abilities. Here the key point is that the method preserves non-agentic coding capabilities even alongside agentic post-training — the improvement is not bought at the expense of overall model quality.

Why this matters for small models

Results on 7B and 8B parameter models are particularly significant: they show that mid-training can compensate for reduced model capacity on agentic tasks that require understanding complex code dependencies. For organizations that cannot afford to run models with tens of billions of parameters, function-aware FIM offers a practical path to stronger coding agents while keeping inference costs in check.