arXiv:2607.16122: CRAFT Method Clusters Grading Rubrics to Reveal Weaknesses in Language Models

How does CRAFT diagnose a language model’s weaknesses?

Vipul Gupta and colleagues, in the paper “CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data” (arXiv:2607.16122), propose a method that turns existing rubric-based evaluation data into a model-specific diagnosis. A rubric is a grading criterion that defines what a model’s answer must contain to be judged correct or complete. CRAFT clusters these criteria into a hierarchical capability tree, so instead of a single aggregate score, it reveals exactly which nodes of that tree are weakest for a particular model.

From diagnosis to targeted fine-tuning data

After identifying the weakest nodes, CRAFT generates targeted fine-tuning data for them — additional training on a narrow set of examples designed to fix precisely that weakness. The method was tested on 4 open-source models, in 2 professional domains (finance and law), and on 13 held-out benchmarks — test sets the models had not seen during training or during rubric creation.

Finance 4/4 versus law 3/4

CRAFT achieves the strongest average score in the finance domain across all 4 tested models, while in the legal domain it achieves the strongest average for 3 of 4 models. This difference suggests that the rubric-clustering approach is more consistent in finance regardless of the chosen model, while in law one model deviates from the pattern that holds for the others.