arXiv:2605.15015 Small Private LM: Competitive Results in Educational Assessment Design with Human-in-the-Loop Recommendations

Chris Davis Jaldi, Anmol Saini, Shan Zhang, Noah Schroeder, Cogan Shimizu, and Eleni Ilkou published a paper on arXiv on May 14, 2026, addressing one of the critical gaps in the current AI-in-education discourse — how to use AI for assessment design with the privacy guarantees that the educational sector demands.

What is the educational assessment design problem?

Generative AI has demonstrated impressive ability to generate pedagogically aligned questions — quiz questions, problem sets, essay prompts targeting specific Bloom’s taxonomy levels. The industry already uses GPT-4, Claude, and Gemini for this task.

The problem: educational data is extremely sensitive. Student responses, learning analytics, curriculum specifics — none of this should end up in cloud API logs that may be used for model training. Cloud-based LLM APIs are a compliance nightmare for schools (FERPA in the US, GDPR Article 8 in the EU, local regulatory frameworks for minors).

What does the paper specifically demonstrate about smaller models?

The authors conduct a systematic comparison of smaller models against larger alternatives:

• Quality dimension — ability to generate questions aligned with Bloom’s taxonomy levels (remember, understand, apply, analyze, evaluate, create)
• Reproducible metrics — a measurement framework that can be independently reproduced, not subjective rater opinions
• Comparison to expert human judgment — model-generated questions evaluated against ratings by expert educators

Findings: smaller models achieve competitive results across quality dimensions. The difference is not as dramatic as often assumed — an appropriately fine-tuned 7–13B parameter model can approximate 70–200B model output for assessment design tasks.

What critical limitation was discovered?

The paper highlights a significant caveat: “model-based evaluations also exhibit systematic inconsistencies and bias relative to expert ratings.” Practical consequences:

• If we use LLM-as-judge for evaluating other LLM outputs, we accumulate bias throughout the entire pipeline
• Models prefer generated questions that resemble their own outputs, not necessarily pedagogically optimal ones
• Apparent quality consensus among different models may be an artifact of shared training data, not real pedagogical validity

What is the main recommendation?

The authors clearly recommend a Human-in-the-Loop approach. Concrete implications:

• Small models as teammates — not as autonomous agents
• Expert review required for final output validation
• Local deployment for privacy preservation, but not to circumvent human review
• Bloom’s taxonomy alignment must be expert-verified, not purely model-judged

The approach is compatible with emerging educational AI policy frameworks — UNESCO, EU Digital Education Action Plan, US Department of Education AI guidelines. All emphasize AI augmentation, not replacement of educational professionals.

What does this mean for the education tech sector?

The paper validates the niche that startups like Khanmigo, Magic School AI, and open-source projects like OpenLLM-In-Education are exploring: small, privacy-respecting models running locally on school infrastructure instead of cloud API calls.

The approach is a commercial fit:

• Schools/universities — privacy compliance without capability compromise
• Edtech vendors — lower compute cost, on-premise deployment option
• Open-source community — fine-tuneable base models (Llama, Qwen, Phi) for educational specialization

The paper fits into the broader 2026 trend of specialized small models for sensitive domains: medical small LMs (Cardio-LLM, MedFlow GraphFlow May 15), legal small LMs, financial small LMs. The one-size-fits-all frontier API model faces competition from specialized small models that better serve regulated sectors with privacy demands.