Google Research: Vantage — AI platform that assesses critical thinking and creativity through conversations with avatars
Why it matters
Google Research in collaboration with NYU presents Vantage, an experimental platform that uses generative AI to assess hard-to-measure human skills such as critical thinking and creativity. AI scoring showed agreement with human experts comparable to inter-expert agreement.
Google Research and New York University (NYU) have developed Vantage, an innovative platform that uses generative AI to assess skills that are traditionally nearly impossible to measure objectively — critical thinking, collaboration, and creativity.
How Vantage works
The system places learners in dynamic conversations with AI avatars that simulate collaborative tasks such as debate preparation or creative pitching. Two key AI components power the assessment:
Executive LLM uses pedagogical rubrics to guide avatar responses and introduce targeted challenges — such as challenging ideas or simulated conflicts — so the learner can demonstrate their skills.
AI Evaluator analyzes conversation transcripts against the same rubrics and generates detailed feedback with visual scores and specific suggestions for improvement.
Validation of results
In a study with 188 participants (ages 18-25) at NYU, AI scoring showed agreement with human experts comparable to inter-expert agreement. In a second study involving creative multimedia tasks, the Pearson correlation between AI and human scores was a high 0.88.
Potential for education
Vantage opens the possibility of integrated “skill layers” in curricula where students receive dual feedback — on both content knowledge and the development of key competencies. This could transform how educational institutions handle the assessment of “soft skills” in the age of AI.