Video-Based AI Instructional Coach

Google’s Gemini 1.5 comes with multimodal capabilities and a 1M-token context window. This makes it possible to chat with very long videos, such as class recordings, and thus to use the model as an observer, data collector, and instructional coach.

---

The model does not currently process sounds, but: 
🧠 It can infer a lot from extracted frames
🔊 Other tools such as TeachFX can be used as a second data point
🛡️ This limitation comes as a strength in terms of privacy

See in the video how easy it is to blur faces with Runway, as well.

🚫 Would I recommend teachers start using Gemini 1.5 now? No. But this is a glimpse into the very near future for schools.

💡Imagine a similar model fine-tuned for this purpose, coupled with in-context learning to align with a school’s T&L philosophy, and connected through knowledge graphs to sub-agents assisting with feedback, goal-setting, and suggestions!

👩‍🏫 Teachers would have access to coaching sessions on-demand, and school leaders would be able to “observe” any class, any day, and get aggregated data to analyze patterns, set objectives, and measure progress.

🤝 This AI integration would, of course, require discussions and a shared understanding and norms, so that it is seen, and used, as a tool for assistance, not surveillance.

✨ As a matter of fact, this technology could help fully integrate traditional “evaluations” into the ongoing improvement, by teachers, of their own practice.