Vetting AI Tools: What Happens When Teachers Lead?

Inside aiEDU's first professional learning community on AI tool evaluation, piloted with University Prep Schools' AI Innovation Lab.

aiEDU's work has never put AI tools first. Our AI Readiness Framework and curricular resources focus on the fundamentals that hold up even as the technology evolves, such as understanding what AI is, judging when and how to use it, and building durable skills that matter with AI or without it. 

For districts navigating this moment, readiness eventually meets a practical question: what makes an AI tool effective for the classroom, and under what circumstances should it be put in front of students? Our approach is a team-oriented, deliberate look at the value and impact of each potential tool. In this pilot, that meant teachers taking the lead on evaluation: the educators closest to how AI actually lands in instruction, and to what AI Readiness asks of their students. 

That was the basis for aiEDU's partnership with University Prep Schools, a charter network of ten schools in Detroit. 

Over four sessions this spring, seven U Prep educators worked through aiEDU's first professional learning community (PLC) on evaluating tools — a small cohort that meets over several months to build a shared practice and bring it back to their schools. Together, we built a structured process for evaluating the AI tools already in use across the network and deciding which deserve to stay.

The educators in this pilot started with a variety of AI experience; many already used AI tools, some fluently. What this PLC provided was the time and structure to figure out what “good” looks like: in our early sessions, we built that vocabulary together, moving from "I like this tool" to naming exactly what it does well, for whom, and at what cost.

The rubrics districts already use for ed tech don't transfer cleanly to AI, which cuts differently across instruction, grade levels, and subject areas. So educators first defined their criteria across the dimensions that matter to a school system — learning impact, privacy, equity, and others — then tested real tools against them in their own classrooms, logging specific observed evidence rather than impressions. Each educator synthesized the evidence into a written recommendation, then traded drafts with a peer who was testing a different tool for structured feedback. 

"Whatever we decided about any one tool, the bigger win was what this process did for our team," says Chris Spencer, U Prep's Director of Data & Innovation, who went through the PLC alongside his colleagues. "Our educators now know how to look at a tool and ask whether it actually builds the skills our students need — and that's a capability we’ll be using for years, regardless of what the tools look like by then."

The framework we piloted with U Prep deliberately doesn't prescribe answers — the technology is too new for anyone to honestly claim a 'correct' decision on a tool. Instead, it's structured as questions that each organization must answer, and we wanted teachers to lead the answers. Through this approach, U Prep's educators came away with insights no outside rubric would have surfaced.

For example, partway through, the group raised a dimension aiEDU's materials hadn't emphasized — the environmental impact and ethics of the companies behind the tools — and named it as a criterion for their future evaluations. In the classroom, one algebra teacher testing a math platform's built-in AI tutor found its value hinged less on the tool than on instruction: students got the most out of it once they were taught to pose their exact question to the AI rather than to a teacher. Through this process, they realized asking the AI a good question is a skill that has to be taught, not assumed.

Similarly, the teachers in our cohort ranked privacy higher than we typically see in external evaluations. Tools that showed real promise were still limited to teacher-facing use until a formal education agreement could make student use appropriate. The deliberative process also drove the cohort to push the district for stronger guardrails, including a clear student AI use policy, rules on what data can never be entered, and disclosure to families when AI is used.

Every tool the cohort evaluated ended in one of four decisions: adopt, pilot, pause, or stop. Underlying those decisions, the teachers determined which distinctions actually had an impact. "Students liked it" isn't the same as "students learned from it" — one tool drove strong engagement with weekly homework, but its impact on learning was less clear. And "this saves teachers time" isn't "this moves student achievement." That's not to say those tools weren't useful — the teachers just wanted to be clear about where the value lies.

In their final session, participants turned those decisions into implementation plans for themselves and their colleagues, naming approved and off-limits uses, equity guardrails, and what they'd expect to see if a tool is actually working.

Our partnership with U Prep is the first step in a long road towards developing tested and scalable frameworks for AI tool evaluation in schools, but this first cohort’s experience gives any districts facing the same questions around tool adoption a few places to start: 

• Begin with the tools already in your classrooms. It’s easier to build muscle around evaluation if you start with the products people actually know. 

• Engage across the system. A decision made only in the IT office or only in one classroom won't hold. 

• Make the criteria your own. Every organization has different priorities, policies, and guardrails, which is why an evaluation process can be supported from outside but never imported from outside.

That last point, especially, is the main takeaway of this pilot. AI Readiness isn't something a school buys or settles with a ratings index. The judgment that makes a system ready — knowing what you need, testing against it, and being willing to say "not proven yet" — is built by the people inside it. What U Prep's educators built this spring is the capacity to make the next hundred tool decisions themselves.

Interested in bringing this PLC to your school or district to help your educators lead on AI tool evaluation? Reach out at programs@aiedu.org.