Course Quality
Course QA Pipeline
A repetitive manual QA process became a scalable first-pass quality system.
EdPlus at ASU
- Python
- Canvas API
- Airtable
- Railway
Measured
43 criteria
Every course checked against 29 critical and 14 foundational standards, 17 resolved by script and 26 by AI with cited evidence.
Measured
100 courses
Audited in 50 minutes on the hybrid deterministic and AI pass, measured on a full production cohort.
Measured
6 to 8 hrs
The manual baseline for a single course review.
The Friction
Manual quality review of a single course takes roughly 6 to 8 hours. Across a catalog of this size, that math does not work. Reviewers were spending their expertise on checks a machine could make.
What I Did First
I shadowed a QA initiate end to end before writing any code, then interviewed the reviewer about the full process. The goal was to find which checks were mechanical and which genuinely needed judgment, because only the first kind should be automated.
What I Built
A prototype that completes a QA initiate form from an .imscc package in 3 to 5 minutes, using Python and Google Gemini. It grew into a batch pipeline that grades courses against WCAG 2.1 AA and ASU design standards, with adaptive rate limiting that keeps requests inside the Canvas API quota.
What Came of It
The prototype was demoed internally and led to me spearheading the technical side of the QA workstream under the AI taskforce. Reviewers now start from a report rather than a blank checklist, and spend their time on the calls a tool cannot make.
Scope and Limits
Still in beta. The benchmark measures the hybrid pass end to end, and deterministic and AI-judged criteria are reported separately, because conflating the two would make an honest number misleading. The pipeline surfaces issues and leaves the fixing to a person.
More systems, and the learning experiences behind them.
or write directly to brent.michael670@gmail.com