Case Study
A Performance System Case Study
Amazon Delivery Central Operations
Problem: L&D lacked a system to gather Level 3/4 data, leading to costly retraining and decisions based on associate anecdotes.
Solution: Architected Project TIES, an end-to-end training evaluation system using Python, Slack APIs, Asana, and AI-powered reporting.
Impact: Increased survey participation by 56% (from 44.49% to 69.17%) by migrating the system from VBA/Outlook to Python/Slack. Identified critical performance gaps: the data revealed that 20% of department SOPs were outdated, shifting the L&D focus from content revision to systemic process improvement. Validated training effectiveness: confirmed that 81% of participants frequently apply their learned skills.
L&D could count training but could not see its effect. The only data available was training counts, requested, pending, assigned, and completed, plus trainer performance. Nothing captured whether associates could do the work a month later, so gaps surfaced as anecdotes and the default response was more training.
Goals were set against Gilbert's Behavioral Engineering Model, so the analysis could separate what people lacked from what the environment failed to provide.
Compare actual task execution against expected task execution.
Distinguish knowledge and skill gaps from information, resource, and incentive gaps.
Feed evidence back into what L&D builds and revises.
Five components, with a human review step before anything reached a senior leader.
Sent initial and escalation outreach, logged sent and error states, linked directly to the relevant SOP, and routed feedback to the SOP creator. Accessibility was built in: keyboard navigation, contrasting colors, alt text on hover, a progress bar, and error pop-ups.
Delivered surveys and logged sent and received messages.
Stored submissions, tracked outreach and metrics, and drove the associate and manager dashboards.
Analyzed the survey data exported from Asana.
Sent AI-generated reports to senior leaders by department, filtered by month and year. L&D reviewed and revised every report before it went out.
V1 ran on VBA with the Outlook API. V3 ran on Python with Slack webhooks and Asana. The switch was made on data, not preference: engagement was measurably higher in Slack than in Outlook, and VBA could not support the accessibility requirements or the interface quality the workflow needed. Participation after the migration confirmed it.
Measured
88%
Felt confident working in-process one month after training.
Measured
81%
Frequently applied the skills they learned.
Measured
85%
Felt supported by their manager and team.
Measured
5%
Reported dissatisfaction.
Measured
44.49% to 69.17%
Survey participation after the move from Outlook to Slack, a 56% increase.
Measured
20%
Of department SOPs were flagged out of date.
Measured
80+ hrs
Of manual reporting eliminated per quarter, per teammate.
Estimated
~$50K / year
Estimated labor savings, calculated as time multiplied by loaded rate. This is an estimate, not a measured finance figure.
Thirty-nine open responses were coded for sentiment.
Measured
33%
Positive
Measured
62%
Neutral
Measured
5%
Negative
The data confirms that over 85% of participants are confident to work in-process one month post-training, apply the skills they learned during training, and feel supported by their manager and team.
However, the data highlights that outdated SOPs and sentiment analysis themes (more hands-on training and outdated training materials) shift the focus from improving L&D content to systemic process improvement: bureaucratic approval processes, pro-active SOP updates, and timely material update requests from leaders.
Getting sign-off to survey associates and report upward took longer than the build.
Asana and Slack API access and permission scopes constrained what could be automated.
Adoption required the L&D team to change how they reported, not just what tool they opened.
Full automation was possible only with wider API access, so reports were generated then reviewed and revised by L&D before sending. I worked with SDEs and data scientists on the tool choices.
2 months
Planning and development of the application.
1 week
Slack integration.
2 weeks
Asana projects, rules, and integration.
The system then ran for 5 months in production.
The internal case-study deck as presented.
Open the Slides in a New TabMore systems, and the learning experiences behind them.
or write directly to brent.michael670@gmail.com