Leading an AI Department: Architecture, a Hiring Bar, and a 93% Spend Cut
An eight-figure DTC brand had four AI engineers, no architecture document, and a founder who could not describe his own infrastructure. Four systems were live and in daily use, which nobody had written down. I came in to lead the department. I set the architecture, wrote the assessment we hire against, and took ownership of what every system costs to run.
- Role
- Head of AI and Automation. Architecture, delivery standards, hiring, and unit cost.
- Tools
- Python ,TypeScript ,Next.js ,PostgreSQL ,n8n ,Claude ,Vercel ,Hetzner ,ffmpeg
Watch the walkthrough.
The problem
Quality was scored by prompts
The video pipeline judged whether an ad was good by asking a model. A prompt can be ignored, edited, or silently drift. Nothing in the pipeline could tell you why one cut passed and another failed, so nobody could improve it on purpose.
Spend had no owner and no ceiling
One shared vendor key, no cap set anywhere, and thirty-seven code paths that could reach a paid call. Twenty-five of those left no receipt of their own, so an audit built from the logging calls would have found the paths that record spend rather than the ones that cause it.
The company did not own its own infrastructure
Subscriptions renewing in personal names, a production database in a personal account, and an engineer who had left the department still holding live credentials. That is a continuity problem before it is a security one.
The solution.
Made quality a number a computer checks
I took the video pipeline apart and rebuilt it in code over a day and a half. I dissected the four reference videos the founder had already approved and turned lighting, speech cadence, volume and background noise into measured values with a corridor. Cuts per minute went from 3.9 to 30.7 and the longest unbroken avatar shot from 68 seconds to 5.4, every value inside the corridor those four files set. All seven seams verified hidden in the output at twenty-one sample points.
Put a spend gate in without taking anyone's work away
The plan on the table was to remove thirteen people's access. That solves the cost problem by also destroying access to work those people had already produced. I changed the shape: everybody keeps their login, only two accounts can spend. The gate sits on the eight library functions that cause a charge rather than on the routes, and it is a code constant rather than a database setting, because seven of the thirteen paused accounts were administrators who could have edited a setting.
Wrote the bar we hire against
Candidates are expected to use AI on every part, so a task with a determinate answer ranks nobody. The assessment gives them a working pipeline with a missing gate and four unanswered questions of unequal consequence. Which one a candidate escalates is the signal. Silence is a hard fail. Twelve candidates are running it now.
Made mutation testing the shipping gate
A test that has never been watched to fail is not evidence. Every fix ships with a control suite and a mutation campaign in which each limb is watched to bite, and a skipped mutation counts as a failure. One module set shipped with 121 mutations, all of which bit.
The impact
Unit cost
- A finished ad-minute costs USD 2.34 on the rebuilt pipeline, measured against the vendor's own ledger and stated at the resolution it was measured
- One system's API bill went from USD 925.51 to USD 64.82 over twelve days, a 93 percent cut, with the cut point tested three separate ways
- Vendor pack pricing audited and rejected: the packs cost two to two and a half times usage-based rates at the volume the business was planning
- One generation stage removed after measurement showed it was 42 percent of the bill
Engineering standards
- One architecture and one set of delivery standards across every system the team ships
- Continuous integration introduced where no repository in the department had any
- Mutation testing as a shipping gate, not a nice-to-have
- A hiring instrument that measures how someone directs AI inside an existing architecture, rather than which language they know
Platform and governance
- Account ownership, credential rotation and offboarding brought under the department rather than left with individuals
- A written onboarding and offboarding pair, built so the revocation list mirrors the grant list and the two change together
- Production systems moved into company-owned accounts, costed and presented as continuity rather than as findings
Have a similar problem?
Tell me what is going on and I will tell you what I would do about it. No obligation.