Guide

How we built an AI board that checks our biggest calls

Every AI model has blind spots, and it can't see its own. So before we ship anything that matters, it goes in front of a board: three AI models from three different companies. Each one reviews it alone before seeing what the others said.

Why one model isn't enough

Ask a model to check its own work and it usually agrees with itself. It checks the plan with the same assumptions it used to write it. A second copy of the same model has the same blind spots. You want a reviewer that was trained differently and isn't trying to be agreeable.

Who sits on the board

How a review runs

1. One brief. The question, the facts, what's been tried, and what a wrong answer costs. Every seat gets the same brief, word for word.
2. Blind round. Each seat answers alone. Nobody sees another answer first, so nobody anchors on it.
3. Joint round when it matters. Each seat reads the others' reports and responds. Real disagreements get argued out instead of averaged away.
4. One decision. The operator reads everything and makes the call. When the seats split, the decision and the reason get written down.
5. I make the final call on the big ones. Money, signatures and anything that can't be undone still come to me.

What goes to the board

• Every security change, before it ships.
• Major client-facing documents, before they go out.
• A weekly R&D session every Monday: what changed in the industry that week, and what we do about it.
• Hard calls with real money or reputation on the line.

Not everything goes to the board. Three models at their highest reasoning settings cost real money and real time, and routine work doesn't need three opinions.

Two times it paid off

The release that didn't ship. We had a security update ready for a client. The seat from our own model family said ship it. The outside seat said hold, and it found two real holes: a way for bad code to get into one of our trusted scripts, and a signature check that a fake certificate could pass. It was right. We held the release and fixed both, and the update doesn't go out until the board signs off on the fix. If we'd only had our own model reviewing, it would have shipped.

The catch the board missed. In our first R&D session, two seats agreed that accounting firms could be a strong customer group. Neither noticed that it ran into a rule we'd already set for ourselves about handling financial data. The operator caught it and changed the plan: firms became referral partners instead of customers. The board doesn't have the last word. It gives you a better first draft of the decision.

Keeping client data out

The outside seats only get briefs with the client details stripped out. It's "Client A," never a name. Before any brief goes out, a check in code reads it and refuses to send it if a client is named. Client details stay with the AI provider our clients already agreed to.

Build your own

1. Pick models from at least two companies. Three is better.
2. Write the brief once and send it to every seat unchanged.
3. Run the first round blind.
4. Tell every seat to disagree when it should. Giving each one a role helps.
5. Pick one decider, and write down the reason whenever the seats split.
6. Save every brief and answer. After a few weeks you'll see which seat catches what.
7. Keep client data away from any model your clients didn't agree to.

This is how we run CommandKey day to day, and it's the same discipline that goes into every agent we build.

Sources

• CommandKey AI: What it does
• CommandKey AI: FAQ