Test Everything, Assume Nothing - Why AI Needs a Lab Coat, Not a Crystal Ball

April 15, 2025

“The greatest danger of Artificial Intelligence is that people conclude too early that they understand it.”- Eliezer Yudkowsky

Fifteen years ago, I was sitting in a Roman Law class, probably trying to stay awake while deciphering 2,000-year-old legal codes. Our professor said something that didn’t quite click at the time - but has everything to do with AI today:

“In ancient Rome, the law evolved through trial and error. It wasn’t perfect. It adapted.”

At the time, I was too busy highlighting Latin phrases to care. But looking back, that simple idea - build the rulebook as you go - has become the foundation of how I approach emerging tech.

Because in today’s world of rapidly evolving AI, the only constant is uncertainty. And the smartest response? Experimentation.

AI Isn’t Magic - It’s Math (With a Messy Middle)

There’s a tendency to treat AI like a magic wand - wave it, and out comes perfectly tagged data, personalized content, and automated everything. But let’s be real. It’s more like a math problem written in disappearing ink… over a moving target.

AI learns from your data - and all the glorious, messy, edge-case-laden context that comes with it. That means:

• It reflects your inconsistencies.
• It inherits your assumptions.
• And it exposes where your systems aren’t quite as tidy as you’d hoped.

Need proof? Just look at what’s happened in the last year. We will keep the names out of it, but you can research this 😅

• A major news outlet launched an AI-written article program - only to discover the bot had plagiarized and fabricated facts.
• A well-known retailer rolled out AI-generated product descriptions, which created some fun and completely unrelated descriptions.
• A prominent legal case saw a lawyer cite fake cases created by ChatGPT - because no one double-checked the output.

The point? AI isn’t magic. It’s a system. And systems, especially ones built on probabilities, need oversight, context, constant feedback, and a TON of experimentation and testing.

Experimentation is Your AI Superpower

Instead of obsessing over when to implement AI, more teams should ask:

What can we test with AI this month? This mindset shift unlocks creative, low-risk ways to learn:

• Personalization pilots: Show 10% of users an AI-curated experience and watch how they behave. Do they click more? Convert faster? Or bounce entirely?
• Pricing trials: Let a machine learning model suggest discounts for aging inventory - then compare performance vs. business-as-usual markdowns.
• Support experiments: Use GenAI to generate first-draft replies for common customer tickets. Do agents respond faster? Are customers happier?

You don’t need a 16-month roadmap. You need a 2-week sprint with a clear hypothesis and success metric. Some tests will flop. Others will surprise you.

But every single one will make your team smarter and sharper for the next round.

You Don’t Need to Have It All Figured Out - Just Be Willing to Try

We’re in a moment where technology is moving faster than most organizations can plan for - and honestly, that can feel overwhelming. But that’s not a reason to hit pause. It’s a reason to shift how we show up. The old motto was:

“Move fast and break things.”

But in the world of AI? It’s more like:

“Move curiously and test things.”

I’ve seen teams spend months crafting the “perfect” AI strategy - only to find that by the time it was ready, the landscape had already shifted. Meanwhile, other teams picked one small use case, ran a few thoughtful experiments, and uncovered insights that led to better questions -and better solutions. You don’t need a master plan to start. You just need a test-and-learn mindset, a safe space to explore, and a willingness to be surprised.

The real edge today isn’t in having all the answers - It’s in building the confidence and capacity to keep asking better questions.

Your Turn: What’s Your AI Lab Testing?

What’s your team experimenting with right now? I would love to chat about it. Ping me, call me, message me. What are some hard-earned lessons, or even your funniest flops - you can drop them in the comments and all have a good laugh and cry. 😅

I think, especially in retail, we need to advocate for building an open lab notebook together - a space for anyone navigating this wild, ever-evolving AI landscape. Because here’s the truth.

Nobody has this all figured out. Not the biggest tech companies, the most seasoned teams, or even the AI models themselves.

The ones who will thrive in this new era aren’t the ones who get it perfect on the first try. They’re the ones who keep showing up. Keep asking questions. And most importantly, keep learning together.

Let’s make this journey a little less lonely, a little more open, and a whole lot more collaborative. We’re all figuring it out - let’s do it out loud.

“The more I help out, the more successful I become. But I measure success in what it has done for the people around me. That’s the real accolade.” - Adam Grant, Give and Take (a must-read for anyone navigating change)

AI can be intimidating - even a little scary. But people don’t have to be. If we lead with curiosity, kindness, and a willingness to share what we’re learning, we won’t just build smarter tools - we’ll build a better industry and community.

Let’s grow, together.