RIP Experimentation, Hello Validation!

We're using AI to build one idea in three months. It should be failing a hundred a day. - Mark Pincus

The obvious thing that’s happening now is that the constraint of ideas being difficult to build has evaporated. And this raises the question: do we even care about experimentation?
Of course we do, because that’s how you validate ideas, and it’s the best way to figure out what’s worth working on. But you can shout this from the rooftops, and if people don’t intrinsically believe there’s value in experimentation, then they won’t run experiments. It’s much more exciting to make and build something than to pause and create small versions for testing, even though that’s exactly what you should do.

So maybe it’s time to rethink digital product ideation as first building something with AI, so we can demo it, see it, feel it, and touch it.

What’s new is that this is where experimentation used to end, not start.

But your demo is not a product yet. Remember, we’re still on day zero, so the idea hasn’t got distribution or validation yet.

Mark Pincus put it well on Lenny Rachitsky‘s Podcast. His view is that AI’s real value for experimentation is being squandered.
“So the way we should be using AI is as a testing machine, a failure machine and a way to vibe code... but build the lowest possible cycled version of your product that you can get signal back on. How are you testing a 100 ideas a day instead of one in three months? I think AI is being used more to build one idea in three months than a hundred ideas in a day.”

So how do we validate this ‘product’?

I’m shifting my thinking towards how we can accelerate validation and compress the time from idea to experimentation.

This is necessary because there’s an explosion of AI-built apps and half-solved problems. I’m sure you’ve all got five or ten half-finished projects, things you’ve spun up an initial version of and then abandoned because it got hard.

And that’s exactly the point at which we need to start figuring out how we test and experiment. How do we get data from customers and potential users to know whether this is a “hell yeah, let’s go” or an “absolutely not, stop it immediately” moment?

That’s what I’m working on right now.
If the AI-generated product IS the idea, then how do we use the proven Pretotyping and rapid experimentation as the underlying method?

It should be embedded and automated, with human-in-the-loop judgement and expertise, so we move quickly from an idea or AI-generated demo to the first experiment worth running. Then use Alberto Savoia‘s modified Bayesian probability to tell us whether it’s passed the pass/fail metric you set.

Maybe it’s time to think more about validating what AI has helped you build. Spend less time relearning how to experiment and innovate the pre-AI way, so you can focus your limited time, resources, and tokens on the right ideas.

I’d love to hear what you think and what you’re seeing in your own business.

And remember, there’s only upside in validating ideas!
Valid idea = make money
Failed idea = save money