How to run a good experiment
You’ve got an operational improvement you’re after, or a new product line idea. Something that feels daunting, but important.[1] You’re agonizing over whether it will “work” and the stakes seem high to get it “right”. After reading my prior post (you did, right? 😉), you know you’ve got to try something. But what next – just jump straight off the cliff? Not quite. Let’s make sure that each action you take is intentionally designed to advance your learning.
Experiments aren't just for scientists in white coats. They're for anyone who's got the guts to question the status quo and say, "Hey, maybe there's a better way to do this!" Whether you're trying to overhaul your company's production process, pursue a new type of customer, or just establish a better morning routine in your household, experimentation is the secret to discovering your way into that imagined future.
Before you begin, get clear with yourself: what are you trying to achieve? Or better yet, what do you need to learn? Understanding where your threshold of knowledge[2] is will help you decide what the most impactful experiment will be. This will inform what type of experiment you want to run:
1. Go and see: This is like playing detective. Bring your figurative magnifying glass. You're not changing anything just yet – rather, you're taking a good, hard look at how things are currently done. Think of it as a fact-finding mission to uncover the nitty-gritty details of your process or market.
2. Exploratory experiment: You might poke at your process, introducing small changes to see how it reacts. You might ask prospective customers discovery questions. This phase is all about forming hypotheses that you then want more information on.
3. Testing a hypothesis: This is where the magic happens. You've got a hunch about how to make things better, and now it's time to put it to the test. You introduce a specific change to your process and see if it pans out like you expected. Or if your idea is a new product, maybe you’re putting a duct-tape-and-coat-and-hanger prototype in front of a representative customer to see what they do.
Now that you've got the lay of the land on the types of experiments possible, let’s walk through how to approach them:
Step 1: Design your experiment – Get specific about what exactly you’re going to try, the steps you will take, and what you expect to happen. Change only one thing at a time, so you can isolate its effect on the result. Smaller, more rapid experiments help you see sooner, faster, while limiting risk (because hardly anything works on the first try). Make sure you know how you will evaluate the results with objectivity (a quantitative metric helps). Write down your prediction up front!
Step 2: Run the experiment – Pour the liquid into the test tube and watch the sparks fly! Run the process or take the step, incorporating the change you planned. Make sure you have a way to observe it (ask a buddy, or record it on camera).
Step 3: Document what happened – And be honest. Did you do what you said, or something else? What were the results? What feedback did you get? Capture this visually so you can better understand gaps between your expectation and reality.
Step 4: Reflect and capture learnings – So, did it “work”? Consider what actually happened compared to your expectation. What was the same, and what was different? What did that help you realize? What came as a surprise? What do you now know about how this thing works, that you didn’t before? These are INSIGHTS! Write them down.
What else do you still need to know? It’s not uncommon for the first few experiments on a given topic to simply reveal more questions. That just means it’s time to move on to designing the next experiment, so you can continue learning. The more rounds you can run, the smarter you’ll be.
So there you have it, folks – you’re on your way to becoming a “process scientist”. Remember, experimentation isn't about being right – it's about learning by putting a tangible change out into the real world to see what actually happens. Only then will you know what's possible. So go forth, brave souls, and unleash your inner mad scientist. The world is your laboratory, so get out there and start experimenting!
[1] See “How to Begin”, by Michael Bungay Stanier (2022).
[2] Threshold of knowledge, the 3 types of experiments, and the 4 high-level steps were first described in “The Toyota Kata Practice Guide”, by Mike Rother (McGraw Hill, 2019). My knowledge about them is credited to Kata Girl Geeks and Kata School Cascadia.