Business Experiments

(Also known as Pilot Projects and Pilot Experiments)

Your boss has asked you to develop a new service that will boost customer retention, and you've come up with several ideas that you think might work.

The problem is, you're not sure which ones will have the biggest impact. And you don't have the time and resources to implement each idea fully. So, which idea should you move forward with?

This is one situation where it's useful to set up business experiments. These allow you to test ideas on a small scale, before you take big risks or commit significant resources to a larger project.

In this article, we examine how to conduct business experiments, and discuss how you can learn from them.

Why Run Business Experiments?

Many businesses don't grow unless they innovate, and innovation comes from developing and implementing new ideas. However, innovation usually carries a risk of some kind: fundamental assumptions may be wrong, new products may not work sufficiently well, customers may not accept your ideas, and so on.

This is why it's so useful to run business experiments. By testing ideas, you can understand how they might perform on a larger scale. You can also learn from your failures and mistakes, and refine your ideas to make them better.

Business experiments can involve anything from very basic tests (for example, changing the welcome message on your website, or testing a new process for dealing with telephone inquiries) to more complex projects (for example, testing a new product or service with a small number of customers).

Business Experiments and Risk

Business experiments can sometimes seem fraught with uncertainty and risk. After all, there's often no way to know whether they will succeed.

However, that's the point. When you do a business experiment, you open yourself up to risk in a very controlled way. Although you clearly want to avoid failure, you're actually willing to fail, so that you can avoid more costly mistakes in the future, and so that you can develop and exploit high-risk opportunities.

In the short term, business experiments may seem too small and too risky to be worth the effort. But, when you look at the benefits over the long term, you'll see that it's often more costly not do to them. By being willing to learn – both from the ideas that succeed and the ones that fail – you'll uncover the successful ideas that will help your business or organization be the very best that it can be.

How Do Business Experiments Work?

To organize an effective business experiment, consider using the approach below. This is based on a process developed by innovation expert Thomas H. Davenport in his Harvard Business Review article, "How to Design Smart Business Experiments."

1. Create a Hypothesis

As with scientific experiments, the first part of the process is to create a hypothesis – this is your prediction of what will happen if your test is a success. For example, "Sending out two extra mailers a month will increase overall revenue" or "Moving the navigation to the left-hand side of the website will keep users on the website for longer."

As you create your hypothesis, think about the following questions:

• Can you measure what happens when you make the change? (You need to be able to "pass or "fail" the test, based on your hypothesis.)
• Does it fit with your team and organization's overall strategy, goals and values?
• Does it add value to what you do?

If the answer to any of these questions is "no," then you may want to amend your hypothesis or consider not doing the experiment at all.

Finally in this step, identify how you'll measure the success of your experiment. Ask yourself, "What does 'success' look like?"

What you measure will depend on your industry and on the type of experiment you're conducting, and it can include metrics such as sales and revenue, new sales leads, visits to key web pages, or more general factors such as ease of use, or feedback from customers or team members.

Again, think about how these measurements will affect your overall goals and strategy. For instance, you might define success as "Increasing website page views." However, this won't be viewed as a success if it causes a drop in overall revenue because people have stopped going to your online store.

2. Design Your Experiment

Now you need to think about how you'll conduct your business experiment. Outline what you'll be testing and how long you'll be testing it for.

Testing a new product or process is relatively simple – you just need to run the experiment for a specific time, and then use the metrics you identified in the previous step to evaluate its success.

However, if you're testing a modification to an existing product or process, you need to compare "like for like" scenarios, so that you can measure the success of your hypothesis. You can do this by using a "control group" and a "treatment group." (This is sometimes called A/B testing or split testing.)

Start by controlling as many variables as you can – this means that you should try to minimize change wherever possible.

Your control group is what you use for your baseline measure. Try to avoid making any changes with this group during your experiment. Do the same with your treatment group, except for the change that you want to test. You can then compare your control group with your treatment group to test your hypothesis.

For instance, imagine that your hypothesis is, "Opening until 10 p.m. will increase profits in retail stores." You choose two stores as the treatment group; these will open until 10 p.m. Your other stores are your control group; these won't change their opening hours. Then, after two weeks, you'll compare each store to see if opening until 10 p.m. increased profits during the test period.

In some cases, it will be impractical to use a control group and treatment group simultaneously. (You wouldn't be able to run the test above if you only had one store, for example.)

When this happens, you can use carefully selected past data as your control group, and then use the data that you get when you make the change as your treatment group. In the example of the store with only one site, you could compare sales this year with sales for the same period last year.

Overall, keep your experiment as simple as you can. The more complex the experiment, the more it will cost to do, the more risks it will involve, and the more time it will take to analyze the results. However, you need to make sure that your experiment will provide meaningful data.

3. Run the Experiment

Once you've designed your business experiment, it's time to take action and run it.

Inform everyone in your organization who is affected by the test before it goes live. They should know how you'll be conducting it, and why it's taking place.

Once the test is live, monitor it to make sure that nothing happens that could distort your findings. For instance, factors such as bad weather could affect sales in the example we highlighted above. If events like this do happen, consider repeating the experiment or removing some of the data from your final analysis.

If you're testing new products and services with customers, you may want to let them know that they're using a test product. This is a useful way to protect your reputation if things go wrong. It also encourages people to give feedback.

4. Analyze Results and Follow Up If Necessary

Once you've conducted your experiment, take some time to analyze the results thoroughly.

First, compare actual performance against your hypothesis:

• What was supposed to happen and what did happen?
• Why was there a difference?
• Was the experiment a success? (Don't be too disappointed if it wasn't!)
• What unexpected consequences happened as a result of your experiment? How could you manage or take advantage of these?

Then think about how you can use what you've learned. Would conducting further experiments help you to get even more value from your idea?

For instance, would opening stores until 9 p.m. instead of 10 p.m. make them more profitable? Or would modifying a product further result in even better feedback from customers?

In some cases, you can take action on what you've learned at the same time that you continue to conduct further experiments. Other times, you may have to conduct follow-up experiments before you can understand the best way forward.