One of the greatest fears of product managers is to create an app that flopped because it's based on untested assumptions. After successfully launching more than 20 products, we're convinced that we've found the right approach for hypothesis-driven development.
In this guide, I'll show you how we validated the hypotheses to ensure that the apps met the users' expectations and needs.
What is hypothesis-driven development?
Hypothesis-driven development is a prototype methodology that allows product designers to develop, test, and rebuild a product until it’s acceptable by the users. It is an iterative measure that explores assumptions defined during the project and attempts to validate it with users’ feedbacks.
What you have assumed during the initial stage of development may not be valid for the users. Even if they are backed by historical data, user behaviors can be affected by specific audiences and other factors. Hypothesis-driven development removes these uncertainties as the project progresses.
Why we use hypothesis-driven development
For us, the hypothesis-driven approach provides a structured way to consolidate ideas and build hypotheses based on objective criteria. It’s also less costly to test the prototype before production.
Using this approach has reliably allowed us to identify what, how, and in which order should the testing be done. It gives us a deep understanding of how we prioritise the features, how it’s connected to the business goals and desired user outcomes.
We’re also able to track and compare the desired and real outcomes of developing the features.
The process of Prototype Development that we use
Our success in building apps that are well-accepted by users is based on the Lean UX definition of hypothesis. We believe that the business outcome will be achieved if the user’s outcome is fulfilled for the particular feature.
Here’s the process flow:
How Might We technique → Dot voting (based on estimated/assumptive impact) → converting into a hypothesis → define testing methodology (research method + success/fail criteria) → impact effort scale for prioritizing → test, learn, repeat.
Once the hypothesis is proven right, the feature is escalated into the development track for UI design and development.
Step 1: List Down Questions And Assumptions
Whether it’s the initial stage of the project or after the launch, there are always uncertainties or ideas to further improve the existing product. In order to move forward, you’ll need to turn the ideas into structured hypotheses where they can be tested prior to production.
To start with, jot the ideas or assumptions down on paper or a sticky note.
Then, you’ll want to widen the scope of the questions and assumptions into possible solutions. The How Might We(HMW) technique is handy in rephrasing the statements into questions that facilitate brainstorming.
For example, if you have a social media app with a low number of users, asking, “How might we increase the number of users for the app?” makes brainstorming easier.
Step 2: Dot Vote to Prioritize Questions and Assumptions
Once you’ve got a list of questions, it’s time to decide which are potentially more impactful for the product. The Dot Vote method, where team members are given dots to place on the questions, helps prioritize the questions and assumptions.
Our team uses this method when we’re faced with many ideas and need to eliminate some of them. We started by grouping similar ideas and use 3-5 dots to vote. At the end of the process, we’ll have the preliminary data on the possible impact and our team’s interest in developing certain features.
This method allows us to prioritize the statements derived from the HMW technique and we’re only converting the top ones.
Step 3: Develop Hypotheses from Questions
The questions lead to a brainstorming session where the answers become hypotheses for the product. The hypothesis is meant to create a framework that allows the questions and solutions to be defined clearly for validation.
Our team followed a specific format in forming hypotheses. We structured the statement as follow:
We believe we will achieve [ business outcome],
If [ the persona],
Solve their need in [ user outcome] using [feature].
Here’s a hypothesis we’ve created:
We believe we will achieve DAU=100 if Mike (our proto persona) solve their need in recording and sharing videos instantaneously using our camera and cloud storage.
Step 4: Test the Hypothesis with an Experiment
It’s crucial to validate each of the assumptions made on the product features. Based on the hypotheses, experiments in the form of interviews, surveys, usability testing, and so forth are created to determine if the assumptions are aligned with reality.
Each of the methods provides some level of confidence. Therefore, you don’t want to be 100% reliant on a particular method as it’s based on a sample of users.
It’s important to choose a research method that allows validation to be done with minimal effort. Even though hypotheses validation provides a degree of confidence, not all assumptions can be tested and there could be a margin of error in data obtained as the test is conducted on a sample of people.
The experiments are designed in such a way that feedback can be compared with the predicted outcome. Only validated hypotheses are brought forward for development.
Testing all the hypotheses can be tedious. To be more efficient, you can use the impact effort scale. This method allows you to focus on hypotheses that are potentially high value and easy to validate.
You can also work on hypotheses that deliver high impact but require high effort. Ignore those that require high impact but low impact and keep hypotheses with low impact and effort into the backlog.
At Uptech, we assign each hypothesis with clear testing criteria. We rank the hypothesis with a binary ‘task success’ and subjective ‘effort on task’ where the latter is scored from 1 to 10.
While we’re conducting the test, we also collect qualitative data such as the users' feedback. We have a habit of segregation the feedback into pros, cons and neutral with color-coded stickers. (red - cons, green -pros, blue- neutral).
The best practice is to test each hypothesis at least on 5 users.
Step 5 Learn, Build (and Repeat)
The hypothesis-driven approach is not a single-ended process. Often, you’ll find that some of the hypotheses are proven to be false. Rather than be disheartened, you should use the data gathered to finetune the hypothesis and design a better experiment in the next phase.
Treat the entire cycle as a learning process where you’ll better understand the product and the customers.
We’ve found the process helpful when developing an MVP for Carbon Club, an environmental startup in the UK. The app allows users to donate to charity based on the carbon-footprint produced.
In order to calculate the carbon footprint, we’re weighing the options of
- Connecting the app to the users’ bank account to monitor the carbon footprint based on purchases made.
- Allowing users to take quizzes on their lifestyles.
Upon validation, we’ve found that all of the users opted for the second option as they are concerned about linking an unknown app to their banking account.
The result makes us shelves the first assumption we’ve made during pre-Sprint research. It also saves our client $50,000, and a few months of work as connecting the app to the bank account requires a huge effort.
.png)
Step 6: Implement Product and Maintain
Once you’ve got the confidence that the remaining hypotheses are validated, it’s time to develop the product. However, testing must be continued even after the product is launched.
You should be on your toes as customers’ demands, market trends, local economics, and other conditions may require some features to evolve.
Our takeaways for hypothesis-driven development
If there’s anything that you could pick from our experience, it’s these 5 points.
1. Should every idea go straight into the backlog? No, unless they are validated with substantial evidence.
2. While it’s hard to define business outcomes with specific metrics and desired values, you should do it anyway. Try to be as specific as possible, and avoid general terms. Give your best effort and adjust as you receive new data.
3. Get all product teams involved as the best ideas are born from collaboration.
4. Start with a plan consists of 2 main parameters, i.e., criteria of success and research methods. Besides qualitative insights, you need to set objective criteria to determine if a test is successful. Use the Test Card to validate the assumptions strategically.
5. The methodology that we’ve recommended in this article works not only for products. We’ve applied it at the end of 2019 for setting the strategic goals of the company and end up with robust results, engaged and aligned team.
Summary
You'll have a better idea of which features would lead to a successful product with hypothesis-driven development. Rather than vague assumptions, the consolidated data from users will provide a clear direction for your development team.
As for the hypotheses that don't make the cut, improvise, re-test, and leverage for future upgrades.
Keep failing with product launches? I'll be happy to point you in the right direction. Drop me a message here.