Growing Consumer startups [Part 3]: Getting to Stickiness Using Experimentsvaidy
March 23, 2016
In this third and final part of the series, we dive into the details of performing experiments to cross the Stickiness stage in a Consumer Startup.
[If you need more context, please start at Part 1 of this series]
Here’s the four stages of experiments: * Step 1: Setup the experiment
Step 2: Build
Step 3: Review results
Step 4: Iterate
Here’s how you execute those steps for a Consumer Startup:
Step 1: Set up the Experiment
1. What is your OMTM? Here’s some examples of metrics that you could track at this stage:
I realize that the above image could be a bit overwhelming. Beyond the empathy stage, the metrics that you need to keep track of depends on two factors:
The stage of your startup (Stickiness, Virality and so on)
The type of your business model: The image above has 6 business models (from Lean Analytics) - that pretty much cover most types of business models. Your startup idea is probably a combo of one or more of those business models. So you should be able to use the above image to get some good starter ideas to come up with an OMTM.
Make sure to grab a copy of the Lean Analytics book - before you move forward. You’ll save a ton of time, money. And also keep your stress levels down :).
Want something quicker? Watch this fantastic (and free!) course on Udemy.
Finally, you could also check out this post I wrote (which summarizes the Udemy course).
A key point to keep in mind is the concept of Good Metrics and Bad Metrics. This is another key concept from Lean Analytics. Once again, the book and the links above do a very good job of covering this topic as well.
2. What is the current value of the OMTM? Measuring your OMTM at this stage will probably need some tooling. Google Analytics should suffice for most situations. KISSMetrics and Mixpanel are also good tools. Check out these posts for a roundup of web and mobile app tools.
3. What is your target value for your OMTM? Using a good benchmark here will help you set a good starting point for the speed of growth of your OMTM that you should aim for. For example, Y Combinator asks its startups to strive for a 7% week on week growth.
4. What is your experiment hypothesis (or “idea”) to help you move forward toward the OMTM goal? We finally come to the juicy stuff :). For a consumer product, your goal is to pass the “ Toothbrush Test”. And this is where the Hooked model is very useful.
It’s prudent to come up with hypotheses that are based on this kind of significant user behavior research, rather than to shoot in the dark and hope that your idea sticks.
Here’s each phase of the hooked cycle:
Your goal is to ensure that your product has an increasing number of strong Hook cycles encoded in its “design”. Here’s a few next steps for you:
Refer to this post to learn more about the Hooked model.
Fill up the Hooked Workbook
This post has an example of how you could craft experiments using the Hooked model at the Stickiness stage.
NOTE We’re not talking about visual design here. This is “product design” - or the way that we intend for the user to interact and navigate through the product. And the kind of deliberate behavioural (“habitual”) changes we want to make in the user, through the design of the product.
5. How will the idea help you to learn more about your customer? Your goal at this stage is to find out how well you’re able to “encode” the empathy you already have with your users into your product. When they interact with your product, are they getting engaged in “predictable” ways? Are they coming back from more? Do they “respond” to your features in predictable ways?
6. How will you measure whether your “idea” is working? You will be measuring both of the following:
Whether your users are interacting with the specific change that you made in the product, as predicted
Whether you’re moving closer to the OMTM goal
If you can connect both a correlation and causation between the above metrics, then you can say with confidence that your hypotheses has been validated.
**7. What is the reporting structure for the above information? **
Use this template to keep record the details of each experiment.
Use this template to keep track of how your OMTM is changing with each experiment
Step 2: Build
To reiterate, the two key parts of this phase are:
Implement the idea
Keeping your measuring process sustainable
1. Implementation As mentioned above, for consumer products, your goal is to implement the Hooked cycle in your product. For this purpose, you need to build a “working” prototype - that is something with enough moving parts to allow multiple users to interact with your product over time. Usually this means that you would have to build a full fledged web or mobile app, but that’s not necessarily true.
Here’s 3 options that you can choose from to do that:
A. Proof-of-Concepts: We’ve been able to build and test a bunch of ideas here at Multunus using a combination of just Google Spreadsheets, Google Forms and Trello. Read this, this and this to learn more..
- While the first link above is a Consumer Product, the last two links above are enterprise products. But the posts will still help you realize that you can actually get a lot done with off the shelf free tools and some scripting.
- While it is less straightforward to implement the Hooked cycle, it is still possible to do this. We’re currently working on a Proof-of-Concept that demonstrates that this is possible - using a combination of the following Slack and Google Spreadsheets. It’s still work-in-progress, but if you’re keen to learn more about this, let us know in the comments.
B. Low Fidelity Prototypes: These are fully working web or mobile apps - but with much less UI polish than would be expected of a mature product. Since the goal here is to figure out how to implement the Hooked cycle better (with an early set of pre-Alpha users), UI polish is not necessary.
So you don’t have to obsess over pixel level perfection or butter smooth animations. You can instead focus on faster iteration and “user behaviour” learning.
C. High Fidelity Products: These are full-fledged apps. I would not recommend that you go for these at the Pre-Stickiness stage - unless your target users would expect a very high level of fidelity from a very early stage product. If you’re tempted to say “Yes! My users do fall in that category”, you’re probably wrong. Unless you’re NASA, start with either a Proof-of-Concept or a Low Fidelity prototype.
**2. Is your measuring process easy? **
Generally, the higher the fidelity of your prototype, the more work you’ll need to do to collect metrics. For example, if you’re going with a Proof-of-Concept with Google Spreadsheets as your “database”, then you have very little additional work to do to collect and analyse your data. Where if you were going with a high fidelity product, you’d need to integrate the API’s from products like MixPanel or KissMetrics to get deep insights about your user’s behaviour.
As I’ve been suggesting all along, the more “work” you have to do as part of your experiment lifecycle, the longer it will take for you to get feedback from your experiments and the less motivated you will be to continue experimenting.
Learning in the pre-stickiness stage is a long-winded, difficult process. So it’s in your best interest to do whatever you can, to allow you to persist over at least a 4-6 month period.
Step 3: Review Results
1. What is the OMTM value, after the experiment? Once you know what you want to measure, and you’ve also got the “instrumentation” taken care of (for gathering your data), then finding the new OMTM value is usually a simple case of doing some spreadsheet analysis.
2. Update the Report with the new OMTM value This is simple enough.
3. How well did your “idea” work? Which hypotheses were validated and which were invalidated? If your results from the first set of experiments are not looking good, don’t worry. They almost never look good, but that doesn’t mean that they were failures. It just means that some of your hypotheses got invalidated. Which is part of the learning process.
4. Is the customer’s behavior more predictable to you after the experiment? This will depend on the answer to the above question. As your experimentation continues, you should be able to do an increasingly better job of implementing the Hooked cycle more successfully - and getting closer to passing the Toothbrush Test.
5. Was there something else (perhaps unexpected) that you discovered during the experiment? These insights are not limited to just Cust Devs. You’ll always have opportunities to discover these. It could be said that the success of your product is highly correlated to the frequency of insights you gather from your experiments. This is a skill that is worth mastering.
6. Are there any new hypotheses that you want to test? Any hypotheses that need updating? This is pretty straightforward. The important thing to realize is that, you’ll need to do this diligently with each experiment. It can get somewhat difficult to do this properly, because you need to be alert to the fact that the value of each experiment is only going to be as good as the hypotheses you feed into it. There might be a tendency to just do the experiments without reviewing the hypotheses carefully. Beware of this and avoid it.
Step 4: Iterate
You’ll need to go through the experimentation process as many times as it takes to get to the target value of your OMTM.
Feed the updated/new hypotheses that you’ve got as the output from your just finished experiment cycle - as the input for the new experiment cycle.
This process will get you closer to the OMTM over a period of time.
Crossing the stickiness stage of your startup is a huge milestone. It is an indicator that you’ve designed your product very well and that you’re now ready to accelerate the growth of your business.
Of course, anything that significant, requires persistence and a carefully crafted process of continuous improvement. This post is an introduction to the latter. All you need to bring in, is the former :).