This is a guest post by Robbert van Geldrop, Chief Venturing Officer at Firmhouse.
How to measures success and progress of your innovation projects and portfolio?
I attended the Innov8rs conference in Madrid last week and had some thorough discussions with other participants on that exact topic. Lots of people had very different ideas, but very few had an actual set of metrics in place at this time. Currently, most projects and programs get scaled or killed by committee over seemingly random heuristics. I think there’s a simple and effective way to measure the progress.
The one metric to measure is Assumptions Tested. This can be on a per-month basis. It’s a very simple metric to track. When you test assumptions and complete experiments, which go with it, you’ll produce learnings. The speed at which teams in innovation projects produce those learnings is a great indicator for the progress of the project. Let’s remind ourselves that Lean is all about reducing waste.
A low amount of assumptions tested might indicate that your team’s innovation project has waste
By analysing how long it takes certain teams to complete an experiment, you’ll be able to identify bottlenecks in the project or across projects. Perhaps certain types of experiments take very long for teams to complete and can be resolved by additional training or investing in tools.
Innovation Accounting In The Early Days
I had a flashback to the startup community many years ago when I and my fellow startup founders were breaking our heads on what could be a good set of metrics for our SaaS business. This was just before the Lean Startup movement took off and the term Innovation Accounting was coined. I hacked my own excel sheet with a cohort analysis of sign-ups per week and how we retained them over time. I automated API connections from our CRM to slice our data around revenue per customer based on either their sign-up date or the channel in which we marketed for them.
We were probably one of the first startups to report about Lifetime Value and Customer Acquisition Costs to our investors. We weren’t expecting that these metrics would later become the key indicators investors would steer on in their portfolio companies, as they do today.
The North Star
Innovation Accounting has gradually become a very common practice in the startup space. Facebook, for instance, lead there by choosing retention and the way that metric curves to find out if it had hit product/market fit. I consider Alex Schultz video about this the root of all the frameworks other thought leaders have conveyed, such as Ash Maurya and Dave McClure. In the core it boils down to measuring one thing and focus your effort on moving the needle on that one. Alex calls this ‘the north star’.
Slide from Alex Schultz presentation showing good (blue) and bad (red)retention curves
Some other great exploration work in the same field has been done by Phil Libin from Evernote. It’s the same pattern and core thought: you measure the very thing which proves that value was delivered. In consumer software as well as social networks this metric turned out to be retention.
Nowadays I’m deeply involved in startup accelerators and corporate innovation programs. I feel like it’s 2010 all over again, but now it’s the innovation managers at large companies who stab in the dark on what to measure in order to gauge and decide on the progress of their innovation projects. I’ve seen very crazy proposals which basically defy the core principle of finding leading indicators, which are not things like revenue, market share or costs.
Tested Assumptions Prove Or Disprove The New Business Model
However, it would be a little too easy to scale or kill projects just on that metric alone. As with Innovation Accounting, you’re always interested in what happened and why some assumptions were validated or invalidated.
Also, the assumptions should be tied to a new business model or a change in the existing business model of the company. So on the flip-side of the Assumptions Tested metric is some notion of what parts of the business model are affected by the outcomes. By looking at your total inventory of assumptions as well as the way they tie into the business model you can always say something sensible about the progress of the innovation project. This can even work without knowing the metrics that particular project might need by itself to determine success. It’s often unclear what metric would capture the value of that particular new idea for the customer, especially in the very early stage of innovation projects.
Pivot or Persevere
Whether you’re in one innovation project or oversee a portfolio of them, you probably have to check with the team on a regular basis if things are working. This is always hard to do as each project has its own dynamics. I think it’s a strong indicator if few learnings are produced or if it basically invalidates the business model. You can have the same discussion with each team, asking questions such as:
- What assumptions were tested?
- What did you learn?
- Does that allow you to move forward on this new business model?
If discussed this with innovation leaders and practitioners in this space. Some people challenge me and claim that innovation is not always about changes in the business model. I think that is rarely the case with innovation projects.
Very often the project is changing at least one part of the business model. For instance, if you’re redesigning your online banking app to improve the onboarding for customers, you probably have some personas in mind who are currently not served well with your current app. This touches the customer segment and customer relationship box in the Business Model Canvas.
The Aggregate Sum Of Assumption Tested
If you run a portfolio of multiple innovation projects the nice thing about this metric is that it can aggregate and serve as an indicator how much insights the company is gathering about new markets, opportunities, customer segments and value propositions. The logbook of those assumptions, experiments and insights can very well become a large company’s most valuable asset.
