How Top Apps Validate New Features to Ensure Success

“Smart people really like to over complicate the problem because you can see four steps ahead, so it’s very tempting to say like, ìHmm let me go build that final version.î But when you do that…you donít get something that actually increases your users metrics.”
– Waseem Daher, Product Manager at Dropbox

It’s kind of funny. On web, we’ve come so far in applying agile methodologies to our businesses. We no longer invest tremendous amounts of time and resources building products we’re unsure about. Instead, businesses are building smarter, utilizing MVPs to take out the guesswork and build better products using data.

At the same time, the way we develop on mobile is disappointing. It’s like we’ve gone back to the dark ages, developing software and investing without validation. Instead of using MVPs and making small nimble changes every few weeks, we’re shipping large releases filled with untested features, crossing our fingers, and hoping that users love them. We know there’s a better way, so how is it that this is still a widespread practice on mobile?

The Problem Is the App Store

The answer is surprisingly simple. The reason most teams move and develop so slowly on mobile is because of the app stores. The app stores are the biggest blockers because everything has to funnel through their arduous review processes. That’s a huge problem because the app stores are inordinately slow.

We’ve gotten here because the App Store reflects Appleís DNA of great products plus big launches. They are a 1980s hardware company thatís mastered that strategy, and when developers build on their platform, they have no choice but to emulate the approach as well.
– Andrew Chen

With an average review time of 7 days, a huge chunk of the development cycle is spent simply waiting for a change to deploy so that you can get meaningful data back from your users. Even the tiniest of changes are subject to reviews. You can’t rollback to previous versions either. That means if something isn’t working in your app, you can’t just revert it to a previous state like on web. You have to re-submit and go through the review process all over again.

“In mobile what do I do? I push a button, it goes to The App Store, the black hole that is Apple, and out comes, in some indeterminate amount of time, my binary. If I’ve made a mistake or if there’s a fatal, or something silly in that app, itís gone. That bullet has left the barrel, and I’m screwed.” – Facebook Release Engineering Director Chuck Rossi

This means your team has to spend even more time making sure there are no bugs due to the uncertainty of being able to release a fix quickly. With so much time and effort spent on each release, using agile methodologies to adapt to a quickly changing market becomes an incredibly difficult task.

So how do top companies deal with the constraints of the app store, yet still build amazing products?

A New Methodology

The top apps have completely changed their development methodologies to adapt and evolve to the changing landscape in mobile.

If you talk to teams such as Uber, Facebook, HotelTonight, Lyft, Yahoo Mail, etc., you’ll see that all of them approach mobile in a very data-driven manner. Instead of deploying large, untested releases every few months, they make small, iterative changes to their app every 1-2 weeks.

They test these changes on small groups of users, get early feedback, and use that data to determine whether their next steps, and deploy immediately if they choose. This allows them to push out tested, validated changes that they’re sure users will love.

Here’s how it works:

The Formula

1. Determine the problem or pain point

The first thing teams do is figure out what the problems or pain points of their users are. They use a combination of quantitative and qualitative data gathered from user testing, analytics, and user feedback to make sure they understand their users’ needs and address the most pressing ones.

2. Form a hypothesis

Once an issue has been identified and prioritized, they form a hypothesis about how to improve the product to better serve user needs. The goal isn’t to find the perfect solution, because this can’t be determined without actual testing on users. Instead, they choose a solution they think will best address the problem.

3. Determine the metrics for success

Without data, it’d be nearly impossible for teams to see the actual effects of the changes that they’re making. That’s why defining the correct metrics are so important. During this stage, teams identify what the best KPIs to measure a change would be and set up additional metrics to track throughout the entire funnel.

4. Build an MVP

The next step is to build a simple MVP that will validate or invalidate the hypothesis. A common pitfall at this stage is trying to do too much. As Waseem mentioned above, the focus here is to keep the scale small so that you’re cheaply testing in order to validate an idea. Iteration and improvement only comes after we know if a change improves our metrics or not.

5. Test and validate

After the MVP is built, they A/B test their changes on a small fraction of their users, so as not to prematurely disturb their entire user base. This allows them to get data that shows exactly how real users are reacting to the changes, and whether or not they have the desired effects. Once they know this, teams then decide whether or not to push forward with a change, or head back to the lab to tweak it.

6. Iterate, Iterate, Iterate

Once teams have data on the efficacy of a change, they can go about investing more heavily into it by instantly deploying it to all of their users, tweaking it, or dropping it altogether. The testing is vital because it allows them to cheaply validate their ideas, without investing heavily resources or affecting many of their users.

7. Repeat

The last step is to repeat the process. The top teams are constantly running through this process to identify changes or features they want to build that will best serve their users. Doing so gives them a framework for continuously improving and optimizing their app to make sure their product is the best on the market. That’s why they’re so successful.

Even within the constraints of the app stores, it’s still very much possible to use agile methodologies and MVPs to create amazing products. The top teams in mobile keep pushing ahead because they’re constantly using A/B testing to gain data to validate changes. This allows them to be agile, and focus their resources on only changes that will have a positive impact on their users.