Over a million developers have joined DZone.

The Registration Test Results Netflix Never Expected

A lot of potential Netflix users claimed that being able to see the vast library would improve sign-up. However, Netflix's test proved this might not be true. Check out the fascinating results.

· Mobile Zone

Visually compose APIs with easy-to-use tooling. Learn how IBM API Connect provides near-universal access to data and services both on-premises and in the cloud, brought to you in partnership with IBM.

“Come on Netflix. Why can’t I browse titles before I sign up?”

It’s a sentiment that many Netflix visitors have expressed at one point or another. And by many, I mean 46% of visitors surveyed. According to a their data, 46% of survey respondents said that making titles of Netflix content available to browse was the 1 thing they wanted to know more about before signing up.

Netflix 46 survey

It makes total sense to let potential customers take a peek right? If you share the movie and show titles they’d be getting from Netflix, they’re probably going to complete signups at a much higher rate.

We know that demonstrating value early in the onboarding process is a fundamental principle in the onboarding process. So how is it that one of the most data driven companies in Silicon Valley doesn’t do this?

Anna Blaylock, a Product Designer at Netflix wondered this too when she first started at the company. She knew that visitors were pining for this, so she posed the question to the team. “Why don’t we allow users to view titles before sign up?”

Netflix Web and Mobile Landing Page

Blaylock wasn’t the only team member with this question in mind. While she wasn’t the first to have the idea, she was the catalyst that sparked the series of A/B tests that followed.

The team was really confident that this change would help improve signups. The only question on their minds was, “How do we best implement it?” They designed a clever series of back-to-back tests that would pit variants against each other to determine the optimal solution.

Netflix Hypothesis

The test would start out with the control (not allowing users to browse titles) vs. Variant 1 (where users could scroll different genres, click on a title, get a synopsis, and see all the titles). The winner of the first test would become the new control and go on to face Variant 2. The process would repeat with variant 3 and the rest of the possibilities until they determined the best onboarding flow. This would allow Netflix to learn quickly, and determine the optimal solution out of the 5 very quickly.

Netflix Bracket

As they ran the experiment, they found that the original page (without any title searching abilities) beat variant 1.

Then it went on to beat variant 2.

And then variant 3.

And 4.

And 5.

Netflix Results

The team was shocked by the results. The survey data and best practices learned had suggested the exact opposite. Everyone had thought that allowing visitors to preview titles would increase signups. So they dug deep to figure out the reasons why the original performed best. Here’s what they learned.

“Don’t Confuse the Meal With the Menu.”

“Netflix is all about the experience,” says Blaylock. Just as a restaurant dining experience isn’t solely about the food on the menu, Netflix’s experience isn’t just about the titles. There’s something magical about being able to just visit the site, have your favorite shows or personalized suggestions pop up, click once with your mouse, and start watching. This incredibly smooth experience isn’t something that a web page can just describe to users, it’s something that has to be shown.

netflix family

Simplify Choices

Each of the variants built for the tests increased the number of choices and possible paths for Netflix visitors to follow. The original only had one choice: Start Your Free Month. Blaylock says that the low barrier to entry outweighs the need for users to see all their content before signing up.

Users Don’t Always Know What They Want

henry ford horses

While survey had shown that 46% of users wanted to browse titles before signing up for Netflix, the tests proved otherwise. Testing reveals our assumptions. That’s why it’s so important to run tests and trust the data. If Netflix had simply followed best practices, they would have been scrambling trying to figure out why their new onboarding was performing so poorly.

A/B testing allowed the team to quickly invalidate their hypotheses, without disrupting the user experience, or significantly affecting their key metrics.

Blaylock left us with one great final quote: “Your assumptions are your windows on the world. Scrub them off every once in awhile, or the light won’t come in.” – Isaac Asimov 

To learn more about how to improve your onboarding process to turn mobile downloads into engaged users, check out our whitepaper of 16 Tested Onboarding Techniques that the top apps use.

This post is adapted from Anna Blaylock’s talk at Manifesto SF.

The Mobile Zone is brought to you in partnership with Strongloop and IBM.  Visually compose APIs with easy-to-use tooling. Learn how IBM API Connect provides near-universal access to data and services both on-premises and in the cloud.


Published at DZone with permission of Kendrick Wang, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}