DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Testing, Tools, and Frameworks
  4. Conducting Usability Tests in Production

Conducting Usability Tests in Production

It's difficult to truly simulate how our apps behave in a production environment. How do we test a feature while also simulating the environment it’s meant to be used in?

Justin Baker user avatar by
Justin Baker
·
Jan. 05, 17 · Opinion
Like (9)
Save
Tweet
Share
4.36K Views

Join the DZone community and get the full member experience.

Join For Free

Usability testing in a real-world environment (production environment) gives us insight into how users actually use our product in their day-to-day lives. It is one thing to run a test in a lab setting, but it is another to have users try features while they are walking, running to the airport, stressed, and sleepy.

No matter how hard we try, it is very difficult to truly simulate how our apps behave in our production environment. We can run focus groups, run beta tests in a beta environment, and test things internally, but how can we truly mimic the real world in an artificial context? In other words, how do we test a feature while also simulating the environment it’s meant to be used in?

A good real-world usability test analyzes features efficiently and accurately collects user feedback to improve the user experience. However, when we test anything in a non-production environment, we are inherently biasing our tests. In a lab-based usability test:

  • Users are overly cognizant of the feature they are testing.
  • Users are unnaturally focused on testing that new feature, ignoring the rest of the product.
  • Users are using fake data or incomplete data and don’t sufficiently utilize actual use cases.
  • It is very hard to test discoverability (i.e., can a user find the feature on their own?).
  • Users tend to ignore distractions like external notifications (Facebook, Skype, texts, etc.) and just focus on the task at hand.
  • Users are in an overly analytical mode, typically looking to give feedback.
  • Users are overly forgiving, looking to please!

Does this mean that running usability tests in non-production environments is a waste of time? Not at all! This is absolutely necessary to identify bugs, check for general usability, and solicit feedback quickly.

However, it should just be one of the steps in a comprehensive usability testing process, one that involves internal, staging, and production usability testing.

Benefits of Usability Testing in Production

The primary purpose of usability testing in production is to assess real-world user behavior while minimizing bias and performance risk. Some more benefits include:

Genuine User Feedback in a Real-World Environment

You receive quantitative insight into your feature’s performance. How well is it scaling? Are users using it? How is it impacting your system? How are your levels of engagement?

You receive contextual insight into your feature’s efficacy. Do users see the new feature? How is it meshing within the context in your existing feature set? Are people using it as intended? Are people using it once and then not using it again?

You receive qualitative feedback. Are users complaining? Are they happy? Are they neutral? Are they submitting support tickets? Are they leaving?

No Opt-In Bias

Users test the feature without knowing that they are part of the test. You can assess how well they use it by using a product like Full Story to record the session or by tracking metrics. Therefore, you get a more representative sample testing the feature rather than just early adopters.

Measuring Actual System Performance 

There is nothing quite like your production environment, where you can have a complex array of nodes, clusters, CDNs, etc. allowing your app to scale. As people start to use the new feature, you can see how it is impacting your actual system (load times, discoverability, caching issues, etc.).

Managing a Usability Test in Production

Of course, testing anything in your production environment is inherently risky and has real-world consequences. If you launch something to everyone just to get feedback and they hate it, then you risk permanently losing those users. Equally bad, you can cripple your entire application with unforeseen scaling and performance issues.

To mitigate this, companies like Facebook, Amazon, and Google collect production feedback by releasing features behind feature flags. While we won’t go into to the specific anatomy of a feature flag, we can go through the methodology behind the release.

LaunchDarkly Usability Testing in Production - User Feedback

If a feature is wrapped in a feature flag, then it gives you control over who sees the feature and when. This means that you can perform targeted and controlled releases using a percentage rollout whereby you can incrementally increase a feature’s visibility to 1%, 5%, and to 100% of your users.

Hence, you can collect production level feedback because you control the level of risk. If a new feature is performing well, then you can keep increasing the percentage rollout. If it is tanking or hurting performance, then you can reduce the rollout or kill it completely.

Therefore, feature flags (also known as feature toggles or config switches) give you full control over the risk of your production releases. You can gather real-world user feedback by separating your feature rollout from your code deployment.

Testing Production (computer science) Usability

Published at DZone with permission of Justin Baker, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • How Do the Docker Client and Docker Servers Work?
  • How To Get Page Source in Selenium Using Python
  • Top 12 Technical Skills Every Software Tester Must Have
  • How and Why You Should Start Automating DevOps

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: