Beta-testing on Windows Phone - some curious facts
Beta-testing on Windows Phone - some curious facts
Join the DZone community and get the full member experience.Join For Free
I recently launched a beta-test of the Visual Studio Achievements for Windows Phone, and in the process I noticed some interesting tendencies, that I tried to describe in this article. Although it is hard to say how many of these can be applied to other beta launches, I am positive that there will be some common points in a wide variety of testing programs.
Most of the feedback comes from developers
Even though the beta testing process might be public to some extent, the most feedback I got was from people who have some developer experience. Users were mostly hesitant to provide criticism, mostly limiting themselves to "Nice work on the app!" Developers, on the other side, tried to break the app workflow and I received pretty detailed replication instructions, screenshots and suggestions as to why the current UI is not exactly usable in specific situations. Also, developers usually submit the most obscure test cases - e.g. "I tested the application by launching it and pressing the same button 15 times in a row"
Announcing the beta on a major WP7 site helps
After the announcement was posted on WPCentral, I gut hundreds of emails from people that wanted to participate in the private beta through the Markteplace - most of them could not be accommodated due to the limited number of available slots. If you have a beta product that you desperately need to test, use the contact form on one of those websites and let them know that there is something on the way that the public needs to know about.
Out of all your private beta testers, only a handful will be really active
From a hundred people that signed up for the beta, only 5 actually had active conversations with me on what should be improved. Going back to the first point, a lot of potential users do not have the background that would let them go in-depth with all the testing processes and some of them might be hesitant to be extremely critical if the application is somewhat functional and does it's job in the most common environment.
Windows Phone beta through the Marketplace needs modifications
A huge problem with beta distribution through the Marketplace is the fact that I cannot submit a modified XAP as a part of the same beta launch. So, if I modify the application, add capabilities and fix bugs, I need to initiate another beta program for the application. This takes more time and is a bit of an inconvenience, as I have to let the users know about another application URL that they should use to download the new version.
Keep track of all the bugs and suggestions online
If the application is open-source, then you can use Github or CodePlex. Even if you don't want to publish the source (or maybe you just want to keep the project private), there are several hosting solutions (such as Assembla) and standalone installations that can help you track the work you still have to do when it comes to improving the beta application. I just had a CodePlex page for VS Achievements for Windows Phone.
Your own testing is not enough
There is a limited number of tests you can do on your own. First and foremost, there are multiple devices out there. People use them with different carriers, with different data speeds and different hardware performance. Putting your application in the hands of those people ensures that the chances the product will work as intended in a majority of cases are higher. One of the feedback points I got was the fact that my application was working fine on a WiFi network, but was failing (even though it got a correct response from the server) on an EDGE connection because it took too much time and some data manipulation mechanism broke. I would've never encountered that situation because I never personally use an EDGE carrier-based network connection.
Every application has its own target audience, and that influences feedback
The fact that 100 beta testers signed up doesn't mean that each and every person out of that group will use the application. There is the novelty factor that might drive some users to sign up for the beta to be among the first to try something, to never use it again later. Visual Studio Achievements for Windows Phone targets a very restricted number of people - those who have Visual Studio, and have the Achievements plugin installed. During the test, not a lot of people had the plugin itself, so the application became an achievement viewer for others. That was one of the reasons why the usage stopped for a part of the audience. Make sure that you are aware of the groups that will most likely use your application and plan accordingly.
The in-app feedback form is rarely used
Even though I integrated a simple feedback mechanism in the About page, it was rarely used. It convinced me that not a lot of people will type long feedback messages directly from the phone but would much rather use a machine with a full-sized keyboard to type their email. Nonetheless, the form was used and I believe it should be there for convenience purposes.
I personally decided that since the application is open-source anyway, I should just distribute the XAP for developers that have dev-unlocked Windows Phone devices. Besides, there is a cheap alternative to AppHub, so developers can experiment without shelling out $99 if they don't intend on publishing their applications in the Marketplace - if they want to test sideloaded XAPs, that would be one of the ways of doing this.
Opinions expressed by DZone contributors are their own.