Pretotyping: a complete example

pretotyping is a technique for testing user reception to a new product, before even building a functional prototype; what is put under questioning in pretotyping is business viability, not technical feasibility.

after reading the pretotype it book in one sit, i set out to try an idea i had for a web service. in this article i share what i learned and the results of my experiment, open to interpretation.

summary

the goal of my activities was to validate the idea of a service which provides links considered controversial with respect to the user's interests; for example, in the field of programming languages, an article on ruby's unique features to a phper, or the wonders of automated refactoring in static languages for a rubyist.

the service is about avoiding the filter bubble : getting stuck in the bubble of our interests and never learn something new from the competition. i called this pretotype who disagreees with me? .

instead of building a real prototype , i launched a weekly newsletter: this approach is an example of a mechanical turk pretotype, where a human (me) does the work of a machine (an algorithm for selecting links, or a platform for crowdsourcing it like dzone does.) it is also an example of re-label , since i borrowed a mailing list service from mailchimp and call it a web service.

i published weekly articles on my blog, which has 1.2k feed subscribers; i extended my reach with links on my google+, facebook page and twitter.

subscriptions

subscriptions mainly came after my blog posts, related to the creation of the service or a new issue of the newsletter.

the trend peaked after the first issue , and is declining ever since, probably as a result of the conversion of all interested blog subscribers.

note that these data (apart from the first which span 5 days instead of the weeks) are relative to a period of one week from one issue to another, so it is really a subscription rate. i am not counting the views on blog pages, where older pages were naturally bound to have more views.

opening rate

according to mailchimp, the percentage of emails that were actually opened is very high with respect to the industry (and all industries): it was always over 50%.

this datum indicates a very selected public , since it consists of the followers of my blog. the potential page views were close to the 1.2k feed subscribers number; the page view i got were in the hundreds, and the final newsletter subscribers was 78; i had 2 unsubscribers during the whole experience.

cohort analysis

googler alberto savoia followed me during this experiment, and suggested a different analysis of the data: by tracking simply the data of a single issue of a newsletter, i would confound new subscribers with older ones.

he suggested a cohort analysis, which divides the subscribers in cohorts: groups with a single characteristic.

in my case, the cohort was defined by the periods of subscription :

• cohort 1: after creation and first issue
• cohort 2: between first and second issue
• cohort 3: between second and third issue
• cohort 4: between third and fourth issue.

the after the fourth issue cohort does not exist because i have no data: the variable provided by mailchimp is the openings number for each newsletter.

since i had various version of each issue depending on the topic (php|c|java...), i aggregated them and retain the number of openings, specific for an issue and for cohort.

thus now i have a trend for each category of user where i see his evolution in time, relative to the time of his subscription.

compare this scenario to twitter's one : the activity in there lowers very much after the first month of usage. however twitter has millions of new subscribers every month, so the one that remain ultimately lead to more activity than for the first month.

we have to look at two things in such an graph:

• how the average cohort behaves in time , so that newer subscriptions don't cover up the disengagement of older users who cease to open mails. if each cohort goes down after a certain period of time, the user isn't sticking with the service.
• how newer cohorts improve with respect to the previous ones (this was not my objective while creating the newsletter however, as the format and content stay constant).

conclusions

this is a full example of pretotyping a web application's concept without writing any code (which costs you your time, focus, and money for hosting), and of analyzing data collected by a third-party service.

what do you think about the collected data? do you think users show a recurring interest? is their subscription rate high enough?