My 2012 Side Projects
My 2012 Side Projects
Join the DZone community and get the full member experience.Join For Free
Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.
ExtJS Sample code
Throughout the year I wrote a couple dozen how-to samples of ExtJS code. This product seems to be poorly documented, although it has improved, and is poorly covered on blogs outside the Sencha forums. This series has proved one of the most popular, and has received most of the comments my blog.
Scraping Flippa data
I wanted to test out the difficulties of scraping, as well as doing some market research. I pulled a small subset of auctions to run some tests. This had challenges that work more work than anticipated – memory management, error handling, and dealing with various page formatting. This got me into looking at Weka and R for data mining (e.g. looking for indicators of good values or spam), and it proved difficult to get statistically significant conclusions from the data.
Scraping ads with PhantomJS
I thought it’d be interested to test the difficulty of scraping advertising, as a market research exercise. This was insprired by Mixrank, and I was thinking of using it to do due diligence on website purchases (e.g. a website with adsense, or a competitor to a website with adsense).
I went to a Solr/Lucene conference and wanted to build a project while there. I wrote a simple ETL script to convert git history to a solr index, and a UI to facet search results by authors or companies. This addresses the interesting problem of resolving which engineer to call when a client calls the front desk, as well as identifying for new engineers who worked on what project.
This is an ongoing side project I’m working on with my father- he wrote several short books of beehive plans, which sell well at conferences and through a major beekeeping magazine. I set up a domain and an online store for him, as an exercise in learning a bit about SEO and marketing, which has been quite fruitful. I have a free account with adzerk, a company that does hosted ad servers. This enabled me to run a couple A/B tests on Amazon products; nothing earth-shattering, but still interesting.
I did some experimentation with R, partly because I like math, and partly to see an alternative way of managing data – one could think of R as a SQL like language that operates on an in-memory database, and it makes for an interesting learning exercise. I did some experimentation with reverse-engineering music (what chords are listed in a song), which is easy to do in some simple cases, and difficult to do well – to make this work effectively requires a very large and broad training set.
I wanted to improve my writing and document side projects. Much to my surprise, people have subscribed to this and follow along. It has also proven to be a good way to get feedback on these projects, as people periodically leave updates or further requests on sample code. I’ve also met a few people and started receiving more recruiter spam, although not enough to be overwhelming.
I’m currently working on a simple project to build an invoicing application for someone, which will let me test out some new APIs, as well as working on a workflow to move from mock-ups to functional prototypes more quickly, as well as getting a better handle on what is available for PDF APIs, testing twilio, and mobile device APIs.
This will be signficant work-wise, and my future JVM based projects will likely move to Scala, so stay tuned.
Published at DZone with permission of Gary Sieling , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.