Continuing Cerebrata journey, in this blog post I will describe how we built our 3rd product – Azure Management Cmdlets. In short this product contains a number of PowerShell Cmdlets for managing Windows Azure.
Why Did We Build It?
Only reason I could think of regarding building this product was “To Save Our Asses” :) and I will tell you the reason why. Folks who were using Cloud Storage Studio were asking for a way in that tool to download contents of an entire storage account. We had the capability of downloading data from a table or a blob container but that’s about it. It sounded like a great feature request but when this request came, there were two problems (and one of them is still there):
- Performance of Cloud Storage Studio: We were still on Cloud Storage Studio 1.0 and there were many performance related problems with that product when it came to downloading all blobs in a blob container. If we were to extend it to download all blobs from all blob containers in a Windows Azure Storage Account, the problem would worsen considerably.
- GUI based application: Cloud Storage Studio is a GUI based application. And downloading storage data from the cloud to local desktop could be a “really” long process. What that means is that if we were to implement this functionality in the tool, the tool would need to be up and running all the time while the contents are being downloaded. At that time what we found is that if we kept the tool running for a long duration of time, the performance starts degrading.
So basically for backing up a storage account we needed something that:
- can run in the background
- can be scripted & scheduled
That was the genesis of this product. We built this product with the sole purpose of providing a way for our users to take a backup of their storage account.
At first we thought about building this as a console application but when I discussed this idea with some of my friends and users, all of them recommended to go PowerShell route. The way they described PowerShell to me is that “it is the new console for .net”. We decided to give it a try.
Also at that time we started rewriting our products (more on version 2.0 of our product in a blog later on) and the way we decided to architect our products gave us a motive to create this product.
So with that we started creating this product.
As I mentioned above the primary driving force behind building this thing was to provide our users with a capability to take a backup of their storage account and save that data locally however as we started developing this product we thought that we already have the functionality for complete Windows Azure Storage management (blobs, tables and queues) so why restrict ourselves to this backup functionality. Why not write something which will allow our users for complete storage management (and more) through this PowerShell interface.
So the team started learning PowerShell (we knew absolutely nothing about it when we started working on this product ) and then started building these Cmdlets while we learnt PowerShell. Like with our other products, we maintained a really fast pace and were out with a closed private beta in a matter of month or so. In September, 2010 we went out with our first private beta release which included a little over 50 PowerShell cmdlets covering all aspects of managing Windows Azure Storage and (obviously) the capability to take a backup of storage account.
In the next iteration, we included support for managing subscription related activities like creating a new cloud service deployment using our Cmdlets.
With this product we followed a rather aggressive approach to releasing and we just did 2 rounds of private beta and then opened up this product for public beta sometime in 2nd week of October. We went through 4 more iterations before we decided to go Live with product. Throughout the phase of private and public beta we kept on improving the functionality and adding new cmdlets. Finally we went live in November 2010 with this product. At that time the product had 100+ cmdlets for managing Windows Azure Storage, Subscriptions, Diagnostics and SQL Azure (and almost doubled the number of cmdlets).
Issues & Resolutions
It looks like we had a pretty smooth ride with this product. It was actually quite opposite. We ran into a number of issues during the beta phase and even after we’ve been live with the product for over 6 months. Luckily most of the issues were technical in nature and they stemmed from the fact that we knew nothing or very little about PowerShell. Some of the issues we ran into:
- PowerShell Naming Convention: It seems that one has to name the cmdlets in a certain way (Verb-Noun). Moreover the noun portion of the name cannot be plural. Furthermore, one needs to pick from certain names. In our initial releases, we completely ignored this. We ended up naming our cmdlets the way we pleased (and found logical). For example, we had a cmdlet which was listing all tables in a storage account. We named that cmdlet Get-Tables. The verb portion was right but we ended up using “Tables” instead of “Table” which was wrong. In fact one of my team member told me that we’re naming our cmdlets incorrectly but out of my ignorance I simply vetoed them (only to eat humble pie later on ).
- PowerShell Real Use: We didn’t completely understand the power and use of PowerShell even after we went live. For example, we didn’t know about “Pipelining” concept there and the power it brings to the table. We were still thinking like regular application developers. For example, we ended up writing a lot of unnecessary data to the console to let the users know about the outcome (e.g. how many tables are there in a storage account) which made our cmdlets rather useless in pipelining scenarios.
So we had issues and we resolved them. I’ve said it over and over again in my previous posts and I will say it again – Never underestimate the power of the community. This time the PowerShell community came to our rescue. Some of the members of the community essentially became our mentors and guides and helped us understand it much better. They not only took time to provide us with valuable feedback but also gave us their inputs on the way we wanted to fix the problems. They gave us the feedback to fix the naming issue in our cmdlets but they didn’t stop there. When I went back to them with revised names or even question about how we should name these cmdlets, they reviewed it and told us the right way to do things. For example, we had a cmdlet which basically creates a read-only copy of the blob (a term known as snapshot blob in Windows Azure community). We knew the functionality the cmdlet does and based on that we originally named it “Snapshot-Blob” (which was incorrect) but we didn’t know the correct name for that. They helped us finding the correct name.
Two of the folks who helped us immensely are: Ravikanth Chaganti, Marco Shaw. I am extremely grateful to these two guys for constantly guiding us throughout the course of development.
There’re some interesting things that I want to share:
- When we started building this product, we thought that we would offer it for free to our existing users primarily because we started this product just to build a backup solution. Only by the time we were ready to go Live with this product, we decided to charge money for it as we saw the potential of this product and the capabilities it offered.
- We ended up building a side product while developing this product. That product is Cmdlet Help Viewer. The reason we built this product was because some of us were (and still are) uncomfortable viewing help in PowerShell console so we decided to build a GUI based help viewer. It’s available on CodePlex and is open source: http://cmdlethelpviewer.codeplex.com/.
There’re two important lessons I learned while developing this product:
Be Humble & Ready To Learn (from others or even yourself): Obviously one does not know everything so be humble and listen to folks who know stuff. Being ignorant initially is fine but if you choose to remain ignorant after being told is just sin in my opinion.
Be Agile: We’ve had issues with the product (every product has those) but we were quite agile about fixing those issues. Once they were brought to our attention, our sole focus was to fix those issues and make sure that they get released as soon as possible.
So this was the story of our Azure Management Cmdlets product. In the next few posts in this series, I’m going to talk more about my journey.