Over a million developers have joined DZone.

Azure Production Issues: Not a Nice Thing to Share with Your Friends

· Cloud Zone

Build fast, scale big with MongoDB Atlas, a hosted service for the leading NoSQL database on AWS. Try it now! Brought to you in partnership with MongoDB.

Last night I got a call from a close friend.

"Our production SQL Server VM at Azure is down, and we cannot provide service to clients."

A short analysis showed that we were in deep trouble:

  1. The server status was "stopped (failed to start)".
  2. Repeated tries to start the server resulted in starting ... and back to the fail message.
  3. Changing instance configuration as proposed in various forums and blogs resulted in the same fail message.
  4. Microsoft claims that everything is OK with its data centers.
  5. Checking the Azure storage container found out that the specific VHD disk was not updated since the server failure.
Since getting back to a cold backup would cause losing too much data, we had to somehow restore the failed server.
The odds were against us. Yet, lucky us, we could do that by restoring the database files from the VHD (VM disk) file that was available at Azure storage. 
How to Recover from the Stopped (Failed to Start) VM Machine?
  1. Start a new instance in the same availability set (that way you can continue using the same DNS name, instead of also deploying a new version of the app servers).
  2. Attach a new large disk to the instance (the failed server disk was 127GB, make sure the allocated disk is larger).
  3. Start the new machine.
  4. Format the disk as a new drive.
  5. Get to your Azure account and download the VHD file from Azure storage. Make sure you download it to the right disk. We found out that the download process takes several hours even when the blob storage is in the same data center as the VM.
  6. Mount the VHD file you downloaded as a new disk.
  7. Extract the database and log files from the new disk and attach them to the new SQL Server instance.
Other Recommendations
  1. Keep your backup files updated and in a safe place.
  2. Keep your database data and log files out of the system disk, so you could easily attach them to other servers.
Bottom Line
When the going gets tough, the tough get going
Keep Performing,

Now it's easier than ever to get started with MongoDB, the database that allows startups and enterprises alike to rapidly build planet-scale apps. Introducing MongoDB Atlas, the official hosted service for the database on AWS. Try it now! Brought to you in partnership with MongoDB.


Published at DZone with permission of Moshe Kaplan, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}