Uploading Large File By Splitting Into Blocks In Windows Azure Blob Storage

In this blog post, we will see how you can upload a large blob in blob storage using Windows Azure SDK for PHP. I must state that I don’t know anything about PHP and did this exercise in order to help somebody out on StackOverflow. What helped me in the process is excellent documentation on PHP’s website and my knowledge on how Windows Azure Blob Storage Service REST API works.

What I realized during this process is that it’s fun to get out of one’s comfort zone (.Net for me) once in a while . It’s extremely rewarding when you accomplish something.

Since I’m infamous for writing really long blog posts , if you’re interested in seeing the final code, either scroll down below to the bottom of this post or head on toStackOverflow. Otherwise please read on .

Since we’re uploading really large files, the procedure would be to split the file in chunks (blocks), upload these chunks and then commit those chunks.

Getting Started

I’m assuming you have installed Windows Azure SDK for PHP. If not you can download it from here: http://www.windowsazure.com/en-us/downloads/?sdk=php. This SDK depends on some external packages. For this blog post, only thing we need is to install Http_Request2 PEAR package which you can download from here: http://pear.php.net/package/HTTP_Request2.

Add Proper Classes

We just have to ensure that we have referenced all the classes we need in our code

<?php 
require_once 'WindowsAzure/WindowsAzure.php';
use WindowsAzure\Common\ServicesBuilder;
use WindowsAzure\Common\ServiceException;
use WindowsAzure\Blob\Models\Block;
use WindowsAzure\Blob\Models\BlobBlockType;
?>

Get Azure Things in place

This would mean creating an instance of “BlobRestProxy” class in our code. For the purpose of this blog, I’m uploading a file in storage emulator.

$connectionString = "UseDevelopmentStorage=true";
$instance = ServicesBuilder::getInstance();
$blobRestProxy = $instance -> createBlobService($connectionString);
$containerName = "[mycontainer]";
$blobName = "[myblobname]";

Here’re the operations we would need to do:

Read file in chunks

To read file in chunks, first we’ll define the chunk size

define('CHUNK_SIZE', 1024*1024);//Block Size = 1 MB

Then we’ll get the file handler by specifying the file name and opening the file

$handler = fopen("[full file path]", "r");

and now we’ll read the file in chunks

while (!feof($handler))
{
    $data = fread($handler, CHUNK_SIZE);
}
fclose($handler); 

Prepare blocks

Before this, there are a few things I want to mention about blocks:

• A file can be split into fifty thousand (50000) blocks.
• Each block must be assigned a unique id (block id).
• All block ids must have the same length. I would encourage you to read myprevious blog for more details regarding this.
• When sending to Windows Azure, each block id must be Base64 encoded.

Based on this, what we’re going to do is assign each block an incrementing integer value and to keep block id length the same, we’ll pad it with zeros so that the length of the block id is 6 characters.

$counter = 1;
$blockId = str_pad($counter, 6, "0", STR_PAD_LEFT);

Then we’ll create an instance of “Block” class and add that block id there with type as “Uncommitted”.

$block = new Block();
$block -> setBlockId(base64_encode($blockId));
$block -> setType("Uncommitted");

Then we add this block to an array. This array will be used in the final step for committing the blocks (chunks).

$blockIds = array();
array_push($blockIds, $block);

Upload Blocks

Now that we have the chunk content ready, we just need to upload it. We will make use of “createBlobBlock” function in “BlobRestProxy” class to upload the block.

$blobRestProxy -> createBlobBlock($containerName, $blobName, base64_encode($blockId), $data);

We would need to do this for each and every block we wish to upload.

Committing Blocks

This is the last step. Once all the blocks have been uploaded, we need to tell Windows Azure Blob Storage to create a blob by adding all blocks we’ve uploaded so far. We will make use of “commitBlobBlocks” function again in “BlobRestProxy” class to commit the block.

$blobRestProxy -> commitBlobBlocks($containerName, $blobName, $blockIds);

That’s it! You should be able to see the blob in your blob storage. It’s that easy .

Complete Code

Here’s the complete code:

<?php 
require_once 'WindowsAzure/WindowsAzure.php';
use WindowsAzure\Common\ServicesBuilder;
use WindowsAzure\Common\ServiceException;
use WindowsAzure\Blob\Models\Block;
use WindowsAzure\Blob\Models\BlobBlockType;
define('CHUNK_SIZE', 1024*1024);//Block Size = 1 MB
try {
     
    $connectionString = "UseDevelopmentStorage=true";
    $instance = ServicesBuilder::getInstance();
    $blobRestProxy = $instance -> createBlobService($connectionString);
    $containerName = "[mycontainer]";
    $blobName = "[myblobname]";
    $handler = fopen("[full file path]", "r");
    $counter = 1;
    $blockIds = array();
    while (!feof($handler))
    {
        $blockId = str_pad($counter, 6, "0", STR_PAD_LEFT);
        $block = new Block();
        $block -> setBlockId(base64_encode($blockId));
        $block -> setType("Uncommitted");
        array_push($blockIds, $block);
        $data = fread($handler, CHUNK_SIZE);
        echo " \n ";
        echo " -----------------------------------------";
        echo " \n ";
        echo "Read " . strlen($data) . " of data from file";
        echo " \n ";
        echo " -----------------------------------------";
        echo " \n ";
        echo "Uploading block #: " . $blockId . " into blob storage. Please wait.";
        echo " \n ";
        echo " -----------------------------------------";
        echo " \n ";
        $blobRestProxy -> createBlobBlock($containerName, $blobName, base64_encode($blockId), $data);
        echo "Uploaded block: " . $blockId . " into blob storage.";
        echo " \n ";
        echo " -----------------------------------------";
        echo " \n ";
        $counter = $counter + 1;
    }
    fclose($handler); 
    echo "Now committing block list. Please wait.";
    echo " \n ";
    echo " -----------------------------------------";
    echo " \n ";
    $blobRestProxy -> commitBlobBlocks($containerName, $blobName, $blockIds);
    echo "Blob created successfully.";
}
catch(Exception $e){
    // Handle exception based on error codes and messages.
    // Error codes and messages are here: 
    // http://msdn.microsoft.com/en-us/library/windowsazure/dd179439.aspx
    $code = $e->getCode();
    $error_message = $e->getMessage();
    echo $code.": ".$error_message."<br />";
}
?>

Summary

As you saw, how insanely easy it is to upload a large file in blob storage using PHP SDK. I didn’t (and still don’t) know anything about PHP but I was able to put this code together in a matter of hours. I think if you’re a PHP developer, you should be able to do so in minutes. I hope you’ve found this information useful. As always, if you find any issues with the post please let me know and I’ll fix it ASAP.