Migrate HDFS Data to Azure
A developer and Hadoop expert runs through the processes he and his team used to transfer their data over network with TLS encryption when switching to Azure.
Join the DZone community and get the full member experience.Join For Free
During the middle of last year, my team decided to move our Hadoop workloads to Azure, including our data and applications. This article provides some of the best practices we used in migrating on-premises HDFS data to Azure HDInsight. Mentioned below are two approaches that we adopted to transfer the data over network with TLS encryption.
The ExpressRoute Azure service uses a private connection between Azure and on-premise data centers (ExpressRoute offers higher security, reliability, and speeds with lower latencies than typical connections over the Internet). We took advantage of Data Factory's native data copy functionality using the integration runtime to migrate the data. Data Factory's self-hosted integration runtime (SHIR) should be installed on a pool of Windows VMs on an Azure virtual network. The VMs can be scaled out to multiple VMs to fully utilize network and storage IOPS or bandwidth.
This approach is recommended when there is a need to migrate huge amounts of data (i.e. data volume > 50 GB) for legacy applications.
Transfer your data to Azure storage over a regular internet connection using any one of several tools, such as: AzCopy, Azure PowerShell, and Azure CLI.
This approach is used to copy smaller volumes of data (i.e. 2-5 GB or less), or, for ad-hoc purposes, PoC environment data, or to transfer data to Azure that does not involve any transformations.
azcopy command copies an entire directory (recursively) using
azcopy and all the underlying files in that directory to an Azure container. The result is a directory in the Azure container by the same name.
az cli command uploads a file named
upld.txt from the local file system to a directory
testdir in an Azure container named
testcon under the storage account
The above set of PowerShell commands uploads a file named
upld.txt from the local file system to a directory called
testdir in the Azure container named
Opinions expressed by DZone contributors are their own.