DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Container Checkpointing in Kubernetes With a Custom API
  • Leveraging Seekable OCI: AWS Fargate for Containerized Microservices
  • Deploying Dockerized Applications on AWS Lambda: A Step-by-Step Guide
  • Strategic Deployments in AWS: Leveraging IaC for Cross-Account Efficiency

Trending

  • Unlocking AI Coding Assistants Part 4: Generate Spring Boot Application
  • A Guide to Developing Large Language Models Part 1: Pretraining
  • Beyond Linguistics: Real-Time Domain Event Mapping with WebSocket and Spring Boot
  • Building Enterprise-Ready Landing Zones: Beyond the Initial Setup
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Mount AWS EFS, NFS or CIFS/Samba volumes in Docker

Mount AWS EFS, NFS or CIFS/Samba volumes in Docker

How to deal with large file systems that must be spanned and preserved against several Docker services using Amazon EFS.

By 
Jeremy Unruh user avatar
Jeremy Unruh
·
Oct. 29, 15 · Tutorial
Likes (6)
Comment
Save
Tweet
Share
28.8K Views

Join the DZone community and get the full member experience.

Join For Free

Recently I had a project where I was dealing with large file systems that had to be spanned and preserved against hundreds of Docker services for computation.  I decided to use Amazon's EFS which was recently announced at re:Invent 2015.

I setup the file system which was easy and attempted to play with it.  I first mounted it manually via NFS4 on one of the Mesos slaves.  It worked and initial tests showed great performance.   Now
the next step was meeting some of my personal requirements before I could call this successful.

My Requirements

  • The file system should be mounted when needed within a Docker container and tore down when no longer in use
  • EFS provides endpoints in each availability zone so I also needed to support dynamic linking based on which slave the container was running in and the corresponding endpoint matching the AZ.
  • Must use NFS4 (This is what EFS supports)
  • Did not want to create or modify existing container images to attempt to mount NFS within the container itself.  This also would mean the container would need to run in privileged mode which I’m against

After analysis, I decided the best choice was to write a plugin myself to extend the Docker engine.

Docker Volume Plugins

Docker volume plugins enable Docker deployments to be integrated with external storage systems. A volume plugin makes use of the -v and --volume-driver flag on the docker run command. For example:

$ docker run -ti -v volumename:/data --volume-driver=some_plugin ubuntu sh

This command passes the volumename through to the volume plugin as a user-given name for the volume.

By having the user specify a volumename, a plugin can associate the volume to an external volume beyond the lifetime of a single container or container host. This can be used, for example, to move a stageful container from one server to another.

Meet Docker-Volume-Netshare Plugin

Before I began coding a plugin I wanted to pre-define some requirements. Here’s some of them below:

  • Write it in Go so it runs at native speed
  • Support NFS 3/4 and CIFS (might come in handy someday)
  • For EFS support the plugin will lookup local EC2 metadata on the slave to find the corresponding AZ and region for building the URI from the File System ID
  • Must support volume management
    • Understand when a volume is already mounted and keep track of its connections
    • If a container is stopped determining based on active connections if the volume mount should be removed
  • Support all of the Docker Plugin API

After designing the plugin I installed it on the slaves (container hosts) and as a result I was able to dynamically mount EFS volumes and link them into the container based on the specified target mount.

Using Netshare:

First thing to do is download the binary for your particular platform at: https://github.com/gondor/docker-volume-netshare

The examples below are just a few and the plugin offers many options.  Let’s start with EFS.

Mounting EFS Volumes

The example below will start a container and mount the specified EFS File System by the FS_ID (assigned by AWS) into a directory called /data in the container.

  1. Run the plugin or add it to systemd
    $ sudo docker-volume-netshare efs
  2. Run a container to test:
    $ docker run -i -t --volume-driver=efs -v aws_fs_id:/data ubuntu /bin/bash

Mounting CIFS/Samba Volumes

The example below will start a container and mount the specified CIFS server/sharename into a directory called /data in the container.

  1. Run the plugin or add it to systemd
    $ sudo docker-volume-netshare cifs --username user --password pass --domain domain
  2. Run a container to test:
    $ docker run -i -t --volume-driver=cifs -v hostname/share:/data ubuntu /bin/bash

Mounting NFS Volumes

  1. Run the plugin or add it to systemd
    $ sudo docker-volume-netshare nfs
  2. Run a container to test:
    $ docker run -i -t --volume-driver=nfs -v hostname/volume:/data ubuntu /bin/bash

Whats Next…

Here are some future enhancements I will be adding to the plugin based on requests and priority:

  • Support for credential files based on hosts or possibly a credential service
  • REST service to show stats on mounted volumes and how many connections
  • RPM packages for enterprise linux
  • Systemd and init based scripts

For more information see full documentation at: https://github.com/gondor/docker-volume-netshare

Network File System Docker (software) AWS File system

Published at DZone with permission of Jeremy Unruh, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Container Checkpointing in Kubernetes With a Custom API
  • Leveraging Seekable OCI: AWS Fargate for Containerized Microservices
  • Deploying Dockerized Applications on AWS Lambda: A Step-by-Step Guide
  • Strategic Deployments in AWS: Leveraging IaC for Cross-Account Efficiency

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!