DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. AWS S3 Storage Gateway Revisited (Part 1)

AWS S3 Storage Gateway Revisited (Part 1)

This dive into AWS S3 Storage Gateway covers what has changed since its inception and its main functions as a file, volume, or VTL gateway.

Greg Schulz user avatar by
Greg Schulz
·
Jun. 16, 17 · Review
Like (0)
Save
Tweet
Share
6.33K Views

Join the DZone community and get the full member experience.

Join For Free

server storage i/o trends

these amazon web services (aws) storage gateway revisited posts are a follow-up to the aws storage gateway test drive and review i did a few years ago (thus why it’s called revisited). as part of a two-part series, the first post looks at what aws storage gateway is, how it has improved since my last review of aws storage gateway along with deployment options. the second post in the series will look at a sample test drive deployment and use.

if you need an aws primer and overview of various services such as elastic cloud compute ( ec2 ), elastic block storage ( ebs ), elastic file service ( efs ), simple storage service ( s3 ), availability zones ( az ), regions and other items check this multi-part series ( cloud conversations: aws ebs, glacier and s3 overview (part i) ).

aws

as a quick refresher, s3 is the aws bulk, high-capacity unstructured and object storage service along with its companion deep cold (e.g. inactive) glacier . there are various s3 storage service classes including standard reduced redundancy storage (rrs) along with infrequent access (ia) that have different availability durability, performance, service level and cost attributes .

note that s3 ia is not glacier, as your data always remains online accessible while glacier data can be off-line. aws s3 can be accessed via its api , as well as via http rest calls, aws tools along with those from third-party’s. third party tools include nas file access such as s3fs for linux that i use for my ubuntu systems to mount s3 buckets and use similar to other mount points. other tools include cloudberry , s3 motion , s3 browser as well as plug-ins available in most data protection (backup, snapshot, archive) software tools and storage systems today.

aws s3 storage gateway and what’s new

the storage gateway is the aws tool that you can use for accessing s3 buckets and objects via your block volume, nas file, or tape-based applications. the storage gateway is intended to give s3 bucket and object access to on-premise applications and data infrastructures functions including data protection (backup/restore, business continuance (bc), business resiliency (br), disaster recovery (dr) and archiving), along with storage tiering to cloud.

some of the things that have evolved with the s3 storage gateway include:

  • easier, streamlined download, installation, deployment
  • enhanced virtual tape library (vtl) and virtual tape support
  • file serving and sharing (not to be confused with elastic file services (efs))
  • ability to define your own bucket and associated parameters
  • bucket options including infrequent access (ia) or standard
  • options for aws ec2 hosted, or on-premise vmware as well as hyper-v gateways (file only supports vmware and ec2)

aws storage gateway three functions

aws storage gateway can be deployed for three basic functions:

aws storage gateway file architecture aws storage gateway file architecture via aws.com

file gateway (nfs nas): files, folders, objects, and other items are stored in aws s3 with a local cache for low latency access to most recently used data. with this option, you can create folders and subdirectory similar to a regular file system or nas device as well as configure various security, permissions, access control policies. data is stored in s3 buckets that you specify policies such as standard or infrequent access (ia) among other options. aws hosted via ec2 as well as vmware virtual machine (vm) for on-premise file gateway.

also, note that aws cautions on multiple concurrent writers to s3 buckets with storage gateway, so check the aws faqs which may have changed by the time you read this. current file share limits (subject to change) include 1 file gateway share per s3 bucket (e.g. a one-to-one mapping between file share and a bucket). there can be 10 file shares per gateway (e.g. multiple shares each with its own bucket per gateway) and a maximum file size of 5tb (same as maximum s3 object size). note that you might hear about object storage systems supporting unlimited size objects which some may do, however generally there are some constraints either on their api front-end, or what is currently tested. view current aws storage gateway resource and specification limits here .

aws storage gateway cached volume architecture aws storage gateway cached volume architecture via aws.com aws storage non-cached e.g. stored volume gateway architecture aws storage gateway non-cached volume architecture via aws.com

volume gateway (block iscsi): leverages s3 with a point in time backup as an aws ebs snapshot. two options exist including cached volumes with low-latency access to most recently used data (e.g. data is stored in aws, with a local cache copy on disk or ssd). the other option is stored volumes (e.g. non-cached) where the primary copy is local and periodic snapshot backups are sent to aws. aws provides ec2 hosted, as well as vms for vmware and various hyper-v windows server based vms.

current storage gateway volume limits (subject to change) include maximum size of a cached volume 32tb, the maximum size of a stored volume 16tb. note that snapshots of cached volumes larger than 16tb can only be restored to a storage gateway volume, they can not be restored as an ebs volume (via ec2). there is a maximum of 32 volumes for a gateway with a total size of all volumes for a gateway (cached) of 1,024tb (e.g. 1pb). the total size of all volumes for a gateway (stored volume) is 512tb. view current aws storage gateway resource and specification limits here . aws storage gateway vtl architecture

aws storage gateway vtl architecture via aws.com

virtual tape library gateway (vtl): supports saving your data for backup/bc/dr/archiving into s3 and glacier storage tiers. being a virtual tape library (e.g. vtl) you can specify emulation of tapes for compatibility with your existing backup, archiving and data protection software, management tools, and processes.

storage gateway limits for tape include minimum size of a virtual tape 100gb, maximum size of a virtual tape 2.5tb, maximum number of virtual tapes for a vtl is 1,500 and total size of all tapes in a vtl is 1pb. note that the maximum number of virtual tapes in an archive is unlimited and the total size of all tapes in an archive is also unlimited. view current aws storage gateway resource and specification limits here .

where to learn more

  • aws storage gateway site
  • aws storage gateway resource limits and specifications and pricing
  • aws storage gateway, getting started , managing volumes , troubleshooting and local console
  • cross-region replication for amazon s3
  • aws (amazon) storage gateway, first, second and third impressions
  • cloud conversations: if focused on cost you might miss other cloud storage benefits
  • data protection diaries
  • cloud conversations: aws overview and primer
  • eight ways to avoid cloud storage pricing surprises
  • cloud and object storage center
  • are more than five nines of availability really possible?
  • how do primary storage clouds and cloud for backup differ?
  • what’s most important to know about my cloud privacy policy?
  • cloud conversations: aws s3 cross region replication storage enhancements
  • s3motion buckets containers objects aws s3 cloud and emccode
  • aws efs elastic file system (cloud nas) first preview look

what this all means

as to which gateway function and mode (cached or non-cached for volumes) depends on what it is that you are trying to do. likewise choosing between ec2 (cloud hosted) or on-premise hyper-v and vmware vms depends on what your data infrastructure support requirements are. overall i like the progress that aws has put into evolving the storage gateway, granted it might not be applicable for all usage cases.

ok, nuff said (for now…).

AWS Amazon Web Services File system Data infrastructure Cloud Object (computer science) Hyper-V Object storage Web Service

Published at DZone with permission of Greg Schulz, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • How to Rescue Your Magento 2 Project
  • Hackerman [Comic]
  • The New DevOps Performance Clusters
  • Promises, Thenables, and Lazy-Evaluation: What, Why, How

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: