Software Defined, Cloud, Bulk, and Object Storage Fundamentals

Welcome to the cloud, Big Data, software defined, bulk, and object storage fundamentals page — part of the objectstoragecenter.com microsite collection of resources. Software defined, bulk, cloud, and object storage exist to support expanding and diverse application data demands. There are various types of cloud, bulk, and object storage, including public services such as Amazon Web Services (AWS) Simple Storage Service (S3), Google, Microsoft Azure, IBM Softlayer, Rackspace among many others. There are also solutions for hybrid and private deployment from Cisco, DDN, Dell EMC, Fujitsu, HDS, HPE, IBM, NetApp, Noobaa, OpenStack, Quantum, Rackspace, Scality, Seagate, Spectra, Storpool, Suse, Swift, and WD among others.

Cloud products and services among others, along with associated data infrastructures including object storage, file systems, repositories and access methods are at the center of bulk, Big Data, big bandwidth, and little data initiatives on a public, private, hybrid and community basis. After all, not everything is the same in cloud, virtual and traditional data centers or information factories from active data to inactive deep digital archiving.

Cloud Object Storage Fundamentals Access and Architectures

There are many facets to object storage, including technology implementation, products, services, access, and architectures for various applications and use scenarios.

• Project or Account – Top of the hierarchy that can represent the owner or billing information for a service where buckets are also attached.
• Region – Location where data is stored. It can include one or more data centers, also known as Availability Zones.

Moving and replicating buckets/containers, subfolders, and objects.

• An Availability Zone (AZ) or data center or server implements durability and accessibility for availability within a region.

Example of Regions and Availability Zones (AZs)

• Bucket or Container – Where objects or sub-folders containing objects are attached and accessed.




• Sub-folder – While object storage can be located in a flat namespace for commonality and organization, some solutions and services support the notion of sub-folders that resemble traditional directory hierarchy.
• Object – Byte (or bit) stream that can be as small as one byte to as large as several Tbytes (some solutions and services support up to 5TByte-sized objects). The object contains an organization's data and metadata. Different solutions and services support sizes from a couple hundred KBytes of metadata to MBytes worth of metadata. Anything from files, videos, images, virtual disks (VMDKs, VHDX), ZIP or TAR files, backup and archive save sets, executable images or ISOs can be considered objects for storage.
• End-point – Where or what your software, application or tool and utilities, along with gateways, attach to for accessing buckets and objects.

A common theme for object storage is flexibility, along with scaling (performance, availability, capacity, economics), along with extensibility — without compromise or complexity. From those basics, there are many themes and variations regarding how data is protected (RAID or no RAID, hardware or software), deployed as a service or as tin-wrapped software (an appliance), optimized for archiving or video serving, or other applications.

Many Facets of Cloud and Object Storage Access

One aspect of object and cloud storage is accessing or using object methods, including application programming interfaces (APIs) vs. traditional block (LUN) or NAS (file) based approaches. Keep in mind that many object storage systems, software, and services support NAS file-based access, including NFS, CIFS, HDFS, among others for compatibility and ease of use.

Likewise, various APIs can be found across different object solutions, software, or services including Amazon Web Services (AWS) Simple Storage Service (S3), HTTP REST based, among others. Other APIs will vary by specific vendor or product, however, and can include iOS (e.g. Apple iPhone and iPad), WebDav, FTP, JSON, XML, XAM, CDMI, SOAP, and DICOM, among others. Another aspect of object and cloud storage is expanded and dynamic metadata.

While traditional file systems and NAS have simple or fixed metadata, object and cloud storage systems, services, and solutions, along with some scale-out file systems, have the ability to support user defined metadata. Specific systems, solutions, software, and services will vary on the amount of metadata, which could range on the low-end from 100s of KBytes to tens or more MBytes.