DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Solving the Kubernetes and Storage Challenge

Solving the Kubernetes and Storage Challenge

Companies who want Kubernetes will want to know how it can solve their issues regarding persistant storage.

Vamsi Chemitiganti user avatar by
Vamsi Chemitiganti
·
Jan. 16, 19 · Presentation
Like (4)
Save
Tweet
Share
15.80K Views

Join the DZone community and get the full member experience.

Join For Free

This article was written jointly with Peter Fray. 

Enterprise Infatuation with Kubernetes

It seems that everyone is infatuated with Kubernetes. Every organization has an active project with Kubernetes or planning one. When adopting Kubernetes, organizations are quickly realizing that many decisions need to be made and that Kubernetes is no simple thing. Some common questions are:

  1. How will I deploy Kubernetes?
  2. Where will I deploy Kubernetes?
  3. Do I want to use public cloud versions of Kubernetes? Are they reliable?
  4. What networking model should our organization use? Calico, Weave, Flannel, other?
  5. How is storage managed and deployed in Kubernetes?
  6. Once we deploy Kubernetes, what about Day 2 operations?

In this blog, we will focus on the first challenge once Kubernetes is up and running: storage.

As we work with large enterprise customers on their container deployments, a few truths begin to emerge. Every customer is looking to standardize on Kubernetes as the standard for container orchestration of enterprise applications, and newer cloud-native applications need all the enterprise add-ons that data centers provide.

Enterprises across the Fortune 5000 are looking to leverage Kubernetes in two modes, the first as a way of hosting greenfield applications, but also for re-architecting and modernizing legacy applications so that they're Kubernetes-based. Kubernetes offers robust features and APIs that provide high availability, scalability, support for running portable containers, and for deployment concerns such as rolling upgrades, canary deployments, etc.

Storage is critical for real-world pod-based infrastructure. Container files are transient in nature which means that on termination or crashing, all user data is lost. Also, containers organized into a pod read to share files and data assets. The Volume abstraction accounts for both scenarios.

Therefore, one of the key considerations for Kubernetes is enterprise storage support. Applications running in containers need to perform a range of data management tasks and CRUD functions. There is a wide range of persistent storage paradigms that Kubernetes needs to support. Storage has to be a first-class citizen for running legacy applications in relational databases, message queues, NoSQL databases etc.

The Kubernetes project at CNCF has helped incubate the CSI (Container Storage Interface) to enable a range of third-party storage vendors (SP) to develop a standard plugin for their product that is CSI API compliant. Once created, this should integrate seamlessly across a number of container orchestration (CO) systems such as Kubernetes, Mesos, etc. that implement the CSI.

Enter Platform9 Managed Kubernetes (PMK)

Platform9 offers the industry's only Kubernetes managed service. The Platform9 Managed Kubernetes (PMK) simplifies and addresses the above challenges for deploying and managing Kubernetes clusters across any infrastructure or cloud. PMK allows for simple deployment of Kubernetes clusters, scaling these clusters, monitoring, upgrading and has a first class service for troubleshooting these deployments. Whether your organization is just dipping your toe into the Kubernetes ecosystem with your first K8s clusters or has hundreds of Kubernetes clusters deployed, Platform9 can manage these environments.

Deploying Kubernetes on Platform9

I have deployed Kubernetes on Platform9 many times. I wanted to create a new environment for testing storage and Platform9 Managed applications. I set up seven physical servers in our datacenter. I installed Ubuntu 16.04 and was ready to install the Kubernetes Cluster.

  1. I downloaded the Platform9 Agent and installed it on each server
  2. Registered the Agent for Platform9 Managed Kubernetes
  3. Followed the Platform9 Install wizard and created a Kubernetes Cluster.

Simple! Kubernetes up and running in 10 minutes.


Once Kubernetes has been deployed and development teams start using the new environments one of the first questions that is asked is: "How do I solve persistent storage?" This is the first and a very common challenge that organizations are facing. While dev teams may want to write 12-factor stateless applications, there will be requirements to manage stateful datasets when stateful services like databases, queues, key-value storage, big data and machine learning are running on Kubernetes. The question about storage will come up, it's just a matter of time, and it will need to be solved in order for organizations to be successful on their Kubernetes journey.

Storage in Kubernetes is complex, and from the Kubernetes documentation, there are many types of storage and they can change from cloud provider to cloud provider:

  1. Volumes
  2. Persistent Volumes
  3. Volume Snapshots
  4. Storage Classes
  5. Volume Snapshot Classes
  6. Dynamic Volume Provisioning

PMK offers complete flexibility across all the above decisions that need to be made when deploying K8s. However, when combined with Portworx the storage decision is greatly simplified.

Enter Portworx

After what seemed to be hours of time with the Google machine and reading Kubernetes documentation about storage, I ran into Portworx, which provides a solution for Kubernetes storage. I later even discovered that Portworx is a Platform9 partner. Already having deployed Kubernetes and getting excited to deploy some applications on my newly-created Kubernetes cluster, I needed a simple, fast solution to provide storage to my containers.

From the Portworx website:

"Portworx is the cloud-native storage company that enterprises depend on to reduce the cost and complexity of rapidly deploying containerized applications across multiple clouds and on-prem environments.

With Portworx, you can manage any database or stateful service on any infrastructure using any container scheduler. You get a single data management layer for all of your stateful services, no matter where they run. Portworx thrives in multi-cloud environments."

In my environment, I have seven servers that I used Platform9 to deploy a Kubernetes cluster with three master nodes and four workers. After a few clicks of the mouse, I navigated through the Portworx documentation to the "Install on Kubernetes - all other distributions" page. I clicked the "Generate a Spec" link.

I was prompted for:

  1. Etcd details. I chose Portworx hosted because that was easier than installing my own and I didn't want to use my Kubernetes etcd.
  2. Selected use the "on-prem" storage and use unmounted volumes. This was a great option for me as I already had 4 50gb unmounted volumes on my worker nodes.
  3. I selected default values for the networking.
  4. Finally, under advanced settings, I selected to enable the UI and Stork, which is a Kubernetes scheduler extension created by Portworx to enable hyperconvergence of pods and storage, as well as cross-cloud migrations.

OK. This seems too easy to be true. It just what I was looking for. I was presented a spec.yaml which I downloaded and executed:

Kubectl apply -f spec.yaml


That couldn't have been easier. After waiting for some time for all the Docker layers to download, I had a complete up and running Portworx environment.

Conclusion

The bottom line is, in a matter of about 30 minutes I was able to install a complete Kubernetes cluster with Platform9 Managed Kubernetes and have a fully functional storage solution. I think the best description of these two tools working together would be described as "simplicity in motion."

After everything was installed I wanted to test this solution to ensure dynamic PVC's were in fact working and usable by my Kubernetes Cluster. So I decided to install some applications that require the use of persistent storage.

I installed:

  1. Minio - Opensource AWS S3 environment for your local datacenter
  2. Jenkins-x - Opensource CI/CD pipeline environment
  3. Prometheus - Opensource Kubernetes Monitoring
  4. Grafana - Opensource graphing engine I used as a front end for Prometheus
  5. EFK Stack - Kubernetes Logging and Monitoring: The Elasticsearch, Fluentd, and Kibana (EFK)

After a half day, I have a highly available Kubernetes cluster running in Platform9 Managed Kubernetes with Platform9 Managed applications and Portworx fully functional. I never had a worry about my storage environment.

Kubernetes Machine learning Docker (software) application Database Big data

Published at DZone with permission of Vamsi Chemitiganti, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • How Do the Docker Client and Docker Servers Work?
  • An Introduction to Data Mesh
  • Understanding gRPC Concepts, Use Cases, and Best Practices
  • Stream Processing vs. Batch Processing: What to Know

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: