With applications moving to the cloud, how can we do without containers! Click above Container Magic Cube to follow me
This article is reproduced from “Container Era”
Author: Orain Xiong
Kubernetes itself includes a powerful storage subsystem with a wide range of use cases. However, when we build a relational database platform using Kubernetes, we face a challenge: establishing data storage. This article discusses how to expand CSI (Container Storage Interface) 0.2.0 while integrating Kubernetes and demonstrates the importance of dynamic expansion.
1
st
Introduction
Container orchestration technology has great potential for development. Developers hope to redesign standalone applications running on virtualized infrastructure and bare metal through open-source solutions.
In terms of scalability and technical maturity, Kubernetes and Docker are at the top of the industry. Relational databases are crucial for migration, but migrating standalone applications to distributed orchestration systems like Kubernetes is quite challenging.
For relational databases, we should focus on their storage capabilities. Kubernetes itself has a powerful storage subsystem with extensive functionality and use cases. However, building data storage when utilizing Kubernetes to construct a relational database platform poses a challenge. Additionally, Kubernetes has some fundamental functionalities that are not yet implemented, especially dynamic expansion. It may sound tedious, but operations such as creating, deleting, mounting, and unmounting are very necessary.
Currently, expansion can only be used with storage providers such as:
-
gcePersistentDisk
-
awsElasticBlockStore
-
OpenStack Cinder
-
glusterfs
-
rbd
To enable this feature, we need to set the feature-gate expandpersistentvolume to True and enable the PersistentVolumeClaimResize admission plugin. Once PersistentVolumeClaimResize is enabled, the ability to resize storage will be activated for storage classes where allowVolumeExpansion is set to True.
Unfortunately, even if the underlying storage provider has this capability, dynamic expansion via Container Storage Interface (CSI) and Kubernetes is still not available.
This article will provide a simplified view of CSI, then introduce how to implement a new volume expansion feature on existing CSI and Kubernetes. Finally, it will demonstrate how to dynamically expand.
2
nd
Container Storage Interface (CSI)
To better understand our subsequent work, we first need to know what the Container Storage Interface is. Currently, the existing storage subsystem in Kubernetes still has many issues. The storage driver code is maintained in the Kubernetes core repository, which complicates testing. Additionally, Kubernetes requires storage vendors to check code into the Kubernetes core repository. Ideally, this requirement should be implemented externally. The design of CSI aims to define an industry standard that allows storage providers compatible with CSI to be used by container orchestration systems that support CSI.
This diagram illustrates a high-level Kubernetes prototype combined with CSI:
-
Introduced three new external components to decouple Kubernetes and storage provider logic
-
The blue arrows represent the conventional method of calling the API server
-
The red arrows represent gRPC calls to the volume driver
See:
https://github.com/container-storage-interface/spec/blob/master/spec.md
3
rd
Extending CSI and Kubernetes
To implement the volume expansion functionality on Kubernetes, we should extend multiple components, including the CSI specification, “In-Tree” volume plugins, external providers, and external connectors.
Extending the CSI Specification
The latest CSI 0.2.0 does not yet define the feature for expanding volumes. Three new RPCs should be introduced, including RequiresFSResize, ControllerResizeVolume, and NodeResizeVolume.
service Controller {rpc CreateVolume (CreateVolumeRequest) returns (CreateVolumeResponse) {}……rpc RequiresFSResize (RequiresFSResizeRequest) returns (RequiresFSResizeResponse) {}rpc ControllerResizeVolume (ControllerResizeVolumeRequest) returns (ControllerResizeVolumeResponse) {}}service Node {rpc NodeStageVolume (NodeStageVolumeRequest) returns (NodeStageVolumeResponse) {}……rpc NodeResizeVolume (NodeResizeVolumeRequest) returns (NodeResizeVolumeResponse) {}}
4
th
Extending “In-Tree” Volume Plugins
To extend the CSI specification, the csiPlugin interface needs to be implemented on Kubernetes. The csiPlugin interface will extend PersistentVolumeClaim to proxy the ExpanderController.
type ExpandableVolumePlugin interface {VolumePluginExpandVolumeDevice(spec Spec, newSize resource.Quantity, oldSize resource.Quantity) (resource.Quantity, error)RequiresFSResize() bool}
5
th
Implementing Volume Drivers
Finally, to abstract the complexity of implementation, we should hard-code the logic for managing separate storage providers into the following functions defined in the CSI specification:
-
CreateVolume
-
DeleteVolume
-
ControllerPublishVolume
-
ControllerUnpublishVolume
-
ValidateVolumeCapabilities
-
ListVolumes
-
GetCapacity
-
ControllerGetCapabilities
-
RequiresFSResize
-
ControllerResizeVolume
Example:
Let’s illustrate this feature with a concrete user case.
-
Create a storage class for the CSI storage provider
allowVolumeExpansion: trueapiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: csi-qcfsparameters: csiProvisionerSecretName: orain-test csiProvisionerSecretNamespace: defaultprovisioner: csi-qcfspluginreclaimPolicy: DeletevolumeBindingMode: Immediate
-
Deploy the CSI volume driver across Kubernetes clusters, including the storage provider CSI -qcfsplugin
-
Create a PVC qcfs-pvc that will be dynamically provided by the storage class csi-qcfs
apiVersion: v1kind: PersistentVolumeClaimmetadata: name: qcfs-pvc namespace: default….spec: accessModes: – ReadWriteOnce resources: requests: storage: 300Gi storageClassName: csi-qcfs
-
Create a MySQL 5.7 instance to use the PVC qcfs-pvc
-
To reflect the exact same production-level scenario, this is divided into two different types of workloads, including:
1. Bulk inserts that cause MySQL to consume more filesystem capacity
2. Increased query requests
-
Dynamically expand the capacity by editing the pvc qcfs-pvc configuration
The integration of Prometheus and Grafana allows us to visualize the corresponding key metrics.
We noticed that the intermediate readings show that the size of MySQL data files slowly increases during bulk inserts. Meanwhile, the bottom readings show that the filesystem expanded twice in about 20 minutes, from 300G to 400G, then to 500G. At the same time, the readings from above indicate that the entire expansion process was completed immediately, with minimal impact on MySQL QPS.
6
th
Conclusion
No matter where infrastructure applications run, databases are always a critical resource. It is crucial to fully support database needs with advanced storage subsystems. This will help drive broader adoption of cloud-native technologies.
Original link:
https://kubernetes.io/blog/2018/08/02/dynamically-expand-volume-with-csi-and-kubernetes/
Recommended Reading
Authoritative Interpretation of Container Storage Standards | CSI White Paper (Part 1)
Authoritative Interpretation of Container Storage Standards | CSI White Paper (Part 2)
Prize-Winning Essay
Talk about your experiences with cloud-native
The Huawei Cloud Container Team is currently collecting stories, technical insights, cloud migration practices/cases, etc.
Participating gives you a chance to win Huawei wristbands, cloud-native professional books, JD E-cards, and other multiple gifts!
Sharing creates greater value; scan the QR code below to participate!
Click below to read the original article and participate in the activity