Transcription
IBM Spectrum Scale:Use cases and (field) lessonslearned with Kubernetes andOpenShift—Harald SeippSenior Technical Staff MemberCenter of Excellence for Cloud Storage
EMEA Storage Competence CenterThe EMEA Storage Competence Center (ESCC) provides Pre- to Post-Sales Storage services to assist youThe EMEACompetenceCenter (ESCC)providesPre- to Post-Salesservices to assist youacrossIBM’sStoragecompleteStorage portfolio(Hardware/ Software/ Solutions)Storagevia:across IBM’s complete Storage portfolio (Hardware / Software / Solutions) via:Advanced Technical Skills (ATS)AdvancedSkills(ATS)Solution Enablement, Architectural GuidanceSkillTransfer,TechnicalNew ProductIntro.,Skill Transfer, New Product Intro., Solution Enablement, Architectural GuidanceLab Services (LS)LabServices (LS)SkillEnablement,Hands-On-Training, Implementation Assistance, Data Migration &SkillEnablement,Performance TuningHands-On-Training, Implementation Assistance, Data Migration &Performance TuningDevelopment & Product Test Systems LabDevelopmentProductTest SystemsProofof Concept&(PoC),Hands-onProductLabTraining Proof of Concept (PoC), Hands-on Product Training Product Field Engineering (PFE)ProductField Engineering(PFE)Lastlevel defectsupportLast level defect supportClient CareClient CarePro-activeCustomer Care ManagementPro-active Customer Care Management Copyright IBM Corporation 20202
Outline Baseline Use Cases ADAS CSP Dev/Test Multi-cloudTopics clients ask for Unsupported platforms ;-) Security / Multi-tenancy Heterogenous environmentsSummaryIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation3
Baseline: Containers Everywhere Multi-Cloud: On-premises and Public Clouds————— Development, DevOps and continuous integration—— Elastic scheduling and auto-scalingImproved resource utilizationSecure isolation and Multi-TenancyPortable and reproducible serviceOne-click Laptop to SupercomputerRe-use of applications and servicesSimplify and accelerate application deploymentMicroservices Architecture——Programming language and technology stack independenceFaster and easier developmentIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation4
Baseline: KubernetesPersistent VolumesDeveloperA Persistent Volume (PV) is a unit ofstorage that has been provisioned by anadministrator or dynamicallyprovisioned via a storage driver/plug-in.Claims a PV from the poolA PersistentVolumeClaim (PVC) is arequest for storage by a user.claimReferences Claims in ContainerPodContainerclaimPVCs consume PV resources.A Storage Class is pre-defined by theadmin to dynamically provision PVs.IBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM CorporationPool of Persistent Volumes5
IBM persistent storage architecture 2017-2019Kubernetes ClusterIBM Storage Enablerfor Containers providesKubernetes FlexVolumeDriversIBM Spectrum Connectprovides the Storagemanagement API andStorage classes for IBMBlock StorageMaster NodesWorker NodeWorkloadWorkloadContainerContainerWorker NodeWorkloadWorkloadContainerContainerIBM Storage Enabler for ContainersIBM Spectrum Scale (client)IBMSpectrumConnect(VM)iSCSI orFCS3 API or S3fsFlexVolumeBlockFile Storage / NASIBM SpectrumScaleIBM Block StorageData PathManagementIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation6
IBM persistent file storage architecture 2019 InterfaceCSIContainer platform ArchRoadmapInstallation IBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM CorporationKubernetes 1.13 CSI v1.1.0Operator-based install DockerHub githubStateless pluginCSI certificationOperatorFull Open Source: de path from SEC to the new CSI plugin.7
Multi-cluster supportKubernetes Cluster with CSIMaster Node(s)Infrastructure NodeInfrastructure NodeWorker NodeWorker ice AMicroservice BCSI - PluginCSI - PluginCSI - PluginCSI - PluginCSI - PluginScale ClientScale ClientScale ClientScale ClientScale ClientScale GUI Server(s)Scale GUI Server(s)Worker NodeSpectrum Scale Client clusterScale NSD ServerScale NSD ServerSpectrum Scale Storage clusterHighspeednetworkScale NSD Server
SEC vs. CSIFeatureStorage Enabler for ContainersCSI Driver 1.0.xStatic ProvisioningNo K8s native static provisioningsupport, but there is way to use anexisting fileset as a PVYesDynamic ProvisioningFileset based (dependent andindependent)Lighweight (directory based) and filesetbased (dependent and independent)Multiple Filesystem SupportYesYesRemote cluster mountsLimited supportYesDeploymentHelm ChartOperatorSpectrum Scale Version5.0.0 5.0.4.1 Platform supportRHEL 7.x on x86 64, ppc64le, s390xRHEL 7.x on x86 64Kubernetes Version1.10, 1.11, and 1.121.13 Openshift Support3.11 via RPQ only4.2Migration path from SEC to CSIN/AManual offline migrationIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation9
Use CasesIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation10
Spectrum Scale use caseat automotive client Containerized platform to train and test the AI for anADAS (Advanced Driver Assistance Systems) projectHigh-bandwidth data ingest (double-digit TB perday) through Spectrum Scale/ESS Sophisticated cloud architecture (see next slide) Skilled admins and developersIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation11
Spectrum Scale use caseat automotive client – cluster architectureRedHat OpenShiftRHELOpenStackVMware ESXx86 Servers Nvidia GPUsESS 5U84StorageIBM Spectrum Scale / IBM Elastic Storage Server (ESS)IBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation12
Spectrum Scale use caseat automotive client – lessons learned To get the Spectrum Scale client up & running–Assigned an additional OpenStack network witha dedicated NIC to the VM – With OpenStack floating IP the Scale ClusterIP was not visible within VMAdjusted OpenStack security groups to allowinbound traffic to the Spectrum Scale portsTo ensure that persistent Pods are placed onthe Spectrum Scale node(s)–Labeled the node and added a nodeSelector tothe persistent Pod deployment configsIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation13
Spectrum Scale use caseat automotive client – lessons learned (cont.) Made Storage Enabler for Containers 2.0.0 work– Re-installed with SEC 2.1.0– Steps are now documented as solution blueprintSEC helm chart 1.0.1 requires container privilegeadjustments for the deployment to succeedExisting data can be integrated thoughadditional storage class for existing filesets–Created a PVC to re-use an existing SpectrumScale fileset (ingest directory)–Ensure sufficient access rights for the containerprocessIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation14
Spectrum Scale use caseat Cloud Service Provider: AIaaS*User requestwith parametersand dataCreate VPN sessionManagement ServerSubmit RequestReturn SessionRouterSpawn ContainerGPU 1GPU 2GPU 3GPU 4IBM HWMgmtPower9 LC922 x86 Server FarmPower9 AC922 Power AICust 1 Cust 2 Cust 3 Cust 4IBMConductorIBM CloudPrivateCust 1Cust 2Cust 3 Cust 4Cust nIIB10 GbitEPower9 LC921 EdgeIB 100 Gbit read/write data sets from local storageCust 1 Cust 2 Cust 3 Cust 4ManagmentTier 3SAS/SATASpectrum Scale FilesystemTier 1NVMeIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM CorporationTier 1NVMeTier 2SSDTier 3SAS/SATADockerRepository* Artificial Intelligence as a service15
Spectrum Scale use caseat CSP: AIaas – lessons learned For the CSP, multi-tenant isolation and data managementis more important than dynamic provisioning–Used (semi-)static provisioning to prevent Fileset sprawl–CSP pre-creates PVs/Filesets with own naming conventions–Better control of tenant-related services (Snapshots,Backup)Integration with 3-Tier concept–Through Fileset placement policies–One Kubernetes Storage Class per Tier–Leveraging CSI driver lightweight volumesCSI Driver works with IBM Cloud Private 3.2.1 on Power–No IBM CSI Driver Support here, CSP is supportingIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation16
Spectrum Scale use caseDev/TestKubernetes Cluster with CSIKubernetes Cluster with CSIMaster Node(s)InfrastructureNodeInfrastructureNodeMaster Node(s)Worker NodeWorker tructureNodeWorker NodeWorker NodeApplicationApplicationCSI - PluginCSI - PluginCSI - PluginCSI - PluginCSI - PluginCSI - PluginCSI - PluginCSI - PluginScale ClientScale ClientScale ClientScale ClientScale ClientScale ClientScale ClientScale ClientStatic PV(Read/Write)Scale GUI Server(s)Scale NSD ServerStatic PV(Read/Write)Scale NSD ServerScale NSD ServerExisting DirectorySpectrum Scale Storage clusterIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation17
Spectrum Scale use caseDev/Test – lessons learned Stronger isolation in Dev/Testclusters beyond Kubernetesnamespace separation–Separate Kubernetes clusterswith different cluster admins–Still single data plane wanted With or without access to samedataIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation18
Spectrum Scale use caseMulticloudMaster Node(s)Home (on-prem)Worker NodesMaster Node(s)Scale CSI - DriverWorker NodesScale CSI - DriverFilesystem1Fileset1Directory1Directory2Mongo DBSync PVsCache (on-prem)AFMAFM(SW)(SW)Scale GUI Server(s)Scale NSD o DBPrefetchScale NSDServersSync PVsMaster Node(s)Cache (cloud)Worker NodesScale CSI - DriverFilesystem1Fileset1Directory1Directory2Mongo DBIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM CorporationScale GUIServer(s)Scale GUIServer(s)PrefetchScale NSDServersSync PVs19
Spectrum Scale use caseMulticloud – lessons learned To service workload on cloud–Single-writer (home site) only–Processed data should be pushed to separatefile systemFor DR purposes–Only one workload container should be up atgiven time–Independent writer can be used, but monitorcloud data outgoing traffic (EGRESS)Can be used with Spectrum Scale on AWS automation–Requires (not-yet-supported) Spectrum Scale GUIIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation20
Topics clients ask for.IBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation21
CSI driverunsupported platforms Operating Systems–Ubuntu 18.04.x InReseNo issues probe testing with plain k8sOrchestration Platforms–IBM Cloud Private See Cloud Service Provider use case Cross-architecture (x86, Power, Z) support in a single k8s cluster To get the CSI driver running––- apiGroups: ['policy']resources: ['podsecuritypolicies']resourceNames: ["ibm-anyuid-hostaccess-psp"]verbs:['use']- apiGroups: ["extensions"]resources: ["podsecuritypolicies"]resourceNames: ["ibm-anyuid-hostaccess-psp"]verbs: ["use"]add ibm-anyuid-hostaccess-psp Pod Security Policy toClusterRole CRsRancher Sold as “plain k8s”, easy deployment & nice GUI To get the CSI driver running–Remove the type: Directory specifier from the gpfsclassic hostPath mount of the ibm-spectrum-scale-csidaemonSet–Add a PodSecurityPolicy to the ibm-spectrum-scale-csi*ClusterRole definitions when running Rancher with PSPenforcementIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation- hostPath:path: /gpfs/fs1type: Directoryname: gpfs-classic- apiGroups:- policyresourceNames:- csi-pspresources:- podsecuritypoliciesverbs:- usearch--apiVersion: policy/v1beta1kind: PodSecurityPolicymetadata:name: csi-pspspec:requiredDropCapabilities:- NET RAWprivileged: falseallowPrivilegeEscalation: ork: truehostPorts:- min: 0max: 65535fsGroup:rule: RunAsAnyrunAsUser:rule: RunAsAnyseLinux:rule: RunAsAnysupplementalGroups:rule: RunAsAnyvolumes:- emptyDir- secret- persistentVolumeClaim- downwardAPI- configMap- projected22- hostPath
Security / Multi-tenancy For maximum container security, considerOpenShift– Uses K8s RBAC plus Security ContextConstraints (plus SELinux ) to restrict containerprocess security impactKubernetes Administrator (kubeadmin) user canaccess all PVs–One could use RBAC /PSP or SCCs to create“less privileged admin” role–For strict isolation use separate Kubernetesclusters Optionally use Kubernetes federation (v2!)IBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation23
Heterogenous environments Multi-platform– X86, IBM Power, IBM ZIBM Cloud Private supports multi-architectureclusters–IBM Storage Enabler for Containers supports allthree platforms–Spectrum Scale supports all three platforms(Linux)IBM PowerSystemsIBM ZOpenShift support for IBM Power and IBM Z iscoming–Initial releases might have limitations regardingRHEL worker node and multi-architecture supportIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation24
Summary and call to action Spectrum Scale provides tight Kubernetes and OpenShiftintegration through–IBM Storage Enabler for Containers (FlexVolume)for Kubernetes 1.13 and OpenShift 3.x–IBM CSI driversfor Kubernetes 1.13 and later and OpenShift 4.x Use cases (ADAS, CSP, Dev/Test and more) are evolving Involve us (ATS / Lab Services)–To discuss your Spectrum Scale Container Platformuse case–If you need help with IBM Storage Orchestration forContainersIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation25
Thank you!Please help us to improve Spectrum Scale withyour feedbackIBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation If you get a survey in your email or a popupfrom the GUI, please respond We read every single reply26
Teilen Siesich mit!Nutzen Sie den Hashtag#IBMStorageDays aufLinkedIn, Twitter und Co. undberichten von unserenVorträgen!Let‘sconnect!Bei Fragen einfach 45835627
Notices and Disclaimers 2020 International Business Machines Corporation. No part ofthis document may be reproduced or transmitted in any formwithout written permission from IBM.U.S. Government Users Restricted Rights — use, duplication ordisclosure restricted by GSA ADP Schedule Contract with IBM.Information in these presentations (including information relating toproducts that have not yet been announced by IBM) has beenreviewed for accuracy as of the date of initial publication and couldinclude unintentional technical or typographical errors. IBM shallhave no responsibility to update this information. This document isdistributed “as is” without any warranty, either express orimplied. In no event, shall IBM be liable for any damage arisingfrom the use of this information, including but not limited to, lossof data, business interruption, loss of profit or loss of opportunity.IBM products and services are warranted per the terms andconditions of the agreements under which they are provided.IBM products are manufactured from new parts or new and usedparts.In some cases, a product may not be new and may have beenpreviously installed. Regardless, our warranty terms apply.”Any statements regarding IBM's future direction, intent orproduct plans are subject to change or withdrawal withoutnotice.Performance data contained herein was generally obtained in acontrolled, isolated environments. Customer examples are presentedas illustrations of how thosecustomers have used IBM products andthe results they may have achieved. Actual performance, cost, savingsor other results in other operating environments may vary.References in this document to IBM products, programs, or servicesdoes not imply that IBM intends to make such products, programs orservices available in all countries in which IBM operates or doesbusiness.Workshops, sessions and associated materials may have beenprepared by independent session speakers, and do not necessarilyreflect the views of IBM. All materials and discussions are provided forinformational purposes only, and are neither intended to, nor shallconstitute legal or other guidance or advice to any individualparticipant or their specific situation.It is the customer’s responsibility to insure its own compliancewith legal requirements and to obtain advice of competent legalcounsel as to the identification and interpretation of any relevant lawsand regulatory requirements that may affect the customer’s businessand any actions the customer may need to take to comply with suchlaws. IBM does not provide legal advice or represent or warrant that itsservices or products will ensure that the customer follows any law.28
Notices and Disclaimers (cont.)Information concerning non-IBM products was obtainedfrom the suppliers of those products, theirpublished announcements or other publicly availablesources. IBM has not tested those products about thispublication and cannot confirm the accuracy ofperformance, compatibility or any other claims related tonon-IBM products. Questions on the capabilities of nonIBM products should be addressed to the suppliers ofthose products. IBM does not warrant the quality of anythird-party products, or the ability of any such third-partyproducts to interoperate with IBM’s products. IBMexpressly disclaims all warranties, expressed orimplied, including but not limited to, the impliedwarranties of merchantability and fitness for apurpose.IBM, the IBM logo, ibm.com and [names of otherreferenced IBM products and services used in thepresentation] are trademarks of International BusinessMachines Corporation, registered in many jurisdictionsworldwide. Other product and service names mightbe trademarks of IBM or other companies. A current listof IBM trademarks is available on the Web at "Copyrightand trademark information" at:www.ibm.com/legal/copytrade.shtmlThe provision of the information contained herein is notintended to, and does not, grant any right orlicense under any IBM patents, copyrights, trademarks orother intellectual property right.29
Bookmarks IBM Spectrum Scale CSI Driver ecenter/en/STXKQY 5.0.4/com.ibm.spectrum.scale.csi.v5r04.doc/bl1csi kc landing.html Storage Enabler for Containers ecenter/SSCKLT 2.1.0/sec kc welcome.html IBM Spectrum Scale CSI Driver si IBM Spectrum Scale CSI Driver video /ibm-spectrum-scale-csi-driver-video-blogs/ IBM Spectrum Scale CSI Driver at ectrum-scale-csi-operator Kubernetes CSI s/IBM Spectrum Scale / Use cases and lessons learned with Kubernetes and OpenShift / 2020 IBM Corporation30
IBM Spectrum Scale Master Nodes Worker Node Worker Node Workload Container Workload Container Workload Container Workload Container IBM Block Storage IBM Spectrum Connect (VM) Kubernetes Cluster Management Data Path IBM Storage Enabler for Containers provides Kubernetes FlexVolume Drivers IBM Spectrum Connect provides the Storage management API .
