Transcription
Mirantis Container CloudReference Architectureversion latest
Mirantis Container Cloud Reference Architectureversion latestContentsCopyright notice1Preface2About this documentation set2Intended audience3Conventions3Technology Preview support scope4Documentation history5Overview6Cloud provider9Release controller12Web UI13Bare metal14Bare metal components14Bare metal networking16IP Address Management17Management cluster networking17Managed cluster networking19Host networking22Extended hardware lt-in load balancing29Kubernetes API load balancing29Services load balancing29VMware vSphere network objects31Kubernetes lifecycle management32LCM custom resources32LCM controller33LCM agent34Helm controller34 2021, Mirantis Inc.Page i
Mirantis Container Cloud Reference ArchitectureIdentity and access managementversion latest36IAM API and CLI36External identity provider integration37Authentication and authorization38Implementation flow38Kubernetes CLI authentication flow39Monitoring40Deployment architecture40Authentication flow44Supported features45Monitored components46Outbound cluster metrics47StackLight proxy49Hardware and system requirementsBaremetal-based cluster5152Reference hardware configuration53System requirements for the seed node54Network fabric55Ceph56OpenStack-based cluster56AWS-based cluster60VMware vSphere-based cluster62Proxy and cache support66Proxy support66Artifacts caching66Mirantis Kubernetes Engine API limitations 2021, Mirantis Inc.68Page ii
Mirantis Container Cloud Reference Architectureversion latestCopyright notice2021 Mirantis, Inc. All rights reserved.This product is protected by U.S. and international copyright and intellectual property laws. Nopart of this publication may be reproduced in any written, electronic, recording, or photocopyingform without written permission of Mirantis, Inc.Mirantis, Inc. reserves the right to modify the content of this document at any time without priornotice. Functionality described in the document may not be available at the moment. Thedocument contains the latest information at the time of publication.Mirantis, Inc. and the Mirantis Logo are trademarks of Mirantis, Inc. and/or its affiliates in theUnited States an other countries. Third party trademarks, service marks, and names mentionedin this document are the properties of their respective owners. 2021, Mirantis Inc.Page 1
Mirantis Container Cloud Reference Architectureversion latestPrefaceThis documentation provides information on how to deploy and operate Mirantis ContainerCloud. About this documentation set Intended audience Conventions Technology Preview support scope Documentation historyAbout this documentation setThe documentation is intended to help operators understand the core concepts of the product.The information provided in this documentation set is being constantly improved and amendedbased on the feedback and kind requests from our software consumers. This documentation setoutlines description of the features that are supported within two latest Cloud Container minorreleases, with a corresponding note Available since release.The following table lists the guides included in the documentation set you are reading:Guides listGuidePurposeReference ArchitectureLearn the fundamentals of Container Cloud reference architectureto plan your deployment.Deployment GuideDeploy Container Cloud of a preferred configuration usingsupported deployment profiles tailored to the demands of specificbusiness cases.Operations GuideDeploy and operate the Container Cloud managed clusters.Release CompatibilityMatrixDeployment compatibility of the Container Cloud componentsversions for each product release.Release NotesLearn about new features and bug fixes in the current ContainerCloud version as well as in the Container Cloud minor releases.QuickStart GuidesEasy and lightweight instructions to get started with ContainerCloud.For your convenience, we provide all guides from this documentation set in HTML (default),single-page HTML, PDF, and ePUB formats. To use the preferred format of a guide, select therequired option from the Formats menu next to the guide title on the Container Clouddocumentation home page. 2021, Mirantis Inc.Page 2
Mirantis Container Cloud Reference Architectureversion latestIntended audienceThis documentation assumes that the reader is familiar with network and cloud concepts and isintended for the following users: Infrastructure Operator Is member of the IT operations team Has working knowledge of Linux, virtualization, Kubernetes API and CLI, and OpenStackto support the application development team Accesses Mirantis Container Cloud and Kubernetes through a local machine or web UI Provides verified artifacts through a central repository to the Tenant DevOps engineers Tenant DevOps engineer Is member of the application development team and reports to line-of-business (LOB) Has working knowledge of Linux, virtualization, Kubernetes API and CLI to supportapplication owners Accesses Container Cloud and Kubernetes through a local machine or web UI Consumes artifacts from a central repository approved by the Infrastructure OperatorConventionsThis documentation set uses the following conventions in the HTML format:Documentation conventionsConventionDescriptionboldface fontInline CLI tools and commands, titles of the procedures andsystem response examples, table titles.monospaced fontFiles names and paths, Helm charts parameters and their values,names of packages, nodes names and labels, and so on.italic fontInformation that distinguishes some concept or term.LinksExternal links and cross-references, footnotes.Main menu menu itemGUI elements that include any part of interactive user interfaceand menu navigation.SuperscriptSome extra, brief information. For example, if a feature isavailable from a specific release or if a feature is in theTechnology Preview development stage.Messages of a generic meaning that may be useful to the user.NoteThe Note block 2021, Mirantis Inc.Page 3
Mirantis Container Cloud Reference Architectureversion latestInformation that prevents a user from mistakes and undesirableconsequences when following the procedures.Caution!The Caution blockWarningThe Warning blockMessages that include details that can be easily missed, butshould not be ignored by the user and are valuable beforeproceeding.List of references that may be helpful for understanding of somerelated tools, concepts, and so on.SeealsoThe See also blockLearn moreUsed in the Release Notes to wrap a list of internal references tothe reference architecture, deployment and operation proceduresspecific to a newly implemented product feature.The Learn moreblockTechnology Preview support scopeThis documentation set includes description of the Technology Preview features. A TechnologyPreview feature provides early access to upcoming product innovations, allowing customers toexperience the functionality and provide feedback during the development process. TechnologyPreview features may be privately or publicly available and neither are intended for productionuse. While Mirantis will provide support for such features through official channels, normalService Level Agreements do not apply. Customers may be supported by Mirantis CustomerSupport or Mirantis Field Support.As Mirantis considers making future iterations of Technology Preview features generallyavailable, we will attempt to resolve any issues that customers experience when using thesefeatures.During the development of a Technology Preview feature, additional components may becomeavailable to the public for testing. Because Technology Preview features are being underdevelopment, Mirantis cannot guarantee the stability of such features. As a result, if you areusing Technology Preview features, you may not be able to seamlessly upgrade to subsequentreleases of that feature. Mirantis makes no guarantees that Technology Preview features will begraduated to a generally available product release. 2021, Mirantis Inc.Page 4
Mirantis Container Cloud Reference Architectureversion latestThe Mirantis Customer Success Organization may create bug reports on behalf of support casesfiled by customers. These bug reports will then be forwarded to the Mirantis Product team forpossible inclusion in a future release.Documentation historyThe documentation set refers to Mirantis Container Cloud GA as to the latest released GA versionof the product. For details about the Container Cloud GA minor releases dates, refer to ContainerCloud releases. 2021, Mirantis Inc.Page 5
Mirantis Container Cloud Reference Architectureversion latestOverviewMirantis Container Cloud is a set of microservices that are deployed using Helm charts and run ina Kubernetes cluster. Container Cloud is based on the Kubernetes Cluster API communityinitiative.The following diagram illustrates an overview of Container Cloud and the clusters it manages:All artifacts used by Kubernetes and workloads are stored on the Container Cloud contentdelivery network (CDN): mirror.mirantis.com (Debian packages including the Ubuntu mirrors) binary.mirantis.com (Helm charts and binary artifacts) mirantis.azurecr.io (Docker image registry)All Container Cloud components are deployed in the Kubernetes clusters. All Container CloudAPIs are implemented using the Kubernetes Custom Resource Definition (CRD) that representscustom objects stored in Kubernetes and allows you to expand Kubernetes API.The Container Cloud logic is implemented using controllers. A controller handles the changes incustom resources defined in the controller CRD. A custom resource consists of a spec thatdescribes the desired state of a resource provided by a user. During every change, a controllerreconciles the external state of a custom resource with the user parameters and stores thisexternal state in the status subresource of its custom resource.The types of the Container Cloud clusters include:Bootstrap cluster Runs the bootstrap process on a seed node. For the OpenStack, AWS, or VMwarevSphere-based Container Cloud, it can be an operator desktop computer. For thebaremetal-based Container Cloud, this is the first temporary data center node. Requires access to a provider back end: OpenStack, AWS, vSphere, or bare metal. 2021, Mirantis Inc.Page 6
Mirantis Container Cloud Reference Architectureversion latest Contains minimum set of services to deploy the management and regional clusters. Is destroyed completely after a successful bootstrap.Management and regional clusters Management cluster: Runs all public APIs and services including the web UIs of Container Cloud. Does not require access to any provider back end. Regional cluster: Is combined with management cluster by default. Runs the provider-specific services and internal API including LCMMachine andLCMCluster. Also, it runs an LCM controller for orchestrating managed clusters andother controllers for handling different resources. Requires two-way access to a provider back end. The provider connects to a backend to spawn a managed cluster nodes, and the agent running on the nodesaccesses the regional cluster to obtain the deployment information. Requires access to a management cluster to obtain user parameters. Supports multi-regional deployments. For example, you can deploy an AWS-basedmanagement cluster with AWS-based and OpenStack-based regional clusters.Supported combinations of providers types for management and regional clustersRegionalclusterproviderBare metalAWSOpenStackvSphereBare metalmanagementcluster AWSmanagementcluster OpenStackmanagementcluster vSphere managementclusterManagement and regional clusters comprise Container Cloud as product. For deploymentdetails, see Deploy a management cluster and Deploy an additional regional cluster sectionsfor the required cloud provider.Managed cluster 2021, Mirantis Inc.Page 7
Mirantis Container Cloud Reference Architectureversion latest A Mirantis Kubernetes Engine (MKE) cluster that an end user creates using theContainer Cloud web UI. Requires access to a regional cluster. Each node of a managed cluster runs an LCMagent that connects to the LCM machine of the regional cluster to obtain thedeployment details. An attached MKE cluster that is not created using Container Cloud. In such case, nodesof the attached cluster do not contain LCM agent. For supported MKE versions that canbe attached to Container Cloud, see Compatibility matrix. Baremetal-based managed clusters support the Mirantis OpenStack for Kubernetes(MOS) product. For details, see MOS documentation.All types of the Container Cloud clusters except the bootstrap cluster are based on the MKE andMirantis Container Runtime (MCR) architecture. For details, see MKE and MCR documentation.The following diagram illustrates the distribution of services between each type of the ContainerCloud clusters: 2021, Mirantis Inc.Page 8
Mirantis Container Cloud Reference Architectureversion latestCloud providerThe Mirantis Container Cloud provider is the central component of Container Cloud thatprovisions a node of a management, regional, or managed cluster and runs the LCM agent onthis node. It runs in a management and regional clusters and requires connection to a providerback end.The Container Cloud provider interacts with the following types of public API objects:Public API object nameContainer Cloud releaseobjectDescriptionContains the following information about clusters: Version of the supported Clustermanagement and regional clustersreleasefora List of supported Cluster releases for the managedclusters and supported upgrade path Description of Helm charts that are installed on themanagement and regional clusters depending on theselected providerCluster release object Provides a specific version of a management, regional, ormanaged cluster. Any Cluster release object, as well as aContainer Cloud release object never changes, only newreleases can be added. Any change leads to a newrelease of a cluster. Contains references to all components and their versionsthat are used to deploy all cluster types: LCM components: LCM agent Ansible playbooks Scripts Description of steps to execute during a clusterdeployment and upgrade Helm controller image references Supported Helm charts description: Helm chart name and version Helm release name Helm values 2021, Mirantis Inc.Page 9
Mirantis Container Cloud Reference ArchitectureCluster object ReferencestheCredentials,ClusterRelease objects.version latestKaaSReleaseand Is tied to a specific Container Cloud region and provider. Represents all cluster-level resources. For example, forthe OpenStack-based clusters, it represents networks,load balancer for the Kubernetes API, and so on. It usesdata from the Credentials object to create theseresources and data from the KaaSRelease andClusterRelease objects to ensure that all lower-levelcluster objects are created.Machine object References the Cluster object. Represents one node of a managed cluster, for example,an OpenStack VM, and contains all data to provision it.Credentials objectPublicKey object Contains all information necessary to connect to aprovider back end. Is tied to a specific Container Cloud region and provider.Is provided to every machine to obtain an SSH access.The following diagram illustrates the Container Cloud provider data flow:The Container Cloud provider performs the following operations in Container Cloud: 2021, Mirantis Inc.Page 10
Mirantis Container Cloud Reference Architectureversion latest Consumes the below types of data from a management and regional cluster: Credentials to connect to a provider back end Deployment instructions from the KaaSRelease and ClusterRelease objects The cluster-level parameters from the Cluster objects The machine-level parameters from the Machine objects Prepares data for all Container Cloud components: Creates the LCMCluster and LCMMachine custom resources for LCM controller and LCMagent. The LCMMachine custom resources are created empty to be later handled by theLCM controller. Creates the the HelmBundle custom resources for the Helm controller using data fromthe KaaSRelease and ClusterRelease objects. Creates service accounts for these custom resources. Creates a scope in Identity and access management (IAM) for a user access to amanaged cluster. Provisions nodes for a managed cluster using the cloud-init script that downloads and runsthe LCM agent. 2021, Mirantis Inc.Page 11
Mirantis Container Cloud Reference Architectureversion latestRelease controllerThe Mirantis Container Cloud release controller is responsible for the following functionality: Monitor and control the KaaSRelease and ClusterRelease objects present in a managementcluster. If any release object is used in a cluster, the release controller prevents the deletionof such an object. dhttps://binary.mirantis.com/releases/ with an existing management cluster.at Trigger the Container Cloud auto-upgrade procedure if a new KaaSRelease object is found:1. Search for the managed clusters with old Cluster releases that are not supported by anew Container Cloud release. If any are detected, abort the auto-upgrade and display acorresponding note about an old Cluster release in the Container Cloud web UI for themanaged clusters. In this case, a user must update all managed clusters using theContainer Cloud web UI. Once all managed clusters are upgraded to the Clusterreleases supported by a new Container Cloud release, the Container Cloudauto-upgrade is retriggered by the release controller.2. Trigger the Container Cloud release upgrade of all Container Cloud components in amanagement cluster. The upgrade itself is processed by the Container Cloud provider.3. Trigger the Cluster release upgrade of a management cluster to the Cluster releaseversion that is indicated in the upgraded Container Cloud release version. TheLCMCluster components, such as MKE, are upgraded before the HelmBundlecomponents, such as StackLight or Ceph.4. Verify the regional cluster(s) status. If the regional cluster is ready, trigger the Clusterrelease upgrade of the regional cluster.Once a management cluster is upgraded, an option to update a managed clusterbecomes available in the Container Cloud web UI. During a managed cluster update, allcluster components including Kubernetes are automatically upgraded to newerversions if available. The LCMCluster components, such as MKE, are upgraded beforethe HelmBundle components, such as StackLight or Ceph.Container Cloud remains operational during the management and regional clusters upgrade.Managed clusters are not affected during this upgrade. For the list of components that areupdated during the Container Cloud upgrade, see the Components versions section of thecorresponding Container Cloud release in Release Notes.When Mirantis announces support of the newest versions of Mirantis Container Runtime (MCR)and Mirantis Kubernetes Engine (MKE), Container Cloud automatically upgrades thesecomponents as well. For the maintenance window best practices before upgrade of thesecomponents, see MKE and MCR Documentation. 2021, Mirantis Inc.Page 12
Mirantis Container Cloud Reference Architectureversion latestWeb UIThe Mirantis Container Cloud web UI is mainly designed to create and update the managedclusters as well as add or remove machines to or from an existing managed cluster. It alsoallows attaching existing Mirantis Kubernetes Engine (MKE) clusters.You can use the Container Cloud web UI to obtain the management cluster details includingendpoints, release version, and so on. The management cluster update occurs automaticallywith a new release change log available through the Container Cloud web UI.The Container Cloud web UI is a JavaScript application that is based on the R
OpenStack-based cluster 56 AWS-based cluster 60 VMware vSphere-based cluster 62 Proxy and cache support 66 Proxy support 66 Artifacts caching 66 Mirantis Kubernetes Engine API limitations 68 Mirantis Container Cloud Reference Architecture version latest 2021, Mirantis Inc. Page ii
