WIXLIB

vSphere with Tanzu Configuration and ManagementDeploy Workloads on Tanzu Kubernetes Clusters387Deploy a Test Workload to a Tanzu Kubernetes ClusterInstall and Run Octant387388Tanzu Kubernetes Service Load Balancer Example389Tanzu Kubernetes Service Load Balancer with Static IP Address Example391Tanzu Kubernetes Service Load Balancer Examples for Local Traffic Policy and Source IPRanges 393Tanzu Kubernetes Ingress Example Using NginxTanzu Kubernetes Storage Class Example395398Tanzu Kubernetes Persistent Volume Claim ExamplesTanzu Kubernetes Guestbook TutorialGuestbook Example YAML Files399401403Using Pod Security Policies with Tanzu Kubernetes ClustersExample Role Bindings for Pod Security PolicyExample Role for Pod Security Policy408410412Deploy TKG Extensions on Tanzu Kubernetes ClustersDownload the TKG Extensions v1.3.1 BundleInstall the TKG Extensions Prerequisites413413414Deploy and Manage the TKG Extension for Fluent Bit LoggingDeploy and Manage the TKG Extension for Contour Ingress418425Deploy and Manage the TKG Extension for Prometheus MonitoringDeploy and Manage the TKG Extension for Grafana MonitoringDeploy and Manage the TKG Extension for Harbor Registry434447454Deploy and Manage the TKG Extension for External DNS Service DiscoveryDeploy AI/ML Workloads on Tanzu Kubernetes ClustersAbout Deploying AI/ML Workloads on TKGS Clusters465470470vSphere Administrator Workflow for Deploying AI/ML Workloads on TKGS Clusters (vGPU)471Cluster Operator Workflow for Deploying AI/ML Workloads on TKGS Clusters484vSphere Administrator Addendum for Deploying AI/ML Workloads on TKGS Clusters (vGPUand Dynamic DirectPath IO) 492Cluster Operator Addendum for Deploying AI/ML Workloads on TKGS Clusters (DLS)49315 Using a Container Registry for vSphere with Tanzu Workloads 496Enable the Embedded Harbor Registry on the Supervisor ClusterLog In to the Embedded Harbor Registry Console497497Download and Install the Embedded Harbor Registry Certificate498Configure a Docker Client with the Embedded Harbor Registry Certificate499Install the vSphere Docker Credential Helper and Connect to the Registry501Push Images to the Embedded Harbor Registry503Purge Images from the Embedded Harbor Registry505Use the Embedded Harbor Registry with Tanzu Kubernetes ClustersVMware, Inc.5068

vSphere with Tanzu Configuration and ManagementUse an External Container Registry with Tanzu Kubernetes Clusters50916 Working with vSphere Lifecycle Manager 515Requirements515Enable vSphere with Tanzu on a Cluster Managed by vSphere Lifecycle ManagerUpgrade a Supervisor Cluster516Add Hosts to a Supervisor Cluster517Remove Hosts from a Supervisor ClusterDisable a Supervisor Cluster51651851817 Updating the vSphere with Tanzu Environment 520About vSphere with Tanzu UpdatesNetwork Topology Upgrade520523Upgrade the NSX-T Network ToplogyUpgrade vSphere Distributed Switch526527Update the Supervisor Cluster by Performing a vSphere Namespaces UpdateSupervisor Cluster Auto Upgrade528529Update the vSphere Plugin for kubectlList of Tanzu Kubernetes releasesUpdate Tanzu Kubernetes Clusters530530531Update a Tanzu Kubernetes Cluster by Upgrading the Kubernetes VersionUpdate a Tanzu Kubernetes Cluster by Changing the VirtualMachineClassUpdate a Tanzu Kubernetes Cluster by Changing the Storage ClassUpdate Tanzu Kubernetes Clusters Using the Patch Method53253453753918 Backing Up and Restoring vSphere with Tanzu 542Considerations for Backing Up and Restoring vSphere with Tanzu542Install and Configure the Velero Plugin for vSphere on the Supervisor ClusterBackup and Restore vSphere Pods Using the Velero Plugin for vSphere544554Install and Configure the Velero Plugin for vSphere on a Tanzu Kubernetes Cluster557Backup and Restore Tanzu Kubernetes Cluster Workloads Using the Velero Plugin for vSphere560Install and Configure Standalone Velero and Restic on a Tanzu Kubernetes Cluster562Backup and Restore Tanzu Kubernetes Cluster Workloads Using Standalone Velero and Restic567Backup and Restore vCenter Server575Backup and Restore NSX-T Data Center57519 Troubleshooting vSphere with Tanzu 576Storage Best Practices and Troubleshooting576Use Anti-Affinity Rules for Control Plane VMs on Non-vSAN DatastoresVMware, Inc.5769

vSphere with Tanzu Configuration and ManagementStorage Policy Removed from vSphere Continues to Appear as Kubernetes Storage Class577Use External Storage with vSAN DirectTroubleshooting Networking578580Register vCenter Server with NSX Manager580Unable to Change NSX Appliance Password580Troubleshooting Failed Workflows and Unstable NSX EdgesCollect Support Bundles for Troubleshooting NSX-TCollect Log Files for NSX-T581581582Restart the WCP Service If the NSX-T Management Certificate, Thumbprint, or IP AddressChanges 582VDS Required for Host Transport Node Traffic583Troubleshooting the NSX Advanced Load BalancerCollect Support Bundles for TroubleshootingTroubleshooting Network Topology Upgrade584584585Upgrade Precheck Fails Due to Insufficient Edge Load Balancer CapacitySupervisor Cluster Workload Namespaces Skipped During UpgradeLoad Balancer Service Skipped During UpgradeTroubleshooting Tanzu Kubernetes Clusters586586Troubleshoot vCenter Single Sign-On Connection Errors586Troubleshoot Subscribed Content Library Errors587587Troubleshoot Workload Deployment ErrorsTroubleshoot Virtual Machine Class Errors588588Restart a Failed Tanzu Kubernetes Cluster Update JobTroubleshooting Workload Management585586Collect a Support Bundle for Tanzu Kubernetes ClustersTroubleshoot Cluster Provisioning Errors585589590Collect the Support Bundle for Workload ManagementTail the Workload Management Log File590590Troubleshoot Workload Management Enablement Cluster Compatibility ErrorsShut Down and Start Up the vSphere with Tanzu Workload DomainVMware, Inc.59159210

vSphere with Tanzu Configuration andManagementvSphere with Tanzu Configuration and Management provides information about configuring andmanaging vSphere with Tanzu by using the vSphere Client. It also provides information aboutusing kubectl to connect to namespaces running on vSphere with Tanzu and run Kubernetesworkloads on designated namespaces.vSphere with Tanzu Configuration and Management provides an overview of the platformarchitecture as well as considerations and best practices for setting up storage, compute,and networking that meet the specific requirements of vSphere with Tanzu. It providesinstructions for enabling vSphere with Tanzu on existing vSphere clusters, creating and managingnamespaces, and monitoring Tanzu Kubernetes clusters that are created by using the VMwareTanzu Kubernetes Grid Service.This information also provides guidelines about establishing a session with the vSphere withTanzu Kubernetes control plane through kubectl, running a sample application, and creatingTanzu Kubernetes clusters by using the VMware Tanzu Kubernetes Grid Service.At VMware, we value inclusion. To foster this principle within our customer, partner, and internalcommunity, we create content using inclusive language.Intended AudiencevSphere with Tanzu Configuration and Management is intended for vSphere administratorswho want to enable vSphere with Tanzu in vSphere, configure and provide namespaces toDevOps teams, as well as manage and monitor Kubernetes workloads in vSphere. vSphereadministrators who want to use vSphere with Tanzu should have basic knowledge of containersand Kubernetes.This information is also intended for DevOps engineers who want to establish a session with thevSphere with Tanzu control plane, run Kubernetes workloads, and deploy Kubernetes clusters byusing the VMware Tanzu Kubernetes Grid Service.VMware, Inc.11

Updated Information1vSphere with Kubernetes Configuration and Management is updated regularly with newinformation and fixes as needed.This table provides the update history of the vSphere with Kubernetes Configuration andManagement.RevisionDescription5 NOV 2021nAdded a link for installing TKG 1.4 Packages on Tanzu Kubernetes clusters provisionedby the Tanzu Kubernetes Grid Service. See Deploy TKG Extensions on TanzuKubernetes Clusters.29 OCT 2021nUpdated the documentation for deploying vGPU workloads on TKGS clusters. SeeDeploy AI/ML Workloads on Tanzu Kubernetes Clusters.nUpdated the Velero Plugin for vSphere installation documentation. See Install andConfigure the Velero Plugin for vSphere on the Supervisor Cluster.21 OCT 202108 OCT 202105 OCT 2021VMware, Inc.nUpdated RBAC examples. See Example Role Bindings for Pod Security Policy.nUpdated the Supervisor Cluster networking. See Supervisor Cluster Networking.nAdded a caution to the networking and enabling workload management prerequisitetopics that DRS should not be disabled on the Supervisor Cluster and that disablingDRS leads to breaking of your clusters.nAdded documentation for deploying AI/ML workloads on vGPU-enabled TKGSclusters. See Deploy AI/ML Workloads on Tanzu Kubernetes Clusters.nUpdated Add PCI Devices to a VM Class in vSphere with Tanzu with information aboutsupport of PCI devices in passthrough mode.nMoved the listing of Tanzu Kubernetes releases to dedicated Release Notes. Refer tothese release notes for all Tanzu Kubernetes release information.nFixed typos and minor doc bugs.nUpdated the Tanzu Kubernetes release version. See List of Tanzu Kubernetes releasesand Provisioning Tanzu Kubernetes Clusters Using the Tanzu Kubernetes Grid Servicev1alpha2 API.nAdded the procedure for installing the Velero Plugin for vSphere in an air-gappedenvironment. For more information, see Install and Configure the Velero Plugin forvSphere on the Supervisor Cluster.nUpdated information about Supervisor Cluster backup and restore. For moreinformation, see Considerations for Backing Up and Restoring vSphere with Tanzu.nFixed typos.Initial release.12

vSphere with Tanzu Concepts2By using vSphere with Tanzu you can turn a vSphere cluster to a platform for running Kubernetesworkloads in dedicated resource pools. Once enabled on a vSphere cluster, vSphere with Tanzucreates a Kubernetes control plane directly in the hypervisor layer. You can then run Kubernetescontainers by deploying vSphere Pods, or you can create upstream Kubernetes clusters throughthe VMware Tanzu Kubernetes Grid Service and run your applications inside these clusters.This chapter includes the following topics:nWhat Is vSphere with Tanzu?nWhat Is a vSphere Pod?nWhat Is a Tanzu Kubernetes Cluster?nWhen to Use vSphere Pods and Tanzu Kubernetes ClustersnUsing Virtual Machines in vSphere with TanzunvSphere with Tanzu User Roles and WorkflowsnHow Does vSphere with Tanzu Change the vSphere Environment?nLicensing for vSphere with TanzuWhat Is vSphere with Tanzu?You can use vSphere with Tanzu to transform vSphere to a platform for running Kubernetesworkloads natively on the hypervisor layer. When enabled on a vSphere cluster, vSphere withTanzu provides the capability to run Kubernetes workloads directly on ESXi hosts and to createupstream Kubernetes clusters within dedicated resource pools.The Challenges of Today's Application StackToday's distributed systems are constructed of multiple microservices usually running a largenumber of Kubernetes pods and VMs. A typical stack that is not based on vSphere with Tanzuconsists of an underlying virtual environment, with Kubernetes infrastructure that is deployedinside VMs, and respectively Kubernetes pods also running in these VMs. Three separateroles operate each part of the stack, which are application developers, Kubernetes clusteradministrators, and vSphere administrators.VMware, Inc.13

vSphere with Tanzu Configuration and ManagementFigure 2-1. Today's Application StackKubernetes WorkloadPodsVolumesServices.DeveloperKubernetes ClusterWorkersControl PlaneETCD.Cluster AdminVirtual EnvironmentVMsNetworksStorage.vSphere AdminThe different roles do not have visibility or control over each other's environments:nAs an application developer, you can run Kubernetes pods, and deploy and manageKubernetes based applications. You do not have visibility over the entire stack that is runninghundreds of applications.nAs a DevOps engineer, you only have control over the Kubernetes infrastructure, without thetools to manage or monitor the virtual environment and resolve any resource-related andother problems.nAs a vSphere administrator, you have full control over the underlying virtual environment, butyou do not have visibility over the Kubernetes infrastructure, the placement of the differentKubernetes objects in the virtual environment, and how they consume resources.Operations on the full stack can be challenging, because they require communication between allthree roles. The lack of integration between the different layers of the stack can also introducechallenges. For example, the Kubernetes scheduler does not have visibility over the vCenterServer inventory and it cannot place pods intelligently.How Does vSphere with Tanzu Help?vSphere with Tanzu creates a Kubernetes control plane directly on the hypervisor layer. As avSphere administrator, you enable existing vSphere clusters for Workload Management, thuscreating a Kubernetes layer within the ESXi hosts that are part of the cluster. A cluster enabledwith Workload Management is called a Supervisor Cluster.VMware, Inc.14

vSphere with Tanzu Configuration and ManagementFigure 2-2. vSphere with TanzuvSphere with TanzuNamespaceKubernetes WorkloadsvSpherePodsCPUVMsKubernetes ual EnvironmentKubernetes LayerESXiNetworkingStoragevSphere AdminHaving a Kubernetes control plane on the hypervisor layer enables the following capabilities invSphere:nAs a vSphere administrator, you can create namespaces on the Supervisor Cluster, calledvSphere Namespaces, and configure them with specified amount of memory, CPU, andstorage. You providevSphere Namespaces to DevOps engineers.nAs a DevOps engineer, you can run workloads consisting of Kubernetes containers on thesame platform with shared resource pools within avSphere Namespace. In vSphere withTanzu, containers run inside a special type of VM called vSphere Pod. You can also deployregular VMs.nAs a DevOps engineer, you can create and manage multiple Kubernetes clusters insidea namespace and manage their lifecycle by using the Tanzu Kubernetes Grid Service.Kubernetes clusters created by using the Tanzu Kubernetes Grid Service are called TanzuKubernetes clusters.nAs a vSphere administrator, you can manage and monitor vSphere Pods, VMs, and TanzuKubernetes clusters by using the vSphere Client.nAs a vSphere administrator, you have full visibility over vSphere Pods, VMs, and TanzuKubernetes clusters running within different namespaces, their placement in the environment,and how they consume resources.Having Kubernetes running on the hypervisor layer also eases the collaboration betweenvSphere administrators and DevOps teams, because both roles are working with the sameobjects.VMware, Inc.15

vSphere with Tanzu Configuration and ManagementWhat Is a Workload?In vSphere with Tanzu, workloads are applications deployed in one of the following ways:nApplications that consist of containers running inside vSphere Pods, regular VMs, or both.nTanzu Kubernetes clusters deployed by using the VMware Tanzu Kubernetes Grid Service.nApplications that run inside the Tanzu Kubernetes clusters that are deployed by using theVMware Tanzu Kubernetes Grid Service.What Is a vSphere Pod?vSphere with Tanzu introduces a new construct that is called vSphere Pod, which is theequivalent of a Kubernetes pod. A vSphere Pod is a VM with a small footprint that runsone or more Linux containers. Each vSphere Pod is sized precisely for the workload that itaccommodates and has explicit resource reservations for that workload. It allocates the exactamount of storage, memory, and CPU resources required for the workload to run. vSphere Podsare only supported with Supervisor Clusters that are configured with NSX-T Data Center as thenetworking stack.Figure 2-3. vSphere PodsESXi HostvSphere PodvSphere PodContainerContainerContainerLinux KernelCPUMemoryLinux KernelStorageCPUMemoryStoragevSphere Pods are objects in vCenter Server, and therefore enable the following capabilities forworkloads:nStrong isolation. A vSphere Pod is isolated in the same manner as a virtual machine. EachvSphere Pod has its own unique Linux kernel that is based on the kernel used in Photon OS.Rather than many containers sharing a kernel, as in a bare metal configuration, in a vSpherePod, each container has a unique Linux kernelVMware, Inc.16

vSphere with Tanzu Configuration and ManagementnResource Management. vSphere DRS handles the placement of vSphere Pods on theSupervisor Cluster.nHigh performance. vSphere Pods get the same level of resource isolation as VMs, eliminatingnoisy neighbor problems while maintaining the fast start-up time and low overhead ofcontainers.nDiagnostics. As a vSphere administrator you can use all the monitoring and introspectiontools that are available with vSphere on workloads.vSphere Pods are Open Container Initiative (OCI) compatible and can run containers from anyoperating system as long as these containers are also OCI compatible.Figure 2-4. vSphere Pod Networking and StoragevSphere PodContainerContainer ImageContainerEphemeral DiskContainer EnginePod EnginePersistent VolumeSpherelethostdvNICNSX vSwitchvSphere Pods use three types of storage depending on the objects that are stored, that areephemeral VMDKs, persistent volume VMDKs, and containers image VMDKs. As a vSphereadministrator, you configure storage policies for placement of container image cache, ephemeralVMDKs, and control plane VMs on the Supervisor Cluster level. On a vSphere Namespace level,you configure storage policies for placement of persistent volumes and for placement of the VMsof Tanzu Kubernetes clusters. See Chapter 10 Using Persistent Storage in vSphere with Tanzu fordetails about the storage requirements and concepts with vSphere with Tanzu.For networking, vSphere Pods and the VMs of the Tanzu Kubernetes clusters created throughthe Tanzu Kubernetes Grid Service use the topology provided by NSX-T Data Center. For details,see Supervisor Cluster Networking.VMware, Inc.17

vSphere with Tanzu Configuration and ManagementvSphere Pods are only supported on Supervisor Clusters that use NSX-T Data Center as theirnetworking stack. They are not supported on clusters that are configured with the vSpherenetworking stack.What Is a Tanzu Kubernetes Cluster?A Tanzu Kubernetes cluster is a full distribution of the open-source Kubernetes containerorchestration platform that is built, signed, and supported by VMware. You can provision andoperate Tanzu Kubernetes clusters on the Supervisor Cluster by using the Tanzu Kubernetes GridService. A Supervisor Cluster is a

Proxy Load Balancer 105 Topologies for Deploying the HAProxy Load Balancer 108 Create a vSphere Distributed Switch for a Supervisor Cluster for Use with HAProxy Load Balancer 116 Install and Configure the HAProxy Load Balancer 117. 5. Configuring and Managing a Supervisor Cluster 122. Prerequisites for Configuring vSphere with Tanzu on a .