Transcription
Top 10 Best Practicesfor vSphere BackupsDate: July. 30, 2019Veeam Version 9.5 Update 4bVMware Version 6.7Hannes KasparickSenior Analyst, Veeam Product Management Team 2019 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.
Top 10 Best Practices for vSphere BackupsExecutive summaryServer virtualization is a widespread practice all around the world. In 2019, VMware is still the market leader in this sectorand many Veeam customers use VMware vSphere as their preferred virtualization platform. This white paper describes bestpractices that are specific for the backup and Availability of VMware vSphere with Veeam Backup & Replication 9.5.It does not include general best practices for Hyper-V and Veeam Agent specifics.IntroductionServer virtualization is a widespread practice all around the world. In 2019, VMware is still the market leader in this sectorand many Veeam customers use VMware vSphere as their preferred virtualization platform. The backup of virtual machines (VMs)on vSphere is only one part of service Availability. Backup is the foundation for restores, so it is essential to have backups alwaysavailable with the required speed. The most important rule as a general best practice in the field of backup is the 3-2-1 Rule.This means having at least three copies of your data: Production data, a first line of backup and a second line of backup.It also recommends storing the backup copies on at least two independent types of media. The “independent” cannotbe overemphasized. Independent means that there is no dependency from a technology perspective. And finally, another copyshould be off site and offline, out of reach of natural disasters, malicious software and unauthorized people. For example, Veeamadded insider protection for Veeam Cloud Connect in Veeam Backup & Replication 9.5 Update 3. Of course, tape is still an optionfor offsite storage for backups.Veeam Backup & Replication helps extend the 3-2-1 Rule to the 3-2-1-0 rule. The zero means zero restore issues, which madepossible with automated restore tests with Veeam SureBackup . SureBackup exists primarily to find logical issues in the backups.An example of this is if someone installed updates but never did a reboot. After a reboot, a blue screen or kernel panic happens.This document describes several best practices with Veeam Backup & Replication and VMware vSphere. These best practicesare dedicated to Veeam and VMware. Note that other hypervisors are not covered in this document.These general best practices include: Having a backup and restore strategy that fits your business needs Doing proper sizing Making sure VSS works in Windows machines Having enough backup spaceThese apply in any case, regardless of whether it’s a VMware, Hyper-V, cloud provider or physical server backup.The first and most important thing to do before planning or implementing any solution is to be certain about the requirements.In an ideal world, the business creates the requirements and tells IT which recovery point objective (RPO) and recovery timeobjective (RTO) is needed. Do they only need backup, or is disaster recovery (DR) also a requirement?With this information, it is possible to size the hardware. That includes the number of CPU cores and the amount of memoryand bandwidth requirements for WAN, LAN and SAN. Finally, it needs a source and backup storage that is fast enoughto achieve the required speed.The next step is the backup itself. Veaam’s application-aware image processing uses Microsoft VSS to achieve applicationconsistent backup of Windows VMs. This mechanism does not use VMware tools quiescing. To make application-awareimage processing work reliably, it is necessary that the VSS writers of the VMs are working properly. 2019 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.3
Top 10 Best Practices for vSphere BackupsNo. 1. Use current versions of Veeam and vSphereThe latest versions of Veeam Backup & Replication improve performance together with VMware vSphere. Veeam Backup& Replication 9.5 has much better performance than earlier versions, especially in vSphere environments. The Veeambroker service and non-VADP backup methods for Hot-Add, direct-NFS and Backup from Storage Snapshots addthe most significant performance improvements.On the other hand, VMware has improved VM snapshot consolidation with ESXi version 6.x. This leads to less VM stunson I/O intensive virtual machines during snapshot commit after a backup.The best practice: Look out for improvements to the latest versions of Veeam Backup & Replication and vSphere.No. 2. Choose your backup mode wiselyWith Veeam Backup & Replication, there are three different transport modes to back up VMs on vSphere. All of themhave their pros and cons and there is no general rule as to which is the best. The environment and requirements will decidewhich one of the following three modes you should choose:1. Network mode or NBD2. Direct Storage Access3. Virtual appliance or “Hot-Add”The properties of each proxy allow the configuration of the options above in the transport mode section. Figure 1 shows this:Figure 1: Transport mode options 2019 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.4
Top 10 Best Practices for vSphere BackupsThe network mode or NBD mode is the easiest way to do VMware backups. The Veeam proxy server uses the ESXi managementport of each ESXi-host to transfer backup data. That makes the setup very simple as it requires no additional storage or VMconfiguration. Additionally, it has very low overhead, which is another advantage. Compared to Hot-Add mode, it does not needany additional Hot-Add mount operations, which saves time. It also does not create additional storage snapshots like Backupfrom Storage Snapshots with integrated storage systems. The coordination of VM and storage snapshots takes time, so the networkmode can even be the fastest for incremental backups in environments with many VMs and a low data change rate.The ESXi management port can become a bottleneck, especially if it is only a 1 Gbit interface. However, with 10 Gbit and betternetwork interface cards this usually isn’t a problem.In case you use ESXi 6.5: VMware enforces encryption of backup traffic via NBD-SSL with this version. Encryption was an optionalsetting before. This reduces the backup speed significantly. With later updates, VMware allowed unencrypted NBD traffic again.Veeam supports this new unencrypted backup via NBD since Backup & Replication 9.5 Update 3.The direct storage access mode backup traffic goes directly from the storage system to the Veeam backup proxy.The backup traffic does not need to go through the ESXi hypervisor and the protocol depends on the storage environment.Usually it is FibreChannel or iSCSI. Direct storage access mode also has the same advantage over Hot-Add as network mode,which means no time-consuming Hot-Add operation. On the other hand, both modes use VADP.VADP is the official API from VMware to back up virtual machines. It has some backup performance implications whichis why Veeam Backup & Replication does not use VADP in three special configurations. These three special configurations are: Backup from Storage Snapshots Direct NFS (similar to direct storage access) Virtual appliance/Hot-AddAvoiding VADP leads to significant backup performance improvements, which is why Hot-Add is becoming more popular. ButHot-Add has one more advantage. In Hot-Add mode, the Veeam backup proxy runs as an additional VM for backups. It mountsthe snapshots of the VMs to backup and sends the traffic over the normal VM network. It does not use the ESXi managementinterface. This fact makes Hot-Add a performant alternative in 1GBit networks where direct storage access backup modes are notpossible.The Hot-Add backup mode is not recommended in general with NFS datastores. The recommendation with NFS is to use directstorage access which results in the direct NFS mode. Direct NFS has no separate option in the UI, it’s just a flavor of direct storageaccess. The reason for this recommendation is that Hot-Add often results in VM stuns if the Veeam proxy does not run on thesame ESXi host as the VM. Veeam KB1681 provides more details in the section titled “for environments with NFS datastores.” Ifyou plan to use “Hot-Add” mode on NFS datastores anyway, apply the following rules and settings: One Hot-Add proxy per ESXi host Set EnableSameHostHotAddMode 1 in HKEY LOCAL MACHINE\SOFTWARE\Veeam\Veeam Backup and ReplicationAs there are diverse options to do backups, you can use the following table to quantify results of each mode and reacha conclusion as to which one is the best for you.The best practice: Test which backup mode fits best to your environment. 2019 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.5
Top 10 Best Practices for vSphere BackupsNo. 3. Plan how to restoreAfter defining the optimal backup mode, it is important to look at the restore mode. Veeam offers 57 recovery scenarios to restoreVMs, files and application objects.First, it is important to know that file and object restore differs from VM or disk restore. Veeam restores files or objects(like Microsoft Exchange e-mails or Microsoft Active Directory objects) over the network. Over-the-network means an RPC(Windows) or SSH (Linux) connection that transfers the data to restore into the VM.As backup is VM snapshot-based as block-level backup, restore of full VMs or virtual disks is also block based. Depending on the restoremode, it makes a difference whether the VM is thick or thin provisioned. The restore modes are the same as the ones for backup(i.e., direct storage access, virtual appliance and network). Additionally, there is Instant VM Recovery combined with StorageVMotion or quick migration.Hot-Add and network mode can restore thick- and thin-provisioned VMs. As already mentioned, the virtual appliance or Hot-Addtransport with version 9.5 mode has improved performance for backup. This is also true for full VM or disk restores with“Hot-Add.” In most scenarios, it makes sense to have at least one Hot-Add proxy available for VM or disk restores.Network mode is usually the slowest way to restore, as it cannot use the full bandwidth.Direct storage access mode has no limitations concerning network bandwidth, but it can only restore thick-provisioned disks.Thin-provisioned disks would be converted on-the-fly to thick disks. As direct storage access mode uses VADP for restores,it is usually not the fastest option. The exception here is restore with direct NFS where Veeam Backup & Replicationdoes not use VADP.To restore a VM or virtual disk, it is not required to fully transfer all data. If the change block tracking information on the production storageis correct, then a restore based on change block tracking is possible. Setting this option can reduce the restore time. The quickrollback option to do this must be manually enabled during restore. Figure 2 shows this:Figure 2: Quick rollback based on change block tracking information 2019 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.6
Top 10 Best Practices for vSphere BackupsInstant VM Recovery is an alternative way for full VM restore. It allows you to instantly boot a VM directly from the backuprepository. The backup repository acts as an NFS datastore that is mounted to an ESXi host. There are two options to transferthe VM data from the repository NFS datastore back to the production datastore: Veeam Quick Migration VMware Storage VMotionSince there are diverse options for full VM restores, you can use the following table to quantify the results of each modeand reach a conclusion as to which one is the best for you.The best practice: Plan and test the restore options depending on your storage and transport modes. If you do not use NFSdatastores, have at least one “Hot-Add” proxy installed as spare.No. 4. Install VMware toolsIn many situations, Veeam Backup & Replication relies on the existence of VMware tools that run in the VMs. Without VMwaretools, it cannot find out, for example, IP addresses or the operating system version. As a result, application-aware imageprocessing will fail.This is because Veeam Backup & Replication cannot detect the IP address and without the IP address Veeam cannot connectto the VM over the network. The fallback mechanism VIX or vSphere API for guest interaction also doesn’t work due to the lackof VMware tools (see No. 10 for more information on VIX). Figure 3 shows this in an example of a failed guest credentials testbecause of missing VMware tools:Figure 3: Failed Application Aware Processing test 2019 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.7
Top 10 Best Practices for vSphere BackupsThe second example is SureBackup tests. Heartbeat and ping tests will fail if VMware tools are not present. For VMware toolsthe #1 rule applies: Keep them up-to-date.The best practice: Install VMware tools and keep them up-to-dateNo. 5. Integrate storage-based snapshotsinto your Availability conceptStorage snapshots aren’t as good as backup, that’s for sure, but they can help to minimize data loss in many situations.Veeam Backup & Replication has integrations with various storage vendors in conjunction with VMware vSphere.Storage integration adds more options for data protection.The first is that Veeam Backup & Replication can open storage snapshots and restore files and objects directly from the storagesnapshot. This allows you, for example, to schedule storage snapshots every 15 minutes without the requirement to createVM snapshots too. Although the every-15-minutes snapshot is not a real backup as it does not meet the 3-2-1 Rule, it does helpto decrease RPO times.Figure 4 shows an example of this concept. It shows Veeam Explorer for Storage Snapshots. The left side shows the storagesnapshots (i.e., the LUNs and the snapshots of one LUN). The right side shows the VMs of each storage snapshot. From thereit is possible to restore VMs with Instant VM Recovery or restore files and application objects. Now imagine the storagedoes snapshots of critical LUNs or volumes every 15 minutes and deletes them after four hours. That means it is possibleto restore data from 15 minutes ago, instead of older data from the last night’s backup.Figure 4: Object restore from storage snapshot 2019 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.8
Top 10 Best Practices for vSphere BackupsThe second advantage of a storage integration with Veeam Backup & Replication is that backups of large, highly transactional VMslike database servers are now possible without the risk of VM stuns during VMware snapshot consolidation. Although the situationis much better with current vSphere versions, it is still the main reason to use storage snapshots.Finally, Backup from Storage Snapshots allows Veeam to use its proprietary data fetcher mechanisms to outperform classic VADPbackups. This is especially relevant for full backups or any backup that has high change rates.The best practice: Use storage integration if you have a storage that has snapshot support for Veeam Backup & ReplicationNo. 6. VMware vSAN backupAs VMware vSAN is getting more popular, it is relevant to keep some specifics in mind. VMware vSAN does not use traditionalstorage protocols. This means that there is no direct storage access or Backup from Storage Snapshots option available.The supported backup modes are virtual appliance/Hot-Add and network mode. With Hot-Add mode, Veeam Backup& Replication backs up VMs relative to the proximity to the VM data. That means the backups occur through the proxyon the host that has the most VM-specific data. To make that work properly, there must be one Hot-Add proxy per ESXihost. Host affinity for the proxy VM rules prevent the VMware Distributed Resource Scheduler (DRS) from moving thoseVMs to other ESXi hosts.That means shorter backup windows, as there is less network traffic and latency. If a VM was on one host and the proxyon a different host, then there is more traffic over the network which adds latency and reduces speed.Veeam Backup & Replication is certified as VMware-ready for vSAN within the data protection category. The VMwareknowledge base article 2149874 and the VMware vSAN HCL have further information.The best practice: Install one Hot-Add proxy per ESXi host if you use virtual appliance mode with VMware Virtual SAN.No. 7. Keep track of your (vSphere) infrastructureFor many years “It just works” was Veeam’s slogan. While this is true for most customers because the default settings are verysolid, it still makes sense to plan a Veeam Backup & Replication deployment for larger installations in detail.vCenter is one of the most critical parts needed for Veeam Backup & Replication to work. If vCenter is down, backups will fail.So, the maintenance windows of vCenter should be planned outside the backup windows. Also, keeping an eye on the vCenter loadand number of connections is a good idea. The network between Veeam Backup-Server and vCenter should be stable!Depending on your environment, backup can put a significant load on your production storage. Multiple GBs per secondare not uncommon and can raise the I/O latency on traditional disk arrays. Veeam Backup & Replication I/O control(commonly known as backup I/O control) throttles backup and restore speed (Figure 5). 2019 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.9
Top 10 Best Practices for vSphere BackupsFigure 5: Storage latency controlStorage latency control assigns or throttles tasks based on the datastore latency values that Veeam gets from vSphere.This happens in two stages. First Veeam stops to assign new backup tasks to a datastore. If the latency still increases,then it will throttle the existing backup tasks. As a result, the backup will take longer, but with less influence on running VMs.With this mechanism, it is possible to do backups during production hours with minimal impact on VMs, applications and users.Storage latency control disables the default setting of a maximum of four VM snapshots per datastore at the same time.This can also lead to performance improvements.The best practice: As Veeam Backup & Replication relies heavily on vCenter, make sure it’s running efficiently, monitorthe load during backup windows and tune as required.No. 8. SecurityVeeam Backup & Replication connects to vCenter to manage backup and restores of VMs. From a security point of view, it isalways a good idea to work with the least privileges required. VMware vCenter offers fine granular permissions to allow backups.The required permissions document contains a detailed description of which permissions to configure for which backup mode.The different backup modes require different permissions. A security-relevant permission for the virtual appliance backup mode isthat it requires the “remove disk” permission.These security considerations can have influence on the choice of the backup mode. It is also possible to restrict specific backupservers (if you have multiple) to specific locations or objects in vCenter.The best practice: Work within the boundaries of the principal of least privilege. 2019 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.10
Top 10 Best Practices for vSphere BackupsNo. 9. Plan your Veeam Backup & Replicationdeployment with Veeam ONEThe Veeam Availability Suite contains a powerful planning tool for Veeam Backup & Replication deployments called Veeam ONE .The Veeam ONE monitor shows the actual status and current issues of the vSphere environment. Relevant issues aroundbackup could be, for example, a high storage latency or old, large, many or orphaned VM snapshots.The Veeam ONE reporter includes the VM configuration assessment report that shows potential backup issues.Typical issues the report shows are: VMware tools not installed Hardware version 4 or earlier Disks that cannot be backed up (e.g., independent disks) Datastores with less than 10% free space Raw device mappings in VMsFixing these issues before running backups prevents further backup issues.The best practice: Use Veeam ONE to plan the Backup & Replication installation.No. 10. Application-aware backup via VIX APIBest practice No. 4 recommends having VMware tools always installed and up-to-date. VMware tools give a Veeam administratorthe chance to do application-aware backups for Windows VMs without a direct network connection to the VM.The preferred way to do application-aware backup is connecting the application proxy via RPC to the VM. This is the fastest way.If network segmentation or firewalls prevent a network communication to the VM, Veeam can use the VIX API or in newervSphere versions (6.5 and newer) the vSphere API for guest interaction. Figure 6 shows the login via VIX marked in orange.Figure 6: Guest credentials test via VIX API 2019 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.11
Top 10 Best Practices for vSphere BackupsVIX or vSphere API for guest interaction does not work out of the box. Veeam KB 1788 describes the requirements in detail.To summarize, it has two requirements: The user account used by Veeam must be a member of the local administrators group If the account is not titled “administrator,” then Windows User Account Control (UAC) must be disabledVIX or vSphere API for guest interaction is the fallback mode if RPC does not work. The result for environment, where mostVMs are not reachable via RPC, is that the backup will take longer because Veeam always tries RPC first. For those environments,it is possible to change the order to “VIX first” with the following registry key on the backup server or guest interaction proxy:HKEY LOCAL MACHINE\SOFTWARE\Wow6432Node\Veeam\Veeam Backup and Replication\DWORD: InverseVssProtocolOrderValue 1To disable (default behavior), value is 0 (false)It is important to know that VIX or vSphere API for guest interaction has some limitations on restore operations. It is onlypossible to restore files but no application items. That means it is not possible to restore Active Directory, Exchange or othersimilar objects this way. It requires network connection for restores. The second thing is that the file is much slower whenyou go via network.Speaking of speed, the VeeamLogShipper service that does SQL log-shipping can also use VIX as fallback mechanismif it cannot reach the repository via network. This can be too slow for most environments. That said, it is recommendedthat SQL log-shipping is done via network.The best practice: Keep in mind the limitations of VIX or vSphere API for guest interaction.ConclusionThe combination of Veeam Backup & Replication with VMware vSphere usually works just right “out of the box.” But there areseveral best practices that can make it work even better. Those best practices are not complicated, they can be configured fastand in quite an easy way.The best practice: Read the full Veeam Backup & Replication Best Practices guide if you plan a larger or complex deployment 2019 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.12
Top 10 Best Practices for vSphere BackupsAbout the AuthorHannes Kasparick has been working in the IT business since 2004. Today Hannes isa member of the Veeam product management team. In the past he managed Linuxand Windows environments as well as infrastructure services like servers, storage,network and firewalls.About Veeam SoftwareVeeam is the leader in Backup solutions that deliver Cloud Data Management. Veeam Availability Platform is the most completebackup solution for helping customers on the journey to achieving success in the 5 Stages of Cloud Data Management. Veeamhas 355,000 customers worldwide, including 82% of the Fortune 500 and 67% of the Global 2,000, with customer satisfactionscores at 3.5x the industry average, the highest in the industry. Veeam’s global ecosystem includes 66,000 channel partners;Cisco, HPE, NetApp and Lenovo as exclusive resellers; and 23,500 cloud and service providers. Headquartered in Baar,Switzerland, Veeam has offices in more than 30 countries. To learn more, visit https://www.veeam.com or follow Veeam onTwitter @veeam. 2019 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.13
added insider protection for Veeam Cloud Connect in Veeam Backup & Replication 9.5 Update 3. Of course, tape is still an option for offsite storage for backups. Veeam Backup & Replication helps extend the 3-2-1 Rule to the 3-2-1-0 rule. The zero means zero restore issues, which made possible with automated restore tests with Veeam SureBackup .
