Transcription
Keys to optimizing yourbackup environment:Veritas NetBackupJeff HarbertStorage ConsultantGlassHouse Technologies, Inc.jharbert@glasshouse.comIntroductionz Audience Profilez Storage Management Interdependencez Case Studyz Backup Architecture and Performance ServerTapeNetworkClientsAudience Responsez Are you a NetBackup Administrator? YesNoz How long have you been using the product? 1. 0-6 months2. 6-12 months3. 12-24 months4. More than 2 years
Case Study – Biotechnology CompanyTechnical Issues Backup architecture unable toscale with increasing storagecapacities and server growth Network architecture unable tosupport throughputs requiredfor backup Lack of NBU / OS tuning:Implementation did not includeoptimization Client-side issues impactingbackup success (OS, antivirus,network, applications)Business Problems Limited budget and staff Reliability and performanceproblems for both hardwareand software Regulatory / data retentionrequirements Disconnect between applicationdesign and storagemanagement Shrinking backup windows Lack of data managementpoliciesNetBackup ARDNetBackupServerSD5411 Enterprise Switch10 00 BASE X1Clients2AMBER AC TIVI TYGREEN LINK OKFL ASHING GREENDISABLED 10/ 100 BASE TX445678910115 6 78 9 1 0 11112233R333R3P OWERMGMTNetworkMediaServerCase StudySolutionTechnical Issues Backup server underpoweredfor size of environment Lacking in CPUs, RAM, andNetwork throughput No media servers to balanceload Upgraded hardware to HPrp7410 Master: 4GBs of RAM, 4CPUs, Gb Ethernet cards Partitioned server into masterand media server Upgraded hosts with more than200 GBs into SAN mediaservers NetBackup not properly tunedfor optimal performance Modified NetBackup server-sidevariables to maximizeperformance
NetBackup Server Performancez Application Performance Tuning Backup Schedule Balancing Specific Tuning VariablesNumber of Backup Jobs Multi-streaming and multiplexing NET BUFFER SZ DISABLE RESOURCE BUSY CLEAN IN BACKGROUND Pre-Process Intervalz NetBackup Performance Tuning Guide UNIX: tmAverage Media Server VolumeData Transferred (GB)1,2001,000800Server 1Server 2Server 3Server 4600400200Server 4Server 3Server 2Server y0NetBackup ARDNetBackupServerSD5411 Enterprise Switch10 00 BASE X12AMBER AC TIVI TYGREEN LINK OKFL ASHING GREENDISABLED 10/ 100 BASE TX445678910115 6 78 9 1 0 11112233R333R3POWERMGMTNetworkMediaServerClients
Case StudySolutionTechnical Issues Veritas Shared Storage Optiondeployed with STK L700 and20 LTO-1 drives Frequent drive problemscaused multiple backupfailures Overall Throughput was lessthan expected Modified SSO to limit drivesharing to smaller number ofservers Updated persistent bindings tominimize impact of serverreboots and SCSI-locks Tape drives were “shoe-shining” due to a lack of data Increased the number of multiplestreams written to each drive Reduced the number of drivesavailable for backup Modified SIZE DATA BUFFERS andNUMBER DATA BUFFERS variableson media serversTape Tuning Strategiesz Storage Unit Fragment Size Rule-of-Thumb 2GBz Multiplexing Data Reduces “shoe-shining” effectToo much can negatively impact restore speed.z In-Line Duplication Requires additional hardware, but enables tapes to besent offsite quickerz Software vs. Hardware Compression Best Practice: Utilize hardware compression unlessnetwork-constrained.Disk-Based Backups with NetBackup v5.0z Enhanced functionality of v5.0 makes disk-basedbackups a viable option.z Disk as a Primary Storage Device Enterprise-class solutions are limited due to tremendousstorage requirements.
Disk-Based Backups with NetBackup v5.0(II)z Disk as a Staging Device v5.0 includes Staging functionality. Leverages speed of disk for backup, with tape forlong-term storageData is automatically moved based on preestablished policies.z Disk as Hybrid-Solution Direct some backups to disk, others to tape.Eliminates additional step of staging, but providesadvantages of both disk and tapeStorage Unit DistributionAmount of Data Transferred for a 7-Day Period (15.4 TB)STK9940A531, 0, 0%STK9940A317, 0.3TB, 2%STK9940A258, 0.4TB, 2%STK9940B317, 1.1TB, 7%STK9940B258, 5.9TB, 39%STK9940B531, 3.6TB, 23%STK9940B532, 4TB, 27%NetBackup ARDNetBackupServerSD5411 Enterprise Switch10 00 BASE X12AMBER AC TIVI TYGREEN LINK OKFL ASHING GREENDISABLED 10/ 100 BASE TX445678910115 6 78 9 1 0 11112233R333R3POWERMGMTNetworkMediaServerClients
Case StudyTechnical IssuesSolution Separate Backup Network was Modified “Required Interface”created to reduce impact oncorporate networkoption to bind client backups toappropriate network interface Clients did not always use backupnetwork for data transfer Intermittent backup failuresfollowing host reboots Hard-code network settings to100 Mb, Full-Duplex NICs and Ports set to “auto-negotiate” Log files indicate that Mediaservers were “waiting for fullbuffers” during networkbackups Tuned NET BUFFER SZVariable on both media serversand clients Slowed backup performanceNetwork Designz Dedicated Private VLAN architectures Logical network segregation & physical networksegregation Use jumbo frames and gigabit VLANs. Improves streaming efficiencies to tapez Multiple NICs for client network load balancing Etherchannel configurationsLoad balancing for network, switches, server HBAsNetwork Design, IIz NetBackup Tuning Variable NET BUFFER SZ: Determines size of networkcommunications buffer used to receive data fromnetwork If set too small, media servers can’t emptybuffers fast enough. If set too large, media servers have to wait fordata. Client settings should match media servers.
Sample Charts Available From GlassHouseBuffer Information SummaryData Buffer Analysis per Media ServerNumber of Times50,00045,000Waited for full bufferWaited for empty 000Waited for empty bufferWaited for full bufferServer 4Media ServersServer 3Server 2Server 10NetBackup ARDNetBackupServerSD5411 Enterprise Switch10 00 BASE X1Clients2AMBER AC TIVI TYGREEN LINK OKFL ASHING GREENDISABLED 10/ 100 BASE TX4456789105 6 78 9 1 0 1111112233R333R3POWERMGMTNetworkMediaServerCase StudyTechnical Issues Oracle Agent was implementedfor Hot Oracle DatabaseBackups Backups of some servers ran out-Solution Implemented Block-LevelIncremental backups of Oracle 30% Faster than standard RMANbackupsof-window Network Appliance Filersbacked up via NFS mounts Full backups took 5 days Implemented NDMP backups todedicated tape drives Testing indicated that 3rd-PartyRemote backups were significantlyslower Backups of Netware clients ranout-of-window Tuned Netware variables perVeritas-recommended settings Reduced backup times by 33%
Client Considerationsz Include/Exclude list size Maintenance is difficult, but necessary.Exclude Open Files, Temp files, etc.z Open File Processingz Block-Level / Flash Backupsz Client options settings Pre-Process Interval (set on master server) Increasing default intervalz Number and size of filesz Frequency of data changeClient Backup PerformanceBackup Jobs over 1 GB in Volume16.012.010.08.06.04.0Client 18Client 16Client 17Client 15Client 14Client 13Client 12Client 11Client 9Client 10Client 7Client 8Client 6Client 4Client 5Client 20.0Client 32.0Client 1Throughput (MB/S)14.0To receive charts similar to those seen inthis presentation specific to yourenvironment go to:www.glasshouse.com/backupFor a “cheat sheet” see thehighlighter in your conference bag.
Questions?Thank you.z See me at Ask the Expert –Tuesday 5:00-6:00 PM
NetBackup Server Tape/SAN/Disk Technology Network Clients NetBackup Architecture Media Server 5411 Enterprise Switch POWER MGMT 4 567 891011 1 1 2 3 10 00 BASE X AMBER GREEN FLASHING GREEN AC TIVI TY LINK O K DISABLED SD HEWLETT Tape/SAN/Disk PACKARD Case Study Solution Upgraded hardware to HP rp7410 Master: 4GBs of RAM, 4 CPUs, Gb Ethernet cards
