WIXLIB

Processors - Dynamic Simultaneous MultithreadingDynamic SMT added to change the number of active threads per core without a system outage§ z/VM 6.4 allows one to dynamically change the number of active threads per core when SMT has been enabled in theSYSTEM CONFIG file.– MULTITHREADING ENABLE TYPE ALL 1– System configuration file statement enables SMT-1 (1 thread per core)§ Potential capacity gains going from SMT-1 to SMT-2– (one to two threads per core) can now be achieved dynamically§ Downgrade from SMT-2 to SMT-1 in extremely rare case that it is not optimal for workloads (response time concernsgreater than capacity gains)– Once z/VM has started, toggle between 1 and 2 threads via CP command SET MT TYPE ALL 2 and query statusand information with CP commands QUERY MULTITHREAD and INDICATE MULTITHREAD§ Requires:– IPL with SMT enabled, but can vary active threads per core– Requires z13, z13s, LinuxONE Emperor or LinuxONE Rockhopper .or newer CPC.§ If more than 32 cores are required per LPAR, can not use SMT -- even with one active thread per core. 2018 IBM Corporation17

Guest Virtual Processors§ Be aware:– Of the m axim um per virtual m achine based on VM version and your hardware.– That various guest operating system s and workloads scale differently§ Recom m endations:– Configure num ber of virtual processors per guest for peak workload, no m ore– Do not define m ore virtual processors to a guest than logical processors defined to the z/VM LPAR– W atch data from z/VM Perform ance Toolkit (or other sim ilar product)§ In z/VM PerfKit, check reports:– FCX100 (CPU)– FCX104 (Privileged Operations)– FCX114 (USTAT / W ait State Analysis by User) 2018 IBM Corporation18

Guest Virtual Processors – What to watch for§ In z/VM PerfKit§ Report FCX100 (CPU)– FCX100 look for Total:Virtual ratio (% CPU:% EMU). The closer to 1.00 the better.§ Report FCX104 (Privileged Operations)– High diagnose x’44’ or x’9C’ rates m ay be an indication of too m any virtual processors– FCX104 thresholds to watch for: x’44’ 50,000/sec x’9C’ 5,000/sec§ Report FCX114 (USTAT / W ait State Analysis by User) % CPU wait should be low 2018 IBM Corporation19

Adjust the ‘SHARE’ of Virtual MP Machines§ The default SHARE setting for all virtual m achines is “Relative 100”:– VM dispatches users by VMDBK– There is one VMDBK per virtual processor defined– A users SHARE setting is divided am ong the defined virtual processors§ Recom m end resetting the SHARE of Virtual MP Machines:– Set SHARE RELATIVE (100 * num ber of virtual CPUs defined)– This m aintains a “level playing field”§ Adjust SHARE of guest virtual m achines from this point, as required:– Increase SHARE to prioritize– Decrease SHARE to penalize§ A virtual m achines SHARE only com es into play when there is contention for resources, prim arily CPU§ As of 6.4, CP now honors SHARE settings m ore accurately than previous releases. For m ore details, seethe SRP article listed at the end of the presentation. 2018 IBM Corporation20

More about shares§ Priorities of users (virtual CPUs) are controlled by the SHARE setting. There are 2 types:– Relative- valid range 1-10000– Absolute- valid range 0.1-100%§ Virtual m achine share setting is divided by the num ber of virtual CPUs and assigned to each virtual CPU.– Minim um relative share per CPU is 1§ Set in the directory or with the CP SET SHARE {userid} com m and– Show the current setting with CP QUERY SHARE {userid}§ Absolute share users given resource first– If sum of absolute shares 99% , norm alized to 99%– Leftover share (m inim um 1% ) available for relative share users§ For relative share users - actual share depends on total relative shares of all virtual CPUs– A virtual CPU gets (vCPU relative share / total relative shares) % 2018 IBM Corporation21

Limiting shares§ There are two kinds of lim its which can be set:– LIMITSOFT– LIMITHARD§ W ith the changes m ade to dispatching and SRM, if you’re using either of these you m ay want to investigateif these are indeed producing the results you m ight be expecting. 2018 IBM Corporation22

Quick Dispatch§ Setting QUICKDSP traditionally would have:– Bypassed System Resource Managem ent controls– Placed a virtual m achine directly into the dispatch list– Made a virtual m achine exem pt from being placed in an eligible list§ QUICKDSP should be reserved for use only with:– Service Virtual Machines perform ing critical functions on behalf of other guests such as TCPIP, RACF, orDTCVSW #– Selected high-priority or key production guests running things like DB2, W ebSEAL, Edge Load Balancer,DynaCache, IBM Directory Integrator, etc.– If you are not already using QUICKDSP for a virtual m achine, you should only do so at therecom m endation of IBM VM Support.§ Interested in learning m ore about how SRM used to function? See the reference m aterials slides at the endfor an excellent detailed explanation by Malcolm Beattie (IBM) . 2018 IBM Corporation23

Linux Runlevels and target states§ Linux has different m odes of operation.– Under system -v they are “runlevels”– Under system d, they are target states 2018 IBM Corporation24

Linux Runlevels and target states§ Linux has different m odes of operation.– Under system -v they are “runlevels” The typical system -v runlevels are:– 0 - Halt the system– 1 - Single-user m ode– 2 - Multi-user m ode (without networking)– 3 - Multi-user m ode– 5 - Multi-user m ode (display m anager, GUI)– 6 - Reboot the system W hen you boot Linux, it will initialize to a pre-defined default runlevel (usually 3 or 5).– Most desktop Linux system s boot into RL 5 by default; users are presented with a GUI.– Most server Linux system s boot into RL 3 by default; users are presented with a CLI.– Query the current runlevel with the com m and: runlevel– You change between them with the com m and: telinit {target #} 2018 IBM Corporation25e.g.telinit 3

Linux Runlevels and target states§ Under system d, they are target states– The typical system d targets are: poweroff.target (or runlevel0.target) - Halt the system rescue.target (or runlevel1.target) - Single-user m ode m ulti-user.target (or runlevel3.target) - Multi-user m ode graphical.target (or runlevel5.target) - Multi-user m ode (display m anager, GUI) reboot.target (or runlevel6.target) - Reboot the system em ergency.target– W hen you boot Linux, it will initialize to a pre-defined default target (usually m ulti-user or graphical). Most desktop Linux system s boot into graphical.target by default; users are presented with a GUI. Most server Linux system s boot into m ulti-user.target by default; users are presented with a CLI.– Check the default target with the com m and systemctl get-default– Set the default target with the com m and systemctl set-default {choice}.target– Change the running target level with the com m and systemctl isolate {choice}.target 2018 IBM Corporation26

Linux Runlevel§ Recom m endation is RL 3 or m ulti-user.target for Linux guests of z/VM§ X services are very costly in term s of CPU cycles and RAM§ W hen im properly configured, graphical subsystem s can pose a potential risk vector such as GDDMlistening on all interfaces§ Production system s should never have full graphics enabled§ W hen necessary, use a lightweight X-server like VNC server instead of full GUI desktop and– Ensure that it is fully disabled once you are done 2018 IBM Corporation27

Unnecessary Guest Virtual Machines§ Shutting down unnecessary guest virtual m achines helps to im prove the overall perform ance of thesystem :– Linux guest virtual m achines alm ost never go dorm ant§ Logoff:– Golden im ages used for cloning– Test m achines and “sand boxes”§ Suspend:– Production guests not necessary during POC testing or benchm arking of another application orworkload– See Chapter 39, Suspending and resum ing Linux in the current Linux on System z Device Drivers,Features, and Com m ands Manual§ Reduce “SHARE” setting for virtual m achines running lower priority workloads 2018 IBM Corporation28

Unnecessary Services/Applications§ There are a num ber of services in Linux that get started at boot depending on:– Distribution– Linux kernel level/version– Installed software packages§ Shutting down unnecessary services and unused applications helps to im prove the overall perform ance ofthe system– Status of services can be queried/changed with the “chkconfig” com m and§ The cron daem on is useful for scheduling events to be kicked off autom atically at a specific tim e or atregular intervals– Running m any guests with identical schedules can cause CPU spikes and stress the z/VM pagingsubsystem : Rem ove unnecessary events from cron Consider switching to anacron for events which m ust rem ain Stagger scheduled kick-off tim e of events – especially for high I/O tasks like backups 2018 IBM Corporation29

z/VM Memory Configuration – Central Storage§ Max 2TB (Initial plus Reserved) – Set in HMC LPAR Activation Profile§ Plan on a virtual to real (V:R) m em ory ratio in the range of 1.5:1 to 3:1– Production system s will typically be closer to the low end of range– Developm ent/Test system s m ay be able to push the upper end of range§ STANDBY m em ory can be added and rem oved dynam ically to central storage:– Storage m ust be defined as “RESERVED” in the LPAR Activation Profile– Under z/VM 7.1, m em ory can be added and rem oved dynam ically.– Under z/VM 6.4 and older, m em ory cannot be rem oved dynam ically, only added.– This is because linux tends to autom atically address all m em ory available to it for use as cache. (CachedRAM is good under x86 distributed, but undesirable for a z/VM environm ent) 2018 IBM Corporation30

z/VM Memory Configuration – Expanded Storage§ As of z/VM 6.3 is no longer supported.§ z14 is the last CPC to even offer it at all in the HMC UI.§ Any storage previously configured as expanded should be com bined into central.§ Convert any expanded storage to central storage (real m em ory) before bringing up z/VM 6.4§ See the STORCONF URL on the Additional Inform ation slide for details 2018 IBM Corporation31

z/VM memory configuration – considerations§ z/VM 6.4 increases supported real m em ory from 1 TB to 2 TB at LPAR level (Central storage)– Individual Virtual Machine (USER / IDENTITY) lim it rem ains at 1TB§ Ensure sufficient dum p space§ Ensure sufficient paging space– Even if not increasing m em ory used, a good tim e to double check space guidelines– Check out the VIR2REAL utility on the z/VM Downloads page (See Additional Inform ation slide) 2018 IBM Corporation32

z/VM memory configuration – considerations§ Support for Virtual Machine (Linux) 1 MB large pages in 6.4– Scale better and im prove guest m em ory m anagem ent– Be m ore fair across users– Honor reserved m em ory settings better– To use this from Linux: Build a kernel containing large page exploitation (this is the default build), add hugepages n kernel param eter (num ber of large pages to be allocated at boot tim e) If desired, set sysctl variable to enable allocating large pages from m oveable m em ory§ Be proactive in writing out m em ory pages to disk– Read and write blocks of pages– Use parallel channel paths (PAV) when available. 2018 IBM Corporation33

Linux virtual memory sizing§ The m axim um supported virtual m achine m em ory allocation is 1TB§ The m ost com m on m istake m ade by custom ers running Linux guests under z/VM is over-provisioningvirtual m em ory due to lack of understanding§ In a com m odity hardware environm ent – including both dedicated servers and other virtualizationsolutions:– Traditional wisdom suggests installing as m uch m em ory as possible– Excess m em ory is used as I/O buffer and file system cache– This cache and buffer is usually required to overcom e the architectural lim itations of the m ost popularcom m odity processor cores Also the reason you typically see a drop off in perform ance at around 80-85% CPU utilization onthese cores. 2018 IBM Corporation34

Linux virtual memory sizing§ The z/VM platform is not a com m odity platform– Using thinking and m ethodologies from x86 does not apply!§ In a virtualized environm ent under z/VM, guests with excessively allocated RAM place unnecessary stresson the VM paging subsystem and can easily m ake guests run VERY POORLY Real m em ory is a shared resource. caching pages in a Linux guest reduces m em ory available to otherLinux guests Larger virtual m em ory requires m ore kernel m em ory for address space m anagem ent Live Guest Relocation takes longer to com plete– Rightsizing Linux m em ory requirem ents on z/VM: Is accom plished through iterative tuning of the STORAGE value for the virtual m achine Monitored with the “free” or “vm stat” com m ands along with /proc/m em info– Reference the following docum ent by Stephen W ehr (IBM): ftp.software.ibm .com /com m on/ssi/sa/wh/n/zsw03049usen/ZSW 03049USEN.PDF 2018 IBM Corporation35

z/VM paging subsystem§ Block paging changes elim inated the benefits of 50% page space, m aking m onitoring very im portant.– No loss of efficiency above 50% page space utilization– Contiguous storage no longer needed for block paging§ Recom m end m onitoring focused on availability versus perform ance (avoid ABEND)– Monitor for rapid growth in page space as well as overall size thresholds§ Early writing's goal is to keep the bottom 10% of the global aging list prewritten.§ W hether written-on-dem and or pre-written, page space is still being used. From a m onitoring perspective,this is all that m atters.§ The closer your m onitoring threshold is to 100% , the m ore autom ation is necessary to avoid an ABEND(how quickly page space can be added).§ Do not m ix page space with any other space on a volum e 2018 IBM Corporation36

z/VM paging subsystem – continued§ Rule Of Thum b - Plan for a DASD page space utilization 90% :– Monitor usage with Q ALLOC PAGE com m and and autom ation– Block page size is the key perform ance indicator: Aim for double digits, 10 or m ore 4K pages per block set Perfkit report - FCX109 (CP Owned Device)§ Use m ultiple channels to spread out I/O to paging devices§ Recom m end using devices of the sam e size/geom etry§ Leverage HYPERPAV for paging devices and use fewer, larger devices– Recom m end enabling via com m and and if no surprises, update system configuration file Com m and: SET PAGING ALIAS ON Configuration file: FEATURES ENABLE PAGING ALIAS Can also be controlled at control unit level 2018 IBM Corporation37

z/VM paging subsystem – continued§ For environm ents where the overcom m it level is low and large am ounts of real m em ory are being used,you will want to consider disabling early writes and keep slot– SET AGELIST EARLYWRITES NO KEEPSLOT NO§ EDEVs as paging drives are an option:– Have observed 1.6 I/Os per em ulated FBA volum e– At slightly higher CPU costs§ Page space calculation guidelines are located in the CP Planning and Adm inistration Manual 2018 IBM Corporation38

Linux swap§ The traditional recom m endation in a dedicated server environm ent is that swap space should be twice them em ory size of a Linux m achine– This does not apply to a z/VM Linux guest: Som e swap space should be defined to prevent Linux from hanging and/or a kernel panic duringunexpected m em ory dem ands Properly sized Linux guests should have m inim al swapping under norm al load§ z/VM offers m ultiple options for swap devices§ Recom m endation:– One or two sm all V-disks (256MB - 512MB)– If necessary, additional m inidisk(s) or dedicated volum e(s) Set priorities in fstab so that the V-disk(s) are used first§ See Additional Inform ation slide for m ore details and test results for various swap device options§ Virtualization Cookbook series covers how to setup VDISK and autom atically form at them at boot tim e 2018 IBM Corporation39

Memory management assistconsiderationsCollaborative Mem ory Managem ent Assist (CMMA)VM Resource Manager Cooperative Mem oryManagem ent (VMRM-CMM)– Som etim es referred to as just “CMM”– Som etim es referred to as “CMM2”– Extends coordination of m em ory and pagingbetween Linux and z/VM to the level of individualpages– VMRM detects when there is m em ory constraintand notifies specific Linux virtual m achines with aCP SMSG– CP reclaim s so-called “unused” pages at a higherpriority, to reduce wasted m em ory– The notified Linux VMs can then take actions toadjust their m em ory utilization in order to relievethis constraint on the system , usually issuing aCP DIAGNOSE X'10' instruction to release pagesof storage.– Bypasses host page writes for unused andvolatile pages, effectively cleaning up disk cachepages– Be sure you test out the effects of CMMA on anyvirtual m achines running Java with large heapsizes or databases, as these kinds of workloadscan be intolerant som etim es. 2018 IBM Corporation– Also im portant to test before using withproduction workloads40

SFS is your friend!§ As of z/VM 6.4 the default location for com ponents is SFS instead of m inidisks– Minim izes having to waste tim e to m ess around resizing (m ove to new & copy contents) m inidisks: This can be disruptive (usually is) Can also create fragm entation on your DASD– You could still elect to override and use m inidisks instead, but why? W hen you install Linux, do you create separate non-LVM volum es of fixed sizes for each individualpath you intend to m ount? Spend m ore of your tim e on optim izing your system instead of m anaging disks 2018 IBM Corporation42

z/VM minidisk cache§ z/VM m inidisk cache is a write-through cache:– Im proves read I/O perform ance, but, it’s not free in term s of system resources§ Not recom m ended for:– Mem ory constrained system s– Non-shared Linux file system s– Linux swap file disks§ Default system settings are less than optim al. Recom m ended settings:– Disable MDC for non-shared Linux m inidisks Code MINIOPT NOMDC operands on the MDISK directory statem ent– Lim it MDC in central storage, am ount depends on usage SETMDCSTORE0M256M– Monitor with Q MDC com m and and/or a perform ance m onitor Perfkit report - FCX103 (Storage Utilization) 2018 IBM Corporation44

Disk performance§ Hardware connectivity choices:– FICON available in 1Gb, 2Gb, 4Gb, 8Gb, 16Gb channel speeds§ SCSI verses ECKD/FBA recom m endations:– ECKD or FBA for z/VM and Linux “/” file system– SCSI LUNs for application dat

learned for IBM z/VM and Linux on IBM Z and LinuxONE 2018 VM Workshop Room 209 Thursday, 28 June 2018 17:30 Paul Novák . Visit the following pages on ibm.com for the full details: . LinuxONE Emperor or LinuxONE Rockhopper .or newer CPC. § If more than 32 cores are required per LPAR, can not--z/VM and Linux on IBM Z .