WIXLIB

Runtime VM Protection ByIntel Multi-Key Total Memory Encryption(MKTME) Kai Huang @ Intel CorporationLINUXCON CONTAINERCON CLOUDOPENBeijing, China, 20181

Legal DisclaimerNo license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for aparticular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.This document contains information on products, services and/or processes in development. All information provided here is subject tochange without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.The products and services described may contain defects or errors known as errata which may cause deviations from publishedspecifications. Current characterized errata are available on request.Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or byvisiting www.intel.com/design/literature.htm.Intel and the Intel logo are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.*Other names and brands may be claimed as the property of others Intel Corporation.2

Agenda MKTME IntroductionMKTME Use CasesMKTME Enabling3

Background: Trusted VM in CloudVM protection by using encryption VM encrypted ‘at-rest’, ‘in-transit’ and ‘runtime’.There has been existing technologies for ‘at-rest’and ‘in-transit’ encryptionQemu TLS support for live migrationQemu encrypted image supportVM runtime encryption requires hardwarememory encryption supportAMD SME/SEVIntel MKTME CloudOrchestrator❶ At-restVM ImageRepoCloudAgent❷ RuntimeVM LaunchCloudAgent❸ In-transitLive MigrationVM Launch Launch VM on ‘Trustiness Verified’ Host Trusted hardware/location, etc.Trusted Cloud SW stack.Intel Arch Compute NodeIntel ArchCompute Node Typical VM Lifecycle in Cloud4

Trusted VM w/ OpenCIT -- OpenStack as ExampleEnterprise Data CenterCloud Service HorizonAppOSOpenStackNovaKB ProxyKey BrokerAppOSP o lic y E n fo r c e m e n tAttestationTrust AgentR e p o r t in ghubV e r ifie rM e a su re m e n tOpenStackBarbicantbootxmAttestation ServerIn t e lTPMO p e n S ta ckIntel Arch w/ TXTCompute NodeIntel Open CIT helps on Host trustiness verification 5

TME & MKTME IntroductionNew AES-XTS engine in data path to external memorybus. Data encrypted/decrypted on-the-fly when entering/leavingmemory.AES-XTS uses physical address as “tweak” Same plaintext, different physical address - different ciphertext.TME (Total Memory Encryption) Full memory encryption by TM E key (CPU generated). Enabled/Disabled by BIOS. Transparent to OS & user apps.MKTME (Multi-key Total Memory Encryption) M emory encryption by using multiple keys. Use upper bits of physical address as keyID (see next)6

MKTME KeyIDsRepurpose upper bits of physical address as KeyID as shown below. Reduces useable physical address bits. Creates “aliases” of physical memory locations: different keyIDs can refer to the same page. Architecturally upto 2 15-1 keyIDs (15 keyID bits). Cache-coherency is not guaranteed for the same page that accessed by different keyIDs. CPU caches are unaware of keyID(still treat keyIDas part of PA)Reported by M SR. Configured by BIOS.KeyID 0 is reserved as TM E’s key (not useable by M KTM E).New PCONFIG instruction to program keyID w/ associated key (see next)MAXPHYADDR - 1630ReservedKeyIDPlatform Addressable MemoryMK TME MAX KEYID BITS7

MKTME KeyID Programming OverviewNew Ring-0 instruction PCONFIG to program the KEYID and associated key Package scoped Supports programming keyID to 4 modes: Using CPU generated random ephemeral key (invisible to SW ) Using SW provided key (tenant’s key) No encryption – plaintext domain Clearing a key (using TM E’s key effectively)Allows SW to specify crypto algorithms Only AES-XTS-128 for initial server intercept8 8

VM Protection & Isolation With MKTMEProtection Use keyID to encrypt VM memory at runtimeIsolation Use different keyIDs for different VM sSoftware Enabling For CPU access, SW sets keyID at PTEs IA page table (host) EPT (KVM) For Device access (DM A) w/ IOMMU: Set keyIDto IOMMU page table Physical DMA: Apply keyIDto PA directly9

Highlights of MKTMEGuests continue to run “without modifications” in MKTME domains: Encrypted with 1) CPU-generated ephemeral key, or 2) the one provided by API (“tenantcontrolled keys”) Virtio, including optimization (direct access to guest memory by kernel) continues to work Direct I/O (including accelerators, FPGA) assignment (including SR-IOV VFs) is availableLive migration can be supported (among platforms that support MKTME)vNVDIMM can be supported w/ limitation (because of physical address “tweak”) Host DIM M configuration cannot be changed cross reboots. Qemu DIM M & vNVDIM M configuration cannot be changed cross VM reboots.10

Agenda MKTME IntroductionMKTME Use CasesMKTME Enabling11

MKTME Enabled Use CasesLaunch Tenant VMs with in-use protection (CPU generated keys)1. Let CSP handle the keys VM image provided by CSP2.Launch Tenant VMs with at-rest and in-use protection with full tenant-control VM image encrypted @ rest with tenant-specific keys Keys in tenant control VM memory isolation with tenant-specific keys Trustiness verified hostAdditional: integrity verification of VM imageUse-case Extension:KeyID Sharing for all VMs launched by single tenant with the same tenant-key (or CPU generated key).12

VM Launch w/ CPU Generated KeysMKTME ephemeral keyTME keyImageRepoVM Launch w/ CPU generated key CSP provided VM imageSecurity Properties w/ VM runtime protection w or w/o at-rest and in-transit protection No Host Trustiness VerificationCloudManager1VM LaunchCloudAgent2Policy EnforcementPolicyAgent3VMVM Launchw/ KeyIDKeyid 3Keyid 3Keyid 3HYPERVISOR (Qemu/KVM)Keyid 0HARDWARE Intel TXT/ TPMIntel MKTMEKeyid 0DRAM*CSP Controlled13

VM Launch w/ Tenant Controlled Keys0Upload VM image encryptedw/ tenant keyImageRepoVM Launch w/ Tenant provided key Tenant provided encrypted VM image Tenant controlled key server Trustiness verified host VM image integrity verified Use TPM to wrap/unwrap tenant-keyCloudManagerTrustinessVerification12VM LaunchMKTME ephemeral keyTenant keyTME keyCloudAgent3Policy EnforcementPolicyAgent4Security Properties w/ VM runtime protection w/ VM at-rest protection w/ or w/o in-transit protection w/ Host trustiness verification w/ VM image integrity verificationWrapped tenant keyKeyServerTenant ControlledVMVM Launchw/ KeyIDKeyID 3KeyID 3KeyID 3HYPERVISOR (Qemu/KVM)AttestationServiceKeyID 1HARDWAREIntel TXT/ TPM Intel MKTMERequest Tenant-keyReturn wrapped keyKeyID 1DRAM*CSP Controlled14

KeyID Sharing Among VMsN e w V M Lau n chw / M ktm e Po licyMktmePolicy {tenant id: UUID ,key type: “ephemeral” “persistent”,Cloud SW makes decision whether to share or not.K eyID P o licyK eyIDVM sPo licy1 : te n an t1 , “e p h e m e ral” ke yID 1V M 1 , V M 2 .Po licy2 : te n an t2 , “p e rsiste n t”, xxxxxx ke yID 2VM 3key server: https://.,allow to share: “yes” “no”}Cloud SWLaunch VMw/ keyIDExa m p le: K eyID sh a rin g is b a sed o n K eyID Po licy: ten a n t id , key typ e, ten a n t key QemuC lo u d SW : M a in ta in s ‘K eyID Po licy-to -K eyID ’ ta b le M a kes keyID sh a rin g d ecisio n a cco rd in g to th e ta b leU p d a tes th e ta b le o n V M la u n ch a n d tea rd o w nCompute Nodem K ey A PI: M K TM E key m a n a g em en t A PIApply keyID toVM memorymKey APILaunch VMKVM15

Agenda MKTME IntroductionMKTME Use CasesMKTME Enabling16

MKTME EnablingHigh LevelQEM ULau n ch V Mw / K e yIDCloud SWG u e s t m e m o ryVMw ith K e y IDG u e s t M e m o ryD ire c t I/O Host kernelmkey APIsKey/KeyID ManagementCore-MM KeyID supportVFIO/IOMMU KeyID supportDMA KeyID supportKVMKeyID setup in EPTQemu Receive KeyID from Cloud SWApply KeyID to guest memoryV irtio /V h o s tL iv e M ig ra tio nm k e y A P IsV F IO /IO M M Uw ith K e y IDKVMC o re -M M c o d ew ith K e y IDM M U N o tifie rV h o s t-k e rn e lK e y ID sfo r H o s tIA -P TD M A w ithK e y IDK e y /K e y IDM anagem entPC O N FIGM K T M E E n g in eK e y ID sfo r D M AV T -dK e y ID sfo r g u e s tEPTS e ttin g K e y ID sin E P TK e y ID sfo r D M AD e v ic eN ew C ode(N IC , S C S I, e tc )K e y ID sE n c ry p te dM e m o ry w ithK e y ID17

MKTME Enabling Current StatusSpecification has been published [1]Core kernel enabling status Some preliminary patches have been upstreamed Feature emulation (CPUID, MSR); PCONFIG Proposal of some components have been sent to community for feedback Key management API: Using kernel key retention service Other components W IP internallyCore-MM keyIDsupport; IOMMU keyIDsupport; DMA keyIDsupport; KVM/Qemu enabling status PoC has been done to prove M KTM E actually works. Depending on core kernel parts ready for formal patches.[1] pdf18

THANKS

Intel Arch Live Migration VM Launch At-rest Runtime In-transit Typical VM Lifecycle in Cloud. 5 Trusted VM w/ OpenCIT -- OpenStack as Example Enterprise Data Center Cloud Service Provider Key Broker OpenStack Barbican Trust Director OpenStack Nova OpenStack Glance OpenStack Horizon App OS App OS App OS TPM Intel Arch w/ TXT