WIXLIB

Address Translation ServicesMichael Krause (HP, co-chair)Mark Hummel (AMD)David Wooten (Microsoft)Copyright 2006, PCI-SIG, All Rights Reserved1

Topics Introduction to Address Translation for DMAProblem StatementAddress Differentiated Memory RequestsTranslation RequestsTransaction CompletionsInvalidation RequestInvalidation CompletionsConfigurationPCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved2

ATS Specification Status The material in this presentation represents thecontents of the 0.7 version of the AddressTranslation Services Specification.9 ions/pciexpress/specification/draft/ats-spec-07 draft060327.pdf The technical details of 0.7 specifications areconsidered to be stable unless a specificproblem is found and needs to be fixed. The draft 0.9 specification will arrive shortly.PCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved3

Introduction to DMA AddressTranslation Address translation and an access check isapplied to DMA request from an IO device. In the Address Translation Services (ATS)specification, the entity doing the translation andchecking is called a Translation Agent (TA).PCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved4

DMA Remapping Illustration100DriverDriveroror OSOS100100305MMUMMU305TATADeviceDevice100MemoryOS orHypervisorPCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved5

DMA Remapping on HighVolume systems Multiple companies have announced support forDMA remapping in future chipsets for thevolume market These systems use tree-structured translationtables that are similar to CPU tables Different tree for each Bus/Dev/Func is possible9 Devices can share a device address space9 A devices can have a dedicated address spacePCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved6

Potential Issues of DMAr Increased latency of accesses9 Might need one or two accesses to find address oftree associated with a BDF9 Might need 3 or 4 accesses to walk the tree Translation caches (ATC or IOTLB) will benecessary to reduce overhead. Caches may not provide good behavior if notsized correctly9 Only two possibilities for sizing caches: too large ortoo small “Untimely” latency may cause issues withisochronous devicesPCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved7

ATS to the “rescue” ATS attempts to mitigate the impact of DMA Remappingby providing ways for Endpoints to participate intranslation cache management9 Device can maintain their own cache of translations – an“Address Translation Cache” (ATC)9 TA provides table-walking services to device to avoid excess bustraffic – also means that translation table format is uniform in asystem Device manages its ATC using its intimate knowledge offuture access pattern9 Look-ahead for isochronous devices to avoid “untimely” tablewalk latencies.9 High-load devices (graphics) don’t thrash ATC in TA.9 Application specific caching in devices – ring buffer9 Enable peer-to-peer in virtualized busPCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved8

New Protocols for ATS Differentiated memory address type – device willbe issuing Requests that use both translatedand non-translated addresses Translation Request – Address TranslationCache (ATC) in device requests a translationfrom central TA Translation Completion – translation is returnedin response to Translation Request Invalidation Request – when change occurs incentral table, need to inform remote ATCs Invalidation Completion – when ATC completesthe invalidation operation, it needs to tell TA.PCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved9

Differentiated Memory Request Previously reserved field in Memory Requestused to differentiate Address Type (AT) inRequest. 0765rFmtx 14 1321Type0 000007r654TC 23210reserved7654 33T EAttrD PRequester ID21ATTag076543210LengthLast DWBE1st DWBEAddress 63:32Address 31:02AT MeaningAT Meaning00b Default (un-translated)10b Translated01b Translation Request11b ReservedPCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reservedr10

Translation Request 0765rFmt0 14 1321Type0 000007r65TCRequester ID4 23210reserved7654 33T E AttrD P r r2AT0 1Tag1076543210Length00 000x xxx0Last DW BE 1st DW BE11111111Un-translated Address [63:32]Un-translated Address [31:02] rTranslation Request is Memory Read Request with ATfield set to 01b.Request is for TA to return the translated reference forthe memory range starting at the location referenced inthe Un-translated Address field.PCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved11

Translation Request (cont.) Always uses 64-bit form of address Length always an even number of dwords up toRCB Device can request translations for a range ofaddresses9 range is determined from Length and SmallestTranslation Unit (STU)*9 Size in bytes of range request is Length/2 * STU*Note: The STU is specified in binary multiples of 4KB andis nominally the same size as a page on the system.PCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved12

Translation CompletionError Completion765rFmt0 0 04 3 1210Type0 101076r5TC4 23210reservedCompleter IDRequester ID7654 33T E AttrD P 0 0BCompl. CStatus M0Tag21076543210Length00 0000 0000rByte Count0000 0000 0000rLower Address000 0000Cpl header for Translation CompletionValueStatusMeaning000bSuccessIn Cpl, means no translation found. Nominally means that table walkdid not reach a leaf (page table pointer) entry.001bUnsupportedRequestTranslation Requests from this Function are not supported by the TA.100bCompleterAbortError in the TA. Translation Request may be retried.AllothersReservedReserved – malformed packetPCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved13

Translation CompletionNormal Completion 0765rFmt1 04 1321Type0 101007r65TC4 23210reservedCompleter IDRequester ID7654 332T E AttrrD P 0 0BCompl.CStatusM0000Tag1076543210LengthByte CountrLower AddressCplD header for Translation CompletionTranslated Address [63:32]Translated Address [31:12]SNReservedUWRTranslation Completion translation entry (1 of N)PCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved14

Translation Completion ( cont.)FieldMeaningSSize of translation: This field is 0b if the translation applies to a 4KB rangeof memory. If this field is 1b, then the translation applies to a range ofmemory that is larger than 4KB. See 3.3.1NNon-snooped accesses: If this field is 1b, then the read and write requeststhat use this translation must not set the No Snoop bit in the Attributefield. If it is zero, then the Endpoint may use other means to determine if NoSnoop should be set.Reserved These bits shall be ignored by the ATC.UUn-translated access Only: When this field is set to 1b in a TranslationCompletion entry, the indicated range may only be accessed using untranslated addresses and the Translated Address field of this TranslationCompletion entry may not be used in a subsequent Read/Write Requestwith AT set to Translated. This value may be cached if R or W is set to 1b.R,WRead, Write – These two fields indicate what the transaction types that areallowed for the requests using the translation. If neither field is Set, then thetranslation is not valid and all the remaining fields of this dword are undefined.A value with R W 0b may not be cached in the ATC.PCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved15

Translation Completion ( cont.) TA can return 0 or more translations9 TA decides9 The TA may not return more than Length dwords May take one or two CplDs to close out the request. In a CplD, each translation entry covers the same sizedrange of addresses.9 Smallest range is STU9 All ranges will have some overlap with requested range9 All ranges in the completion must be the same size– If completion in two CplDs, then ranges in both must have samesize.9 Different completions can have different range size.PCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved16

Range DeterminationAddress Bits63:18 17 16 15 14 13 12STranslationRange Sizein 0111164Kxx011111128K Starting with S, look for first 0b (bit N) Range size is 2 (N 1) bytesPCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved17

Translation CompletionsMultiple CplDs TA may split a translation completion any time Length isgreater than 2 Since Length is no larger than allowed by RCB,completion will not take more than two CplDs In all CplD:9 Byte Count indicates bytes remaining to complete request,including the bytes in the “current” CplD9 Length indicates the number of dwords in the current Cpld. For Translation Completions, in the first CplD, LowerAddress is set to:(000 0000b) – (Length * 4)Example: if first CplD contains 2 translations the Lower Address is:(000 0000b) – (00 0000 0100b * 100b) (111 0000b)PCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved18

Translation CompletionsMissing CplDs If Bytes (Length * 4), then this is a first CplD ofa two CplD Completion. Else if Bytes (Length * 4) then this is amalformed TLP Else, if Lower Address Byte Count is not amultiple of RCB, then this is a second CplD oftwo; and if previous Transaction CompletionCplD with same Tag was not received, then aCplD is missing.PCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved19

InvalidationCopyright 2006, PCI-SIG, All Rights Reserved20

Invalidation Purpose9 Maintain consistency between TA and ATC Mechanism9 Invalidate Request Message9 Invalidate Complete MessagePCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved21

Invalidate Request 0765RFmt114 1321Type1 0010076R5TC4 23210ReservedRequester ID7654 33T E AttrD P 000 0 0Device ID2107654 3210Length00 0000 0001Message CodeITag0000 0001ReservedRReservedUn-translated Address [63:32]Un-translated Address [31:12]S ReservedMSGD packet with 4 DW of header and 2 DW of data9 Message code 0000 0001b9 Route by Device ID ITag9 Invalidate tag – Uniquely identifies each Invalidate Request Un-translated Address9 Starting address of block to be invalidated S9 Size bit (same encoding as in Translation Request)PCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved22

Invalidation Range Constraints Each Invalidate Request specifies a singlememory Region9 Power of 2 in size9 Naturally aligned9 Must be at least as big as a STU (StandardTranslation Unit)9 S field and least significant address bits encode rangesizePCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved23

Invalidate Completion 07R65 4Fmt01 1321Type1 0010076R5TC4 23210Reserved7654 33T E AttrD P 00Requester ID210RReservedDevice IDReserved7654 321000 0000 0000Message Code0000 0010CCITag Vector MSG packet with 4 DW of header9 Message code 0000 00109 Route by Device Id ITag Vector9 Invalidation tag vector – Uniquely identifies each Invalidate Completion9 Multiple completions may be compressed into a single response CC9 Completion Count – Indicates number of completions that have beensent for the corresponding request (0 8 completions)9 Always 1 for single TC devicesPCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved24

Invalidation CompletionSemantics Properties to satisfy9 Maintain consistency9 Prevent silent data corruption9 Prevent data leakage Invalidate Complete must not be returned until:9 Stale address translation is flushed from ATC9 Conflicting outstanding writes are pushed to the TA9 Conflicting outstanding reads are either completed ortagged for discardPCI-SIG Developers ConferenceCopyright 2006, PCI-SIG, All Rights Reserved25

Invalidation Flow – Single TC1) Invalidate Request TC 0 ITag 3Root Complex2) Flush matching ATCentries. Drain or discardconflicting Reads .DeviceAATTCCTTAAPosted Write TC 1PCI-SIG Developers Conference3) Invalidate Completion TC 1 ITagV 1000b CC 1Copyright 2006, PCI-SIG, All Rights Reserved26