WIXLIB

Differential signalingPositive signal(single-ended)(non-inverted)1500 mVCommon modevoltage (thelevel is not veryimportant)1200 mV900 mV0VNegative signal(single-ended)(inverted)1010111500 mV1200 mVCommon modevoltage900 mV0VDifferential signal01011600 mVpositive - negative1500-900 600 mV0Vor900-1500 -600 mV-600 mV9/30/20031Differentialsignal isimmune tonoise commonto both singleended signals101SAS general overview011page 34

Physical link rate Each direction runs 1.5 Gbps or 3.0 Gbps (150 MB/sec or 300MB/sec)–Both directions use the same physical link rateDual simplex (full duplex) operation – 600 MB/sec total bandwidth Example: peak bandwidth needs of an HBA with 8 phys –9/30/20032400 MB/sec half duplex, 4800 MB/sec full duplexSAS general overviewpage 35

SAS devices and ports SASdevices containNarrow portports Ports contain phys Ports are virtualconstructs–DeviceDeviceNarrow portphyphyphyphyDeviceWide portGroups of phy with thesame SAS address,attached to anothergroup of phys with thesame SAS addressphyWide portphyDevicephyWide portphyphyWide portphyEach horizontal line represents a differential signal pair9/30/2003SAS general overviewpage 36

SAS addressEach SAS port and expander device has a worldwide unique 64bit SAS address Same namespace as the Fibre Channel Port Name Byte\Bit01276543210NAA (5h)IEEE Company ID (24 bits)3456Vendor-Specific Identifier (40 bits)79/30/2003SAS general overviewpage 37

End devices Enddevice is a SAS device that is not an expanderdevice Sample end devices–HBA – 8 phys One SAS address for all 8 phys– Potentially all one (very) wide port One SAS address for 4 phys, another SAS address for 4 phys– Guarantees at least two ports– Good match for 4-wide connectors Eight SAS addresses–Disk drive - 2 phys Separate SAS address for each phy– Guarantees two ports– Never a wide port9/30/2003SAS general overviewpage 38

Expander devicesExpander devicecontains expanderports May contain SASdevices too (e.g. forenclosuremanagement) Each expanderdevice has a SASaddress Each expander phyhas a phy identifierunique within thatexpander device ExpanderdeviceSMPtargetportExpander port(internal/optional)with virtualphysSAS portExpander al)Either SSP, STP and SMPSAS initiator ports, SAS target ports, orexpander ports9/30/2003SAS device(internal/optional)SAS general overviewExpander port(narrow port)ExpanderphySATASATA deviceportpage 39

Expander device types – edge vs. fanoutExpander deviceSAS addressAn expander device is an edge expander device or afanout expander deviceEdge expander device Edge––expander deviceAlways part of an “edge expander device set”May perform subtractive routing Fanout––Fanout expander deviceexpander deviceNever does subtractive routingUsually supports larger tables for table routing Topologies9/30/2003described laterSAS general overviewpage 40

DomainsA s An ATAdomaincontains aSATA hostand aSATAdevice Service deliverysubsystemwith port(s)SAS ortATA domain(informative)Note: When expander devices are present, SAS target ports may belocated in SAS devices contained in expander devices.SAS general overviewpage 41

SAS domain bridged to ATA Pinitiatorport(s)STPSMPinitiatorport(s)SMPService deliverysubsystemwith (s)9/30/2003STPSTPtargetport(s)SAS domainSTPSTP/SATAbridge(s)SATAATA deviceportATA domain(informative)SMPSMPtargetport(s)SAS general overviewpage 42

SAS devices in multiple SAS domains9/30/2003SAS general overviewpage 43

Edge expander device set Setof edge expander devices 128 SAS addresses per set Typically bounded by a subtractive port (to a fanoutexpander device, or to another edge expander deviceset) Edge expander devices uses table routing and directrouting “downstream” and subtractive routing“upstream” Wide links between expanders are allowed No loops9/30/2003SAS general overviewpage 44

Expander topologies Maximumof one fanout expander device in a SASdomain If no fanout expander, maximum of two edge expanderdevice sets (attached via subtractive decode ports) End devices may be attached at any level––Directly to fanout expander deviceAny level edge expander device Widelinks possible between any two devices No loops No multiple paths9/30/2003SAS general overviewpage 45

Edge expander device set and fanout expanderdeviceOne fanoutexpanderdeviceFanout expander deviceEdgeexpanderdevicesetAny of thephysical linkscould be wideEdgeexpanderdeviceEdgeexpanderdeviceEnd deviceEdgeexpanderdevicesetEdgeexpanderdeviceEnd deviceEdge expander deviceEnd deviceMaximumof 128SASaddressesperedgeexpanderdevicesetEnd device128 edge expander device sets9/30/2003SAS general overviewpage 46

Two edge expander device sets9/30/2003SAS general overviewpage 47

Expander routing attributes and methodsEach expander phy has an expander routing attribute The attribute determines the routing methods the expander useswith each phy Routing attribute When attached toEnd deviceDirectExpander deviceTableEnd deviceExpander deviceSubtractiveEnd deviceRouting method usedDirectTable DirectDirectSubtractive ( Direct)DirectDirect route requests to the attached SAS port through this phy Table route requests that match in routing table through this phy Subtractive route unresolved requests through this phy 9/30/2003SAS general overviewpage 48

Expander route table contentsPhy identifierA phy identifier for each expander phyof the expander device.Expander route table012.NN number of phys - 101Expander route entry2.Includes:! Routed SAS address! Enable/disable bitMExpander route indexAn expander route index for eachexpander route entry.M expander route indexes - 19/30/2003SAS general overviewpage 49

Connections Connection temporary association between an initiatorphy and a target phy–Source phy transmits an OPEN address frame–Expanders route it to a matching destination phyDestination phy replies with an OPEN ACCEPT primitiveConnection is establishedBoth sides exchange CLOSE primitives to close––– Contains a destination SAS address Connectionsare addressed to ports but established phy-to-phy N-wide ports may establish N connections at a time (toup to N other ports) Wide ports may establish multiple connections to otherwide ports simultaneously9/30/2003SAS general overviewpage 50

Connection examplesFanout expander device2-wide link4-wide linkEdgeEdgeexpanderexpanderdevicedeviceset2-wide link1-wide linkEdgeEdgeexpanderexpanderThe greendevicedeviceEdgeexpanderdevicesetEdge expander devicerequest isblockeduntil the redconnectionfinishesEnd device9/30/2003End deviceSAS general overviewEnd deviceEnd devicepage 51

Connection rules Connectionsare addressed to SAS ports but areestablished from phy to phy Wide ports may establish multiple connections at a time(to up to one per phy) to different destinations Wide ports may establish multiple connections to otherwide ports simultaneously (wide initiator port to widetarget port)––9/30/2003SAS disk drives will offer two narrow portsOnly HBAs and RAID controllers will offer wide portsSAS general overviewpage 52

Rate matching (connection rate)Lets 1.5 Gbps and3.0 Gbps SAS portscommunicate– Especially 3.0Gbps initiatorsand 1.5 GbpsSATA drives On faster physicallinks, insert ALIGNprimitives to slowdown the effectivedata rate to matchthe slowest physicallink in theconnectionPhysical link A3,0 Gbpsnegotiated physicallink rate EnddevicePhy(insertsALIGNs)Physical link C3,0 Gbpsnegotiated physicallink rateExpander deviceExpander vesALIGNs)(insertsALIGNs)1,5 Gbpsconnectionrate1,5 Gbpsconnectionrate1,5 GbpsconnectionrateSample dwords on physical links (from left to right) during a 1,5 Gbps connection:TimePhysicallink APhysicallink BPhysicallink C9/30/2003Physical link B1,5 Gbpsnegotiated physicallink rateALIGNdword 0ALIGNdworddworddword 1ALIGNdword 0ALIGNSAS general overviewdwordALIGNdword 2ALIGNdword 1dword 0ALIGNdworddword 2dword 1ALIGNpage 53

Phy layer9/30/2003SAS general overviewpage 54

Encoding 8b10bcoding converts 8-bit bytes into 10-bitdata characters for transmission on the wire8 bits Reasons––––Clock recoveryDC balanceSpecial charactersError detection Mapping––010 bits0255done with two simple logic blocks5b6b and 3b4bFull table in SAS and SATA Invented Used byby IBM in 1983Fibre Channel, Gigabit Ethernet,1394b, and many other standards9/30/2003SAS general overview1023page 55

Dwords Byte 8 bits (xxh) Character 10 bits (Dxx.y or Kxx.y) Dword 4 characters (or 4 bytes, depending oncontext)––––40 bits on the wireUsually represents 32 bits of dataA dword may flip disparity or leave it the sameA dword is either a data dword, a primitive, or an invaliddwordfirst second third fourthchar char char char9/30/2003SAS general overviewpage 56

Primitives 1control character and 3 data characters–First character is K28.5 (for SAS primitives), K28.3 (forSATA primitives), or K28.6 (special SATA error primitive) K28.6 primitive serves as an invalid dword for SATA–Last three characters are data charactersEndianness does not matter both SAS and SATA primitives always have the controlcharacter first on the wire–first second third fourthK28.5 Dxx.y Dxx.y Dxx.yK28.3 Dxx.y Dxx.y Dxx.yK28.6 Dxx.y Dxx.y Dxx.y9/30/2003SAS general overviewpage 57

OOB signals OOB–signal is a pattern of idle times and burst timesIdle time (and negation time) Differential 0 V (Positive signal negative signal) No transitions (DC idle)–Burst time Transmitted as a burst of ALIGN(0) primitives Received as presence of edges (whether they are valid ALIGNsis irrelevant) Designedto be detectable by analog squelch detectionlogic Length of idle time distinguishes between OOB signals–9/30/2003COMINIT, COMWAKE, and COMSASSAS general overviewpage 58

SAS OOB sequenceTransmit and receive COMINITTransmit COMSAS– If COMSAS received, physical link is SAS to SAS– If COMSAS is not received, physical link is SAS to SATA There are a few hot-plug situations where COMINIT leadsdirectly to COMSAS; this is allowed 9/30/2003SAS general overviewpage 59

SAS to SATA OOB sequenceIf COMSAS is not received, physical link is SAS to SATA SAS OOB sequence morphs into the SATA OOB sequence Since the SAS device drives COMWAKE, it looks like a SATAhost Only supports SATA device not SATA host 9/30/2003SAS general overviewpage 60

SAS speed negotiation sequence Slow-to-fastBoth phys runsame set ofspeednegotiationwindows1.5, then 3.0,then 6.0 (ifneeded), etc.until they find:– a supportedrate; then– a (faster) nonsupported rateLast windowreturns to thehighestsupported ratedetected9/30/2003SAS general overviewpage 61

Link layer9/30/2003SAS general overviewpage 62

Clock skew management Slow down the transmitter by including dwords that can be deleted– ALIGN (and NOTIFY for SAS) primitives stuck in– Throttles the transmitter to be slower than the slowest receiver SATA (and SAS STP) requires 2 per 256 dwords (0.7812%)– Must always appear in pairs SAS requires 1 per 2048 dwords (0.0488%)Receiver throws away incoming ALIGNs (and NOTIFYs for SAS)–transmitter dword e 9/30/2003dword fdword ereceiverALIGNdword fSAS general overviewdword gdword gdword hdword idword hpage 63

Connections Establishedbetween a SAS initiator phy and a SAStarget phy Three types of connections–––SSP (Serial SCSI protocol)SMP (Serial management protocol)STP (Serial ATA Tunneling protocol) Basic–––sequence1. Transmit OPEN address frame2. Receive AIP (arbitration in progress) primitives ifexpanders are involved3. Receive final result OPEN ACCEPT primitive – connection established OPEN REJECT primitive – rejected, go back to idle SATA9/30/2003has no concept of connectionsSAS general overviewpage 64

Connection – basic sequenceSOAFOPENaddress 8 data dwordsEOAFframeidledwords idle dwords 9/30/2003SAS general overviewAIPidle dwordsAIPidle dwordsAIPidle dwords OPEN ACCEPTor OPEN REJECTpage 65

Arbitration fairness and deadlocks SASimplements a Least Recently Used (LRU) arbitrationfairness scheme––Each OPEN address frame includes the age of a requestIn case of ties, SAS addresses are used to pick the highestpriority SASimplements deadlock detection and recovery usingPartial Pathway timers––9/30/2003When expanders detect a hang, one request is backed offto break the likely deadlockSAS addresses are eventually used to decide which requestis the lowest priority and gets backed offSAS general overviewpage 66

Closing a connection Exchange CLOSEprimitive sequencesend-to-end to close aconnectionSSP and SMP requiretraffic be stopped first– CLOSE ignored bySAS phy statemachines if not atthe correct timeSTP connectionshould be idle beforeclosing– Not enforced by SLstate machineExpanders alwayshonor CLOSEregardless ofconnection state9/30/2003SAS general overviewpage 67

SSP overview Inside an SSP connection Phys exchange SSP frames––SSP frame SOF primitive, data dwords, EOF primitiveEach frame results in an ACK or NAK primitive Credit-based–flow controlPermission to send a frame must be granted with RRDYprimitives Fullduplex–SSP frames can be sent in both directions simultaneouslyIndependent credit for each direction–(so can ALIGNs and NOTIFYs)– ACK,NAK, and RRDY primitives may be interjectedamong frame dwords9/30/2003SAS general overviewpage 68

SSP interlocked frames example Showing frames only in one directionSendframeWait forACK orNAKSendframeWait forACK orNAK9/30/2003Interlocked frameEOFidle dwords ACK or NAKidle dwords SOFInterlocked frameEOFidle dwords ACK or NAKSAS general overviewpage 69

SSP non-interlocked frames example Showing frames in one directionNon-interlocked frameEOF, SOF, non-interlocked frame1EOF, SOF, non-interlocked frame2EOF, SOF, non-interlocked frame3EOFidle dwords142349/30/2003SAS general overviewACK or NAK, idle dwordsACK or NAK, idle dwordsACK or NAK, idle dwordsACK or NAK, idle dwordspage 70

SMP overview Only–an initiator can open an SMP connectionTarget not allowed to open an initiator Insidean SMP connection Two SMP frames are transferred–––9/30/2003SMP frame SOF primitive, data dwords, EOF primitive1. Initiator transmits one SMP REQUEST frame to target2. Target transmits one SMP RESPONSE frame to initiatorSAS general overviewpage 71

SMP exampleSMP initiator phySMP target phyOPEN ACCEPTSendrequestframeSOFSMP REQUEST frameEOFidle dwords SendresponseframeSOFSMP RESPONSE frameEOFidle dwords (optional)CLOSECLOSE9/30/2003SAS general overviewpage 72

STP and SATA overview InSATA, SATA host and SATA device just communicatedirectly – no connections In SAS, once an STP connection is open, STP initiatorand STP target communicate as if they were SATA hostand SATA device directly attached on a physical link–Extra latency introduced as dwords flow through expanders Half–– duplexSATA never transmit frames in both directions at one timeUsually the frame goes one way and R IP primitives go theother wayNote: SAS prefixes the SATA primitive names with “SATA ”––9/30/2003SATA SOF (used in STP) is different than SOF (used in SSP)In this presentation, prefix not always usedSAS general overviewpage 73

SATA basic frame transmissionIdleRequestpermissionSATA SYNCSATA X RDYSATA SYNCIdleSATA R RDYSendframeWait forterminationIdleGrantpermissionSATA SOFData dwordsSATA EOFSATA WTRMSATA R IPSATA R OK or SATA R ERRAcknowledgeframe receiptSATA SYNCSATA SYNC9/30/2003Receipt inprogressSAS general overviewIdlepage 74

Transport layer9/30/2003SAS general overviewpage 75

SSP framesOne common SSPframe format– Frame header: 24bytes– Information Unit: 0to 1024 bytes– Fill bytes: 0 to 2bytes– CRC: 4 bytes Frame format isbased on FibreChannel 9/30/2003ByteField(s)0Frame Type1 to 3Hashed Destination SAS address4Reserved5 to 7Hashed Source SAS address8 to 9Reserved10Reserved11ReservedRetransmitNumber of FillBytes12 to 15Reserved16 to 17Tag18 to19Target Port Transfer Tag20 to 23Data Offset24 to mInformation Unitm to (n-3)Fill bytes, if needed(n-3) to nCRCSAS general overviewRsvdpage 76

SSP frame types Information Units are nearly identical to Fibre ChannelProtocol (FCP)– Separate TASK frame is new to SASInformationCommand Unit field DirectionDescriptionsizeCOMMAND 28 to 284 I to T Send a commandTASKI to T Send a task management function28XFER RDYT to I Request write data12I to T or Write data (I to T) or read data (TDATA1 to 1024T to I to I)Send SCSI status (for commands) or24 toRESPONSET to I task management response (for task1024management functions)9/30/2003SAS general overviewpage 77

SSP task management frame sequenceSSP initiator portSend TaskManagementRequest ()Received TaskManagementFunctionExecuted ()SSP target portTASK frameRESPONSE frametime9/30/2003TaskManagementRequestReceived ()TaskManagementFunctionExecuted ()timeSAS general overviewpage 78

SSP non-data command sequence9/30/2003SAS general overviewpage 79

SSP write command sequenceSSP initiator portSend SCSICommand ()SSP target portCOMMAND frameXFER RDY frameSSPinitiator portreplies toXFER RDYwith one ormore DATAIUsCommandCompleteReceived ()DATA frameRESPONSE frametime9/30/2003SCSI Command Received ()SSP target portsendsXFER RDYframes andreceives DATAframes until allwrite data hasData Out Received ()beentransferredReceive Data Out ()Send Command Complete ()timeSAS general overviewpage 80

SSP read command sequenceSSP initiator portS

Serial Attached SCSI Test and Integration Serial Attached SCSI Product Availability November 2002 Serial Attached SCSI standard to T10 letter ballot May 2003 Serial Attached SCSI to ANSI public review Serial Attached SCSI Product Design SAS-1.1 for bug fixes SAS-2 for 6.0 Gbps September 2003 Serial Attached SCSI to ANSI publication 4Q03 .