Transcription
ATLAS ShruggedPat O’Toole(with apologies to Ayn Rand and John Galt)
ATLASzzzzzzz“Ask The Lead AppraiserS”A scenario-based email forum used to elicitopinions on “interesting” topicsDistributed to all 400 LAs and 1500 non-LAsLimited to one pageMultiple choice format - ample room for commentsResults are compiled and published by PACT withno intellectual property rights retainedSEI is just another recipient of the data – they donot sponsor or influence ATLAS in any way.2
ATLAS ScenariosScenario number (# LA’s / # non-LA’s)#1 – Bidirectional traceability (46/45)#2 – Process descriptions/measures (40/17)#3 – Applicability of SAM (66/75)#4 – Interpretational issues (79)3
ATLAS Scenarios#1 – Bidirectional traceability (46/45)Manage descriptions/measuresRequirements#2 – Process(40/17)#3 – Applicability ofObtainSAM (66/75)ManageObtain anCommitmentRequirementsUnderstandingto#4 – Interpretationalissues eability nsistenciesBetween ProjectWork andRequirementsTraceability Matrix4
ATLAS #1 – Bidirectional TraceabilityzA project in a SCAMPI A appraisal is:z estimatedto be 30,000 person hoursz 14 months into its 18 month schedulez preparing to initiate system testingzNo alternative practices for this project.5
Question 11. For this project, which selection bestrepresents your view of model expectationswith respect to REQM SP1.4?A.B.C.D.E.Vertical and horizontal traceabilityEither vertical or horizontal traceabilityOnly vertical traceabilityOnly horizontal traceabilityOther6
Answer 1: Model ExpectationsSelected choice:A: Vertical and horizontalB: Vertical or horizontalC: Vertical onlyD: Horizontal onlyE: Other/NoneLeads Non-Leads70%84%13%0%13%11%0%4%4%0%7
Question 22. What characterization (FI, LI, PI, NI)is most appropriate if there is:A. Ample evidence of vertical traceabilitybut no evidence of horizontal traceability?B. Ample evidence of horizontal traceabilitybut no evidence of vertical traceability?8
Answer 2a - CharacterizationsAmple evidence of vertical traceabilitybut no evidence of horizontal traceabilitySelected choice:A. FIB. LIC. PID. NILeads20%39%36%5%Non-Leads15%32%47%6%9
Answer 2b - CharacterizationsAmple evidence of horizontal traceabilitybut no evidence of vertical traceabilitySelected choice:A. FIB. LIC. PID. NILeads7%11%66%16%Non-Leads2%9%66%23%10
Question 33. For each of the following, pleaseindicate if you consider it to be:A. Vertical TraceabilityB. Horizontal TraceabilityC. NeitherD. BothE. I don’t have a clue! (Don’t know)11
Answer 3a – Traceability TypeHigh-level business requirementsare traceable to feature requirementsSelected choice:A: VerticalB: HorizontalC: NeitherD: BothE. Don't KnowLeads95%5%0%0%0%Non-Leads86%2%0%12%0%12
Answer 3b – Traceability TypeTraceability is maintained amonginterdependent functional requirementsSelected choice:A: VerticalB: HorizontalC: NeitherD: BothE. Don't KnowLeads4%82%2%9%2%Non-Leads9%74%2%16%0%13
Answer 3c – Traceability TypeEach of the 500 system test cases liststhe specific requirement(s) being testedSelected choice:A: VerticalB: HorizontalC: NeitherD: BothE. Don't KnowLeads78%16%2%4%0%Non-Leads67%21%4%9%0%Note: Traceability ACROSS the life cycle is “vertical;”And we wonder why there are interpretational issues!14
Answer 3d – Traceability TypeSystem requirements are traceable tothe group(s) to which they are allocatedSelected choice:A: VerticalB: HorizontalC: NeitherD: BothE. Don't KnowLeads51%29%16%0%4%Non-Leads32%39%14%16%0%15
Answer 3e – Traceability TypeTechnical requirements are traceableto specific elements in the WBSSelected choice:A: VerticalB: HorizontalC: NeitherD: BothE. Don't KnowLeads51%29%16%4%0%Non-Leads40%35%16%7%2%16
ATLAS #1 NotezThe SEI’s website contains answers to“Frequently Asked Questions” (FAQ)zBidirectional traceability is .html#Q318See partial text on next slideOnly 1 of nearly 100 respondents (a leadappraiser) mentioned the SEI FAQ!17
SEI FAQ regarding TraceabilityVertical traceability identifies the origin of items (e.g.,customer needs) and follows these same items as they travelthrough the hierarchy of the WBS to the project teams andeventually to the customer. When the requirements aremanaged well, traceability can be established from the sourcerequirement to its lower level requirements and from the lowerlevel requirements back to their source.Horizontal traceability is also important, but it is notrequired to satisfy bidirectional traceability. Horizontaltraceability identifies the relationships among related itemsacross work groups or product components for the purpose ofavoiding potential conflicts. For example, horizontaltraceability would follow related requirements across two workgroups working on two associated components of a product.18
ATLAS Scenarios#1 – Bidirectional traceability (46/45)#2 – Process descriptions/measures (40/17)#3 – Applicability of SAM (66/75)#4 – Interpretational issues (79)19
ATLAS #2: Process DescriptionsWhen conducting a ML2 appraisal, theorganization has no documented processdescriptions, and organizational personnelcorroborate this.20
Question 1Would you document a weakness regarding the lack of processdescriptions?YesNoLead Appraisers79% 21%Non-Leads87% 13%Do you perceive this to be a goal-threatening weakness?Lead AppraisersNon-LeadsYes65%73%No35%27%21
ATLAS #2: Metric SpecificationsWhen conducting a ML2 appraisal theorganization employs project and productmeasures, but no process measures.Organizational personnel corroborate this.They DO plan and track the processactivities associated with REQM, PP, PMC,etc., but they have not implemented anyprocess measures as suggested by theGP2.8 example boxes.22
Question 2Would you document a weakness regarding the lack of processmeasures?YesNoLead Appraisers51% 49%Non-Leads80% 20%Do you perceive this to be a goal-threatening weakness?Lead AppraisersNon-LeadsYes18%53%No82%47%23
ATLAS #2 – How Much Is Enough?When conducting a ML2 appraisal, the org has specifiedonly 4 measures: SLOC, Earned Value, Peer ReviewDefects, and Test Defects. The specifications arecomplete and cover all of the MA SG1 specific practices.In addition to the 4 specified measures, the org andprojects capture and use many more measures, but nospecs exist for these additional measures.Org personnel contend that the specified measures arethose that were most recently introduced (throughout thepast year). The unspecified measures were already wellestablished and used consistently.24
Question 3Would you document a weakness regarding the limited numberof specified measures?YesNoLead Appraisers70% 30%Non-Leads53% 47%Do you perceive this to be a goal-threatening weakness?YesNoLead Appraisers27% 73%Non-Leads33% 67%25
ATLAS Scenarios#1 – Bidirectional traceability (46/45)#2 – Process descriptions/measures (40/17)#3 – Applicability of SAM (66/75)#4 – Interpretational issues (79)26
ATLAS #3, Scenario 1The project team is incorporating an “OpenSource” component. The source code wasposted on the originator’s website with anindication that it can be used without restriction,free of charge and “at your own risk.”The originator also indicated she retains nointellectual property rights with respect to thecomponent, nor any responsibility for itsongoing support/maintenance.27
Question 1Must SAM be applied with respect to theOpen Source component?YesLead Appraisers35%Non-lead Appraisers 19%No65%81%28
ATLAS #3, Scenario 2The customer’s SOW requires that youincorporate an unmodified version ofcomponent X which is available solely fromCompany Y.According to the SOW, the customer willnegotiate X’s acquisition cost, maintenancefees, and license fees with Company Y.29
Question 2Must SAM be applied with respect to Company Y?Lead AppraisersNon-lead AppraisersYes28%34%No72%66%Must SAM be applied with respect to the customer?YesNoLead Appraisers50% 50%Non-lead Appraisers 64% 36%30
ATLAS #3, Scenario 3The solution that your very small companyintends to provide to your customer includesa laser jet printer supplied by Very BigCompany.It is off-the-shelf and no modifications arerequired.31
Question 3Must SAM be applied with respect toVery Big Company?YesLead Appraisers52%Non-lead Appraisers 51%No48%49%32
ATLAS Scenarios#1 – Bidirectional traceability (46/45)#2 – Process descriptions/measures (40/17)#3 – Applicability of SAM (66/75)#4 – Interpretational issues (79)33
ATLAS #4 – Interpretational IssueszWhich 3 ML2 specific practices are mostlikely to encounter interpretational issues?Interpretational Issues - ML2 Response Data by RankPAPracticeREQM SP 1.4-2PPSP 2.3-1MA SP 1.1-1PPSP 1.2-1CM SP 3.2-1Practice TitleMaintain BidirectionalTraceability of RequirementsPlan for Data ManagementEstablish MeasurementObjectivesEstablish Estimates of WorkProduct and Task AttributesPerform Configuration AuditsPercent of Rank Cum. % ofNumber ofML2withinML2ML2Respondents Respondents ML2 448%34
ATLAS #4 – Interpretational IssueszWhich 3 ML3 specific practices are mostlikely to encounter interpretational issues?Interpretational Issues - ML3 Response Data by RankPAPracticePractice TitleEstablish OperationalConcepts and ScenariosEstablish Guidelines forDAR SP 1.1-1Decision AnalysisEvolve Operational ConceptsTS SP 1.2-2and ScenariosEstablish a Technical DataTS SP 2.2-3PackageValidate Requirements withRD SP 3.5-2Comprehensive MethodsRDSP 3.1-1Number of Percentage of Rank Cum. % ofML3ML3withinML3Respondents Respondents ML3 29%35
ATLAS #4 – Interpretational IssueszWhich 3 ML4/5 specific practices are mostlikely to encounter interpretational issues?Interpretational Issues - ML4/5 Response Data by RankPAPracticeOPPSP 1.5-1Practice TitleNumber of Percent ofRank Cum. % ofML3ML3withinML3Respondents Respondents ML3 ResponsesEstablish ProcessPerformance Models2649%119%QPM SP 1.3-1Select the Subprocesses thatWill Be Statistically Managed1325%228%QPM SP 1.2-1Compose the Defined Process1223%337%1121%445%361019%552%OPPSP 1.4-1OPP SP 1.1-1Establish ProcessPerformance BaselinesSelect Processes
ATLAS #4 – Interpretational IssueszWhich 1 Generic Practice is most likely toencounter interpretational issues?Interpretational Issues - GP Response Data by RankPAPracticeGP 2.8GP 2.2GP 3.2GP 2.9GP 2.7Practice TitleNumber ofPercent of Rank Cum. % ofML3ML3withinML3Respondents Respondents ML3 ResponsesMonitor and Control theProcessPlan the ProcessCollect ImprovementInformationObjectively EvaluateAdherenceIdentify and Involve %458%79%566%37
ConclusionzThe SEI’s lead appraiser upgrade trainingincluded a module on model interpretation issues.The conclusion drawn by the SEI VisitingScientist that authored that section is:“Model interpretation issues will always exist.For the benefit of the lead appraiser communityand that of our constituents, such issues needto be identified, discussed, resolved andcommunicated.”38
Questions?To be added to the ATLAS distribution list, send an mail to:Pat O’ToolePACT.otoole@att.net(And don’t hesitate to email suggestions for other “interesting” topics!)39
ATLAS Shrugged Pat O’Toole (with apologies to Ayn Rand and John Galt) 2 ATLAS z“Ask The Lead AppraiserS ” zA scenario-based email forum used to elicit opinions on “interesting” topics zDistributed to all 400 LAs and 1500 non-LAs zLimited to one page zMultiple choice format - ample room for comments zResults are compiled and published by PACT with no intellectual property rights .
