B. Test Results: Trouble Analysis Facilitation Interface .

Transcription

BellSouth – GeorgiaSTP Final ReportB.Test Results: Trouble Analysis Facilitation Interface (TAFI) FunctionalTest of Resale Lines (M&R-12)1.0DescriptionThe objective of the TAFI Functional Test of Resale Lines was to validate theexistence of TAFI trouble reporting and screening functionality for telephonenumber (TN) assigned resale service customers in accordance with the CLECTAFI User Guide (User Guide). This test cycle was executed in BellSouth's TAFIproduction environment by exercising a defined set of functions associated withtrouble management activities against resale test bed accounts. Scenarios testingthese functions were executed via a LAN-to-LAN connection and via dial-upaccess in order to evaluate differences in system response times associated withthe methods of access.The functional elements specifically targeted by this test include the entry andresolution of trouble reports, query and receipt of status reports, access to testcapabilities, access to trouble history, and error conditions. TAFI functionalitywas evaluated in conjunction with the documentation provided addressing itsuse. In addition, TAFI usability was considered as part of this test.2.0MethodologyThis section summarizes the test methodology.2.1Business Process DescriptionSee Section VI, “M&R Overview” for a description of BellSouth's TAFI interface.2.2ScenariosTAFI functionality was tested by manually processing Maintenance and Repair(M&R) related scenarios in TAFI via both dial-up and LAN-to-LAN connections.The transactions used in this evaluation were chosen to test the applicable TAFIfunctions across the line types specified in Table VI-2.1 and were not intended todemonstrate statistical significance. The following table lists the scenarios usedto test each of the functions included in the TAFI functionality test.Table VI-2.1: TAFI Functional ScenariosScenarioNumberScenario Description1Business customer with resale POTS line is experiencing problems with their three-way callingvertical feature.2Business customer with resale POTS line is experiencing problems with their call waitingMarch 20, 2001VI-B-1Published by KPMG Consulting, Inc. Confidential. For BellSouth, KCI, and Georgia Public Service Commission use.

BellSouth – GeorgiaScenarioNumberSTP Final ReportScenario Descriptionvertical feature.3Business customer with resale POTS line is experiencing problems with their call forwardingand three- way calling vertical features.4Business customer with resale POTS line is hearing other conversations on their line.5Residential customer with resale POTS line is hearing a roaring sound on their line as well asexperiencing no dial tone at times.6Business customer with resale POTS line is experiencing problems with their call forwardingand three- way calling vertical features.7Residential customer with resale POTS line is getting a wrong number when making anoutgoing call.8Business customer with resale POTS line is experiencing troubles with their three-way callingvertical feature as well as transmission problems.9Residential customer with resale POTS line is experiencing problems with their speed callingvertical feature.10Business customer with resale POTS line is experiencing a problem with their inside wiring orjack.11Business customer with resale POTS lines is experiencing troubles with incoming calls on twolines.12Business customer with resale POTS lines is experiencing transmission troubles on two lines.13Business customer with resale POTS lines is experiencing trouble making outgoing calls on twolines.14Business customer with resale POTS lines is experiencing physical trouble on two lines.15Business customer with resale POTS lines is experiencing dial tone related troubles on two lines.16Business customer with resale POTS lines is experiencing troubles with incoming calls on twolines.2.3Test Targets & MeasuresThe test targets were TAFI and the CLEC TAFI User Guide (Issue 1, March, 2000and Issue 2, April, 2000). The CLEC TAFI User Guide is provided to CLECpersonnel attending BellSouth’s CLEC TAFI training class. It is also availableonline at the BellSouth Interconnection site at ides p.html. This manual is both a training tool and areference tool. The TAFI training provided to the CLECs is a two-day coursewith a standard charge for each participant.March 20, 2001VI-B-2Published by KPMG Consulting, Inc. Confidential. For BellSouth, KCI, and Georgia Public Service Commission use.

BellSouth – GeorgiaSTP Final ReportProcesses, sub-processes, and evaluation measures are summarized in thefollowing table. The last column, “Test Cross-Reference,” indicates where theparticular measures are addressed in Section 3.1 "Results & Analysis.”Table VI-2.2: Test Target Cross-ReferenceProcessTrouble reportsSub-ProcessCreate trouble reportEvaluation MeasureTest Cross-ReferencePresence ofFunctionalityAccuracy of ResponseTAFI UsabilityM&R-12-1-1Presence ofFunctionalityAccuracy of ResponseTAFI UsabilityM&R-12-1-2Presence ofFunctionalityAccuracy of ResponseTAFI UsabilityM&R-12-1-3Presence ofFunctionalityAccuracy of ResponseTAFI UsabilityM&R-12-1-4Presence ofFunctionalityAccuracy of ResponseTAFI UsabilityM&R-12-1-5Enter and RetrieveTrouble Reports fromQueuesPresence ofFunctionalityAccuracy of ResponseTimeliness of ResponseTAFI UsabilityM&R-12-1-6Execute SupervisorFunctionsPresence ofFunctionalityAccuracy of ResponseTimeliness of ResponseTAFI UsabilityM&R-12-1-7Presence ofFunctionalityAccuracy of ResponseTAFI UsabilityM&R-12-1-8Modify trouble reportCreate repeat reportCreate subsequent reportEnter Multiple TroubleReports (MTRs)Close Trouble 12-2-2M&R-12-3-7M&R-12-1-8M&R-12-3-8March 20, 2001VI-B-3Published by KPMG Consulting, Inc. Confidential. For BellSouth, KCI, and Georgia Public Service Commission use.

BellSouth – GeorgiaProcessSTP Final ReportSub-ProcessCancel Trouble ReportAccess to testcapabilityInitiate port and loopport testView port and loop-porttest resultsDownstreamSystem ReportsRetrieve LMOS recentstatus reportObtain customer linerecord (BOCRIS)Obtain Predictor resultsView DLR (Display LineRecord)View SOCS pendingorderEvaluation MeasureTest Cross-ReferencePresence ofFunctionalityAccuracy of ResponseTAFI UsabilityM&R-12-1-9Presence ofFunctionalityAccuracy of ResponseTimeliness of ResponseTAFI UsabilityM&R-12-1-10Presence ofFunctionalityAccuracy of ResponseTimeliness of ResponseTAFI UsabilityM&R-12-1-11Presence ofFunctionalityAccuracy of ResponseTimeliness of ResponseTAFI UsabilityM&R-12-1-12Presence ofFunctionalityAccuracy of ResponseTimeliness of ResponseTAFI UsabilityM&R-12-1-13Presence ofFunctionalityAccuracy of ResponseTimeliness of ResponseTAFI UsabilityM&R-12-1-14Presence ofFunctionalityAccuracy of ResponseTimeliness of ResponseTAFI UsabilityM&R-12-1-15Presence ofFunctionalityAccuracy of ResponseTimeliness of ResponseTAFI -8M&R-12-3-16March 20, 2001VI-B-4Published by KPMG Consulting, Inc. Confidential. For BellSouth, KCI, and Georgia Public Service Commission use.

BellSouth – GeorgiaSTP Final ReportProcessAccess errorreportsTrouble historyEvaluation MeasureHost request errorsPresence ofFunctionalityAccuracy of ResponseTAFI UsabilityM&R-12-1-17Presence ofFunctionalityAccuracy of ResponseTimeliness of ResponseTAFI UsabilityM&R-12-1-18TAFI UsabilityM&R-12-3-19Retrieve Trouble HistoryGeneral2.4Sub-ProcessTAFI UsabilityTest R-12-2-9M&R-12-3-18Data SourcesThe data for M&R12 were collected during participation in a TAFI trainingcourse, through interviews with BellSouth personnel, through reviews ofBellSouth documentation, and through the execution of functional test scenariosin BellSouth’s TAFI production environment. The data collected for M&R12 aresummarized in the table below.TableVI-2.3: Data Sources for TAFI Functional TestDocumentFile NameLocation in WorkPapersSourceBellSouth ResidentialRepair Center InterviewSummaries andApprovalsNo Electronic CopyM&R-1-A-2BLS/KCIBellSouth Business RepairCenter InterviewSummaries andApprovalsNo Electronic CopyM&R-1-A-3BLS/KCITAFI Online HelpNo Electronic CopyM&R-12-A-3BLSExcerpts of TAFIArchitecture from theCLEC TAFI SpecificationsdocumentNo Electronic CopyM&R-1-A-5BLSFunctional Test ApproachStatementsNo Electronic CopyM&R-12-A-5KCIFunctional Test Logs:LAN-to-LANNo Electronic CopyM&R-12-A-6KCIFunctional Test Logs:Dial-UpNo Electronic CopyM&R-12-A-7KCIFunctional Test Logs:SOCSNo Electronic CopyM&R-12-A-8KCIMarch 20, 2001VI-B-5Published by KPMG Consulting, Inc. Confidential. For BellSouth, KCI, and Georgia Public Service Commission use.

BellSouth – GeorgiaSTP Final ReportDocumentFile NameLocation in WorkPapersSourceScreen Prints: LAN-toLANNo Electronic CopyM&R-12-A-9KCIScreen Prints: Dial-UpNo Electronic CopyM&R-12-A-10KCIScreen Prints: SOCSNo Electronic CopyM&R-12-B-11KCILegacy Access Times forCLEC TAFI and BLS TAFIReport (April 2000)No Electronic CopyM&R-12-B-12BLSCLEC TAFI User Guide(Issue 1)Clec101g.pdfM&R-8-A-16BLSCLEC TAFI User Guide(Issue 2)Clec trn.pdfM&R-8-A-16BLSCLEC TAFI User Guide(Issue 3)Gtaff001.pdfM&R-8-A-16BLS2.4.1 Data Generation/VolumesThis test relied on the results expected as a result of the examination of the CLECTAFI User Guide and the submission of trouble-related transactions through theTAFI interface. This test did not rely on volume testing.2.5Evaluation MethodsIn preparation for the functional testing of resale lines, information gatheredduring the interviews with BellSouth Customer Service Associates (CSAs),Maintenance Administrators (MAs), and management personnel from theResidential Repair Center (RRC) and Business Repair Center (BRC) wasreviewed. This test cycle was executed by exercising a defined set of TAFIfunctions associated with trouble management activities against test bedaccounts1. The CLEC TAFI User Guide and M&R test bed data were used tomanually process the 16 test scenarios, using TAFI, as documented in Section2.2. During testing, other functionality such as edit rules and designed errors(e.g., invalid entries, cancels, and repeat troubles) were checked. These 16scenarios comprised the input used to test the 19 functions defined in Table VI2-2 on business and residential resale lines.The following steps outline the test approach.1. The CLEC TAFI User Guide was reviewed to determine how to process each ofthe functional tests associated with the 16 M&R scenarios defined in Section2.2.1SeeSection IV, “Pre-Ordering, Ordering & Provisioning Overview” for a description of the test bed usedfor this evaluation.March 20, 2001VI-B-6Published by KPMG Consulting, Inc. Confidential. For BellSouth, KCI, and Georgia Public Service Commission use.

BellSouth – GeorgiaSTP Final Report2. Paper-based Functional Test Approach Statements including expected resultsfor each scenario were completed using the CLEC TAFI User Guide. As part ofthis process, KCI considered the usability of the CLEC TAFI User Guide,commenting on attributes such as ease of use and clarity. As each M&Rscenario was used to test multiple functions, multiple Approach Statementswere often created for each scenario.3. The statements created in Step 2 were used to provide the key data entered inthe TAFI system during test execution. However, due to the decision treelogic embedded in TAFI, the exact data required to perform some of thefunctions could not be predetermined for the Functional Test ApproachStatements by referencing the user manual. Therefore, the User Guide wasactively utilized in conjunction with the data from the paper forms duringtest execution.4. In order to prevent technicians from being inappropriately dispatched andinterrupting BellSouth operations, KCI, with BellSouth’s concurrence, tookthe following steps for each trouble report created: The phrase TST TCKT DN DISP / PLS IGNR was placed in the narrativesection of each trouble report.The commitment time was set at a date one month out.The CLEC contact number posted on each report was 404-954-5715, aworking number that connected to a KCI tester's desk.5. During test execution, Functional Test Logs were utilized to document stepstaken by KCI, as well as system responses. Appropriate categories ofevaluation criteria were considered as these system responses and commentswere recorded.6. As part of the data entry process in Step (3), TAFI fields were validated toensure that invalid data were flagged, and that required fields werepopulated.2.6Analysis MethodsThe M&R-12 TAFI Functional Test of Resale Lines included a checklist ofevaluation criteria developed by KCI during the initial phase of the BellSouth Georgia OSS Evaluation. These evaluation criteria provided the framework ofnorms, standards, and guidelines for the TAFI Functional Test of Resale Lines.The data collected from transaction processing were analyzed relative to theevaluation criteria referenced above.3.0Results SummaryThis section identifies the evaluation criteria and test results.March 20, 2001VI-B-7Published by KPMG Consulting, Inc. Confidential. For BellSouth, KCI, and Georgia Public Service Commission use.

BellSouth – Georgia3.1STP Final ReportResults & AnalysisThe results of this test are presented in the table below. Definitions ofevaluation criteria, possible results, and exceptions are provided in Section II.Table VI-2.4: Evaluation Criteria and Results - Presence of FunctionalityTest CrossReferenceEvaluation CriteriaResultCommentsM&R-12-1-1The user is able to entera trouble report usingTAFI and receive asatisfactory response.SatisfiedTAFI was used to create 30 troubletickets and responded as expected 30times.M&R-12-1-2The user is able tomodify a trouble reportusing TAFI and receivea satisfactory response.Satisfied"Modify" is not a formal functionavailable in TAFI. Rather,modifications to a trouble ticket areperformed through the creation of asubsequent report or through editfunctions in the trouble report screenduring initial trouble report creation.Edit rules, in terms of required fields,were specifically tested in sixscenarios and six satisfactoryresponses were received. In order totest this function, KCI entered datainto fields incorrectly. In theseinstances, TAFI automatically flaggedthe field tested with the cursor andprovided an instructive comment inthe status field. Similarly, KCI leftsome required fields blank in order totest TAFI. As described above, TAFIflagged the required field with thecursor and provided instructivecommentary in the status field.In all instances where modificationswere made, TAFI responded asexpected.M&R-12-1-3The user is able to createa repeat report usingTAFI and receive asatisfactory response.SatisfiedTAFI was used to create 16 repeatreports and responded as expected in14 instances. In two instances, slightdiscrepancies relative to the expectedresults were noted due to specialcircumstances. According to theCLEC TAFI User Guide, TAFIautomatically denotes a report as arepeat if there has been anothertrouble reported on the line within theMarch 20, 2001VI-B-8Published by KPMG Consulting, Inc. Confidential. For BellSouth, KCI, and Georgia Public Service Commission use.

BellSouth – GeorgiaTest CrossReferenceSTP Final ReportEvaluation CriteriaResultCommentslast 30 days. However, in bothinstances, a trouble was entered andclosed; yet in follow-up entries of thesame telephone number (TN), thereports were not recognized as repeatreports. This occurred because thetickets had been closed by the user, asprompted by TAFI, prior to troublereport creation. As an actual troubleticket was never created in the LMOSsystem, no record of a prior troubleexisted to denote the 'new' reports asrepeat reports.M&R-12-1-4The user is able to createa subsequent reportusing TAFI and receivea satisfactory response.SatisfiedTAFI was used to create 30subsequent reports and responded asexpected 30 times.M&R-12-1-5The user is able to entermultiple trouble reports(MTRs) using TAFI andreceive a satisfactoryresponse.SatisfiedTAFI was used to enter 12 multipletrouble reports for accountsexperiencing problems on multiplelines using the new method detailedin the CLEC TAFI User Guide (Issue 2,April 2000). Of the 12 multiple troublereport transactions attempted, sixwere submitted successfully while sixwere unsuccessful. For each of the sixunsuccessful transactions, KCI wasable to create the "parent" ticket butunable to link the "child" report to theparent. LMOS errors and othermessaging indicating that no linksexisted were received. As a result, KCIissued Exception 50.In response to this exception, BLSincluded additional information inIssue 3 of the CLEC TAFI User Guide,dated May 2000, stating under whatcircumstances the MTR function isnot appropriate.A review of the CLEC TAFI User Guide(Issue 3, May 2000) revealed thatadditional language explaining thesepoints to the TAFI user has beenincorporated as indicated by BLS.See Exception 50 for additionalinformation on this issue. ThisMarch 20, 2001VI-B-9Published by KPMG Consulting, Inc. Confidential. For BellSouth, KCI, and Georgia Public Service Commission use.

BellSouth – GeorgiaTest CrossReferenceSTP Final ReportEvaluation CriteriaResultCommentsexception is closed.March 20, 2001VI-B-10Published by KPMG Consulting, Inc. Confidential. For BellSouth, KCI, and Georgia Public Service Commission use.

BellSouth – GeorgiaTest CrossReferenceSTP Final ReportEvaluation CriteriaResultCommentsM&R-12-1-6The user is able to enterand retrieve troublereports from the queuein TAFI and receive asatisfactory response.SatisfiedTAFI was used to enter 14 troublereports into the queue, 11 manuallyand three automatically. 14 reportswere successfully removed from thequeue, 12 manually and twoautomatically.M&R-12-1-7The user is able toexecute supervisorfunctions within TAFIand receive asatisfactory response.SatisfiedTAFI was used to execute supervisorfunctions such as reviewing andreassigning queued reports on fourlines and responded as expected foreach line.M&R-12-1-8The user is able to closea trouble report usingTAFI and receive asatisfactory response.SatisfiedTAFI was used to close 35 troubletickets and responded as expected 35times.2M&R-12-1-9The user is able tocancel a trouble reportusing TAFI and receivea satisfactory response.SatisfiedTAFI was used to cancel 32 troubletickets and responded as expected 32times.M&R-12-1-10The user is able toconduct a port andloop-port test (alsoknown as MechanizedLoop Tests (MLT)) usingTAFI and receive asatisfactory response.SatisfiedTAFI was used to conduct 18Mechanized Loop Tests (MLT) andresponded as expected 18 times.MLTs are not run for subsequentreports, which is indicated on page194 of the User Guide (Issue 3 May,2000).M&R-12-1-11The user is able to viewport and loop-port test(MLT) results usingTAFI and receive asatisfactory response.SatisfiedTAFI was used to view 18 MLT testresults and responded as expected 18times.M&R-12-1-12The user is able toretrieve a LMOS recentstatus report and receivea satisfactory response.SatisfiedTAFI was used to retrieve eight LMOSrecent status reports and retrievedeight reports as expected.KCI was unable to close six additional trouble tickets using the TAFI application during functionaltesting. Instead, these tickets were manually closed by calling the BellSouth Resale Maintenance Center(BRMC). As described in Exception 10 (M&R 1-1-8), which is now closed, TAFI cannot be used tocancel/close subsequent trouble reports that were in dispatched status. Each of these six trouble ticketswere subsequent trouble reports in dispatched status.2March 20, 2001VI-B-11P

For BellSouth, KCI, and Georgia Public Service Commission use. Scenario Number Scenario Description vertical feature . 3 Business customer with resale POTS line is experiencing problems with their call forwarding and three - way calling vertical features. 4 Business customer with r esale POTS line is hearing other conversations on their line. .