Transcription
rdefineandprovidea sampleusein an itprocess,canbe forthe
In this guidebook, the term "audit" specificallyrefers to an SQAtechnique that is used to examine the conformance of adevelopment process to procedures and the conformance of productsto standards. An SQAaudit also can examine the conformance ofthe actual status of the development activity to the reportedstatus.The term "audit" is used to describe a number ofadditional software activities;however, due to their differentpurpose and focus, they are not addressed in this guidebook. Forexample, the Functional Configuration Audit (FCA)and PhysicalConfiguration Audit (PCA)are configuration management(CM)activities.Quality (Engineering) Audits and Safety Audits aretechnical activitiesthat evaluate a software product againstQuality Engineering and Safety requirements. These types ofaudits are not covered in this guidebook.II.CONCEPTSANDDEFINITIONSAn SQAaudit is an activity that is performed to determine theadherence to, and adequacy of, a project's established softwaredevelopment standards and procedures and the effectiveness oftheir implementation. As used in this guidebook, the mainobjective of an SQAaudit is to determine the adherence toestablished standards and procedures; checking their adequacy oreffectiveness is a secondary objective that usually is notrequested of an auditor.In the NASASoftware Assurance Guidebook, standards are definedas "the established criteriato which software products arecompared." Software standards include documentation standards,design standards, and coding standards. In that guidebook,procedures are defined as the "established criteriato which thedevelopment and control processes are compared." Procedures,then, are the step-by-step directions that are to be followed toaccomplish somedevelopment or control process; for example, CMor nonconformancereporting and corrective action (NRCA). Inother words, standards and procedures are requirements forsoftware management,engineering, and assurance; SQAauditsverify their existence and assess a project's compliance withthem.SQAaudits also can compare the actual status of a product withreported status.Status auditing is most effective if there areobjective and consistent criteriafor evaluating the level ofproduct completeness. For example, Unit Development Folders(UDFs)have a cover sheet for recording the progress of a unitthrough its development stages; the folder contains the actualproduct. If a project uses UDFs, then an audit can compare theactual product to the cover sheet and to the progress report.The actual processes and products examined by an audit will varydepending on the objective of the audit.The objective of theaudit can vary, and is determined by the organization that calledfor the audit.A general audit provides a comprehensiveoverview, while a limited audit might be an examination ofcertain procedures, such as CM, or a check on a certainrequirement, such as "Are coding standards being followed?"
An audit may be described as internal or external, depending onthe organization of origin of the auditor(s).An internal auditis an audit conducted by the SQAstaff of the software developer.Internal audits are intended to be preventative in nature; todetect problems before they becomemajor.An external audit is one performed by an independent auditor whois outside of the developing organization.External audits aremost often requested by the acquiring organization, as a meansofobtaining an independent opinion about the work in progress.External audits tend to be more comprehensive in nature thaninternal audits, and usually encompassa broad area of thedevelopment activity.Such audits usually are requested becausethe acquirer is uncertain of the effectiveness of the internalprogram or because of lack of information and fears about thequality of performance on the part of the developer. Anadvantage of an external audit is that the auditor may be moreobjective about a project than an internal auditor; however, anexternal auditor must spend more time learning about the projectand its development process.III.CONDUCTINGAN SQAAUDITAn SQAaudit has four phases: planning and preparation, the sitevisit,reporting, and follow-up.During the planning andpreparation phase, the auditor gains an understanding of theproject.Based on the scope of the audit, the auditor determinesthe specific questions that need to be answered, as well as thepersons to be interviewed and the records and products to beexamined to answer the questions. The interviews are conducted,and records and products are examined during the site visit.Thereporting phase consists of the exit debriefing of the auditedproject, the preparation of a written report on the audit, andclarifyingissues and providing related information as needed.Follow-up is done by the project, as the problems anddeficiencies found in the audit are remedied. Follow-up mayinclude reauditing to assess the adequacy of the remedies.The activitiesconducted during the phases vary depending on thelife cycle phase of the project being audited and the scope ofthe audit.The activitiesalso vary depending on whether theaudit is external or internal; an external audit requires moreextensive preparation and should examine a more comprehensivesample of material than an internal audit.Each of the four phases of an audit is described in the followingsections.The activitiesof each phase are described as if ageneral, external audit is to be done since this results in thegreatest detail.Someof the activitiesmay be superfluous to aninternal SQAaudit and may be omitted.A.Audit Planning and PreparationA general SQAaudit should be planned carefully to examine all ofthe software engineering, management,and assurance processes andall of their products. Software managementprocesses includestatus reporting and CM. Engineering processes include analysis,design, and code. Assurance processes include verificationand
validation(V&V)and NRCA. Products include documents and code.If the scope of the audit is more limited, then planning will bewithin the defined limits.A limited audit might examine onlyone of the processes or a limited set of products. Activitiesduring the planning and preparation phase are similar for allaudits, but preparation for a limited audit is focused on theidentified process or product.As a first step, the auditor should understand the objective ofthe software development project and what products are to beproduced. The auditor needs to know what the contract requiresin the way of deliverable software and documentation, and what,if any, requirements exist for management,engineering, andassurance practices.One source of this information may be thestatement of work and other contract documents. Once it is clearwhat is being developed and what the contract requires, theauditor should review managementdocumentation, such as thesoftware management,development, and assurance plans tounderstand the processes that will be used to develop and controlthe products. Then the developer's standards and proceduresmanual should be reviewed to determine the quality standards andthe detailed procedures planned to be applied to the software andthe development process. From this background information, theauditor should be able to understand the developer's softwaredevelopment process.The auditor also should review somerecent status reports fromthe developer. These reports will furnish information on thestage of completeness of products and may contain information asto problem areas.After background familiarizationand a look at project status,the auditor should define the areas that will require the mostcareful and detailed attention, i.e., the processes or productsthat seemto be in somedifficultyor whose status is in doubt.These areas may be identified by the status reports, discussionswith the acquirer of the software (if it is the acquirer who hasrequested the audit), review of nonconformancereports, and theresults of previous audits.Once the auditor understands the project and has identified theareas of concentration, he/she should develop a checklist.Achecklist is a list of items to be examined and questions to beasked. Each checklist should be tailored for the specificproject being audited and its life cycle phase and should reflectthe scope of the audit.A more comprehensive and less detailedchecklist is required for a general audit; a limited auditrequires a checklist that is more detailed in specific areas.Guidance on preparing a checklist is given in Chapter VI. Achecklist is intended to provide the auditor with a "road map"during the site visit.It must be complete, so that the auditorcan know that sufficientinformation has been gathered if all ofthe checklist items are completed. The checklist questions helpdefine the individuals with whomthe auditor wishes an interviewand the types of records that the auditor will examine.The auditor should schedule the site visitto the project through
its assurance staff or other suitable contact after thepreparation is done and the checklist prepared. During thiscontact with the project, the auditor should specify the intentof the audit, the records to be examined, and which people theauditor wishes to interview.People to be interviewed willinclude managers, selected developers, CMstaff, assurance staff,and testers.Copies of the checklist may be furnished toincrease the project's understanding. The project should beprepared to provide the auditor with a convenient working areathat includes normal office facilities,access to all productsand records, and interviews with the identified individuals.B.The Site VisitThe purpose of the audit site visit is to collect the datanecessary to assess that the required products are beingproduced, the degree to which they conform to applicablestandards, how well procedures are being followed, and that thereported status corresponds to the actual status.The audit isintended to uncover any significantdeviation from standards,procedures, or reported status so that corrective action can betaken. The auditor uses two basic techniques: interviews withproject staff and examination of documentation and records.The site visit should begin with an entrance briefing, involvingthe auditor and key project staff.During this briefing, theauditor should describe the focus of the audit, and identify theinterviews to be conducted and the records to be examined. Theentrance briefing may also be used by the project to brief theauditor on its processes, key staff members,and current status.Time for questions and answers should be included. The auditoralso should assure the project that an exit interview will beheld where the auditor will present preliminary findings to theproject and the project may provide any additional information tothe auditor.This preliminary exchange of information cansignificantlyhelp to allay the fears of the project and tosmooth the course of the site visit.After the entrance briefing, the auditor should proceed with thegathering of information.It is useful to begin the informationgathering process with interviews, during which the auditor triesto understand the realitiesbehind the documentedplans andprocedures. The auditor should learn which individuals carry outa procedure, approve a change or fix, keep project records, etc.Each individual should be asked to describe his/her perceptionsof and interactions with the process. The auditor should takenotes, annotate or develop procedural flow diagrams, askquestions to clarify,and makeit his/her objective to clearlyunderstand the process. In particular,the auditor should bealert for indications of shortcuts or abbreviations to theprocedure. During interviews, the auditor must rememberthatdata are being gathered, and that conclusions should wait untilall of the facts are in. This provides a clearer understandingof the actual processes used on the project and easescommunications with the staff.The checklist developed duringthe preparation phase is used to guide the discussions during theinterview.
Once the auditor is sure that the processes and procedures areunderstood as they really exist, he/she should begin examiningthe tangible parts of the project: its products and records.Products consist of requirements and design documentation,including unit development folders, user manuals, code, etc.Records consist of memorandaand forms that document the eventsin the life of a product. They comefrom CM, NRCA,and V&V,amongothers.i.Records ExaminationThe auditor examines records to see if a procedure is beingcorrectly followed.Record examination is described below interms of the principal processes that SQAaudits examine: CM,NRCA,and V&V. Similar activitieswould be used in theexamination of other sets of records. CMAuditDuring an audit of CM, the auditor should look at the completechange control cycle, beginning with the initialprocessing of achange request; through analysis of impact and dispositioning;design, code, and testing; updating of documentation; submissionof the modified products to the library;and closure of thechange request. Records to be examined include the changerequests as processed by the ChangeControl Board, the workauthorizing documents issued as a result of approved changes, thecode and documentation products that are intended to reflect theapproved changes, and the program library records that capturethe changes to code and data. Throughout the audit, the auditorshould be alert for and document any evidence of unauthorizedchanges.The records should show the authorization of each change, theproduct(s) to be changed, and the version numbers of the changedproduct. Muchof the auditor's attention should be devoted tothe Program Library or equivalent, since this is where thevarious versions of products and the change documents controllingthose versions are stored. The auditor should check the productsin the library to ensure that documentation is up-to-date withcode changes. The auditor should check the version numbering andidentificationschemes, and the control documents. The recordsshould demonstrate that there are adequate security measures inplace to prevent loss and unauthorized changes. The auditorshould verify that every item of code and documentation in theprogram library was properly received. NRCAAuditWhenauditing the NRCAsystem, the auditor should look at thecomplete cycle. The auditor should review the nonconformancereports that are filed, to assure that they are completely andcorrectly filled out. The disposition process and board actionsshould be recorded, usually on the sameform. Thenonconformancesthat result in product changes should be trackedto the product, and evidence should be gathered that changes are
made, tested or reviewed, and approvals for issuance are granted.The NRCAprocedures will parallel those used in CM, and can beaudited in muchthe sameway, especially when it comesto theprogram library.In both cases (CMand NRCA),the auditor shouldpay particular attention to corrected products to assure thatthey stillsatisfy requirements and standards. V&VAuditAn audit of V&Vprocedures should include a check of theverificationmatrix or equivalent, to assure that everyrequirement has a test and every test checks a requirement. Testplans should be adequate, specifying the test environment, testprocedures, and the expected results for each test.Testprocedures should be clear and detailed.Test plans andprocedures should be reviewed and approved.The auditor should verify from SQArecords that test procedureswere followed and that all nonconformancesobserved duringtesting are recorded in the NRCAsystem. In addition to testing,the auditor should assess other methods of V&V, if used. Forexample, if inspections or another form of peer reviews are usedto find problems, the auditor should verify that the records ofthe review show that they were done and that corrections andchanges agreed to in the review are madein the product.2.Product ExaminationThe intent of examination of products is two-fold: to see ifstandards are being followed, and to see if status is accuratelyreported. Documentsare measuredagainst documentationrequirements to makesure that all required documents exist, andagainst documentation standards to ensure that they have thecorrect content and style.The auditor must read enough of thedocuments to form an opinion on the above; that is, the auditormust be able to determine that a document presented as showingthe design indeed contains design information.On the otherhand, the auditor is not responsible for the technicalcorrectness of the documents and should not spend time trying toascertain if the documents are correct.Code also is examined to determine if it meets standards. Codestandards are likely to include rules for internal documentation,size of modules, styling formats, and other such items that theauditor can verify.Rules for coding constructs or variablenaming conventions are more difficultto verify.If the projecthas a code standards checker, the auditor may run it on somecode. If the standards checker is to be run at a certain step inthe development process, or if peer reviews are used to verifycoding standards, the auditor must have access to those records.Products also are examined to compare their status with thatreported. Documentsreported as complete, for example, shouldcontain all of the sections given in the table of contents (whichmay be prescribed by a documentation standard), should be signedby the approving authorities,and should contain few, if any, ToBe-Determined (TBDs)items. Code implementation usually goes
through the steps of detailed design, code, peer review, and unittest.A module that is reported as complete should have gonethrough all of the above steps, should meet the coding standards,and should have whatever approvals are required.The UnitDevelopment Folder or equivalent should contain all of theevidence to look at status of coding.3.SamplingDuring the process of checking records and products, the auditorusually cannot examine each and every item; therefore, somesampling process must be used. The auditor must decide on samplesizes that can be accommodatedin the site visit.The samplesizes must be balanced between completeness of coverage (someitems from each product or set of records) and depth of coverage(numberof items from a specific product or set of records) . Ifthe focus of the audit is limited, the sample size can be largerfor the specific product or processes that are to be covered. Indeciding on sample sizes, the auditor must allow time to followup in more depth in areas where the initialsample indicatesproblems. The specific products or records to be included in thesample should be chosen by some "randomizing" method, and theproject staff should not be informed in advance which items willbe examined and which will not.C.Audit ReportingOnce the interviews and record examination have been completed,initialresults should be shared with the staff of the auditedproject during an exit interview.The exit interview provides anopportunity to clear up misunderstandings and allows projectstaff to present any information that they feel the auditorfailed to consider.In addition, project staff learn immediatelyabout the problems that have been found and can begin makingplans to correct them.After adjusting the initialresults to reflect the informationgathered in the exit interview, the auditor prepares a writtenfinal report.The report should be organized to highlight themost significantresults, addressing both problems andcommendations, and should include a general narrative of theaudit.An example table of contents for an audit report is shownin Appendix A. The audit report should be addressed to themanagementofficialwho arranged for the audit, if the audit isexternal; or directed as required by procedures, if internal.The objective of the audit report is to present a clear pictureof the status of a development activity or a facet of theactivity to project management. The report must be clear,objective, and factual.In somecases, the auditor will findthat, while procedures are being followed or standards are beingmet, the procedures or standards are not effective in producing aquality product. It is the responsibilityof the auditor to notethe specific problems caused by the procedure and/or standard andinclude them in the report.In general, however, problems thatthe auditor identifiesshould be related to project orcontractually-requiredprocedures and standards; the auditor's
opinion of their desirabilityshould not affectevaluation of the adherence to them.D.his/herFollow-upWhile the auditor's role is essentially finished after producingthe audit report, actions to resolve deficiencies identified inthat report must be taken by project management. Problems thatare feasible and reasonable to correct should be converted toaction items and assigned to appropriate individuals.Arationale should be developed for those that are not to becorrected.It is the responsibilityof the developers to improvetheir processes in response to deficiencies identified by theaudit.The changes should be tracked to ensure they occur andare effective and the closure of action items should bedocumented. In manycases, the best way to determine if theproblems have been solved is through a follow-up audit.IV.SQAAUDITSCHEDULINGA.Routine SchedulingInternal SQAaudits should be scheduled frequently enough toidentify potential problems so that no surprises develop forproject management. They should be scheduled routinely duringthe life cycle, particularlyaround life cycle phase transitions.The most effective internal audit programs schedule frequentaudits of small areas of project activity.Frequent auditing,combined with other SQAmonitoring activities,would assureproject managementthat the actual status of the project isknown, vis-a-vis standards, procedures, and schedules.External audits require more planning and interview time, but arescheduled muchless frequently.The most important time for anexternal SQAaudit is at the start of the implementation phase.This audit assures that the developer's standards and proceduresare implemented in a manner appropriate for the project and thatthey are being followed.A second important time in a project'slife cycle is the beginning of system integration.An externalaudit helps to assure that the software is ready for integration,that test plans and procedures are in place, and that proceduresfor control of the software are not short-circuited.Projectsthat are in trouble or have no internal audit function shouldhave more frequent external audits.Another factor to consider in the scheduling of audits, eitherinternal or external, is the results of previous audits.EachSQAaudit should include a review of the results and action itemsfrom any previous audits to confirm closure.If there were anumber of problems and action items, audits should be scheduledmore frequently.Projects that follow their procedures, meettheir standards, and are accurate in reporting schedule andstatus need less frequent auditing.B.SQAAudits in Responseto Warning SignsSomeprojects may show indicationsof problems in the development
process. Whenwarning signs appear, the acquirer should considerconducting an external audit as part of its response. The samewarning signs can be used by the software provider to step up orevaluate the effectiveness of its internal audit program.The audit program should be intensifiedany of the following signs:ifthe project exhibitsFrequent schedule/milestone changes. Inconsistency of the developer's organizational structurewith original plans or apparent inconsistency with the structureor functionalityof the products to be produced. Unexplained fluctuation of project staffover-staffing comparedto estimates.level or under- or Increases in the number of TBDitems and action itemswithout adequate progress in solutions. The inabilityor unwillingness of the developer to provideadequate and accurate information on project status, schedules,and plans. laterContinual delay of scheduled software system capabilitiesreleases/versions.to Unreasonable numbers of nonconformancesor change requests;for example, a large number unresolved, or a sudden increase innumbers. An "unreasonable number" might be a suspiciously smallamount of nonconformancesfor a complex system.There may be other indications that are apparent to projectmanagementin specific cases. An experienced project manager'sintuitionthat something may be wrong is a warning sign thatshould be heeded. An external audit is a cost effective way foran acquirer to ascertain the real product status and realprocesses being used by a developer; developer managementshouldhave an ongoing audit program to assure that no surprises are instore for them.C.Announcing AuditsAdequate notificationof audits should be provided to thedevelopers for a number of reasons. Unannounced(surprise)audits are disruptive and demoralizing to the development staffand should be avoided. The intent of an audit program should beto help promote conformance with standards and procedures and thereporting of accurate status, not to "catch in the act" those"guilty" of violations.An announcedschedule of audits allowsproper preparation in terms of having required documentationavailable and being prepared to answer the auditor's questions.V. SQAAUDITSDURINGTHESOFTWARELIFE CYCLEA.Software Concept and InitiationDuring the concept and initiationPhasephase, the software concept is
lopan
sshouldauditingactua
The NASA Software Assurance Guidebook classifies the software quality assurance (SQA) audit as a fundamental quality assurance technique. It is the intent of this guidebook to further define audits, describe the audit process, and provide a sample checklist that can be tailored for use in an audit. The guideb
