Transcription
APRINCE2 and Test ManagementPRINCE2 is a project management method that is generally applicable toprojects. The name PRINCE stands for PRojects IN Controlled Environments. The PRINCE2 method was developed by the Central Computer andTelecommunications Agency (CCTA) as a process-based method that anyone can use. The CCTA is part of the British government and sets out“best practice” work methods. The method is based on practical experience with various management methods and pays particular attention tochanges in environmental factors that might influence the success of a project.PRINCE2 is the standard in many organisations in Europe for managing(ICT) projects.The test management approach described in this book bears many similarities to the PRINCE2 approach. Test management can therefore function perfectly well in an ICT project driven by PRINCE2. Within the practice of project execution, the two methods can effectively reinforce eachother.A.1 Similarities Between PRINCE2 and TestManagementWe describe briefly here how a number of main principles of PRINCE2 compare with the test management approach described in this book. We alsodiscuss how test management fits into a PRINCE2 project.The PRINCE2 principles are summarised below, followed by a descriptionof the way in which test management corresponds with these.1. The instigation of the project is a specified and measurable business case.Evaluation of this business case in the course of the project can lead tosuspension of the project.In the test management approach, the test process is set up and controlled by measures based on a risk analysis and on the requirements.
256A PRINCE2 and Test ManagementBoth of these factors take the business case as the starting point. Testmanagement can therefore offer considerable added value to the evaluation of the business case during the project. The test processes are set upin such a way that they can generate relevant and targeted managementinformation, on progress and quality of the end product, among otherthings. With the aid of this information, the client (the business caseowner) can determine whether the business case of the project is stillvalid. With an objective and quantified insight into the status of the endproduct, an estimate can be made, for example, of how much more timeand money it will cost to realise an acceptable level of the informationsystem. If the product is delivered too late, or if it appears to be tooexpensive, this may be reason for the client to halt the project.2. The business representation in a project plays a vital role that is not without duties.A fixed part of the test management approach is carrying out a stakeholder analysis at the start of the project. In the analysis, the interestsof all the parties involved in the project are surveyed. Subsequently inthe test strategy, this is explicitly translated into the tasks, responsibilities and authorities that the parties are to be assigned to in the projectexecution. The results of this participation are established in as concretea form as possible. In this way, the input of the business into the testingis secured, just as it is in PRINCE2.3. Risk management is one of the core processes within PRINCE2.Risk management also takes a central place in the test management philosophy: as with PRINCE2, a sound risk analysis is carried out at thestart of a test project. The risk approach of test management forms thebasis of the substantive management of the test project. And it mainlyconcerns the product risks: which risks are directly related to the information system.Besides technical risks, this also concerns business-related risks. Withthese, the product risk analysis forms the essential input for the teststrategy. The test manager monitors these product risks through progressmanagement. He must also monitor the project risks that relate to theproject itself.4. The planning process is product oriented and uses a product breakdownstructure, among other things.The product-oriented planning of the test management approach alignsperfectly with this. Planning on the basis of products is also an important principle of test management, since it allows concrete results to bedelivered at an early stage. Testing focuses first on those products thatare the most important to the client as regards risk and added value.With these priorities, test management is able to create “benefit-based”reports. These provide the client with an objective picture of the qualityof the products.
A.3 More PRINCE22575. Plans are iterative in nature.The test management approach also develops plans in an iterative way.The specific approach that is used within test management is based on“evolutionary planning” (Gilb, 1988). With this, learning experiencesduring the test project are used towards the most realistic planning oftime possible for the remainder of the course of the test project.6. Explicit separation of project initiation (in PRINCE2 the IP phase) andproject management (in PRINCE2 the BF phase).Test management makes a clear distinction between activities during thepreparation of a test project and those during the execution of it. See theTest Management Model.A.2 Position of a Test Project Within a PRINCE2ProjectPRINCE2 projects have their own structure, based on a client–vendor relationship. An important premise of this is that there is always a client whowill specify the required project result, who will use the project result, andwho will most probably pay for the project. This makes it clear to the testorganisation where the responsibilities lie, who can be called to account onwhich points, and what the escalation paths are.Test management in a PRINCE2 project will take the place of whatin PRINCE2 terms is called “Team Management”. Within that team (orproject) the test organisation can be set up in accordance with the structurein the test management approach. The exchange of information with the restof the project then runs according to the PRINCE2 approach. The teammanager (in this case the test manager) will then make concrete agreementswith the client or programme manager concerning the results to be delivered.A.3 More PRINCE2In various chapters of this book where the phases of the Test ManagementModel are discussed, references are made here and there to PRINCE2.If you wish to know more about PRINCE2, you will find information in books(CCTA, 2002) or on the official PRINCE2 website: www.ogc.gov.uk/prince.
BChecklist of Product RisksThis checklist contains a large collection of questions and answers that canhelp the test manager in determining the product risks concerning the implementation of an information system.The questions are divided into the following categories, based on the mostusual groups of stakeholders (see Chapter 5, Section 5.2.1): end users;marketing;support department (such as a helpdesk);IT department;internal security.An example is provided per product risk together with the quality attributewithin ISO 9126 that best applies to the relevant risk.The last column is meant to indicate the priority per product risk underthe MoSCoW rules (i.e. Must test, Should test, Could test or Won’t test).For example:
260B Checklist of Product RisksB.1 Checklist of Product Risks – End UsersNo.1.1Product riskindicationExampleShould the in-Should it be checkedput be checkedwhether all the inputfields comply with theprescribed ''type''by the information system?Quality attribute(1809126)Accuracy(numeric, alphanumerle, date), lengthand style (bold, underlined, italics, etc.)?Can correct values beentered, and are incorrect values rejected?(Syntax)1.21.3Should the cor-Should it be checkedrelation between inputwhether the prescribedcorrelation between in-fields in the information sys-put fields is correctand complete? (Se-tem bechecked?mantic)Are the usersexperiencedHow easily can the endusers work with a newwith the userversion of the information system?interface of nce(MoSCoW)
B.1 Checklist of Product Risks – End Users1.4Are the errormessages andother messagesdisplayed bythe informationsystem clear?Do the error messagesUnderstandabilityand other displayedmessages provide theinformation needed bythe users to understand which fault hasoccurred, or which action has been carriedout arul/or which action they themselvesshould take?1.5Are the helpfacilities clear?Mter starting up a helpfacility, can the enduser operate it easily?Learnability1.6Is the menustructure clear?Is it clear to the endusers how they cannavigate through theinformation systemUnderstandabilityand which functionsthey can performwith it?1.7Are the screensDo the end usersclear and legible?quickly understandhow and where theyAttractivenessshould carry out particular actions? Do thescreens have a layoutthat appeals to the endusers?1.8Are the gener-Is the layout of theated overviewsclear?overviews clear to theend users?Understandability261
2621.9B Checklist of Product RisksDo the overviews containDo the overviews provide the informationthe required information?that the end users require to carry out theirSuitabilitytasks?1.10Is the usermanual clear?Does the user manualprovide the users withLea.rnabilityadequate and useful information?1.11Are there linksWhen a car insurancewith other systems?policy is concluded, isnotification autom.atically sent to the gov-Interoperabilityernment departmentfor road transport?1.12Are there al-For example, for call-ternatives forprocessing databack notes. If no alter-if the information system isnot available?1.13Should the system be avail-able all day?Recoverabilitynatives are available, itmust be possible to recover the systemquicklyNo down time shouldoccur in a system thatMatwityis critical to the company's operation24 hours a day1.14Should the information sys-If text in the information system has to betem be available in varioustranslated, mistakesmay be created. Differ-languages?ent languages can havedifferent address forms,Operabilityfor example1.15Does peakload of the in-formation systern. occurduring theday?At peak use, the response time of the information system maybe longerTime behaviour
B.1 Checklist of Product Risks – End Users1.16temcomplyH a customer is waiting at the cotwterwhile the system is be-Should the information sys-1.171.18with a particu-ing used, it is impor-lar responsetime?tant that the waitingDo many usersuse the systemWith simultaneous useof data, problems mayTime behaviourtime is not too longsim.ultane-arise concerning con-ously?sistency of the databaseCan the usersadapt the system to suitFor example, the usersthemselves adapt thelayout of reports ortheir 19formation sys-Is the system suited tothe daily tasks of thetem fit withinend users?Does the in-Suitabilityexisting stan-dards andprocedures?1.20Is the information system in-Financial details oftenrequire a high degreetended forof accuxa.cyAccuracyprocessing financial details?1.21Are the detailsin the informa-tion systemconfidential?With some informationsystems, the data mustSecuritybe UBed in confidence,e.g. bank account details1.22Does the in-For example, mission-formation sys-critical systems, suchtem influencethe phy.Ucalas heart monitoringenvironment?equipmentutili-Maturity263
2641.23B Checklist of Product RisksDoes the in-With a secondary sys-formation sys-tem., an occasional dis-tem concern aruption is probably lessprimary or aserious than with asecondary system for theprimary systemMaturitybusiness prooess?1.24Axe the procedures for thebusiness process stable?1.25Does the information system affect1.26If these often change,there is a risk that theinformation system isUI18uitable for thechanged processSuitabilityFor example, a newquotation system mayCo-existenceinfluence the printingother departments?channelsHow manyWith many users, theResourcepeople use theinformation systemsationsystem?utili-will require more processing capacity1.27How many difcess the sys-These users willprobably use the system in different ways.tem?Have allowances beenferent users ac-Suitabilitymade for this in theuser interface? For example, diverse menustructures and screens1.28Does the in-With online systems,formation sys-higher demands aremade on the user inter-tem have a.n1.29online or abatch function?faceIs the entireDoes a breakdown inthe information systembusiness process covered bythe informationsystem?affect the whole of thecompany process?OperabilityRecoverability
B.1 Checklist of Product Risks – End Users1.301.311.321.331.34Is the genera-Management uses thetion of management infer-information for takingimportant decisionsmation theand will require a highmost importantdegree of accuracy infunction of theinformationsystem?this informationAre the usersexperienced inIf the users previouslydid everything manu-the use of in-ally, more effort will beformation sys-tems?required in trainingand assisting themHow much ofWith many changes,the existingfunctionalityfrom the previ-the chances of errorsare increased and thedegree of end usersous version hasacceptance may be re-been changed?ducedShould the information sys-Companies are obligedFunctionalityto retain transactioncompliancetem store his-details for a. certaintorical data?number of yearsDoes the information sys-For example, calculation of premiums in antem makecomplex calcu-insurance bilityAccuracy265
2661.35B Checklist of Product RisksCan other information systems influencethe operationof the system?If another system goesdown, can the targetsystem continue operating?Are other systemsCo-existencegiven priority as re-gards processing if several systems are activesimultaneously?1.361.37Is the processing of inputcould be done later iftime-critical?it is not time-criticalIs the status ofDoes the informationsystem indicate that itthe informationsystem alwaysclear?Further processingTime behaviourUnderstandabilityis busy processing?Is a message displayedwhen a transaction iscompleted or if a timeout occurs?1.38Is the information system replacing an existing system?The new system shouldReplaceabilitybe capable of supplyingthe existing functionality. Regression testingis important in that.,.1.39Should it beVarying the input in-possible to operate the fnnc-strument for repetitivetions of the informationsystem in various ways?repetitive strain injuryfunctions can prevent(RBI)Operability
B.2 Checklist of Product Risks – Marketing267B.2 Checklist of Product Risks – MarketingNo.2.1Product riskindicationExampleIs the infor-For example, sales ofmation systeminsurance via theused for sellingInternetQuality attribute(1809126)Operabilityproducts?2.2Should the in-Internet applicationsformation sys-can run on various operating systems, e.g.tern run onvarious plat-2.3AdaptabilityNetsca.pe and Internetforms?ExplorerDoes the in-For example, PersonalData Protection ActFunctionalitycomplianceShould clientsWhen a system is soldInstallabilitybe able to in-to customers, it goesformation systern complywith all thelegal privacyregulations?2.42.5stall the in-to a variety of end us-formation sys-ers with varying levelsternthemselves?of knowledge and ex-Is there a geographical listribution ofthe information system?perienceIs the system usedonly locally or, for example, worldwide?lnteroperabilityImportance(MoSCoW)
268B Checklist of Product RisksB.3 Checklist of Product Risks – Support Department
B.4 Checklist of Product Risks – IT Department269B.4 Checklist of Product Risks – IT DepartmentNo.4.14.2Product riskindicationExampleIs aThis description aids theSUIIUD.8r-Quality attribute(1809126)rised functiondescriptionmaintenance departmentwith change imple-available?mentationAre back-upscreated?If the system goesAnalysabilityFault tolerancedown, can the backupbe reinstalled?4.3In the event ofbreakdown,should the information sys-Is there a maximumtime during which thesystem is allowed tobe unavailable?Recoverabilitytem be restoredwithin a cer-tain time?4.4Is there a contingency plan ifthe hardwarethat runs theinformationsystem fails?Is there a shadow system available thatcan take over thetasks?Fault tolerance4.5Has the information systemUsing technology thatis new to the organi-Maturitybeen createdwith new technology?sation can be thecause of a lot of errorsImportance(MoSCoW)
2704.64.7B Checklist of Product RisksHow do weCan the system revertdeal with inter-Recover abilityrupted transac-to the situation as itwas before the trans-tiona?action was started?Axe there contingency plansHave we consideredan emergency seenario?Recover abilityIs the information systemCan the users config-Adaptabilitysensitive to set-their own preferencesfor when theinformation systern goes down?4.84.9ure the system totings on end(e.g. resolution andusers' PCs?screen size)?Are data fromother informa-Are allowances madefor synchronisation oftion systemsthe data betweenstored in thisboth systems?Intero abilitysystem?4.10Should the in-The existing infr&-formation sys-structure may slowtern operatedown when anotherwithin the existing infra-system is added on. Itis also possible thatstructure?there is insufficientdisk space available inthe existing infra-structureCo-existence
B.4 Checklist of Product Risks – IT Department4.11Axe changes tothe infrastructure planned?If changes to the infrastructure will affectthe system, theyAdaptabilityshould be accommo-dated easily withinthe system4.124.13Axe create,read, updateand delete(CRUD) functions implemented for thevarious entities?If these functions arenot present for all theentities, some testAre correla-If a customer is dis-tions betweenplayed showing ac-entitieschecked uponthe removal ofan entity?4.144.15cases cannot be carried outcounts attached, itto delete the customerwithout administeringthese accountsDo these system components fit within thethird partiesused?information system,or are there, for example, big differencesin the layout of data?Is there a fallback scenario ifIs an emergency planavailable?tation of theinformationsystem fails?Accuracyshould not be possibleAre systemcomponents ofthe i.mplemen-TestabilitySuitabilityFault tolerance271
272B Checklist of Product RisksB.5 Checklist of Product Risks – Internal Security
B.5 Checklist of Product Risks – Internal Security5.2Axe passwordsused?Axe there any rulesconcerning thesepasswords: length,Securitychange frequency,etc.?5.3Are the accessattempts moni-tared?5.45.5Is a log kept thatregisters who is trying to access thesystem and whetherthis person isauthorised or not?SecurityAre internalDo the developersMaintainabilitydevelopmentmaintain these stan-compliancestaodards used?dards?Are there linksMany links meanthat there are moreopportunities forto outside ofthe organisation?Securityhackers to enter thesystem5.6Are transa.c-Locking the infor-tions locked?mation makes itSecuritymore difficult to useit for illegitimatepurposes5.7Is an audit trailWith many fman-required for thecial information sys-information sys-tems, an audit trailtern?is a legal require-mentAnalysability273
CTemplate for Risk- & Requirement-BasedTestingThe template below provides an overview of the information that is importantwithin risk- and requirement-based testing (RRBT).This table can be used in decision making.If, for example, a requirement changes, the test manager can see to whichproduct risk it is linked. He can also see the consequences it will have for thetest condition(s). Are any adjustments to the test conditions required?The table can also form the basis of the reports. See Chapter 12.The testers can link an issue to the relevant product risk during theexecution of the test via the test condition. With this, a tester can determinethe initial priority of an issue.The case from Chapter 2 has been used to give an idea of the way inwhich this table is completed.
276C Template for Risk- & Requirement-Based lderRequirementTestcondition1Customercannotperform atran81IDtionFunctionalityMust testEndCustomer ableto perform atransaction viaown bankIt is possible to performatransactionat ownbank, usingan existingpin codeuserIt is notpossible toperform atransactionat ownbank whenwring aninvalid pincodeCustomer ableto perform atransaction viaother bankCustomer canchoose from setamountsCustomer canselectamount/choiceof banknotes
DQuality Attributes According to ISO9126Quality attributeFunctionalityDescriptionThe capability of the software product toprovide functions which meet stated andimplied needs when the software is usedunder specified conditionsSuitabilityThe capability of the software product to providean appropriate set of functions for specified tasksand user objectivesAccuracyThe capability of the software product to providethe right or agreed results or effects with theneeded degree of precisionInteroperabilityThe capability of the software product to interactwith one or more specified systemsSecurityThe capability of the software product to protectinformation and data so that unauthorisedpersons or systems cannot read or modify themand authorised persons or systems are not deniedaccess to themFunctionality complianceThe capability of the software product to adhereto standards, conventions or regulations in lawsand similar prescriptions relating to functionalityReliabilityThe capability of the software product tomaintain a specified level of performancewhen used under specified conditionsMaturityThe capability of the software product to avoidfailure as a result of faults in the softwareFault toleranceThe capability of the software product tomaintain a specified level of performance in casesof software faults or of infringement of itsspecified interfaceRecoverabilityThe capability of the software product toreestablish a specified level of performance andrecover the data directly affected in the case of afailureReliability complianceThe capability of the software product to adhereto standards, conventions or regulations relatingto reliability
278D Quality Attributes According to ISO9126UsabilityThe capability of the software product tobe understood, learned, used and attractiveto the user, when used under specifiedconditionsUnderstandabilityThe capability of the software product to enablethe user to understand whether the software issuitable, and how it can be used for particulartasks and conditions of useLearnabilityThe capability of the software product to enablethe user to learn its applicationOperabilityThe capability of the software product to enablethe user to operate and control itAttractivenessThe capability of the software product to beattractive to the userUsability complianceThe capability of the software product to adhereto standards, conventions, style guides orregulations relating to usabilityEfficiencyThe capability of the software product toprovide appropriate performance, relativeto the ammmt of resources used, UDderstated conditionsTime behaviourThe capability of the software product to provideappropriate response and processing times andthroughput rates when performing its function,under stated conditionsResource utilisationThe capability of the software product to useappropriate amounts and types of resources whenthe software performs its function under statedconditionsEfficiency complianceThe capability of the software product to adhereto standards or conventions relating to efficiency
D Quality Attributes According to ISO9126Maintainability279The capability of the software product tobe modified. Modifications may includecorrections, improvements or ad.aption ofthe software to changes in environment,and in requirements and functionalspecificationsAnalysabilityThe capability of the software product to bediagnosed for deficiencies or causes of failures inthe software, or for the parts to be modified to beidentifiedChangeabilityThe capability of the software product to enable aspecified modification to be implementedStabilityThe capability of the software product to avoidunexpected effects from modifications of thesoftwareTestabilityThe capability of the software product to enablemodified software to be validatedMaintainability complianceThe capability of the software product to adhereto standards or conventions relating tomaintainabilityPortabilityThe capability of the software product tobe transferred from one environment toanotherAdaptabilityThe capability of the software product to beadapted for different specified environmentswithout applying actions or means other thanthose provided for this purpose for the softwareconsideredInstallabilityThe capability of the software product to beinstalled in a specified environmentCo-existenceThe capability of the software product to co-existwith other independent software in a commonenvironment sharing common resourcesReplaceabilityThe capability of the software product to be UBedin place of another specified software product forthe same purpose in the same environmentPortability complianceThe capability of the software product to adhereto standards or conventions relating to portabilityReferenceISO/IEC 9126-1, Software engineering – Software product quality – Part 1:Quality model, International Organization of Standardization, 2001.
ETemplate for Test PlanThis appendix shows the subjects that should be contained within a test plan.This applies equally to a project test plan and a detailed test plan. Generalaspects such as version management and configuration management are notincluded in this.A brief description is provided on the substance of each subject.E.1 Management SummaryProvide a summary of the test plan here. Usual matters to include are: Reason for the project;Description of the project;Time lines;Costs, etc.E.2 IntroductionWhat is this test plan about? Reflect the general structure of the document.E.2.1 Documentation UsedTest Plan DocumentationProvide exhaustive reference to the source documentation used for this testplan, such as: Quick scan (inventory of the project using the Test Management Model);Planning;Test strategy;Risk analysis.
282E Template for Test PlanBear in mind general documentation, such as: Project plan;QA plan;Configuration management plan;Relevant policies within the company.The test plan should always refer to a higher level plan. If this template isused for a detailed test plan (e.g. Functional Acceptance Test) refer here tothe project test plan.Documentation for the System to be TestedProvide a description of the “test basis” here. Think about things such as: Requirement specifications;Functional design;Technical design;User manual;Installation manual.E.2.2 Standards and ProceduresWhich standards and/or procedures will be used? Think about such standards as ISO 9126, ITIL, etc.E.3 The Test AssignmentE.3.1 ClientIndicate who the client is.E.3.2 SupplierIndicate who, if applicable, the supplier is.E.3.3 Goal of the AssignmentProvide a clear interpretation of the brief obtained from the client. Highlightthe aim of the project and the result to be achieved.The result to be achieved should be measurable. Establish agreement onhow this is to be measured. The project should comply with the SMART principle: the assignment should be Specific, Measurable, Acceptable, Realisticand Timely.
E.3 The Test Assignment283E.3.4 ScopeWhich test levels and tests are planned, what is explicitly not being testedand why not ?System Aspects to be TestedProvide a description here of (the parts of) the information system, includingthe interfaces that are to be tested within the scope of the project. Bear inmind also the risks and importance of the system parts. These are defined inthe test strategy.System Aspects not to be TestedIndicate what explicitly falls outside the scope of the project.E.3.5 Suspension Criteria and Resumption RequirementsIt may be the case that the documentation or software supplied is of poorquality or even incomplete. If so, there is little point in starting with thetest analysis or test execution. Specify here, therefore, the criteria for thepossible postponement of a part of the testing, or even the entire test. Makea connection to the entry criteria for the various test levels.Also specify the test activities that should be repeated when testing isresumed following a period of postponement.E.3.6 Test Project DeliverablesWhich products do you intend to produce, and when? Provide a descriptionand not a plan. Also indicate what the clients should do with the deliverables(for information, for approval) and how they should do it (they are not testexperts).Deliverables of a test project include: Test plan;Test specifications/analysis;Procedures;Test logs;Test issue report.E.3.7 Discharging the TestWhen is the testing complete? Who discharges it?Also indicate the acceptance criteria that have to be met.
284E Template for Test PlanE.3.8 Starting Points and PreconditionsIndicate the conditions that must be met to allow the test project to succeed, orthat have to be created to allow it to run smoothly. Be as explicit as possible.E.3.9 Risks and Risk CountermeasuresThese can be copied from the project risk analysis. Provide an overview hereof the five (plus or minus two) project risks with the highest priority. Be sureto refer to the document with the complete risk analysis.Plans often outline risks, without including risk countermeasures. Risksare also often mentioned that are really more tasks for the test manager anddo not belong in the risk column. It is not necessary in a detailed test planto describe the risks for which the test manager is responsible. However, ina project test plan, those risks should be described explicitly, in the firstplace to make the risks clear to the client (as part of prospect management)and secondly because the test manager may be responsible for preventivemeasures while someone else is responsible for corrective measures. The
Risk management is one of the core processes within PRINCE2. Risk management also takes a central place in the test management phi-losophy: as with PRINCE2, a sound risk analysis is carried out at the start of a test project. The risk approach of test management forms the basis of the substantive management of the test project. And it mainly
![Managing successful projects with PRINCE2 [PDF] 6th ed.,](/img/18/1-principles.jpg)
