WIXLIB

SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011Software Evaluation: Criteria-based AssessmentMike Jackson, Steve Crouch and Rob BaxterCriteria-based assessment is a quantitative assessment of the software in terms of sustainability,maintainability, and usability. This can inform high-level decisions on specific areas for softwareimprovement.A criteria-based assessment gives a measurement of quality in a number of areas. These areas arederived from ISO/IEC 9126-1 Software engineering — Product quality1 and include usability,sustainability and maintainability.The assessment involves checking whether the software, and the project that develops it, conformsto various characteristics or exhibits various qualities that are expected of sustainable software. Themore characteristics that are satisfied, the more sustainable the software. Please note that not allqualities have equal weight e.g. having an OSI-approved open source licence is of more importancethan avoiding TAB characters in text files.In performing the evaluation, you may want to consider how different user classes affect theimportance of the criteria. For example, for Usability-Understandability, a small set of well-defined,accurate, task-oriented user documentation may be comprehensive for Users but inadequate forDevelopers. Assessments specific to user classes allow the requirements of these specific user classesto be factored in and so, for example, show that a project rates highly for Users but poorly forDevelopers, or vice versa.Scoring can also be affected by the nature of the software itself e.g. for Learnability one couldenvisage an application that has been well-designed, offers context-sensitive help etc. andconsequently is so easy to use that tutorials aren’t needed. Portability can apply to both the softwareand its development infrastructure e.g. the open source software OGSA-DAI2 can be built, compiledand tested on Unix, Windows or Linux (and so is highly portable for Users and User-Developers).However, its Ruby test framework cannot yet run on Windows, so running integration tests wouldinvolve the manual setup of OGSA-DAI servers (so this is far less portable for Developers and,especially, Members).The assessment criteria are grouped as follows.CriterionSub-criterionNotes – to what extent is/does the software UsabilityUnderstandabilityEasily n?BuildabilityStraightforward to build on a supported system?InstallabilityStraightforward to install on a supported system?LearnabilityEasy to learn how to use its functions?IdentityProject/software identity is clear and unique?CopyrightEasy to see who owns the project/software?LicencingAdoption of appropriate iappropriate,well-structureduser1

SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011GovernanceEasy to understand how the project is run and thedevelopment of the software managed?CommunityEvidence of current/future community?AccessibilityEvidence of current/future ability to download?TestabilityEasy to test correctness of source code?PortabilityUsable on multiple platforms?SupportabilityEvidence of current/future developer support?AnalysabilityEasy to understand at the source level?ChangeabilityEasy to modify and contribute changes to developers?EvolvabilityEvidence of current/future development?InteroperabilityInteroperable with other required/related software?The rest of this document covers each category in greater depth, with lists of questions that we useat the Software Sustainability Institute when compiling detailed software evaluation reports.2

SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011UsabilityUnderstandabilityHow straightforward is it to understand:Yes/No, supporting comments if warranted What the software does and its purpose?The intended market and users of thesoftware? The software’s basic functions? The software’s advanced functions?High-level description of what/who the softwareis for is available.High-level description of what the software doesis available.High-level description of how the software worksis available.Design rationale is available – why it does it theway it isDescriptions of intended use cases are available.Case studies of use are available.DocumentationLooking at the user documentation, what is itsYes/No, supporting comments if warranted Quality? Completeness? Accuracy? Appropriateness? Clarity?Provides a high-level overview of the software.Partitioned into sections for users, user-developersand developers (depending on the software).States assumed background and expertise of thereader, for each class of user.Lists resources for further information.Further information is suitable for the level of thereader, for each class of user.Is task-oriented.Consists of clear, step-by-step instructions.Gives examples of what the user can see at eachstep e.g. screen shots or command-line excerpts.For problems and error messages, the symptomsand step-by-step solutions are provided.Does not use terms like “intuitive”, “user friendly”,“easy to use”, “simple” or “obviously”, unless as3

SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011part of quotes from satisfied usersStates command names and syntax, says whatmenus to use, lists parameters and error messagesexactly as they appear or should be typed.Uses teletype-style fonts for commandline inputs and outputs, source code fragments,function names, class names etc.For Java, the package names of classes are statedthe first time a class is mentioned.English language descriptions of commands orerrors are provided but only to complement theabove.Plain-text files (e.g. READMEs) use indentation andunderlining (e.g. and ---) to structure the text.Plain-text files (e.g. READMEs) do not use TABcharacters to indent the text.API documentation e.g. JavaDoc or Doxygen,documents APIs completely e.g. configuration files,property names etc.Is held under version control alongside the code.Is on the project web site.Documentation on the project web site makes itclear what version of the software thedocumentation applies to.BuildabilityHow straightforward is it to:Yes/No, supporting comments if warranted Meet the pre-requisites for building thesoftware on a build platform? Build the software on a build platform?Web site has instructions for building thesoftware.Source distributionsbuilding the software.haveinstructionsforAn automated build (e.g. Make, ANT, customsolution) is used to build the software.Web site lists all third-party dependencies thatare not bundled, along with web addresses,suitable versions, licences and whether these aremandatory or optional.Source distributions list all third-partydependencies that are not bundled, along withweb addresses, suitable versions, licences andwhether these are mandatory or optional.Dependencymanagementisusedtoautomatically download dependencies (e.g. ANT,4

SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011Ivy, Maven or custom solution).All mandatory third-party dependencies arecurrently available.All optional third-partycurrently available.dependenciesareTests are provided to verify the build hassucceeded.InstallabilityHow straightforward is it to:Yes/No, supporting comments if warranted Meet the pre-requisites for the software on atarget platform? Install the software onto a target platform? Configure the software following installationfor use? Verify the installation for use?Note that in some cases build and install may beone and the same.Web site has instructions for installing thesoftware.Binary distributions haveinstalling the software.instructionsforWeb site lists all third-party dependencies thatare not bundled, along with web addresses,suitable versions, licences and whether these aremandatory or optional.Binary distributions list all third-partydependencies that are not bundled, along withweb addresses, suitable versions, licences andwhether these are mandatory or optional.Dependencymanagementisusedtoautomatically download dependencies (e.g. ANT,Ivy, Maven or custom solution).All mandatory third-party dependencies arecurrently available.All optional third-partycurrently available.dependenciesareTests are provided to verify the install hassucceeded.When an archive (e.g. TAR.GZ or ZIP) isunpacked, it creates a single directory with thefiles within. It does not spread its contents allover the current directory.When software is installed, its contents areorganised into sub-directories (e.g. docs fordocumentation, libs for dependent libraries) as5

SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011appropriate.All source and binary distributions contain aREADME.TXT with project name, web site,how/where to get help, version, date, licenceand copyright (or where to find this information),location of entry point into user doc.All GUIs contain a Help menu with commands tosee the project name, web site, how/where toget help, version, date, licence and copyright (orwhere to find this information), location of entrypoint into user doc.All other content distributed as an archivecontains a README.TXT with project name, website, nature, how /where to get help, date.Installers allow user to select where to installsoftware.Uninstallers uninstall every file or warns user ofany files that were not removed and where theseare.LearnabilityYes/No, supporting comments if warrantedHow straightforward is it to learn how toachieve: Basic functional tasks? Advanced functional tasks?A getting started guide is provided outlining abasic example of using the software.Instructions are provided for many basic usecases.Instructions are provided supporting all usecases.Reference guides are provided for all commandline, GUI and configuration options.API documentation is provided for userdevelopers and developers.6

SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011Sustainability and maintainabilityIdentityYes/No, supporting comments if warrantedTo what extent is the identity of theproject/software clear and unique both withinits application domain and generally?Project/software has its own domain name.Project/software has a logo.Project/software has a distinct name within itsapplication area. A search by Google on thename plus keywords from the application areathrows up the project web site in the first pageof matches.Project/software has a distinct nameregardless of its application area. A search byGoogle on the name plus keywords from theapplication area throws up the project website in the first page of matches.Project/software name does not throw upembarrassing “did you mean ” hits on Google.Project/software name does not violate anexisting trade-mark.Project/software name is trade-marked.CopyrightYes/No, supporting comments if warrantedTo what extent is it clear who wrote the softwareand owns its copyright?Web site states copyright.Web site states who developed/develops thesoftware, funders etc.If there are multiple web sites then these all stateexactly the same copyright, licencing andauthorship.Each source code file has a copyright statement.If supported by the language, each source code filehas a copyright statement embedded within aconstant.Each source code file has a licence header.LicencingHas an appropriate licence been adopted?Yes/No, supporting comments if warrantedWeb site states licence.Software (source and binaries) has a licence.Software has an open source licence.7