WIXLIB

speaking, a coding module includes or describeseverything that is needed in order to perform acertain kind of markup of spoken languagecorpora. A coding module prescribes whatconstitutes a coding, including the representationof markup and the relations to other codings.Thus, the MATE coding module is a proposalfor a standardised description of codingschemes.The above-mentioned five annotation levelsand the issues to do with cross-level annotationwere selected for consideration in MATEbecause they pose very different markupproblems. If a common framework can beestablished and shown to work for those levelsand across them, it would seem likely that theframework will work for other levels as well.For each annotation level, one or more existingcoding schemes were selected to form the basisof the best practice coding modulesimplemented in the MATE Workbench (Mengelet al., 2000). Common to the selected codingschemes is that these are among the most widelyused coding schemes for their respective levelsin current practice, each having been used byseveral annotators and for the annotation ofmany dialogues. Since all MATE best practicecoding schemes are expressed in terms of codingmodules, they should contain sufficientinformation for use by other annotators. Theiruniform description in terms of coding modulesmakes it easy for the annotator to work onmultiple coding schemes and/or levels, and tocompare schemes since these all contain thesame categories of information. The use ofcoding modules also facilitates use of the sameset of software tools and enables the sameinterface look-and-feel independently of level.2.2ToolingThe engineering objective of MATE has been tospecify and implement a generic annotation toolin support of the markup framework and theselected best practice coding schemes. Severalexisting annotation tools were reviewed early onto gather input for MATE workbenchspecification (Isard et al., 1998). Building onthis specification, the MATE markup frameworkand the selected coding schemes, a java-basedworkbench has been implemented (Isard et al.,2000) which includes the following majorfunctionalities:The MATE best practice coding modules areincluded as working examples of the state of theart. Users can add new coding modules via theeasy-to-use interface of the MATE codingmodule editor.An audio tool enables listening to speech filesand having sound files displayed as a waveform.For each coding module, a default stylesheetdefines how output to the user is presentedvisually. Phenomena of interest in the corpusmay be shown in, e.g., a certain colour or inboldface. Users can modify style sheets anddefine new ones.The workbench enables information extractionof any kind from annotated corpora. Queryresults are shown as sets of references to thequeried corpora. Extraction of statisticalinformation from corpora, such as the number ofmarked-upnouns,isalsosupported.Computation of important reliability measures,such as kappa values, is enabled.Import of files from XLabels and BAS Partiturto XML format is supported in order todemonstrate the usefulness of importing widelyused annotation formats for further work in theWorkbench. Similarly, a converter fromTranscriber format (http://www.etca.fr/CTA/gip/Projets/Transcriber/) to MATE formatenables transcriptions made using Transcriber tobe annotated using the MATE Workbench.Other converters can easily be added. Export tofile formats other than XML can be achieved byusing style sheets. For example, informationextracted by the query tool may be exported toHTML to serve as input to a browser.On-line help is available at any time.The first release of the MATE Workbenchappeared in November 1999 and was madeavailable to the 80 members of the MATEAdvisory Panel from across the world. Sincethen, several improved versions have appearedand in May 2000 access to executable versionsof the Workbench was made public. The MATEWorkbench is now publicly available both in anexecutable version and as open source softwareat http://mate.nis.sdu.dk. The Workbench is stillbeing improved by the MATE consortium, sonew versions will continue to appear. Adiscussion forum has recently been set up at theMATE web site where colleagues are invited toask questions and provide information from theirexperience with the Workbench, including the

new tools they have added to the MATEWorkbench to enhance its functionality.We have no exact figures on how many usersare now using the workbench but we know thatthe MATE workbench is already being used byand is being considered for use in severalEuropean and national research projects.3The MATE Markup FrameworkThe MATE markup framework is a conceptualmodel which basically prescribes (i) how filesare structured, for instance to enable multi-levelannotation, (ii) how tag sets are represented interms of elements and attributes, and (iii) how toprovide essential informationsemantics, coding purpose etc.3.1onmarkup,Files, elements and attributesWhen a coding module has been applied to acorpus, the result is a coding file. The coding filehas a header which documents the codingcontext, such as who annotated the file, when,and the experience of the annotator, and a bodywhich lists the coded elements. Figure 1 showsan example of how annotated communicationproblems are displayed to the user in the MATEWorkbench. Figure 2 shows an excerpt of theinternal representation of the file.Figure 1. A screen shot from the MATE Workbench showing a dialogue annotated withcommunication problems (top left-hand panel). Guidelines for cooperative dialogue behaviour areshown in the top right-hand panel. Communication problems are categorised as types of violations ofthe cooperativity guidelines. Violation types are shown in the bottom right-hand panel. Notes may beadded as part of the annotation. Notes are shown in the bottom left-hand panel.Figure 2. Excerpt of the internal XMLrepresentation of the annotated dialogue shownin Figure 1. The tags will be explained inSection 3.1.1 below.As shown in Figure 2, the annotated filerepresentation is simply a list of references tothe transcription file. The underlying filestructure idea is depicted in Figure 3 whichshows how coding files (bottom layer) refer to atranscription file and possibly to other codingfiles, cf. entry 5 in the coding module in Figure4. A transcription (which is also regarded as acoding file) refers to a resource file listing theraw data resources behind the corpus, such assound files and log files. The resource fileincludes a description of the corpus: purpose ofthedialogues,dialogueparticipants,experimenters, recording conditions, etc. Abasic, sequential timeline representation of the

spoken language data is defined. The timelinemay be expressed as real time, e.g. inRaw dataSound filesLog filesResource fileTranscriptions(coding files)Coding filesmilliseconds, or as numbers indicating, e.g., thesequence of utterance starts.Other data, e.g.video or picturesText filesList of blemsPhonetictranscriptionDialogue actsProsodyFigure 3. The raw corpus data are listed in the resource file to which transcriptions refer. Coding filesat levels other than transcription refer to a transcription and only indirectly to the raw data. Codingfiles may refer to each other.Given a coding purpose, such as to identify allcommunication problems in a particular corpus,and a coding module, the actual coding consistsin using syntactic markup to encode the relevantphenomena found in the data. A coding isdefined in terms of a tag set. The tag set isconceptually specified by, and presented to, theuser in terms of elements and attributes, cf. entry6 in the coding module in Figure 4. Importantly,workbench users can use this markup directlywithout having to know about complex formalstandards, such as SGML, XML or TEI.3.1.1 ElementsThe basic markup primitive is the element (aterm inherited from TEI and SGML) whichrepresents a phenomenon such as a particularphoneme, word, utterance, dialogue act, orcommunication problem. Elements haveattributes and relations to each other both withinthe current coding module and across codingmodules. Considering a coding module M, themarkup specification language is described as: E1 .En : The non-empty list of tag elements. For each element Ei, the following propertiesmay be defined:1. Ni : The name of Ei .Example: u 2. Ei may contain a list of elements Ej fromM.Example: u may contain t : u t Example /t /u 3. Ei has mi attributes Aij , where j 1 . mi .Example: u has attributes who and id,among others.4. Ei may refer to elements in coding moduleMj , implying that M references Mj .Example: a dialogue act coding may referto phonetic or syntactic cues.A concrete example is the coding module forcommunication problems which, i.a., has theelement comprob , cf. the XML representationin Figure 2. comprob has, i.a., the attributes iduref and vtype . uref is a reference to an utterancein the transcription coding. vtype is a referenceto a type of violation of a guideline in theviolation type coding. Due to the inflexibility ofXML, this logical structure has to be representedslightly differently internally in the workbench.Thus, the uref corresponds to the first href in

Figure 2 while vtype is wrapped up in a newelement and corresponds to the second href.3.1.2 AttributesAttributes are assigned values during coding.For each attribute Aij the type of its values mustbe defined. There are standard attributes, userdefined attributes, and hidden attributes, asfollows.Standard attributes are attributes prescribed byMATE. id [mandatory]: ID. The element id iscomposed of the element name and a machinegenerated number.Example: id n 123Time start and end are optional. Elements musthave time information, possibly indirectly byreferencing other elements (in the same codingmodule or in other modules) which have timeinformation. TimeStart [optional]: TIME. Start of event. TimeEnd [optional]: TIME. End of event.User-defined attributes are used to parametriseand extend the semantics of the elements theybelong to. For instance, who is an attribute ofelement u designating by whom the utteranceis spoken. There will be many user-definedattributes (and elements), cf., e.g., the uref andvtype mentioned above.Hidden attributes are attributes which the userwill neither define nor see but which are used forinternal representation purposes. An example isthe following of coding elements which mayrefer to utterances in a transcription but whichdepend on the technical programming choice ofthe underlying, non-user related representation:ModuleRefs CDATA ‘href:transcription#u’See Figure 2 for a concrete example from theMATE Workbench.3.1.3 Attribute standard typesThe MATE markup framework proposes a set ofpredefined attribute value types (attributes aretyped) which are supported by the workbench.By convention, types are written in capitals. Theincluded standard types are: TIME: in milliseconds, as a sequence ofnumbers, or as named points on the timeline.Values are numbers or identifiers, and thedeclaration of the timeline states how tointerpret them.Example: time 123200 dur 1280 (these arederived values, with time TimeStart , and dur TimeEnd – TimeStart ). HREF[MODULE,ELEMENTLIST]:HereMODULE is the name of another codingmodule, and ELEMENTLIST is a list of namesof elements from MODULE. When applied asconcrete attribute values, two parameters mustbe specified:- The name of the referenced coding file whichis an application of the declared MODULEcoding module.- The id of the element occurrence that isreferred to.The values of this attribute are of the ion, u) allows an attribute used as,e.g., OccursIn ”base#u 123” , where base is acoding file using the transcription module andu 123 is the value of the id attribute of a uelement in that file.Example:Forthedeclarationwho:participant]an actualoccurrence may look like who ”#participant2”HREF[transcription,where the omitted coding file name byconvention generically means the currentcoding file.The concept of hyper-references together withparameters referencing coding modules (seepoint 5 in Figure 4) is what enables codingmodules to handle cross-level markup. ENUM: A finite closed set of values.Values are of the form: “(“ Identifier ( “ ”Identifier )* “)”Example: time(year month day hour) allowssuchastime day.attributesThe user may be allowed to extend the set, butnever to change or delete values from the set. TEXT: Any text not containing ‘“’ (which isused to delimit the attribute value).Example: The declaration desc TEXT allowsuses such as: event desc ”Door is slammed” . ID: Automatically generated id for the element.Only available in the automatically addedattribute id.

3.2Coding modulesIn order for a coding module and the dialoguesannotated using it to be usable andunderstandable by people other than its creator,some key information must be provided. TheMATE coding module which is the central partof the markup framework, serves to capture thisinformation. A coding module consists of the tenitems shown in Figure 4.1. Name of the module.2. Coding purpose of the module.3. Coding level.4. The type of data source scoped by themodule.5. References to other modules, if any. Fortranscriptions, the reference is to a resource.6. A declaration of the markup elements andtheir attributes. An element is a feature, or typeof phenomenon, in the corpus for which a tag isbeing defined.7. A supplementary informal description ofthe elements and their attributes, including:a. Purpose of the element, its attributes,and their values.b. Informal semantics describing how tointerpret the element and attribute values.c. Example of each element and attribute.8. An example of the use of the elements andtheir attributes.9. A coding procedure.10. Creation notes.Figure 4. Main items of the MATE codingmodule.Some coding module items have a formal role,i.e. they can be interpreted and used by theMATE workbench. Thus, items (1) and (5)specify the coding module as a namedparametrised module or class which builds oncertain other predefined modules (no cyclesallowed). Item (6) specifies the markup to beused in the coding. All elements, attributenames, and ids have a name space restricted totheir module and its coding files, but arepublicly referrable by prefixing the name of thecoding module or coding file in which theyoccur. Other items provide directives andexplanations to users. Thus, (2), (3) and (4)elaborate on the module itself, (7) and (8)elaborate on the markup, and (9) recommendscoding procedure and quality measures. (10)provides information about the creation of thecoding module, such as by whom and when.In the following, we show an abbreviatedversion of a coding module for communicationproblems to illustrate the 10 coding moduleentries.Name: Communication problems.Coding purpose: Records the different ways inwhich generic and specific guidelines areviolated in a given corpus. A communicationproblems coding file refers to a problem typecoding file as well as to a transcription.Coding level: Communication problems.Data sources: Spoken human-machine dialoguecorpora.Module references: Module Basic orthographic transcription; Module Violation types.Markup declaration:ELEMENT comprobATTRIBUTESvtype: REFERENCE(Violation types, vtype)wref: REFERENCE(Basic orthographictranscription, (w,w) )uref: REFERENCE(Basic orthographictranscription, u )caused by: REFERENCE(this, comprob)temp: TEXTELEMENT noteATTRIBUTESwref: REFERENCE(Basic orthographictranscription, (w,w) )uref: REFERENCE(Basic orthographictranscription, u sproducedbyinadequate system utterance design we use theelement comprob. It refers to some kind ofviolation of one of the guidelines listed in Figure1, top right-hand panel. The comp rob elementmay be used to mark up any part of the dialoguewhich caused, or might cause, a communicationproblem. Thus, comprob may be used to annotateone or more words, an entire utterance, or evenseveral utterances in which an actual or potentialcommunication problem was detected. Thecomprob element has five attributes in addition tothe automatically added id.

The attribute vtype is mandatory. vtype is areference to a description of a guidelineviolation in a file which contains the differentkinds of violations of the individual guidelines.Either wref or uref must be indicated. Boththese attributes refer to an orthographictranscription. wref delimits the word(s) whichcaused or might cause a communicationproblem, and uref refers to one or more entireutterances which caused or might cause aproblem.We stop the illustration here due to spacelimitations. The full description is available in(Mengel et al. 2000).Example:In the following snippet of a transcription fromthe Sundial corpus: u id "S1:7-1-sun" who "S" flight informationbritish airways good day can I help you /u communication problems are marked up asfollows: comprob id "3" vtype "Sundial problems#SG4-1"uref "Sundial#S1:7-1-sun"/ We do not exemplify note here and do notshow the violation type coding file due to spacelimitations. However, note that once a codingmodule is specified in the MATE workbench,the user does not have to bother about themarkup shown in the example above. The userjust selects the utterance to mark up and thenclicks on the violation type palette, or, in case itis a new type, clicks on the violatedcooperativity guideline which means that a newviolation type is added and text can be entered todescribe it, cf. Figure 1.Coding procedure: We recommend to use thesame coding procedure for markup ofcommunication problems as for violation typessince the two actions are tightly connected. As aminimum, the following procedure should befollowed:1. Encode by coders 1 and 2.2. Check and merge codings (performed bycoders 1 and 2 until consensus).Creation notes:Authors: Hans Dybkjær and Laila Dybkjær.Version: 1 (25 November 1998), 2 (19 June1999).Comments: For guidance on how to identifycommunication problems and for a collection ofexamples the reader is invited to look at(Dybkjær 1999).Literature: (Bernsen et al. 1998).The MATE Workbench allows its users tospecify a coding module via a coding moduleeditor. A screen shot of the coding module editoris shown in Figure 5.Figure 5. The MATE coding module editor.

4Early Experience and Future WorkThe MATE markup framework has been wellreceived for its transparency and flexibility bythe colleagues on the MATE Advisory Panel.The framework has been used to ensure acommon description of coding modules at theMATE coding levels and has turned out to workwell for all these levels. We therefore concludethat the framework is likely to work for otherannotation levels not addressed by MATE. Theuse of a common representation and a commoninformation structure in all coding modules atthe same level as well as across levels facilitateswithin-level comparison, creation and use ofnew coding modules, and working at multiplelevels.On the tools side, the markup framework hasnot been fully exploited as intended, i.e. as anintermediate layer between the user interfaceand the internal representation. This means thatthe user interface for adding new codingmodules, in particular for the declaration ofmarkup, and for defining new visualisations isstill sub-optimal from a usability point of view.The coding module editor which is used foradding new coding modules, represents a majorstep forward compared to requiring users towrite DTDs. The coding module editorautomatically generates a DTD from thespecified markup declaration. However, theXML format used for the underlying filerepresentation has not been hidden completelyfrom the editor’s interface. Peculiarities and lackof flexibility in XML have been allowed toinfluence the way in which users must specifyelements and attributes, making the process lesslogical and flexible than it could have been. It ishigh on our wish list to repair this shortcoming.As regards coding visualisation, XSLT-likestyle sheets are used to define how codings aredisplayed to the user. Writing style sheets,however, is cumbersome and definitely notsomething users should be asked to do to definehow codings based on a new coding moduleshould be displayed. We either need a style sheeteditor comparable to the coding module editor asregards ease of use, or, alternatively, acompletely new interface concept should beimplemented to replace the style sheets andenable users to easily define new visualisations.It is high on our wish-list to better exploit themarkup framework in the Workbenchimplementation in order to achieve a better userinterface.Other frameworks have been proposed but toour knowledge the MATE markup framework isstill the more comprehensive framework around.An example is the annotation frameworkrecently proposed by Bird and Liberman (1999)which is based on annotation graphs. These arenow being used in the ATLAS project (Bird etal., 2000) and in the Transcriber tool (Geoffroiset al., 2000). The annotation graphs serve as anintermediate representation layer in agreementwith the argument above for having anintermediate layer of representation between theuser interface and the internal representation.Whilst Bird and Liberman do not considercoding modules or discuss the interface from ausability point of view, they present tion and time line reference. The twoframeworks may, indeed, turn out tocomplement each other nicely.AcknowledgementsWe gratefully acknowledge the support for theMATE project provided by the ng Programme. We would also like tothank all MATE partners. Without the veryconsiderable joint efforts of the projectconsortium it would not have been possible tobuild the MATE Workbench.ReferencesNote on MATE deliverables: like the MATEWorkbench, these are all ob

The MATE best practice coding modules are included as working examples of the state of the art. Users can add new coding modules via the easy-to-use interface of the MATE coding module editor. An audio tool enables listening to speech files and having sound files displayed as a waveform.