Transcription
Message Sense Maker: Engineering a Tool Set for Customer RelationshipManagementDmitri RoussinovSchool of Accountancy and Information ManagementCollege of Business, Arizona State University,Box 873606, Tempe, AZ 85287-3606dmitri.roussinov@asu.eduAbstractTo determine the important trends and issues inthousands of comments from customers andmake strategic decisions about businessoperations, managers must go over thesemessages manually and try to make sense ofthem in a time consuming and tedious manner.There is an urgent need for technologies thathelp improve the efficiency of customer messagemanagement.Wedevelopnewissueidentification techniques based on clusteringand context aware similarity networks to enablemanagers to discover knowledge in textmessages. We engineer a tool set specifically forexploring short text messages in the context ofcustomer relationship management. In thispaper, we report a proof of concept prototypecalled Message Sense Maker that can assistmanagers to map the overall sentiment ofWe furthercustomers semi-automatically.1justify the choice of particular technologies andvalidate our system through a field study of acustomer support center in a large university.Keywords: customer relationship management, computermediated communication, information retrieval, textclustering, text mining, Internet.1.IntroductionManaging customer relationships requires effectivecommunications with customers frequently so thatcompanies can monitor customers’ sentiment towards theproducts and services provided by the companies. Onesuch means of communication is customer survey inwhich customers rate the products or services in someclosed or open questions. Beside numerical ratings,companies also value customers’ comments in free texts1This paper is an extension to Roussinov and Zhao, Making Sense ofCRM Messages: an Interactive Toolset, AIS 2002 AmericasConference on Information Systems, August 9-11, 2002, Dallas, TX.J. Leon ZhaoDepartment of Management Information SystemsCollege of Business and Public AdministrationUniversity of Arizona, Tucson, AZ 85721lzhao@bpa.arizona.edusince they will help companies to discover potentialproblems in the products or services. In recent years,customer survey data comes often in digital messages viaWeb survey forms. Making sense of thousands ofcustomer messages is an important task in customerrelationship management.One of the tasks of a customer relationship manager isto collect feedback from customers and analyze the textmessages for changes in customer sentiments. Thisrequires a customer relationship manager to sip throughhundreds or even thousands of text messages to search formajor shifts in customer opinions. This is very timeconsuming and extremely tedious. As a result, much ofthe valuable customer feedback is not fully utilized.Up to our knowledge, no summarizing techniqueshave been studied in CRM applications. In this paper, wepresent a toolset called Message Sense Maker (MSM) thatis designed to assist CRM managers in analyzingthousands of customer messages and discoveringrecurrent issues and trends. Our tool integrates existingstate of the art artificial intelligence technologies into auser-friendly environment.This tool would address the well-known problem ofinformation overload in the context of computer mediatedcommunication (CMC) [11], [9]. Hiltz & Turoff [11]suggested imposing structure on the data as a possibletechnical remedy to the information overload problem inCMC context. Several studies have explored this avenueby automatically summarizing CMC messages byrepresenting them with a list of most representative topics[3], using concept maps [15], or clustering messages intosemantically homogeneous groups[17]. The commonbelief behind those approaches is that automated contextsummarizing techniques can reduce the cognitive load ofcommunication participants even if manual postprocessing is still required.Our research is based on the business environment inthe Center for Computing and Information Technology ofa major university (CCIT).CCIT providescomprehensive services to hundreds of units and tens ofthousands of users throughout the university community.Its services include e-mail & computer accounts,resources for teaching, campus telephone service,resources for research, resources for administrativesystems, and campus networking. One of the importantmissions of CCIT is to improve the quality of services.To accomplish this mission, CCIT must conduct customerProceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)0-7695-1874-5/03 17.00 2002 IEEE
surveys periodically and report the results of analysis ofthese customer surveys to the upper management.Customer relationship management is important alsobecause CCIT needs to manage customer perception ofthe importance of CCIT services. For instance, manyuniversity employees benefit from improved CCITservices; however, in their feedback, many those peoplesay that their do not use CCIT services at all. Thisfinding made the CCIT management realize that theyneed to do a better internal marketing and committedresources for this cause. A CRM group was put in placepermanently to work with the help desk and the strategicplanning group.A customer relationship manager is considered abridge between the customers and the company managers,which must identify important trends in customersentiment and communicate them convincingly to othercorporate managers. The corporate managers then needto take actions to mitigate the problems.The task of customer feedback analysis requiresimportant several steps:(1) Gather the customer feedback messages through callcenter or Web survey.(2) Read the customer messages (sometimes repeatedly)to discover sensitive and recurrent themes.(3) Summarize the recurrent themes while reading themessages.(4) Categorize the customer messages to support themost important themes the manager(s) consider asissues.(5) Compare with previous customer relationshipinitiatives to identify improvements in customersentiments.(6) Propose corporate actions and estimate resources andimpacts of such actions.Steps (2) to (5) are very time consuming and maketake days of work to go over a few hundreds of messages.As our interviews with CRM managers indicated, goingover thousands of messages is close to impossible becauseof time constraints. Consequently, any tools that can helpspeed up the message analysis process and improve thequality of message categorization will be invaluable tocustomer relationship management.One of our research goals is to develop new techniquesfor analyzing customer survey messages and validatingthe prototype system in the CCIT environment. Currently,we have collected 1438 CRM messages collected in theperiod of several years by the CCIT CRM group.We are currently working with the CCIT CRMmanagers in order to evaluate the accuracy of each of theautomated techniques involved. A field study will be ournext step once all the algorithms and parameters are tunedbased on the data that we currently have or will collect infuture.In the rest of the paper, we present the design andimplementation of a message analysis toolset forcustomer relationship management, which we refer to asMessage Sense Maker (MSM). One of our uniquecontributions is the application of a novel text processingtechnique called context sensitive similarity networks[18].2.At the highest level the toolset can be viewed asconsisting of the components performing the followingtasks:1) Identifying semantic descriptors of the messages.2) Identifying semantic similarity relationships betweendescriptors through web mining.3) Clustering Messages into Issues4) Categorizing new messages into previously identifiedissues.The sections below present more details.2.1Identifying semantic descriptorsIn order to evaluate automatically the similaritybetween messages or to categorize messages into preexisting categories, each of the messages has to bedigitally encoded. Since there has not been anyconvincing evidence so far that any other way is betterthan the commonly accepted “bag of words” approach[19] and the resulting Vector Space model, we opted foradopting them. Thus, each message is represented bywords and phrases that it contains through the processcalled automatic indexing which extracts all words andphrases occurring more than once in the collection. Uponremoving words from a “stop-word” (like “the”, “a”,“on”, “in”) the messages are encoded according to VectorSpace Model [19]. Each coordinate in the vector spacecorresponds to a possible term (word or phrase), set to 1 ifa term is present in the document, and to 0 otherwise. Wepreserve only the specified number of the most frequentterms for computational efficiency. According to [3],[15]who studied collections of meeting messages that aresimilar in size and style to the CRM messages, thisapproach seems the most promising. In addition, wesuggest that manual cleaning of context bearing terms is avery useful step, which we have implemented using MSAccess database shown on Figure 1. A CRM managerwho works with MSM has three options: 1) discard a termas non descriptive (“not useful”) (e.g. TECHNCIALSUPPORT is too general and not useful in this contextsince all messages are related to technical supportanyway), 2) identify a term as a definitely descriptive(“Useful”), e.g. TELEPHONE, 3) do not provide anyfeedback on a term (default option). Once, the user isfinished, the system gives higher weights to thedescriptive terms in the vector space representation of themessages, which results in those terms influencingProceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)0-7695-1874-5/03 17.00 2002 IEEEMessage sense maker
clustering and categorization decisions more than theother terms.Figure 1. Manual refinement of descriptive terms.2.2Mining semantic similarity network fromthe WebClustering short text messages imposes a severelimitation on using existing text clustering andrepresentation techniques because they typically expectthe documents to be sufficiently large and share at leastsome terms in order to be treated as semantically similar.The notorious vocabulary problem [8] caused by the factthat different people use different words for same or verysimilar concepts, makes automated clustering extremelydifficult and the outcome not always intuitive. Afterstudying customer messages we observed that thevocabulary problem is clearly demonstrated in our CRMdata, for example the words user, customer, teacher,student, people refer to the same concept of a customer.Thus, messages mentioning customers or users would nothave been necessary placed into the same cluster byMSM, if it did not use the semantic similarity network toalleviate the vocabulary problem as explain in the sectionsbelow.While solving the problem in the most general settingdoes not seem to be feasible in the nearest future, wealleviate it by using an Organizational Concept Space(OCS) [23] framework. OCS includes a so-calledsimilarity network, a collection of similarity relationshipsbetween the important concepts (words and phrases). Forexample, OCS would note the similarity betweencustomer and user.Roussinov & Zhao [18]presented andempirically validated Web mining approach that iscapable of discovering semantic relationships betweenspecified concepts, and as a result, helps to organizemessages produced during electronic meetings supportedby Group Decision Support Systems. In their study, OCSwas successfully “mined” from the World Wide Web.MSM also uses the OCS approach to alleviatethe vocabulary problem. Below, we detail the NCERATESYSTEMSSLOWTECHNOLOGYTable 1. Top 30 most frequent concepts in theCRM data set in order of frequency.2.5.1. Representing the business context. Accordingto [18], OCS is effective only in specific business context,e.g. inside one organization, or one specific meeting. E.g.in our CRM context of the words customer and studentare almost synonyms, which may not be the case in themore context of education. MSM represents the contextautomatically by the top most frequently occurringconcepts (words or phrases). Table 1 lists 30 suchconcepts for our CRM data.2.5.2. Downloading context specific pages from theWeb for mining. In order to perform effective datamining, Message Sense Maker automatically downloadsthousands of pages from the Web that are semanticallyclose to the identified context. It constructs queries for theunderlying commercial search engine (www.altavista.comin the current implementation) in such a way that thematching document have to include the top 30 conceptsand very likely to include the other concepts describingthe context. Table 2 presents a fragment of such queries.The spider component included in Message Sense Makerdownloads 200 pages from each query. More details onhow to automatically interface with commercial searchengines can be found in [18]. STAFF CAMPUS STUDENTS EXCELLENT TRAINING CLASSES PE CAMPUS STAFF STUDENTS EXCELLENT TRAINING CLASSES PE STUDENTS STAFF CAMPUS EXCELLENT TRAINING CLASSES PE EXCELLENT STAFF CAMPUS STUDENTS TRAINING CLASSES PE TRAINING STAFF CAMPUS STUDENTS EXCELLENT CLASSES PE CLASSES STAFF CAMPUS STUDENTS EXCELLENT TRAINING PE PERSON STAFF CAMPUS STUDENTS EXCELLENT TRAINING CL STUDENT STAFF CAMPUS STUDENTS EXCELLENT TRAINING CTable 2. A fragment of queries generated byMessage Sense Maker for AltaVista searchengine from the top 30 most frequent conceptsin the CRM data set.Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)0-7695-1874-5/03 17.00 2002 IEEE
term:COLLABORATIONrelated:COLLABORATIVE 3.877262e-001COLLABORATIVE SYSTEMS 3.381566e-001SYSTEMS N 1.341769e-001FACILITATORS 1.665471e-001term:HARDWARErelated:NETWORK 2.911064e-001SUPPORT 2.496007e-001SYSTEMS 3.189309e-001term:MEETINGrelated:FACILITATORS 2.335312e-001INFORMATION 2.949944e-001MEETINGS 6.949747e-001term:REMOTErelated:EXAMPLE 2.259506e-001NETWORK 2.501757e-001SYSTEMS 2.463332e-001term:VOICErelated:TECHNOLOGIES 2.254068e-001TECHNOLOGY 2.486785e-001NETWORKS 2.637258e-001SYSTEMS 2.067017e-001Table 3. A fragment of a listing with minedsimilarity relationships from [18].Many researchers and practitioners believe that theWorld Wide Web is a gold mine filled with usefulinformation. Indeed, such vast amount of textual andmultimedia information was not available for researchersbefore the mid 90s. According to Lyman and Varian [14],the Web currently contains more than 2.5 billion of pages,consisting of at least 10 terabytes of textual information.Although there exist multiple definitions, in this study,under "Web mining" we mean automated discovering ofsemantic associations between the specified terms. For acomprehensive review of the Web mining literature,please refer to Cooley et al. [4].2.5.3. Indexing mining collection. The HTML pagesare converted into plain text using and truncated to thefirst 20,000 bytes of text in order to avoid overly longweb pages. They are processed by the same automaticindexing procedure as mentioned in the preceding sectionand represented by vectors. We use 0/1 encoding insteadof popular TF-IDF weighting for the mined collection andnormalize the vectors to the unit length afterwards. Sinceconversion is a one-pass algorithm, it is runs relativelyquickly so downloading is currently the only bottleneckfor performance. However, if performed in parallel, it canbe sped up considerably [16], so that the entire processcan be implemented to run in real time. Also, if searchengine index is available through Application Interface(API), the downloading would not be necessary at all.Currently, many commercial search engines starting toallow this kind of API.2.5.4. Performing data mining in the downloadedcollection. It has been known for a long time that therelationships between concepts (words or phrases) can bediscovered by their co-occurrence in the same documentsor in the vicinity of each other within documents. Firth, aleading figure in British linguistics during the 1950s,summarized the approach with the memorable line: “Youshall know a word by the company it keeps.” [7] Theclassical work of van Rijsbergen [20] initiated the use ofco-occurrence information for text retrieval andcategorization. To obtain the numerical value ofsimilarity, we used the previously suggested [20], [19]formula:Sij ti tj / ti tj ,where Sij is the similarity between terms i and j; ti andtj are vectors representing occurrences of the terms i and jin the documents in the mined collection. In order tomake computation more tractable Message Sense Makertruncates for each term the mined relationships to onlythree (3) other most strongly related terms. A fragment ofthe mined Concept Space from a different dataset(Roussinov & Zhao, 2002) is shown in as example Table3, where each term is followed by 3 most closely termsalong with the similarity between them.2.5.5. Applying Similarity Network. As it is done in[18], Message Sense Maker modifies each message vectorV by the following transformation: V V a O V,where O is the matrix representing OCS, and a is theadjustment factor. The product O x V is the vector Vmultiplied by the matrix S defined by a common linearalgebra.Tr P Cu Te B Co H10000.500Traffic (Tr)01000.400Problem (P)00100000Current (Cu)00010000Technology (Te)00001000Bandwidth (B).5.4000100Cost (Co)0000001.2Hardware (H)000000.21Figure 2. An example of Similarity Network andits matrix representation from [18].For example, the message “bandwidth concerns -impact of remote collaboration” originally representedby concepts BANDWIDTH, CONCERNS, IMPACT,REMOTE and COLLABORATION would also receive theProceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03)0-7695-1874-5/03 17.00 2002 IEEENNetwork (N)
concept NETWORKS if Message Sense Maker capturedsemantic similarity between the concepts of NETWORKSand BANDWIDTH (as shown in Figure 2 below.This way the message would be automaticallyevaluated as similar to the message “Effectivetransmission of video over networks”and would likely to be placed in the same cluster ofnetwork related messages even though those twomessages do not share any words in common. Thisexample illustrates the alleviation of the vocabularyproblem.It is worth noting that even when a is small and theresulting modifications are small, the resulting accuracyincrease may be still significant. This is because withoutthe modifications, most messages did not share anycommon terms and had the same similarity (0, if dotproduct used to compute it). This poses a problem forclustering (and other) algorithms that have to resolvemany "ties" in order to form clusters. Even small changesin the coordinates can break those "ties" in the rightdirection, thus considerably helping the clusteringalgorithm to make more accurate choices.Windows. Each concept is placed on a new line, andrelated concepts immediately follow and are indicated byindentation (e.g. TRAINING is related to CLASSES etc.).The initial relationships are built automatically throughthe co-occurrence based text mining [18]. Then, a CRMmanager can refine them.*** New Issue: CLASSESI attended two lecture/classes for computing duringthe month of November but I'm not sure if they were CCITclasses: 1) Denise Warren - Web Design, 2) CopyrightLaws (Web) They were both excellent. I look forward tomore of the same.In general, my interactions have been verysatisfactory. I am thankful to have an efficient and easyaccess to the internet. However, I was really disappointedwhen you quit offering your free classes for Macintoshusers. There are many of us who use Macs on campus andmuch prefer them to IBM. P
say that their do not use CCIT services at all. This finding made the CCIT management realize that they need to do a better internal marketing and committed resources for this cause. A CRM group was put in place permanently to work with the
