WIXLIB

The Methodology of Search Log Analysisaction). Therefore, for SLA, interaction is a moremechanical expression of underlying informationneeds or motivations.How is SLA Used?Researchers and practitioners have used SLA(usually referred to as TLA in these studies) toevaluate library systems, traditional information retrieval (IR) systems, and more recentlyWeb systems. Transaction logs have been usedfor many types of analysis; in this review, wefocus on those studies that centered on or aboutsearching. Peters (1993) provides a review of TLAin library and experimental IR systems. Someprogress has been made in TLA methods sincePeters’ summary (1993) in terms of collection andability to analyze data. Jansen and Pooch (2001)report on a variety of studies employing TLA forthe study of Web search engines and searchingon Web sites. Jansen and Spink (2005) providea comprehensive review of Web searching TLAstudies. Other review articles include Kinsellaand Bryant (1987) and Fourie (2002).Employing TLA in research projects, Meister and Sullivan (1967) may be the first to haveconducted and documented TLA results, andPenniman (1975) appears to have published one ofthe first research articles using TLA. There havebeen a variety of TLA studies since (c.f., BaezaYates & Castillo, 2001; Chau, Fang, & Sheng,2006; Fourie & van den Berg, 2003; Millsap &Ferl, 1993; Moukdad & Large, 2001; Park, Bae,& Lee, 2005).Several papers have discussed the use of TLAas a methodological approach. Sandore and Kaske(1993) review methods of applying the results ofTLA. Borgman, Hirsch, and Hiller (1996) comprehensively review past literature to identifythe methodologies that these studies employed,including the goals of the studies. Several researchers have viewed TLA as a high-level designedprocess, including Copper (1998). Other researchers, such as Hancock-Beaulieu, Robertson, and102Nielsen (1990), Griffiths, Hartley, and Willson(2002), Bains (1997), Hargittai (2002), and Yuanand Meadows (1999), have advocated using TLAin conjunction with other research methodologiesor data collection. Alternatives for other data collection include questionnaires, interviews, videoanalysis, and verbal protocol analysis.How is SLA Critiqued?Almost from its first use, researchers have critiqued TLA as a research methodology (Blecic etal., 1998; Hancock-Beaulieu et al., 1990; Phippenet al., 2004). These critiques report that transactionlogs do not record the users’ perceptions of thesearch, cannot measure the underlying information need of the searchers, and cannot gauge thesearchers’ satisfaction with search results. In thisvein, Kurth (1993) reports that transaction logscan only deal with the actions that the user takes,not their perceptions, emotions, or backgroundskills.Kurth (1993) further identifies three methodological issues with TLA, which are: execution,conception, and communication. Kurth (1993)states that TLA can be difficult to execute due tocollection, storage, and analysis issues associatedwith the hefty volume and complexity of the dataset (i.e., significant number of variables). Withcomplex datasets, it is sometimes difficult to develop a conceptual methodology for analyzing thedependent variables. Communication problemsoccur when researchers do not define terms andmetrics in sufficient detail to allow other researchers to interpret and verify their results.Certainly, any researcher who has used TLAwould agree with these critiques. However, uponreflection, these are issues with many, if notall, empirical methodologies (McGrath, 1994).Further, although Kurth’s critique (1993) is stillgenerally valid, advances in transaction loggingsoftware, standardized transaction log formats,and improved data analysis software and methods have addressed many of these shortcomings.

The Methodology of Search Log AnalysisCertainly, the issue with terms and metrics stillapply (Jansen & Pooch, 2001).As an additional limitation, transaction logsare primarily a server-side data collection method;therefore, some interaction events (Hilbert &Redmiles, 2001) are masked from these loggingmechanisms, such as when the user clicks on theback or print button on the browser software, orcuts or pastes information from one window toanother on a client computer. Transaction logs also,as stated previously, do not record the underlying situational, cognitive, or affective elementsof the searching process, although the collectionof such data can inform system design (Hilbert& Redmiles, 1998).What are the Tools to Support SLA?In an effort to address these issues, HancockBeaulieu, Robertson, and Nielsen (1990) developed a transaction logging software package thatincluded online questionnaires to enhance TLA ofbrowsing behaviors. This application was able togather searcher responses via the questionnaires,but it also took away the unobtrusiveness (one ofthe strengths of the method) of the transactionlog approach. Some software has been developedfor unobtrusively logging client-side types ofevents, for example, the Tracker research package (Choo, Betlor, & Turnbull, 1998; Choo &Turnbull, 2000), the Wrapper (Jansen, Ramadoss,Zhang, & Zang, 2006), and commercial spywaresoftware systems.In other tools for examining transaction logdata, Wu, Yu, and Ballman (1998) present SpeedTracer, which is a tool for data mining Web serverlogs. However, given that transaction log data isusually stored in ASCII text files, relational databases or text-processing scripts work extremelywell for TLA. Wang, Berry, and Yang (2003) useda relational database, as did Jansen, Spink, andSaracevic (2000) and Jansen, Spink, and Pederson (2005). Silverstein, Henzinger, Marais, andMoricz (1999) used text processing scripts. Allapproaches have advantages and disadvantages.With text processing scripts, the analysis can bedone in one pass. However, if additional analysisneeds to be done, the whole dataset must be reanalyzed. With the relational database approach,the analysis is done in incremental portions; andone can easily add additional analysis steps, building from what has already been done.In another naturalistic study, Kelly (2004)used WinWhatWhere Investigator, which is aspy software package that covertly “monitors”a person’s computer activities. Spy software hassome inherent disadvantages for use in user studies and evaluation including granularity of datacapture and privacy concerns. Toms, Freund,and Li (2004) developed the WiIRE system forconducting large scale evaluations. This systemfacilities the evaluation of dispersed study participants; however, it is a server-side applicationfocusing on the participant’s interactions withthe Web server. As such, the entire “study” mustoccur within the WiIRE framework.There are commercial applications for generalpurpose (i.e., not specifically IR) user studies.An example is Morae 1.1 sp) offered byTechSmith. Morae provides extremely detailedtracking of user actions, including video captureover a network. However, Morae is not specifically tailored for information searching studiesand captures so much information at such a finegranularity that it significantly complicates thedata analysis process.How to Conduct TLA for WebSearching Research?Despite the abundant literature on TLA, thereare few published manuscripts on how actuallyto conduct it, especially with respect to SLAfor Web searching. Some works do providefairly comprehensive descriptions of the methodsemployed including Cooper (1998), Nicholas,Hunteytenn, and Lievestey (1999), Wang, Berry,103

The Methodology of Search Log Analysisand Yang (2003), and Spink and Jansen (2004).However, none of these articles presents a processor procedure for actually conducting TLA in sufficient detail to replicate the method. This chapterattempts to address this shortcoming building onwork presented in (Jansen, 2006).SLA PROCESSNaturally, research questions need to be articulated to determine what data needs to be collected.However, search logs are typically of standardformats due to previously developed softwareapplications. Given the interactions between users and Web browsers, which are the interfacesto Web search engines, the type of data that onecan collect is standard. Therefore, the SLA methodology provided in this chapter is applicable toa wide range of studies.SLA involves the following three major stages,which are: Data Collection: The process of collectingthe interaction data for a given period in atransaction log;Preparation: The process of cleaning andpreparing the transaction log data for analysis; andAnalysis: The process of analyzing theprepared data.Data CollectionThe research questions define what informationone must collect in a search log. Transaction logsprovide a good balance between collecting a robustset of data and unobtrusively collecting that data(McGrath, 1994). Collecting data from real userspursuing needed information while interactingwith real systems on the Web affects the type ofdata that one can realistically assemble. If one isconducting a naturalistic study (i.e., outside of thelaboratory) on a real system (i.e., a system used by104actual searchers), the method of data monitoringand collecting should not interfere with the information searching process. In addition to the loss ofpotential customers, a data collection method thatinterferes with the information searching processmay unintentionally alter that process.Fields in a Standard Search LogTable 1 provides a sample of a standard search logformat collected by a Web search engine.The fields are common in standard Websearch engine logs, although some systems maylog additional fields. A common additional fieldis a cookie identification code that facilitatesidentifying individual searchers using a commoncomputer. A cookie is a text message given bya Web server to a Web browser. The cookie isstored on the client machine.In order to facilitate valid comparisons andcontrasts with other analysis, a standard terminology and set of metrics (Jansen & Pooch, 2001) isadvocated. This standardization will help addressone of Kurth’s critiques (1993) concerning thecommunication of SLA results across studies.Others have also noted terminology as an issue inWeb research (Pitkow, 1997). The standard fieldlabels and descriptors are presented below.A searching episode is a series of searchinginteractions within a given temporal span by asingle searcher. Each record, shown as a row inTable 1, is a searching interaction. The format ofeach searching interaction is: User Identification: The IP address of theclient’s computer. This is sometimes alsoan anonymous user code address assignedby the search engine server, which is ourexample in Table 1.Date: The date of the interaction as recordedby the search engine server.The Time: The time of the interaction asrecorded by the search engine server.

The Methodology of Search Log AnalysisTable 1. Snippet from a Web Search Engine Search Loguser identificationdatethetimesearch 8:50Sphagnum Moss Harvesting New Jersey 3c77fb7f95c425/Apr/200404:08:541’personalities AND gender ANDeducation’125/Apr/200404:08:54dmr 0404:08:55“Mark 9426bawdy poems”04:08:56gay 08:58skin 00404:08:59Pink Floyd cd label cover :09:00freie stellen /Apr/200404:09:00Capablity Maturity Model 9:01ana cleonides paulo fontouraNote: Bolded items are intentional errors Search URL: The query terms as enteredby the user.Web search engine server software normallyalways records these fields. Other common fieldsinclude Results Page (a code representing a set ofresult abstracts and URLs returned by the searchengine in response to a query), Language (the userpreferred language of the retrieved Web pages),Source (the federated content collection searched,also known as Vertical), and Page Viewed (theURL that the searcher visited after entering thequery and viewing the results page, which is alsoknown as click-thru or click-through).Data PreparationOnce the data is collected, one moves to thedata preparation stage of the SLA process. Fordata preparation, the focus is on importing thesearch log data into a relational database (orother analysis software), assigning each recorda primary key, cleaning the data (i.e., checkingeach field for bad data), and calculating standardinteraction metrics that will serve as the basis forfurther analysis.Figure 1 shows the Entity – Relation (ER)diagram for the relational database that willbe used to store and analyze the data from oursearch log.An ER diagram models the concepts and perceptions of the data and displays the conceptualschema for the database using standard ER notation. Table 2 presents the legend for the schemaconstructs names.Since search logs are in ASCII format, onecan easily import the data into most relationaldatabases. A key thing is to import the data inthe same coding schema in which it was recorded(e.g., UTF-8, US-ASCII). Once imported, eachrecord is assigned a unique identifier or primarykey. Most modern databases can assign this au-105

The Methodology of Search Log AnalysisFigure 1. ER Scheme Diagram Web Search Logcidterm idtotqtotsearch urluidqry lengthqidcoocthetim ebooleansearching ep isode(1, n) C o occur(0, n) Q ueryoperatorcom posed of(1, n) T erm s(0, 1) Q uery O ccurrences(0, n) Q uery Totalterm soccurrence sterm idtermTable 2. Legend for ER Schema Constructs for Search Log.Entity NameSearching Episodesa table containing the searching interactionsbooleandenotes if the query contains Boolean operatorsoperatorsdenotes if the query contains advanced query operatorsq lengthquery length in termsqidprimary key for each recordqtotnumber of results pages viewedsearcher urlquery terms as entered by the searcherthetimetime of day as measured by the serveruiduser identification based on IPTermstable with terms and frequencyterm IDterm identificationtermterm from the query settfreqnumber of occurrences of term in the query setCooc106Constructtable term pairs and the number of occurrences of those pairsterm IDterm identificationcidthe combined term identification for a pair of termstotnumber of occurrences of the pair in the query settfreq

The Methodology of Search Log Analysistomatically on importation, or one can assign itlater using scripts.Cleaning the DataOnce the search log data is in a suitable analysissoftware package, the focus shifts to cleaning thedata. Records in search logs can contain corrupteddata. These corrupted records can be as a resultof multiple reasons; but they are mostly relatedto errors when logging the data. In the exampleshown in Table 1, one can easily spot these records(additionally these records are bolded), but manytimes a search log will number millions if notbillions of records. Therefore, a visual inspectionis not practical for error identification. From experience, one method of rapidly identifying mosterrors is to sort each field in sequence. Since theerroneous data will not fit the pattern of the otherdata in the field, these errors will usually appearat the top of, bottom of, or grouped together ineach sorted field. Standard database functionsto sum and group key fields such as time and IPaddress will usually identify any further errors.One must remove all records with corrupted datafrom the transaction log database. Typically, thepercentage of corrupted data is small relative tothe overall database.Parsing the DataUsing the three fields of The Time, User Identification, and Search URL, common to all Websearch logs, the chronological series of actionsin a searching episode is recreated. The Webquery search logs usually contain queries fromboth human users and agents. Depending on theresearch objective, one may be interested in onlyindividual human interactions, those from common user terminals, or those from agents. For therunning example used in this chapter, we willconsider the case of only having an interest inhuman searching episodes. To do this, all sessionswith less than 101 queries are separated into anindividual search log for this example.Given that there is no way to accurately identifyhuman from non-human searchers (Silverstein etal., 1999; Sullivan, 2001), most researchers usingWeb search log either ignore it (Cacheda & Viña,2001) or assume some temporal or interaction cutoff (Montgomery & Faloutsos, 2001; Silversteinet al., 1999). Using a cut-off of 101 queries, thesubset of the search log is weighted to queriessubmitted primarily by human searchers in anon-common user terminal, but 101 queries ishigh enough not to introduce bias by too low of acut-off threshold. The selection of 101 is arbitrary,and other researchers have used a wide varietyof cut-offs.There are several methods to remove theselarge sessions. One can code a program to countthe session lengths and then delete all sessionsthat have lengths over 100. For smaller log files(a few million or so records), it is just as easy todo with SQL queries. To do this, one must firstremove records that do not contain queries. Fromexperience, search logs may contain many suchrecords (usually on the order of 35 to 40 percent ofall records) as users go to Web sites for purposesother than searching.Normalizing Searching EpisodesWhen a searcher submits a query, then views adocument, and returns to the search engine, theWeb server typically logs this second visit with theidentical user identification and query, but with anew time (i.e., the time of the second visit). This isbeneficial information in determining how manyof the retrieved results pages the searcher visitedfrom the search engine, but unfortunately, it alsoskews the results in analyzing how the query levelof analysis. In order to normalize the searchingepisodes, one must first separate these result pagerequests from query submissions for each searching episode. An example of how to do this can befound in the SQL query #00 (Appendix A).107

The Methodology of Search Log AnalysisFrom a tbl main, this will create a new tabletbl searching episodes which contains a countof multiple submissions (i.e., qtot) from eachsearcher within each record as shown in Figure2. This collapses the search log by combiningall identical queries submitted by the same userto give the unique queries in order to analyzesessions, queries and terms, and pages of results(i.e., tbl searching episodes). Use the completeun-collapsed sessions (i.e., tbl main) in order toobtain an accurate measure of the temporal lengthof sessions. The tbl searching episodes will nowbe used for the remainder of our TLA. Use SQLquery #01, Appendix A to identify the sessionswith more than 100 records. Then, one can deletethese records from tbl searching episodes usingthe SQL delete query #02, Appendix A.In SLA, many times one is interested in termsand term usage, which can be an entire study initself. In these cases, it is often cleaner to generateseparate tables that contain each term and theirfrequency of occurrence. A term co-occurrencetable that contains each term and its co-occurrencewith other terms is also valuable for understanding the data. If using a relational database, onecan generate these tables using scripts. If usingtext-parsing languages, one can parse these termsand associated data out during initial processing.Figure 2. Records of Searching Episodes with Number of Duplicate Queries (qtot) Recorded108

The Methodology of Search Log AnalysisWe see these as tbl terms and tbl cooc in ourdatabase (see Figure 1 and Table 2).There are already several fields in our database,many of which can provide valuable information(see Figure 1 and Table 2). From these items, onecan calculate several metrics, some of which takea long time to compute for large datasets.term pairs within the data set. Many times, arelatively low frequency term pair may be stronglyassociated (i.e., if the two terms always occurtogether). The mutual i

methodology to conduct transaction log analysis for the study of Web searching is described. The strengths and shortcomings of search log analysis are discussed. rEVIEW OF LItErAtUrE What is a search Log? Not surprisingly, a search log is a file (i.e., log) of the communications (i.e., transactions) between a system and the users of that system.