WIXLIB

K.J. Cios, G. William Moore / Artificial Intelligence in Medicine 26 (2002) 1–245Table 2Possible outcomes of a testHypothesis positiveHypothesis negativeTest result positiveTest result negativeTPFPFNTNare incorrectly classified as belonging to class 3. Thus, the overall accuracy is only 20%(only 2 out of 10 test examples are correctly classified). True positives are in bold inTable 1.When only two outcomes (positive and negative) of a test are possible, three evaluationcriteria can be used for measuring the effectiveness of the generated rules. There are fourpossibilities, as shown in Table 2.Where true positive (TP) indicates the number of correct positive predictions (classifications); true negative (TN) is the number of correct negative predictions; false positive(FP) is the number of incorrect positive predictions; and false negative (FN) is the numberof incorrect negative predictions.The three measures are:sensitivity ¼specificity ¼TPTP100% ¼100%hypothesis positiveTP þ FNTNTN100% ¼100%hypothesis negativeFP þ TNpredictive accuracy ¼TP þ TNTP þ TN100% ¼100%totalTP þ TN þ FP þ FNSensitivity measures the ability of a test to be positive when the condition is actuallypresent, or how many of the positive test examples are recognized. In other words, thesensitivity measures how often you find what you are looking for. It goes under a variety ofnear-synonyms: false-negative rate, recall, Type II error, b error, error of omission, oralternative hypothesis.Specificity measures the ability of a test to be negative when the condition is actually notpresent, or how many of the negative test examples are excluded. In other words thespecificity measures how often what you find is what you are looking for. It goes under avariety of near-synonyms: false-positive rate, precision, Type I error, a error, error ofcommission, or null hypothesis. Predictive accuracy gives an overall evaluation. A highlevel of confidence can be placed only for results that give high values for all threemeasures.Sensitivity/specificity results are illustrated on data from Table 1. For the first twoclasses, the sensitivity, specificity, and predictive accuracy are all 100% (if we use thethreshold of acceptance at 0.9). For class 1 the sensitivity is 4/4, specificity is 6/6, andpredictive accuracy is 10/10. For class 3 the sensitivity is 100% (2/2), but specificity is 0%(0/8) and predictive accuracy is 20% (2/10). The results suggest that the rules generatedfrom the first two classes are correct (since all three values are high) in recognizing test

6K.J. Cios, G. William Moore / Artificial Intelligence in Medicine 26 (2002) 1–24Table 3Hypothetical results on 27 test examplesCorrect diagnosisClassified as AClassified as BClassified as CClassified as DABCD40001700102010110examples, but the rules for class 3 are not. In other words, if the rules generated for oneclass fail to describe a class this failure has no effect on recognition of other classes, asopposed to the accuracy test.As another example, let us suppose that we have the results shown in Table 3, which isknown as a confusion matrix. Table 4 shows how the sensitivity, specificity, and accuracyare calculated [12]. For comparison, the accuracy test would give just one, quite misleading,result: 23/27 ¼ 85%.Finally, the details of performing a complete medical workup may be slightly differentfor each patient, since details of consent and ethical management vary from patient topatient. Therefore, the analysis may be considered subjective by some standards. Furthermore, a sensitivity/specificity analysis must be formulated as an appropriate yes–noquestion, which is sometimes fairly challenging in medical investigations. A carelesslyformulated yes–no question may become a self-fulfilling prophecy, such as the ‘‘have youstopped beating your spouse’’ question, in which either a yes or no answer automaticallyimplies a history of spousal abuse. Reluctance to use the sensitivity/specificity measures oferror analysis in medical data mining may be due to many factors: the expectation that theresults will not appear very convincing, and will therefore be not publishable or grantfundable; and the burdensome, expensive, and sometimes imprecise process of evaluatingeach case in the study.2.4. Poor mathematical characterization of medical dataAnother unique feature of medical data mining is that the underlying data structures ofmedicine are poorly characterized mathematically, as compared to many areas of thephysical sciences. Physical scientists collect data which they can put into formulas,equations, and models that reasonably reflect the relationships among their data. Onthe other hand, the conceptual structure of medicine consists of word descriptions andimages, with very few formal constraints on the vocabulary, the composition of images, orTable 4Calculation of sensitivity/specificity testSensitivitySpecificityAccuracyABCD57% (4/7)100% (20/20)89% (24/27)100% (7/7)95% (19/20)96% (26/27)67% (2/3)96% (23/24)93% (25/27)100% (10/10)88% (15/17)93% (25/27)

K.J. Cios, G. William Moore / Artificial Intelligence in Medicine 26 (2002) 1–247the allowable relationships among basic concepts. The fundamental entities of medicine,such as inflammation, ischemia, or neoplasia, are just as real to a physician as entities suchas mass, length, or force are to a physical scientist; but medicine has no comparable formalstructure into which a data miner can organize information, such as might be modeled byclustering, regression models, or sequence analysis. In its defense, medicine must contendwith hundreds of distinct anatomic locations and thousands of diseases. Until now, thesheer magnitude of this concept space was insurmountable. Furthermore, there is somesuggestion that the logic of medicine may be fundamentally different from the logic of thephysical sciences [7,29,30,55]. However, it may now happen that faster computers and thenewer tools of data mining and knowledge discovery (DMKD) may overcome this priorobstacle.2.5. Canonical formIn mathematics, a canonical form is a preferred notation that encapsulates all equivalentforms of the same concept. For example, the canonical form for one-half is 1/2, and there isan algorithm for reducing the infinity of equivalent expressions, or aliases, namely 2/4, 3/6,4/8, 5/10, . . . , down to 1/2. Agreement upon a canonical form is one of the features of anymature intellectual discipline. For example, the importance of a canonical form becameapparent to the dictionary writers of the 18th century, who realized that one could notprepare a dictionary without consistent orthography. The variable orthography, say, of 14thcentury poet, Geoffrey Chaucer, could not be supported by the need to have each individualEnglish word appear and be defined in only one place in the dictionary. Investigatorsworking with medical text have reached the same conclusion [47].Unfortunately, in biomedicine, even elementary concepts have no canonical form. Forexample, the canonical form for even a simple idea, such as: ‘‘adenocarcinoma of colon,metastatic to liver’’, has no consistent form of expression. The individual medical words, ofcourse, all have a unique spelling and meaning; but the following distinct expressions (andmany others, easy to imagine) are all medically equivalent:Colon adenocarcinoma, metastatic to liver;Colonic adenocarcinoma, metastatic to liver;Large bowel adenocarcinoma, metastatic to liver;Large intestine adenocarcinoma, metastatic to liver;Large intestinal adenocarcinoma, metastatic to liver;Colon’s adenocarcinoma, metastatic to liver;Adenocarcinoma of colon, with metastasis to liver;Adenocarcinoma of colon, with liver metastasis;Adenocarcinoma of colon, with hepatic metastasis.What about even more complex ideas? What about size-quantifiers (e.g. 2.5 cmmetastasis to the liver), logical-quantifiers (for some, for every, etc.), cardinality (threemetastases to the liver), ordinality (the third-largest metastasis in the liver), conditionals (ifthere is a liver metastasis . . .), logical-not, logical-and, logical-or, etc.? If there is nocanonical form for equivalent ideas in biomedicine, then how are indexes and statisticaltables constructed, when these data mining methods depend upon equivalent concepts

8K.J. Cios, G. William Moore / Artificial Intelligence in Medicine 26 (2002) 1–24being tabulated together? Some of these canonical form issues will be addressed in theemerging XML standards for biomedical data in Section 3.3.3. Ethical, legal, and social issuesBecause medical data are collected on human subjects, there is an enormous ethical andlegal tradition designed to prevent the abuse of patients and misuse of their data. The majorpoints of the ethical, legal, and social issues in medicine may be organized under fiveheadings:"""""Data ownershipFear of lawsuitsPrivacy and security of human dataExpected benefitsAdministrative issues3.1. Data ownershipThere is an open question of data ownership in medical data mining. In legal theory,ownership is determined by who is entitled to sell a particular item of property [32]. Since itis considered unseemly to sell human data or tissue, the question of data ownership inmedicine is similarly muddled. The corpus of human medical data potentially available fordata mining is enormous. Thousands of terabytes are now generated annually in NorthAmerica and Europe. However, these data are buried in heterogeneous databases, andscattered throughout the medical care establishment, without any common format orprinciples of organization. The question of ownership of patient information is unsettled,and the object of recurrent, highly publicized lawsuits and congressional inquiries. Doindividual patients own data collected on themselves? Do their physicians own the data?Do their insurance providers own the data? Some HMOs now refuse to pay for patientparticipation in clinical treatment protocols that are deemed experimental. If insuranceproviders do not own their insurees’ data, can they refuse to pay for the collection andstorage of the data? If the ability to process and sell human medical data is unseemly, thenhow should the data managers, who organize and mine the data, be compensated? Orshould this incredibly rich resource for the potential betterment of humankind be leftunmined?3.2. Fear of lawsuitsAnother feature of medical data mining is a fear of lawsuits directed against physiciansand other health-care providers. Medical care in the USA, for those who can afford it, is thevery best. However, US medical care is some 30% more expensive than that in Canada andEurope, where quality is comparable; and US medicine also has the most litigiousmalpractice climate in the world. Some have argued that this 30% surcharge on USmedical care, about US 1000 per capita annually, is mostly medico-legal: either direct

K.J. Cios, G. William Moore / Artificial Intelligence in Medicine 26 (2002) 1–249legal costs, or else the overhead of ‘‘defensive medicine’’, i.e. unnecessary tests ordered byphysicians to cover themselves in potential future lawsuits. In this tense climate, physiciansand other medical data-producers are understandably reluctant to hand over their data todata miners. Data miners could browse these data for untoward events. Apparent anomaliesin the medical history of an individual patient might trigger an investigation. In many cases,the appearance of malpractice might be a data-omission or data-transcription error; and notall bad outcomes in medicine are necessarily the result of negligent provider behavior.However, an investigation inevitably consumes the time and emotional energy of medicalproviders. For exposing themselves to this risk, what reward do the providers receive inreturn?3.3. Privacy and security of human dataAnother unique feature is privacy and security concerns. For instance, US federal rulesset guidelines for concealment of individual patient identifiers. At stake is not only apotential breach of patient confidentiality, with the possibility of ensuing legal action; butalso erosion of the physician–patient relationship, in which the patient is extraordinarilycandid with the physician in the expectation that such private information will never bemade public. By some guidelines, concealment of identifiers must be irreversible. A relatedprivacy issue may apply if, for example, crucial diagnostic information were to bediscovered on patient data, and a patient could be treated if one could only go backand inform the patient about the diagnosis and possible cure. In some cases, this action maynot be taken. Another issue is data security in data handling, and particularly in datatransfer. Before the identifiers are concealed, only authorized persons should have access tothe data. Since transferring the data electronically via the Internet is insecure, the identifiersmust be carefully concealed even for transfers within a single medical institution from oneunit to another.On the other hand, it has been noted in recent US federal documents [49–51], that thereare at least two legitimate research needs for re-identification of de-identified medical data:first, there is a need to prevent accidental duplicate records on the same patient fromskewing research conclusions; second, there may be a compelling need to refer to original(re-identified) medical records to verify the correctness or to obtain additional informationon specific patients. These special requirements could be managed by appropriateregulatory agencies, but they could not be met at all if the data are completely anonymous.There are four forms of patient data identification:" Anonymous data are data that were collected so that the patient-identification wasremoved at the time the information was collected. For example, a block of tissue maybe taken from an autopsy on a patient with a certain disease, to serve as control tissueblock in the histology laboratory. The patient’s identifiers are not recorded at the time ofspecimen collection, and thus can never be recovered." Anonymized data are data that are collected initially with the patient-identifiers, whichare subsequently, irrevocably removed. That is, there can never be a possibility ofreturning to the patient’s record and obtaining additional information. This researchpractice has been common in the past. However, anonymized data, as described above,

10K.J. Cios, G. William Moore / Artificial Intelligence in Medicine 26 (2002) 1–24could be accidentally duplicated, and could not be verified for corrections or additionaldata." De-identified data are data that are collected initially with the patient-identifiers, whichare subsequently encoded or encrypted. The patient can be re-identified under conditions stipulated by an appropriate agency, typically an Institutional Review Board(IRB)." Identified data can only be collected under significant review by the institution, federalguidelines, etc. with the patient giving written informed consent.Even for public Internet distribution, identifier-encrypted data which enter the databaseonly once are fairly safe from attackers. For example, in the Johns Hopkins AutopsyResource [31], a publicly posted Internet resource that lists over 50 000 deceased patients,each deceased patient enters the database only once, and is contributed by a singleinstitution with an IRB-approved encryption procedure. On the other hand, data frommultiple institutions are only as secure as the procedures from the least-secure contributinginstitution. Also, data from a single institution, in which there are multiple updates of thepublic database over time, are also less secure from a determined attacker.There are a variety of encryption protocols suitable for such purposes [4,42]:" double-brokered encryption;" one-time-pad encryption (lookup table);" public–private encryption.The emerging US federal paradigm for using de-identified medical data for researchpurposes is minimal risk. That is, if one employs only data that are collected in the ordinarydiagnosis and treatment of patients, and there is no change in patient management as aresult of the research, including no pressure on the patient to accept or refuse certainmanagement, and no call-back for additional data that might upset the patient or next-ofkin, then the only risk of using such data is the loss of confidentiality to the patient. This iscalled minimal risk data, and may be possible to use in research projects with a simpleexemption from the IRB. There was a well-publicized case of a prominent researcher at amajor institution a few years ago who called a family in order to verify certain dataregarding a deceased patient under study; this is not allowed under the minimal riskparadigm.3.4. Expected benefitsAny use of patient data, even de-identified, must be justified to the IRB as having someexpected benefits. Legally and ethically one cannot perform data analysis for frivolous ornefarious purposes. However, the Internet is the cheapest and most convenient way todistribute data, and the most accessible to the public which may have legitimate reasons foraccess. For example, there may be rare-disease interest groups, medical watchdog groups,or even investigators with unconventional scientific perspectives, who have reasonableclaims to mine the data, but who could not mount the financial and administrative resourcesto mine privately held databases. How is this conflict between public access and frivoloususe of public human data to be resolved? There is as yet no answer to this question.

K.J. Cios, G. William Moore / Artificial Intelligence in Medicine 26 (2002) 1–24113.5. Administrative issuesThe emerging US federal guidelines for patient privacy specify a number ofadministrative policies and procedures that would not ordinarily be required fornon-medical data mining [41]. There must be policies to evaluate and certify thatappropriate security measures are in place in the research institution. There must belegal contracts between the organization and any outside parties given access toindividually identifiable health information, requiring the outside parties to protectthe data. There must be contingency plans for response to emergencies, including a databackup plan and a disaster recovery plan. There must be a system of information accesscontrol that includes policies for the authorization, establishment, and modification ofdata access privileges. There must be an ongoing internal review of data-accessrecords, in order to identify possible security violations. The organization must ensuresupervision of personnel performing technical systems maintenance activities in orderto maintain access authorization records, to ensure that operating and maintenancepersonnel have proper access, to employ personnel security procedures, and to ensurethat system users are trained in system security. There must be termination proceduresthat are performed when an employee leaves or

Keywords: Medical data mining; Unique features of medical data mining and knowledge discovery; Ethical; Security and legal aspects of medical data mining 1. Introduction This article emphasizes the uniqueness of medical data mining. This is a position paper, in which the authors' intent, based on their medical and data mining experience, is .